2024-01-20T10:54:56.801 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-20T10:54:56.802 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --name yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621 -- /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/orig.config.yaml 2024-01-20T10:54:56.848 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-20T10:54:56.947 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621 branch: main description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7522621' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: a00c9940991c5e1e9ab595faa3f6e1281d8ae167 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: a00c9940991c5e1e9ab595faa3f6e1281d8ae167 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: a00c9940991c5e1e9ab595faa3f6e1281d8ae167 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 587 sha1: a00c9940991c5e1e9ab595faa3f6e1281d8ae167 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a00c9940991c5e1e9ab595faa3f6e1281d8ae167 targets: smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtYQuegvVyO6UZG27OP9qpqVkjKdMu9D5GDJXb4WYgvaW0gqlywdzyv+IURvYFDwwphGS9JaMvWDYVc0Logf+c= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNf/dmXNPIdSY8E4I0n7Bmhwc9mJyxV0TODuUn9bQpvSl5ohdkYiIji26ylANr2L59U4qZ8itxKut7Q+UOPlBjU= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-19_15:25:09 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-20T10:54:56.947 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa; will attempt to use it 2024-01-20T10:54:56.948 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks 2024-01-20T10:54:56.948 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-20T10:54:56.950 INFO:teuthology.task.internal:Checking packages... 2024-01-20T10:54:56.976 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167' 2024-01-20T10:54:56.976 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-20T10:54:56.976 INFO:teuthology.packaging:ref: None 2024-01-20T10:54:56.976 INFO:teuthology.packaging:tag: None 2024-01-20T10:54:56.976 INFO:teuthology.packaging:branch: main 2024-01-20T10:54:56.976 INFO:teuthology.packaging:sha1: a00c9940991c5e1e9ab595faa3f6e1281d8ae167 2024-01-20T10:54:56.976 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-01-20T10:54:57.161 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-772.gd7e2a322 2024-01-20T10:54:57.163 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-20T10:54:57.177 INFO:teuthology.task.internal:no buildpackages task found 2024-01-20T10:54:57.177 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-20T10:54:57.223 INFO:teuthology.task.internal:Saving configuration 2024-01-20T10:54:57.246 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-20T10:54:57.256 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-20T10:54:57.285 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-20 10:49:57.727375', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtYQuegvVyO6UZG27OP9qpqVkjKdMu9D5GDJXb4WYgvaW0gqlywdzyv+IURvYFDwwphGS9JaMvWDYVc0Logf+c='} 2024-01-20T10:54:57.306 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-20 10:49:57.726009', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNf/dmXNPIdSY8E4I0n7Bmhwc9mJyxV0TODuUn9bQpvSl5ohdkYiIji26ylANr2L59U4qZ8itxKut7Q+UOPlBjU='} 2024-01-20T10:54:57.306 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-20T10:54:57.317 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-01-20T10:54:57.317 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-01-20T10:54:57.317 INFO:teuthology.run_tasks:Running task console_log... 2024-01-20T10:54:57.414 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff6ad1b5310>, signals=[15]) 2024-01-20T10:54:57.415 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-20T10:54:57.423 INFO:teuthology.task.internal:Opening connections... 2024-01-20T10:54:57.423 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T10:54:57.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:54:57.521 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T10:54:57.522 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:54:57.627 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-20T10:54:57.634 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-01-20T10:54:57.651 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-01-20T10:54:57.651 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-01-20T10:54:57.707 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-01-20T10:54:57.707 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-01-20T10:54:57.707 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-20T10:54:57.708 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-20T10:54:57.709 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-01-20T10:54:57.737 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-01-20T10:54:57.754 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-01-20T10:54:57.754 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-01-20T10:54:57.811 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-01-20T10:54:57.812 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-20T10:54:57.813 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-20T10:54:57.813 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-20T10:54:57.813 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-01-20T10:54:57.837 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-20T10:54:57.845 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-20T10:54:57.870 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-20T10:54:57.870 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-01-20T10:54:57.873 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-01-20T10:54:57.889 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-20T10:54:57.896 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-20T10:54:57.896 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-01-20T10:54:57.929 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-01-20T10:54:58.269 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-20T10:54:58.331 INFO:teuthology.run_tasks:Running task kernel... 2024-01-20T10:54:58.345 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-20T10:54:58.346 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-20T10:54:58.346 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-20T10:54:58.346 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-20T10:54:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-20T10:54:58.363 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-01-20T10:54:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-20T10:54:58.364 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-01-20T10:54:58.422 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-361.el9.x86_64 2024-01-20T10:54:58.422 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-361.el9.x86_64 2024-01-20T10:54:58.422 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-361.el9.x86_64 2024-01-20T10:54:58.422 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-01-20T10:54:58.422 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-361.el9.x86_64 2024-01-20T10:54:58.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-01-20T10:55:00.543 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2024-01-20T10:55:00.563 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2024-01-20T10:55:01.905 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-01-20T10:55:02.151 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-01-20T10:55:03.679 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-01-20T10:55:05.867 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-01-20T10:55:06.455 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-01-20T10:55:07.984 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-01-20T10:55:12.618 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 5.9 MB 00:01 2024-01-20T10:55:12.688 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 5.9 MB 00:02 2024-01-20T10:55:14.147 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2024-01-20T10:55:14.186 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 15 kB 00:00 2024-01-20T10:55:15.551 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-01-20T10:55:16.965 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 20 MB 00:02 2024-01-20T10:55:20.481 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-01-20T10:55:21.787 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-01-20T10:55:22.655 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-20T10:55:22.655 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-20T10:55:22.752 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T10:55:22.761 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T10:55:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T10:55:22.761 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T10:55:22.761 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T10:55:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-409.el9 CentOS-BaseOS 6.0 M 2024-01-20T10:55:22.761 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T10:55:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-409.el9 CentOS-BaseOS 21 M 2024-01-20T10:55:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-409.el9 CentOS-BaseOS 39 M 2024-01-20T10:55:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-409.el9 CentOS-BaseOS 33 M 2024-01-20T10:55:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T10:55:22.762 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T10:55:22.762 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T10:55:22.762 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-01-20T10:55:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T10:55:22.762 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 99 M 2024-01-20T10:55:22.762 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 122 M 2024-01-20T10:55:22.762 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T10:55:23.175 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-409.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-01-20T10:55:23.600 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-409.el9.x86_64.rpm 25 MB/s | 21 MB 00:00 2024-01-20T10:55:23.944 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-20T10:55:23.944 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-20T10:55:24.047 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T10:55:24.053 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T10:55:24.053 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T10:55:24.053 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T10:55:24.053 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-409.el9 CentOS-BaseOS 6.0 M 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 5.14.0-409.el9 CentOS-BaseOS 21 M 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 5.14.0-409.el9 CentOS-BaseOS 39 M 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core x86_64 5.14.0-409.el9 CentOS-BaseOS 33 M 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-01-20T10:55:24.054 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T10:55:24.055 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 99 M 2024-01-20T10:55:24.055 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 122 M 2024-01-20T10:55:24.055 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T10:55:24.233 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-5.14.0-409.el9.x86_64.rpm 27 MB/s | 39 MB 00:01 2024-01-20T10:55:24.522 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): kernel-5.14.0-409.el9.x86_64.rpm 13 MB/s | 6.0 MB 00:00 2024-01-20T10:55:24.642 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-core-5.14.0-409.el9.x86_6 23 MB/s | 33 MB 00:01 2024-01-20T10:55:24.645 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T10:55:24.645 INFO:teuthology.orchestra.run.smithi134.stdout:Total 53 MB/s | 99 MB 00:01 2024-01-20T10:55:24.646 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T10:55:24.922 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): kernel-core-5.14.0-409.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-01-20T10:55:25.009 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T10:55:25.010 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T10:55:25.481 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): kernel-modules-5.14.0-409.el9.x86_64.rpm 27 MB/s | 39 MB 00:01 2024-01-20T10:55:25.914 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): kernel-modules-core-5.14.0-409.el9.x86_6 24 MB/s | 33 MB 00:01 2024-01-20T10:55:25.918 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T10:55:25.918 INFO:teuthology.orchestra.run.smithi110.stdout:Total 53 MB/s | 99 MB 00:01 2024-01-20T10:55:25.918 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T10:55:26.030 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T10:55:26.031 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T10:55:26.281 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T10:55:26.281 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T10:55:27.336 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T10:55:27.337 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T10:55:28.431 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T10:55:29.089 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-409.el9.x86_64 1/4 2024-01-20T10:55:29.251 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-20T10:55:29.744 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T10:55:30.562 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-core-5.14.0-409.el9.x86_64 1/4 2024-01-20T10:55:30.692 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-20T10:55:32.031 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-20T10:55:32.230 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-20T10:55:33.459 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-20T10:55:33.636 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-20T10:55:36.946 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-20T10:55:37.056 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:55:38.300 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-20T10:55:38.490 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:55:41.789 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:55:43.180 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:56:10.856 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:56:10.857 INFO:teuthology.orchestra.run.smithi134.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-20T10:56:10.857 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T10:56:10.924 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:56:11.370 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:56:11.371 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-409.el9.x86_64 1/4 2024-01-20T10:56:11.371 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-20T10:56:11.371 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-20T10:56:11.810 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:56:11.810 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T10:56:11.810 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T10:56:11.810 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-409.el9.x86_64 2024-01-20T10:56:11.810 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-409.el9.x86_64 2024-01-20T10:56:11.810 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-409.el9.x86_64 2024-01-20T10:56:11.810 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-409.el9.x86_64 2024-01-20T10:56:11.811 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T10:56:11.811 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T10:56:12.341 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-01-20T10:56:12.418 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:56:12.418 INFO:teuthology.orchestra.run.smithi110.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-20T10:56:12.419 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T10:56:12.464 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-20T10:56:12.465 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-20T10:56:12.465 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-409.el9.x86_64 2024-01-20T10:56:12.466 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-20T10:56:12.466 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-20T10:56:12.466 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-01-20T10:56:12.466 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-01-20T10:56:12.466 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-01-20T10:56:12.487 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:56:12.573 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-20T10:56:12.574 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-20T10:56:12.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-01-20T10:56:12.607 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-01-20T10:56:12.607 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-20T10:56:12.608 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T10:56:12.609 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-01-20T10:56:12.610 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-20T10:56:12.610 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-409.el9.x86_64 2024-01-20T10:56:12.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-20T10:56:12.884 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:56:12.884 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-409.el9.x86_64 1/4 2024-01-20T10:56:12.884 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-20T10:56:12.884 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-20T10:56:13.185 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-01-20T10:56:13.336 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-20T10:56:13.336 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T10:56:13.336 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T10:56:13.337 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-409.el9.x86_64 2024-01-20T10:56:13.337 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core-5.14.0-409.el9.x86_64 2024-01-20T10:56:13.337 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-5.14.0-409.el9.x86_64 2024-01-20T10:56:13.337 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core-5.14.0-409.el9.x86_64 2024-01-20T10:56:13.337 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T10:56:13.337 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T10:56:13.870 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-01-20T10:56:13.992 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-20T10:56:13.993 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-20T10:56:13.993 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-409.el9.x86_64 2024-01-20T10:56:13.993 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-20T10:56:13.993 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-20T10:56:13.993 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-20T10:56:13.993 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-01-20T10:56:13.993 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-01-20T10:56:14.101 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-20T10:56:14.102 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-20T10:56:14.102 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.06 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 68.el9 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Group : Unspecified 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 8264254 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-20T10:56:14.136 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : builder@centos.org 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T10:56:14.137 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2024-01-20T10:56:14.138 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-20T10:56:14.138 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 5.14.0-409.el9.x86_64 2024-01-20T10:56:14.138 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-20T10:56:14.553 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-20T10:56:14.579 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-01-20T10:56:14.582 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-01-20T10:56:14.597 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.XVSgsZcRD1 2024-01-20T10:56:14.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XVSgsZcRD1 2024-01-20T10:56:14.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.XVSgsZcRD1 2024-01-20T10:56:14.769 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-01-20T10:56:14.808 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.XVSgsZcRD1 is 6KB 2024-01-20T10:56:14.856 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.XVSgsZcRD1 2024-01-20T10:56:14.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-01-20T10:56:14.934 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-20T10:56:14.934 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-20T10:56:14.935 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-20T10:56:14.935 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64.conf 2024-01-20T10:56:14.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64 2024-01-20T10:56:15.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-01-20T10:56:16.221 INFO:teuthology.orchestra.run.smithi110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-20T10:56:16.270 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-01-20T10:56:16.272 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-01-20T10:56:16.287 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.BPqOzqK0Hn 2024-01-20T10:56:16.288 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BPqOzqK0Hn 2024-01-20T10:56:16.363 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.BPqOzqK0Hn 2024-01-20T10:56:16.501 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.BPqOzqK0Hn is 6KB 2024-01-20T10:56:16.548 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.BPqOzqK0Hn 2024-01-20T10:56:16.562 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2024-01-20T10:56:16.635 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-20T10:56:16.635 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-20T10:56:16.635 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-20T10:56:16.635 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64.conf 2024-01-20T10:56:16.636 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64 2024-01-20T10:56:16.795 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-01-20T10:56:45.123 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-20T10:56:45.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-20T10:56:45.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:56:46.799 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-20T10:56:46.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-20T10:56:46.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:57:03.548 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-01-20T10:57:05.340 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-01-20T10:57:12.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-20T10:57:12.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:57:14.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-20T10:57:14.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:57:15.616 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-01-20T10:57:17.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-01-20T10:57:27.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-20T10:57:27.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:57:29.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-20T10:57:29.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:57:46.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-01-20T10:57:47.840 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-01-20T10:58:01.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-20T10:58:01.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:58:01.503 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-01-20T10:58:01.951 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-20T10:58:01.951 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-409.el9.x86_64"... 2024-01-20T10:58:01.951 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-01-20T10:58:01.965 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-409.el9.x86_64 2024-01-20T10:58:01.965 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-409.el9.x86_64 vs 5.14.0-409.el9.x86_64 2024-01-20T10:58:01.965 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-20T10:58:01.966 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-20T10:58:02.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-20T10:58:02.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T10:58:02.977 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-20T10:58:02.977 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-20T10:58:03.151 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-01-20T10:58:03.195 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-01-20T10:58:03.228 DEBUG:teuthology.parallel:result is None 2024-01-20T10:58:03.635 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-20T10:58:03.636 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-409.el9.x86_64"... 2024-01-20T10:58:03.636 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-01-20T10:58:03.650 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-409.el9.x86_64 2024-01-20T10:58:03.650 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-409.el9.x86_64 vs 5.14.0-409.el9.x86_64 2024-01-20T10:58:03.650 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-20T10:58:03.650 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-20T10:58:04.651 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-20T10:58:04.651 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-20T10:58:04.761 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-01-20T10:58:04.798 DEBUG:teuthology.parallel:result is None 2024-01-20T10:58:04.798 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-20T10:58:04.805 INFO:teuthology.task.internal:Creating test directory... 2024-01-20T10:58:04.806 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-20T10:58:04.808 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-20T10:58:04.832 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-20T10:58:04.839 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-20T10:58:04.887 INFO:teuthology.task.internal:Creating archive directory... 2024-01-20T10:58:04.888 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-20T10:58:04.891 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-20T10:58:04.934 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-20T10:58:04.942 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-20T10:58:04.942 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-20T10:58:04.969 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-20T10:58:05.023 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-20T10:58:05.026 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-20T10:58:05.040 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-20T10:58:05.043 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-20T10:58:05.045 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-20T10:58:05.053 INFO:teuthology.task.internal:Configuring sudo... 2024-01-20T10:58:05.054 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-20T10:58:05.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-20T10:58:05.098 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-20T10:58:05.113 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-20T10:58:05.114 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-20T10:58:05.140 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-20T10:58:05.163 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-20T10:58:05.255 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-20T10:58:05.326 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-20T10:58:05.375 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-20T10:58:05.416 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T10:58:05.416 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-20T10:58:05.482 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-20T10:58:05.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-20T10:58:05.574 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-20T10:58:05.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-20T10:58:05.698 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T10:58:05.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-20T10:58:05.764 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-01-20T10:58:05.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-01-20T10:58:05.849 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-20T10:58:05.856 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-20T10:58:06.395 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-20T10:58:06.403 INFO:teuthology.task.internal:Starting timer... 2024-01-20T10:58:06.404 INFO:teuthology.run_tasks:Running task pcp... 2024-01-20T10:58:06.439 INFO:teuthology.run_tasks:Running task selinux... 2024-01-20T10:58:06.489 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-01-20T10:58:06.538 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-01-20T10:58:06.539 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-01-20T10:58:06.591 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-01-20T10:58:06.593 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-20T10:58:06.593 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-01-20T10:58:06.617 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-01-20T10:58:06.617 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-01-20T10:58:06.637 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-01-20T10:58:06.637 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-01-20T10:58:06.638 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-20T10:58:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-20T10:58:06.686 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-20T10:58:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-20T10:58:06.713 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-20T10:58:06.714 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-01-20T10:58:06.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-01-20T10:58:06.789 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-20T10:58:06.799 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-20T10:58:06.804 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-20T10:58:06.918 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-20T10:58:06.933 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-20T10:58:06.934 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi110.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-20T11:02:04.576 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-01-20T11:02:04.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-20T11:02:04.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T11:02:04.666 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-01-20T11:02:04.742 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-20T11:02:04.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-20T11:02:04.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-20T11:02:04.838 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-01-20T11:02:04.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-20T11:02:04.877 INFO:teuthology.run_tasks:Running task clock... 2024-01-20T11:02:04.888 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-20T11:02:04.888 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-20T11:02:04.889 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-20T11:02:04.891 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-20T11:02:04.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-20T11:02:04.920 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-20T11:02:04.936 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-20T11:02:04.949 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-20T11:02:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-01-20T11:02:04.966 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-20T11:02:04.978 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-01-20T11:02:04.994 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-20T11:02:04.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-01-20T11:02:05.009 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-01-20T11:02:05.010 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-20T11:02:05.026 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-20T11:02:05.042 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-20T11:02:05.060 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-01-20T11:02:05.063 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-20T11:02:05.063 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-01-20T11:02:05.063 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:02:05.064 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:02:05.064 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:02:05.064 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:02:05.091 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-01-20T11:02:05.095 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-20T11:02:05.095 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-01-20T11:02:05.095 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:02:05.095 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:02:05.095 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:02:05.095 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:02:05.096 INFO:teuthology.run_tasks:Running task install... 2024-01-20T11:02:05.104 DEBUG:teuthology.task.install:project ceph 2024-01-20T11:02:05.104 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167'}} 2024-01-20T11:02:05.104 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167'} 2024-01-20T11:02:05.104 INFO:teuthology.task.install:Using flavor: default 2024-01-20T11:02:05.110 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-01-20T11:02:05.110 INFO:teuthology.task.install:extra packages: [] 2024-01-20T11:02:05.110 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167', 'tag': None, 'wait_for_package': False} 2024-01-20T11:02:05.111 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a00c9940991c5e1e9ab595faa3f6e1281d8ae167 2024-01-20T11:02:05.113 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167', 'tag': None, 'wait_for_package': False} 2024-01-20T11:02:05.113 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a00c9940991c5e1e9ab595faa3f6e1281d8ae167 2024-01-20T11:02:05.257 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/centos/9/flavors/default/ 2024-01-20T11:02:05.257 INFO:teuthology.task.install.rpm:Package version is 19.0.0-763.ga00c9940 2024-01-20T11:02:05.325 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/centos/9/flavors/default/ 2024-01-20T11:02:05.325 INFO:teuthology.task.install.rpm:Package version is 19.0.0-763.ga00c9940 2024-01-20T11:02:05.374 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-20T11:02:05.375 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:02:05.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-20T11:02:05.403 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-01-20T11:02:05.403 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-20T11:02:05.456 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-20T11:02:05.457 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:02:05.457 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-20T11:02:05.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-20T11:02:05.484 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-01-20T11:02:05.484 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a00c9940991c5e1e9ab595faa3f6e1281d8ae167/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-20T11:02:05.552 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-20T11:02:05.552 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-20T11:02:05.619 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-01-20T11:02:05.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-01-20T11:02:05.632 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-20T11:02:05.698 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-01-20T11:02:05.699 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-01-20T11:02:05.852 INFO:teuthology.orchestra.run.smithi134.stdout:73 files removed 2024-01-20T11:02:05.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-01-20T11:02:05.939 INFO:teuthology.orchestra.run.smithi110.stdout:73 files removed 2024-01-20T11:02:05.966 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw 2024-01-20T11:02:06.668 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 171 kB/s | 80 kB 00:00 2024-01-20T11:02:06.687 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 196 kB/s | 80 kB 00:00 2024-01-20T11:02:07.072 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 28 kB/s | 9.5 kB 00:00 2024-01-20T11:02:07.093 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2024-01-20T11:02:07.469 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-01-20T11:02:07.506 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-01-20T11:02:07.888 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2024-01-20T11:02:07.916 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2024-01-20T11:02:09.475 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-01-20T11:02:09.489 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-01-20T11:02:21.916 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 672 kB/s | 8.0 MB 00:12 2024-01-20T11:02:29.291 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-01-20T11:02:33.433 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 5.9 MB 00:00 2024-01-20T11:02:35.084 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-01-20T11:03:15.877 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 509 kB/s | 20 MB 00:40 2024-01-20T11:03:20.599 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-01-20T11:03:23.125 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:03:23.133 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:03:23.133 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:03:23.133 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:03:23.133 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:03:23.133 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:19.0.0-763.ga00c9940.el9 ceph 8.6 M 2024-01-20T11:03:23.133 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-01-20T11:03:23.133 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 3.4 M 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 3.1 M 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:19.0.0-763.ga00c9940.el9 ceph 5.1 M 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:19.0.0-763.ga00c9940.el9 ceph 19 M 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:19.0.0-763.ga00c9940.el9 ceph 25 k 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 742 k 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-20T11:03:23.134 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 499 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 4.8 M 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:19.0.0-763.ga00c9940.el9 ceph 45 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:19.0.0-763.ga00c9940.el9 ceph 130 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:19.0.0-763.ga00c9940.el9 ceph 162 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:19.0.0-763.ga00c9940.el9 ceph 320 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:19.0.0-763.ga00c9940.el9 ceph 297 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:19.0.0-763.ga00c9940.el9 ceph 99 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-20T11:03:23.135 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:23.136 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:03:23.136 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:03:23.136 INFO:teuthology.orchestra.run.smithi110.stdout:Install 27 Packages 2024-01-20T11:03:23.136 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 2 Packages 2024-01-20T11:03:23.136 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:23.136 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 55 M 2024-01-20T11:03:23.136 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:03:24.542 INFO:teuthology.orchestra.run.smithi110.stdout:(1/29): ceph-base-19.0.0-763.ga00c9940.el9.x86_ 3.8 MB/s | 5.1 MB 00:01 2024-01-20T11:03:24.600 INFO:teuthology.orchestra.run.smithi110.stdout:(2/29): ceph-selinux-19.0.0-763.ga00c9940.el9.x 431 kB/s | 25 kB 00:00 2024-01-20T11:03:24.926 INFO:teuthology.orchestra.run.smithi110.stdout:(3/29): ceph-common-19.0.0-763.ga00c9940.el9.x8 11 MB/s | 19 MB 00:01 2024-01-20T11:03:24.968 INFO:teuthology.orchestra.run.smithi110.stdout:(4/29): libcephfs2-19.0.0-763.ga00c9940.el9.x86 2.0 MB/s | 742 kB 00:00 2024-01-20T11:03:25.060 INFO:teuthology.orchestra.run.smithi110.stdout:(5/29): ceph-radosgw-19.0.0-763.ga00c9940.el9.x 4.6 MB/s | 8.6 MB 00:01 2024-01-20T11:03:25.085 INFO:teuthology.orchestra.run.smithi110.stdout:(6/29): libradosstriper1-19.0.0-763.ga00c9940.e 3.1 MB/s | 499 kB 00:00 2024-01-20T11:03:25.118 INFO:teuthology.orchestra.run.smithi110.stdout:(7/29): python3-ceph-argparse-19.0.0-763.ga00c9 772 kB/s | 45 kB 00:00 2024-01-20T11:03:25.144 INFO:teuthology.orchestra.run.smithi110.stdout:(8/29): python3-ceph-common-19.0.0-763.ga00c994 2.2 MB/s | 130 kB 00:00 2024-01-20T11:03:25.178 INFO:teuthology.orchestra.run.smithi110.stdout:(9/29): python3-cephfs-19.0.0-763.ga00c9940.el9 2.7 MB/s | 162 kB 00:00 2024-01-20T11:03:25.203 INFO:teuthology.orchestra.run.smithi110.stdout:(10/29): python3-rados-19.0.0-763.ga00c9940.el9 5.3 MB/s | 320 kB 00:00 2024-01-20T11:03:25.245 INFO:teuthology.orchestra.run.smithi110.stdout:(11/29): python3-rbd-19.0.0-763.ga00c9940.el9.x 4.3 MB/s | 297 kB 00:00 2024-01-20T11:03:25.270 INFO:teuthology.orchestra.run.smithi110.stdout:(12/29): python3-rgw-19.0.0-763.ga00c9940.el9.x 1.5 MB/s | 99 kB 00:00 2024-01-20T11:03:25.387 INFO:teuthology.orchestra.run.smithi110.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-01-20T11:03:25.412 INFO:teuthology.orchestra.run.smithi110.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-01-20T11:03:25.429 INFO:teuthology.orchestra.run.smithi110.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-01-20T11:03:25.480 INFO:teuthology.orchestra.run.smithi110.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-01-20T11:03:25.505 INFO:teuthology.orchestra.run.smithi110.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-01-20T11:03:25.530 INFO:teuthology.orchestra.run.smithi110.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-01-20T11:03:25.608 INFO:teuthology.orchestra.run.smithi110.stdout:(19/29): librgw2-19.0.0-763.ga00c9940.el9.x86_6 7.5 MB/s | 4.8 MB 00:00 2024-01-20T11:03:25.789 INFO:teuthology.orchestra.run.smithi110.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 191 kB/s | 26 kB 00:00 2024-01-20T11:03:25.814 INFO:teuthology.orchestra.run.smithi110.stdout:(21/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-01-20T11:03:25.839 INFO:teuthology.orchestra.run.smithi110.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 982 kB/s | 49 kB 00:00 2024-01-20T11:03:25.865 INFO:teuthology.orchestra.run.smithi110.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-01-20T11:03:25.923 INFO:teuthology.orchestra.run.smithi110.stdout:(24/29): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-01-20T11:03:25.957 INFO:teuthology.orchestra.run.smithi110.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.1 MB/s | 838 kB 00:00 2024-01-20T11:03:25.982 INFO:teuthology.orchestra.run.smithi110.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-01-20T11:03:26.024 INFO:teuthology.orchestra.run.smithi110.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-01-20T11:03:26.249 INFO:teuthology.orchestra.run.smithi110.stdout:(28/29): librbd1-19.0.0-763.ga00c9940.el9.x86_6 12 MB/s | 3.1 MB 00:00 2024-01-20T11:03:26.383 INFO:teuthology.orchestra.run.smithi110.stdout:(29/29): librados2-19.0.0-763.ga00c9940.el9.x86 7.9 MB/s | 3.4 MB 00:00 2024-01-20T11:03:26.389 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:03:26.389 INFO:teuthology.orchestra.run.smithi110.stdout:Total 17 MB/s | 55 MB 00:03 2024-01-20T11:03:26.389 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:03:26.419 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:03:26.419 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:03:26.889 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:03:26.889 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:03:28.273 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:03:28.369 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-20T11:03:28.415 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-01-20T11:03:28.464 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-01-20T11:03:28.843 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-01-20T11:03:28.872 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:19.0.0-763.ga00c9940.el9.x86_64 5/31 2024-01-20T11:03:28.964 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:19.0.0-763.ga00c9940.el9.x86_64 5/31 2024-01-20T11:03:29.050 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:19.0.0-763.ga00c9940.el9.x86_64 6/31 2024-01-20T11:03:29.082 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 7/31 2024-01-20T11:03:29.128 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 7/31 2024-01-20T11:03:29.156 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-01-20T11:03:29.192 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:19.0.0-763.ga00c9940.el9.x 9/31 2024-01-20T11:03:29.260 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:19.0.0-763.ga00c9940.el9.x86_64 10/31 2024-01-20T11:03:29.284 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:19.0.0-763.ga00c9940.el9.x86_64 11/31 2024-01-20T11:03:29.349 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:19.0.0-763.ga00c9940.el9.x86_64 11/31 2024-01-20T11:03:29.455 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-20T11:03:29.483 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-20T11:03:29.524 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-20T11:03:29.556 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-20T11:03:29.577 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-20T11:03:29.610 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-20T11:03:29.918 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-20T11:03:29.948 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 19/31 2024-01-20T11:03:30.037 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 19/31 2024-01-20T11:03:30.069 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:19.0.0-763.ga00c9940.el9.x86_64 20/31 2024-01-20T11:03:30.108 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-20T11:03:30.531 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-20T11:03:30.558 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:19.0.0-763.ga00c9940.el9.x86_64 23/31 2024-01-20T11:03:30.621 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:19.0.0-763.ga00c9940.el9.x86_64 23/31 2024-01-20T11:03:30.688 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:19.0.0-763.ga00c9940.el9.x86_64 24/31 2024-01-20T11:03:30.719 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:19.0.0-763.ga00c9940.el9.x86 25/31 2024-01-20T11:03:32.814 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 26/31 2024-01-20T11:03:32.849 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 26/31 2024-01-20T11:03:33.369 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 26/31 2024-01-20T11:03:33.402 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 27/31 2024-01-20T11:03:33.500 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 27/31 2024-01-20T11:03:33.500 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-20T11:03:33.500 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-20T11:03:33.501 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:33.536 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:19.0.0-763.ga00c9940.el9.x86_64 28/31 2024-01-20T11:03:43.060 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:19.0.0-763.ga00c9940.el9.x86_64 28/31 2024-01-20T11:03:43.060 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-01-20T11:03:43.060 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-01-20T11:03:43.060 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-01-20T11:03:43.060 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-01-20T11:03:43.060 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-01-20T11:03:43.060 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-01-20T11:03:43.061 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-01-20T11:03:43.061 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:43.894 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:19.0.0-763.ga00c9940.el9.x86_64 29/31 2024-01-20T11:03:43.918 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-763.ga00c9940.el9.x86_64 29/31 2024-01-20T11:03:43.918 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:03:43.918 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-20T11:03:43.919 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-20T11:03:43.919 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-20T11:03:43.919 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:43.920 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-20T11:03:43.997 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-20T11:03:43.997 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-20T11:03:45.134 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-20T11:03:45.134 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 1/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 2/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:19.0.0-763.ga00c9940.el9.x86_64 3/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:19.0.0-763.ga00c9940.el9.x86_64 4/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 5/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:19.0.0-763.ga00c9940.el9.x86_64 6/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:19.0.0-763.ga00c9940.el9.x86_64 7/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:19.0.0-763.ga00c9940.el9.x 8/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:19.0.0-763.ga00c9940.el9.x86 9/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:19.0.0-763.ga00c9940.el9.x86_64 10/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:19.0.0-763.ga00c9940.el9.x86_64 11/31 2024-01-20T11:03:45.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:19.0.0-763.ga00c9940.el9.x86_64 12/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:19.0.0-763.ga00c9940.el9.x86_64 13/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-20T11:03:45.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-20T11:03:45.137 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-20T11:03:45.137 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-20T11:03:45.137 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-20T11:03:45.137 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-20T11:03:45.137 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-20T11:03:45.137 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:19.0.0-763.ga00c9940.el9.x86_64 28/31 2024-01-20T11:03:45.137 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-20T11:03:45.137 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 30/31 2024-01-20T11:03:46.144 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.145 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.146 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:46.147 INFO:teuthology.orchestra.run.smithi110.stdout: re2-1:20211101-3.el9.x86_64 2024-01-20T11:03:46.148 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-20T11:03:46.148 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:46.148 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:03:46.525 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test 2024-01-20T11:03:47.078 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:27 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:03:47.229 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:03:47.230 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:03:47.230 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:03:47.230 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:03:47.230 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:03:47.230 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:19.0.0-763.ga00c9940.el9 ceph 46 M 2024-01-20T11:03:47.230 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T11:03:47.230 INFO:teuthology.orchestra.run.smithi110.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-20T11:03:47.230 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:19.0.0-763.ga00c9940.el9 ceph 163 k 2024-01-20T11:03:47.230 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-20T11:03:47.231 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-20T11:03:47.231 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-20T11:03:47.231 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:47.231 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:03:47.231 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:03:47.231 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-01-20T11:03:47.231 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:47.231 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 47 M 2024-01-20T11:03:47.232 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 201 M 2024-01-20T11:03:47.232 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:03:47.422 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-01-20T11:03:47.458 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.7 MB/s | 63 kB 00:00 2024-01-20T11:03:47.501 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-01-20T11:03:47.542 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-01-20T11:03:47.634 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): libcephsqlite-19.0.0-763.ga00c9940.el9.x 407 kB/s | 163 kB 00:00 2024-01-20T11:03:49.518 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): ceph-test-19.0.0-763.ga00c9940.el9.x86_6 20 MB/s | 46 MB 00:02 2024-01-20T11:03:49.520 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:03:49.521 INFO:teuthology.orchestra.run.smithi110.stdout:Total 20 MB/s | 47 MB 00:02 2024-01-20T11:03:49.521 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:03:49.541 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:03:49.541 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:03:49.931 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:03:49.931 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:03:50.513 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:03:50.582 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-20T11:03:50.617 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-20T11:03:50.678 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-20T11:03:50.726 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-20T11:03:50.754 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:19.0.0-763.ga00c9940.el9.x86_64 5/6 2024-01-20T11:03:55.128 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:19.0.0-763.ga00c9940.el9.x86_64 5/6 2024-01-20T11:03:55.204 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:19.0.0-763.ga00c9940.el9.x86_64 6/6 2024-01-20T11:03:57.633 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-2:19.0.0-763.ga00c9940.el9.x86_64 6/6 2024-01-20T11:03:57.634 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:19.0.0-763.ga00c9940.el9.x86_64 1/6 2024-01-20T11:03:57.634 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:19.0.0-763.ga00c9940.el9.x86_64 2/6 2024-01-20T11:03:57.634 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-20T11:03:57.634 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-20T11:03:57.634 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-20T11:03:58.022 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-20T11:03:58.022 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:58.023 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:03:58.023 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:58.023 INFO:teuthology.orchestra.run.smithi110.stdout: jq-1.6-16.el9.x86_64 2024-01-20T11:03:58.023 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:03:58.023 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-20T11:03:58.023 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-20T11:03:58.023 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-20T11:03:58.023 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:58.023 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:03:58.337 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph 2024-01-20T11:03:58.893 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:38 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:03:59.051 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:03:59.054 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-01-20T11:03:59.054 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:03:59.054 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-01-20T11:03:59.054 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:03:59.054 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:19.0.0-763.ga00c9940.el9 ceph 6.6 k 2024-01-20T11:03:59.054 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T11:03:59.054 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:19.0.0-763.ga00c9940.el9 ceph 2.1 M 2024-01-20T11:03:59.054 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:19.0.0-763.ga00c9940.el9 ceph 1.5 M 2024-01-20T11:03:59.054 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 245 k 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:19.0.0-763.ga00c9940.el9 ceph 4.7 M 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:19.0.0-763.ga00c9940.el9 ceph 17 M 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-20T11:03:59.055 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-20T11:03:59.056 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout:Install 36 Packages 2024-01-20T11:03:59.057 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:03:59.058 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 30 M 2024-01-20T11:03:59.058 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 106 M 2024-01-20T11:03:59.058 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:07.620 INFO:teuthology.orchestra.run.smithi110.stdout:(1/36): ceph-19.0.0-763.ga00c9940.el9.x86_64.rp 41 kB/s | 6.6 kB 00:00 2024-01-20T11:04:08.045 INFO:teuthology.orchestra.run.smithi110.stdout:(2/36): ceph-mon-19.0.0-763.ga00c9940.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-01-20T11:04:08.103 INFO:teuthology.orchestra.run.smithi110.stdout:(3/36): ceph-mds-19.0.0-763.ga00c9940.el9.x86_6 3.3 MB/s | 2.1 MB 00:00 2024-01-20T11:04:08.145 INFO:teuthology.orchestra.run.smithi110.stdout:(4/36): ceph-mgr-19.0.0-763.ga00c9940.el9.x86_6 2.2 MB/s | 1.5 MB 00:00 2024-01-20T11:04:08.212 INFO:teuthology.orchestra.run.smithi110.stdout:(5/36): ceph-mgr-modules-core-19.0.0-763.ga00c9 2.2 MB/s | 245 kB 00:00 2024-01-20T11:04:08.271 INFO:teuthology.orchestra.run.smithi110.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-01-20T11:04:08.296 INFO:teuthology.orchestra.run.smithi110.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-01-20T11:04:08.313 INFO:teuthology.orchestra.run.smithi110.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 759 kB/s | 32 kB 00:00 2024-01-20T11:04:08.330 INFO:teuthology.orchestra.run.smithi110.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-01-20T11:04:08.347 INFO:teuthology.orchestra.run.smithi110.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-01-20T11:04:08.372 INFO:teuthology.orchestra.run.smithi110.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-01-20T11:04:08.489 INFO:teuthology.orchestra.run.smithi110.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 8.1 MB/s | 1.1 MB 00:00 2024-01-20T11:04:08.514 INFO:teuthology.orchestra.run.smithi110.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 729 kB/s | 103 kB 00:00 2024-01-20T11:04:08.540 INFO:teuthology.orchestra.run.smithi110.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-20T11:04:08.565 INFO:teuthology.orchestra.run.smithi110.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-20T11:04:08.590 INFO:teuthology.orchestra.run.smithi110.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-01-20T11:04:08.924 INFO:teuthology.orchestra.run.smithi110.stdout:(17/36): ceph-osd-19.0.0-763.ga00c9940.el9.x86_ 19 MB/s | 17 MB 00:00 2024-01-20T11:04:08.949 INFO:teuthology.orchestra.run.smithi110.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 58 kB/s | 22 kB 00:00 2024-01-20T11:04:09.125 INFO:teuthology.orchestra.run.smithi110.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 283 kB/s | 151 kB 00:00 2024-01-20T11:04:09.200 INFO:teuthology.orchestra.run.smithi110.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 158 kB/s | 43 kB 00:00 2024-01-20T11:04:09.283 INFO:teuthology.orchestra.run.smithi110.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 129 kB/s | 11 kB 00:00 2024-01-20T11:04:09.334 INFO:teuthology.orchestra.run.smithi110.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 447 kB/s | 172 kB 00:00 2024-01-20T11:04:09.367 INFO:teuthology.orchestra.run.smithi110.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-01-20T11:04:09.409 INFO:teuthology.orchestra.run.smithi110.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-01-20T11:04:09.451 INFO:teuthology.orchestra.run.smithi110.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-01-20T11:04:09.485 INFO:teuthology.orchestra.run.smithi110.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-01-20T11:04:09.560 INFO:teuthology.orchestra.run.smithi110.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-01-20T11:04:09.585 INFO:teuthology.orchestra.run.smithi110.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 779 kB/s | 358 kB 00:00 2024-01-20T11:04:09.611 INFO:teuthology.orchestra.run.smithi110.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 628 kB/s | 79 kB 00:00 2024-01-20T11:04:09.669 INFO:teuthology.orchestra.run.smithi110.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-01-20T11:04:09.703 INFO:teuthology.orchestra.run.smithi110.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 979 kB/s | 90 kB 00:00 2024-01-20T11:04:09.761 INFO:teuthology.orchestra.run.smithi110.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2024-01-20T11:04:09.787 INFO:teuthology.orchestra.run.smithi110.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-01-20T11:04:09.820 INFO:teuthology.orchestra.run.smithi110.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-01-20T11:04:09.862 INFO:teuthology.orchestra.run.smithi110.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-01-20T11:04:09.987 INFO:teuthology.orchestra.run.smithi110.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-01-20T11:04:09.994 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:09.994 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.7 MB/s | 30 MB 00:10 2024-01-20T11:04:10.014 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:10.059 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:10.059 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:10.470 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:10.471 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:11.840 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:11.891 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-20T11:04:11.923 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-20T11:04:11.951 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-20T11:04:12.021 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-20T11:04:12.066 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-20T11:04:12.099 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-20T11:04:12.199 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-20T11:04:12.269 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-20T11:04:12.313 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-20T11:04:12.385 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-20T11:04:12.439 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-20T11:04:12.477 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-20T11:04:12.534 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-20T11:04:12.602 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-20T11:04:12.791 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-20T11:04:12.840 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-20T11:04:12.870 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-20T11:04:12.905 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-20T11:04:12.943 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-20T11:04:13.019 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-20T11:04:13.053 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-20T11:04:13.086 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-20T11:04:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-20T11:04:13.162 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-20T11:04:13.196 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-20T11:04:13.288 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-20T11:04:13.327 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-20T11:04:13.399 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-20T11:04:13.538 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-20T11:04:13.656 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-20T11:04:13.814 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-763.ga00c9940.el9.n 31/36 2024-01-20T11:04:13.835 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 32/36 2024-01-20T11:04:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 32/36 2024-01-20T11:04:13.859 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:04:13.859 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-20T11:04:13.860 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-20T11:04:13.860 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-20T11:04:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:15.354 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:19.0.0-763.ga00c9940.el9.x86_64 33/36 2024-01-20T11:04:15.381 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:19.0.0-763.ga00c9940.el9.x86_64 33/36 2024-01-20T11:04:15.381 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:04:15.381 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-20T11:04:15.381 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-20T11:04:15.381 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-20T11:04:15.381 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:15.821 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:19.0.0-763.ga00c9940.el9.x86_64 34/36 2024-01-20T11:04:15.844 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:19.0.0-763.ga00c9940.el9.x86_64 34/36 2024-01-20T11:04:15.844 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:04:15.844 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-20T11:04:15.844 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-20T11:04:15.844 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-20T11:04:15.844 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:16.055 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:19.0.0-763.ga00c9940.el9.x86_64 35/36 2024-01-20T11:04:16.078 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:19.0.0-763.ga00c9940.el9.x86_64 35/36 2024-01-20T11:04:16.078 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:04:16.078 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-20T11:04:16.078 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-20T11:04:16.078 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-20T11:04:16.078 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:16.144 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:19.0.0-763.ga00c9940.el9.x86_64 36/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-2:19.0.0-763.ga00c9940.el9.x86_64 36/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:19.0.0-763.ga00c9940.el9.x86_64 1/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:19.0.0-763.ga00c9940.el9.x86_64 2/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 3/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:19.0.0-763.ga00c9940.el9.x86_64 4/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:19.0.0-763.ga00c9940.el9.x86_64 5/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-763.ga00c9940.el9.n 6/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-20T11:04:17.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-20T11:04:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-20T11:04:17.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-20T11:04:17.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-20T11:04:17.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-20T11:04:17.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-20T11:04:17.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-20T11:04:17.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-20T11:04:17.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-20T11:04:17.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-20T11:04:17.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-20T11:04:17.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-20T11:04:17.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-20T11:04:18.823 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-20T11:04:18.823 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:18.823 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:18.823 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:04:18.823 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:04:18.823 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:04:18.823 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:04:18.823 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: lua-5.4.4-4.el9.x86_64 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-20T11:04:18.824 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-20T11:04:18.825 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-20T11:04:18.826 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-20T11:04:18.826 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-20T11:04:18.826 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-20T11:04:18.826 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-20T11:04:18.826 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:18.826 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:19.126 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base 2024-01-20T11:04:19.689 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:59 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:19.785 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:04:19.835 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:19.835 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:04:19.835 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:19.910 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install cephadm 2024-01-20T11:04:20.465 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:00 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:20.613 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:20.614 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:20.614 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:04:20.614 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:20.614 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:20.614 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 742 k 2024-01-20T11:04:20.614 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:20.614 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:20.614 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:20.614 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:04:20.615 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:20.615 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 742 k 2024-01-20T11:04:20.615 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 748 k 2024-01-20T11:04:20.615 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:20.913 INFO:teuthology.orchestra.run.smithi110.stdout:cephadm-19.0.0-763.ga00c9940.el9.noarch.rpm 2.4 MB/s | 742 kB 00:00 2024-01-20T11:04:20.914 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:20.914 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.4 MB/s | 742 kB 00:00 2024-01-20T11:04:20.914 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:20.919 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:20.920 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:20.925 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:20.926 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:20.997 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:21.413 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:04:21.501 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:04:21.937 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:04:22.247 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:04:22.247 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:22.248 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:22.248 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:04:22.248 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:22.248 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:22.469 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache 2024-01-20T11:04:23.035 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:03 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:23.185 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-763.ga00c9940.el9 ceph 145 k 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 145 k 2024-01-20T11:04:23.186 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 436 k 2024-01-20T11:04:23.187 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:23.528 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-19.0.0-763.ga00c994 424 kB/s | 145 kB 00:00 2024-01-20T11:04:23.529 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:23.529 INFO:teuthology.orchestra.run.smithi110.stdout:Total 422 kB/s | 145 kB 00:00 2024-01-20T11:04:23.529 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:23.537 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:23.537 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:23.589 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:23.589 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:23.620 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 61 kB/s | 8.0 MB 02:13 2024-01-20T11:04:23.728 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:23.788 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-763.ga00c9940.e 1/1 2024-01-20T11:04:23.812 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-763.ga00c9940.e 1/1 2024-01-20T11:04:23.812 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:04:23.812 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-20T11:04:23.812 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:24.840 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-763.ga00c9940.e 1/1 2024-01-20T11:04:24.840 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:24.840 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:24.841 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:04:24.841 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:24.841 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:25.095 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr 2024-01-20T11:04:25.670 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:05 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:25.771 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:04:25.821 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:25.822 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:04:25.822 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:25.889 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-dashboard 2024-01-20T11:04:26.443 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:06 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:26.593 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 1.7 M 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 28 k 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 15 k 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-20T11:04:26.594 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:26.595 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:26.595 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:26.595 INFO:teuthology.orchestra.run.smithi110.stdout:Install 5 Packages 2024-01-20T11:04:26.595 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:26.595 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.0 M 2024-01-20T11:04:26.595 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 8.5 M 2024-01-20T11:04:26.595 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:26.875 INFO:teuthology.orchestra.run.smithi110.stdout:(1/5): ceph-prometheus-alerts-19.0.0-763.ga00c9 76 kB/s | 15 kB 00:00 2024-01-20T11:04:26.975 INFO:teuthology.orchestra.run.smithi110.stdout:(2/5): ceph-grafana-dashboards-19.0.0-763.ga00c 95 kB/s | 28 kB 00:00 2024-01-20T11:04:27.059 INFO:teuthology.orchestra.run.smithi110.stdout:(3/5): ceph-mgr-dashboard-19.0.0-763.ga00c9940. 4.5 MB/s | 1.7 MB 00:00 2024-01-20T11:04:27.084 INFO:teuthology.orchestra.run.smithi110.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 147 kB/s | 31 kB 00:00 2024-01-20T11:04:27.201 INFO:teuthology.orchestra.run.smithi110.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 840 kB/s | 188 kB 00:00 2024-01-20T11:04:27.204 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:27.204 INFO:teuthology.orchestra.run.smithi110.stdout:Total 3.3 MB/s | 2.0 MB 00:00 2024-01-20T11:04:27.204 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:27.211 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:27.211 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:27.268 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:27.269 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:27.505 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:27.590 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-20T11:04:27.623 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-20T11:04:27.652 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-763.ga00c9940.el9.no 3/5 2024-01-20T11:04:27.885 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-763.ga00c9940.el9.n 4/5 2024-01-20T11:04:27.921 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-763.ga00c9940.el9.noarch 5/5 2024-01-20T11:04:28.458 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-763.ga00c9940.el9.noarch 5/5 2024-01-20T11:04:28.459 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-763.ga00c9940.el9.n 1/5 2024-01-20T11:04:28.459 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-763.ga00c9940.el9.noarch 2/5 2024-01-20T11:04:28.459 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-763.ga00c9940.el9.no 3/5 2024-01-20T11:04:28.459 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-20T11:04:28.736 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-20T11:04:28.736 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:28.736 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:28.736 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:04:28.737 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:04:28.737 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:04:28.737 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-20T11:04:28.737 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-20T11:04:28.737 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:28.737 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:28.985 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-20T11:04:29.537 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:09 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:29.689 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:29.690 INFO:teuthology.orchestra.run.smithi110.stdout:=========================================================================================== 2024-01-20T11:04:29.690 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout:=========================================================================================== 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 7.4 M 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-20T11:04:29.691 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-20T11:04:29.692 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-20T11:04:29.692 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:29.692 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:29.692 INFO:teuthology.orchestra.run.smithi110.stdout:=========================================================================================== 2024-01-20T11:04:29.692 INFO:teuthology.orchestra.run.smithi110.stdout:Install 12 Packages 2024-01-20T11:04:29.692 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:29.692 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 37 M 2024-01-20T11:04:29.693 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 209 M 2024-01-20T11:04:29.693 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:29.958 INFO:teuthology.orchestra.run.smithi110.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 236 kB/s | 32 kB 00:00 2024-01-20T11:04:29.991 INFO:teuthology.orchestra.run.smithi110.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2024-01-20T11:04:30.034 INFO:teuthology.orchestra.run.smithi110.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 849 kB/s | 34 kB 00:00 2024-01-20T11:04:30.084 INFO:teuthology.orchestra.run.smithi110.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-01-20T11:04:30.251 INFO:teuthology.orchestra.run.smithi110.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-01-20T11:04:30.326 INFO:teuthology.orchestra.run.smithi110.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-01-20T11:04:30.351 INFO:teuthology.orchestra.run.smithi110.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-01-20T11:04:30.493 INFO:teuthology.orchestra.run.smithi110.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 802 kB 00:00 2024-01-20T11:04:30.660 INFO:teuthology.orchestra.run.smithi110.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-01-20T11:04:30.685 INFO:teuthology.orchestra.run.smithi110.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-01-20T11:04:30.902 INFO:teuthology.orchestra.run.smithi110.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-7 6.9 MB/s | 7.4 MB 00:01 2024-01-20T11:04:30.961 INFO:teuthology.orchestra.run.smithi110.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 819 kB/s | 245 kB 00:00 2024-01-20T11:04:30.963 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:30.963 INFO:teuthology.orchestra.run.smithi110.stdout:Total 29 MB/s | 37 MB 00:01 2024-01-20T11:04:30.976 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:31.000 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:31.001 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:31.384 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:31.384 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:32.081 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:32.168 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-20T11:04:32.198 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-20T11:04:32.292 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-20T11:04:32.327 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-20T11:04:32.510 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-20T11:04:32.542 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-20T11:04:32.619 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-20T11:04:33.258 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-20T11:04:33.371 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-20T11:04:34.621 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-20T11:04:35.644 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-20T11:04:35.668 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-763.ga00c99 12/12 2024-01-20T11:04:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-763.ga00c99 12/12 2024-01-20T11:04:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-763.ga00c99 1/12 2024-01-20T11:04:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-20T11:04:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-20T11:04:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-20T11:04:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-20T11:04:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-20T11:04:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-20T11:04:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-20T11:04:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-20T11:04:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-20T11:04:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-20T11:04:38.585 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-20T11:04:38.586 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-20T11:04:38.587 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-20T11:04:38.587 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:38.587 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:38.940 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-rook 2024-01-20T11:04:39.505 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:19 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:39.659 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:39.660 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-20T11:04:39.660 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:04:39.660 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-20T11:04:39.660 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:39.660 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 51 k 2024-01-20T11:04:39.660 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T11:04:39.660 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-20T11:04:39.660 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-20T11:04:39.660 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:39.661 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-20T11:04:39.662 INFO:teuthology.orchestra.run.smithi110.stdout:Install 13 Packages 2024-01-20T11:04:39.662 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:39.662 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.0 M 2024-01-20T11:04:39.662 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 26 M 2024-01-20T11:04:39.663 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:40.140 INFO:teuthology.orchestra.run.smithi110.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 209 kB/s | 24 kB 00:00 2024-01-20T11:04:40.165 INFO:teuthology.orchestra.run.smithi110.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-01-20T11:04:40.208 INFO:teuthology.orchestra.run.smithi110.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-01-20T11:04:40.233 INFO:teuthology.orchestra.run.smithi110.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-01-20T11:04:40.258 INFO:teuthology.orchestra.run.smithi110.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-01-20T11:04:40.283 INFO:teuthology.orchestra.run.smithi110.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-01-20T11:04:40.309 INFO:teuthology.orchestra.run.smithi110.stdout:(7/13): ceph-mgr-rook-19.0.0-763.ga00c9940.el9. 178 kB/s | 51 kB 00:00 2024-01-20T11:04:40.518 INFO:teuthology.orchestra.run.smithi110.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-01-20T11:04:40.543 INFO:teuthology.orchestra.run.smithi110.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-01-20T11:04:40.685 INFO:teuthology.orchestra.run.smithi110.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-01-20T11:04:40.719 INFO:teuthology.orchestra.run.smithi110.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 493 kB/s | 202 kB 00:00 2024-01-20T11:04:40.802 INFO:teuthology.orchestra.run.smithi110.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2024-01-20T11:04:40.894 INFO:teuthology.orchestra.run.smithi110.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-01-20T11:04:40.898 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:40.898 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-01-20T11:04:40.898 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:40.911 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:40.911 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:41.006 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:41.007 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:41.526 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:41.697 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-20T11:04:41.744 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-20T11:04:41.799 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-20T11:04:41.838 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-20T11:04:41.878 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-20T11:04:41.971 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-20T11:04:42.073 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-20T11:04:42.122 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-20T11:04:42.843 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-20T11:04:42.894 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-20T11:04:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-20T11:04:42.976 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-20T11:04:42.998 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:19.0.0-763.ga00c9940.el9.noarch 13/13 2024-01-20T11:04:43.775 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-763.ga00c9940.el9.noarch 13/13 2024-01-20T11:04:43.775 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:19.0.0-763.ga00c9940.el9.noarch 1/13 2024-01-20T11:04:43.775 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-20T11:04:43.775 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-20T11:04:43.775 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-20T11:04:43.775 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-20T11:04:43.776 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-20T11:04:43.776 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-20T11:04:43.776 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-20T11:04:43.776 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-20T11:04:43.776 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-20T11:04:43.776 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-20T11:04:43.776 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-20T11:04:44.135 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-20T11:04:44.135 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:44.135 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-20T11:04:44.136 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:44.137 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:44.383 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-cephadm 2024-01-20T11:04:44.961 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:24 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:45.125 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:45.125 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-20T11:04:45.125 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 133 k 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-01-20T11:04:45.126 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:45.127 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 6.8 M 2024-01-20T11:04:45.127 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 33 M 2024-01-20T11:04:45.127 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:45.355 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-20T11:04:45.489 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-01-20T11:04:45.606 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): ceph-mgr-cephadm-19.0.0-763.ga00c9940.el 312 kB/s | 133 kB 00:00 2024-01-20T11:04:45.631 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 407 kB/s | 58 kB 00:00 2024-01-20T11:04:45.681 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-01-20T11:04:45.731 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 406 kB/s | 51 kB 00:00 2024-01-20T11:04:45.733 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:45.734 INFO:teuthology.orchestra.run.smithi110.stdout:Total 11 MB/s | 6.8 MB 00:00 2024-01-20T11:04:45.734 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:45.743 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:45.743 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:45.835 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:45.835 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:46.136 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:46.303 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-20T11:04:46.358 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-20T11:04:46.875 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-20T11:04:46.952 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-20T11:04:47.013 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-20T11:04:47.046 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-763.ga00c9940.el9.noarch 6/6 2024-01-20T11:04:47.875 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-763.ga00c9940.el9.noarch 6/6 2024-01-20T11:04:47.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/6 2024-01-20T11:04:47.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-20T11:04:47.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-20T11:04:47.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-20T11:04:47.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:48.178 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:48.422 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse 2024-01-20T11:04:48.509 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 800 kB/s | 19 MB 00:23 2024-01-20T11:04:48.975 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:28 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:49.127 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:49.127 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:49.127 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:04:49.127 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:49.127 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:49.127 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:19.0.0-763.ga00c9940.el9 ceph 886 k 2024-01-20T11:04:49.128 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T11:04:49.128 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-20T11:04:49.128 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:49.128 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:49.128 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:49.128 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-01-20T11:04:49.128 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:49.129 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 964 k 2024-01-20T11:04:49.129 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.8 M 2024-01-20T11:04:49.129 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:49.242 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 695 kB/s | 79 kB 00:00 2024-01-20T11:04:49.559 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-fuse-19.0.0-763.ga00c9940.el9.x86_6 2.0 MB/s | 886 kB 00:00 2024-01-20T11:04:49.560 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:49.560 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.2 MB/s | 964 kB 00:00 2024-01-20T11:04:49.561 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:49.581 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:49.581 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:49.656 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:49.656 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:49.845 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:49.969 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-20T11:04:50.042 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 2/2 2024-01-20T11:04:50.719 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 2/2 2024-01-20T11:04:50.720 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 1/2 2024-01-20T11:04:51.042 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-20T11:04:51.042 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:51.042 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:51.042 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-20T11:04:51.042 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:51.043 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:51.322 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-volume 2024-01-20T11:04:51.894 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:31 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:52.045 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:52.045 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:52.045 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-20T11:04:52.045 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 262 k 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 262 k 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.2 M 2024-01-20T11:04:52.046 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:52.297 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-volume-19.0.0-763.ga00c9940.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2024-01-20T11:04:52.298 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:52.298 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.0 MB/s | 262 kB 00:00 2024-01-20T11:04:52.298 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:52.306 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:52.306 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:52.343 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:52.343 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:52.564 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:52.633 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:04:52.648 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:04:52.648 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:04:52.648 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-20T11:04:52.648 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:53.562 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.9 MB 00:01 2024-01-20T11:04:53.655 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:04:53.655 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:53.655 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:53.655 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:04:53.655 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:53.655 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:53.961 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel 2024-01-20T11:04:54.532 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:34 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:54.687 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:54.688 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:54.688 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:04:54.688 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:54.688 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:54.689 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:19.0.0-763.ga00c9940.el9 ceph 127 k 2024-01-20T11:04:54.689 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:54.689 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:54.689 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:54.689 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:04:54.689 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:54.689 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 127 k 2024-01-20T11:04:54.689 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 456 k 2024-01-20T11:04:54.689 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:54.912 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-19.0.0-763.ga00c9940.el9.x86_64. 568 kB/s | 127 kB 00:00 2024-01-20T11:04:54.913 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:54.913 INFO:teuthology.orchestra.run.smithi110.stdout:Total 564 kB/s | 127 kB 00:00 2024-01-20T11:04:54.913 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:54.918 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:54.918 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:54.975 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:54.975 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:55.111 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:55.210 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:04:55.360 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-01-20T11:04:55.678 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:04:56.037 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:04:56.037 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:56.037 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:56.037 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:04:56.037 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:56.037 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:56.220 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 27 MB/s | 20 MB 00:00 2024-01-20T11:04:56.281 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2 2024-01-20T11:04:56.854 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:36 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:56.955 INFO:teuthology.orchestra.run.smithi110.stdout:Package libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:04:57.006 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:57.007 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:04:57.007 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:57.094 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs-devel 2024-01-20T11:04:57.647 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:37 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:57.799 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:57.800 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:57.800 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:04:57.800 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:57.800 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:04:57.800 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:19.0.0-763.ga00c9940.el9 ceph 31 k 2024-01-20T11:04:57.800 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:57.800 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:04:57.800 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:04:57.801 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:04:57.801 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:57.801 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 31 k 2024-01-20T11:04:57.801 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 140 k 2024-01-20T11:04:57.801 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:04:57.958 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs-devel-19.0.0-763.ga00c9940.el9.x86_64 198 kB/s | 31 kB 00:00 2024-01-20T11:04:57.959 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:04:57.959 INFO:teuthology.orchestra.run.smithi110.stdout:Total 197 kB/s | 31 kB 00:00 2024-01-20T11:04:57.959 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:04:57.962 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:04:57.962 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:04:57.997 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:04:57.997 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:04:58.078 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:04:58.189 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:04:58.562 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:04:58.825 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:04:58.826 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:58.826 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:04:58.826 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:04:58.826 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:04:58.826 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:58.991 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2 2024-01-20T11:04:59.543 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:39 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:04:59.644 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:04:59.695 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:04:59.696 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:04:59.696 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:04:59.771 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1 2024-01-20T11:05:00.322 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:40 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:00.422 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:05:00.473 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:00.474 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:05:00.474 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:05:00.546 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados 2024-01-20T11:05:01.105 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:41 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:01.204 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rados-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:05:01.229 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-01-20T11:05:01.254 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:01.255 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:05:01.255 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:05:01.329 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw 2024-01-20T11:05:01.882 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:41 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:01.982 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rgw-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:05:02.033 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:02.034 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:05:02.034 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:05:02.104 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs 2024-01-20T11:05:02.660 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:42 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:02.762 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-cephfs-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:05:02.814 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:02.815 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:05:02.815 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:05:02.889 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd 2024-01-20T11:05:03.440 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:43 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:03.541 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rbd-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:05:03.593 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:03.594 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:05:03.594 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:05:03.662 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse 2024-01-20T11:05:03.737 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-763.ga00c9940.el9 ceph 8.6 M 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 3.4 M 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 3.1 M 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T11:05:03.743 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-763.ga00c9940.el9 ceph 5.1 M 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-763.ga00c9940.el9 ceph 19 M 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-763.ga00c9940.el9 ceph 25 k 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 742 k 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 499 k 2024-01-20T11:05:03.744 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-763.ga00c9940.el9 ceph 4.8 M 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:19.0.0-763.ga00c9940.el9 ceph 45 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-763.ga00c9940.el9 ceph 130 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-763.ga00c9940.el9 ceph 162 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-763.ga00c9940.el9 ceph 320 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-763.ga00c9940.el9 ceph 297 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-763.ga00c9940.el9 ceph 99 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-20T11:05:03.745 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:03.746 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:05:03.746 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:03.746 INFO:teuthology.orchestra.run.smithi134.stdout:Install 27 Packages 2024-01-20T11:05:03.746 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-01-20T11:05:03.746 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:03.746 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 55 M 2024-01-20T11:05:03.746 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:05:04.216 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:44 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:04.369 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:04.370 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:04.370 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-01-20T11:05:04.370 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:04.370 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:05:04.370 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:19.0.0-763.ga00c9940.el9 ceph 85 k 2024-01-20T11:05:04.370 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:04.370 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:05:04.370 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:04.370 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:05:04.371 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:04.371 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 85 k 2024-01-20T11:05:04.371 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 231 k 2024-01-20T11:05:04.371 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:05:04.787 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-19.0.0-763.ga00c9940.el9.x86_64.rpm 205 kB/s | 85 kB 00:00 2024-01-20T11:05:04.787 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:04.788 INFO:teuthology.orchestra.run.smithi110.stdout:Total 204 kB/s | 85 kB 00:00 2024-01-20T11:05:04.788 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:05:04.795 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:05:04.795 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:05:04.821 INFO:teuthology.orchestra.run.smithi134.stdout:(1/29): ceph-radosgw-19.0.0-763.ga00c9940.el9.x 12 MB/s | 8.6 MB 00:00 2024-01-20T11:05:04.850 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:05:04.850 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:05:04.871 INFO:teuthology.orchestra.run.smithi134.stdout:(2/29): ceph-selinux-19.0.0-763.ga00c9940.el9.x 502 kB/s | 25 kB 00:00 2024-01-20T11:05:04.955 INFO:teuthology.orchestra.run.smithi134.stdout:(3/29): libcephfs2-19.0.0-763.ga00c9940.el9.x86 8.7 MB/s | 742 kB 00:00 2024-01-20T11:05:04.987 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:05:05.014 INFO:teuthology.orchestra.run.smithi134.stdout:(4/29): libradosstriper1-19.0.0-763.ga00c9940.e 8.3 MB/s | 499 kB 00:00 2024-01-20T11:05:05.085 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:05:05.289 INFO:teuthology.orchestra.run.smithi134.stdout:(5/29): ceph-base-19.0.0-763.ga00c9940.el9.x86_ 4.3 MB/s | 5.1 MB 00:01 2024-01-20T11:05:05.356 INFO:teuthology.orchestra.run.smithi134.stdout:(6/29): librgw2-19.0.0-763.ga00c9940.el9.x86_64 14 MB/s | 4.8 MB 00:00 2024-01-20T11:05:05.372 INFO:teuthology.orchestra.run.smithi134.stdout:(7/29): python3-ceph-argparse-19.0.0-763.ga00c9 540 kB/s | 45 kB 00:00 2024-01-20T11:05:05.407 INFO:teuthology.orchestra.run.smithi134.stdout:(8/29): python3-ceph-common-19.0.0-763.ga00c994 2.5 MB/s | 130 kB 00:00 2024-01-20T11:05:05.432 INFO:teuthology.orchestra.run.smithi134.stdout:(9/29): python3-cephfs-19.0.0-763.ga00c9940.el9 2.7 MB/s | 162 kB 00:00 2024-01-20T11:05:05.465 INFO:teuthology.orchestra.run.smithi134.stdout:(10/29): python3-rados-19.0.0-763.ga00c9940.el9 5.3 MB/s | 320 kB 00:00 2024-01-20T11:05:05.499 INFO:teuthology.orchestra.run.smithi134.stdout:(11/29): python3-rbd-19.0.0-763.ga00c9940.el9.x 4.3 MB/s | 297 kB 00:00 2024-01-20T11:05:05.504 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:05:05.524 INFO:teuthology.orchestra.run.smithi134.stdout:(12/29): python3-rgw-19.0.0-763.ga00c9940.el9.x 1.7 MB/s | 99 kB 00:00 2024-01-20T11:05:05.641 INFO:teuthology.orchestra.run.smithi134.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2024-01-20T11:05:05.666 INFO:teuthology.orchestra.run.smithi134.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-01-20T11:05:05.691 INFO:teuthology.orchestra.run.smithi134.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-01-20T11:05:05.742 INFO:teuthology.orchestra.run.smithi134.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-01-20T11:05:05.767 INFO:teuthology.orchestra.run.smithi134.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-01-20T11:05:05.792 INFO:teuthology.orchestra.run.smithi134.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-01-20T11:05:05.804 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:05:05.804 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:05.804 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:05:05.804 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:05.804 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:05.805 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:05:05.935 INFO:teuthology.orchestra.run.smithi134.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-01-20T11:05:05.976 INFO:teuthology.orchestra.run.smithi134.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2024-01-20T11:05:06.018 INFO:teuthology.orchestra.run.smithi134.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-01-20T11:05:06.030 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror 2024-01-20T11:05:06.060 INFO:teuthology.orchestra.run.smithi134.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-01-20T11:05:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:(23/29): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-01-20T11:05:06.178 INFO:teuthology.orchestra.run.smithi134.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-01-20T11:05:06.203 INFO:teuthology.orchestra.run.smithi134.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-01-20T11:05:06.261 INFO:teuthology.orchestra.run.smithi134.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-01-20T11:05:06.582 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:46 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:06.595 INFO:teuthology.orchestra.run.smithi134.stdout:(27/29): librbd1-19.0.0-763.ga00c9940.el9.x86_6 9.4 MB/s | 3.1 MB 00:00 2024-01-20T11:05:06.735 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:05:06.736 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:19.0.0-763.ga00c9940.el9 ceph 3.1 M 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 3.1 M 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 13 M 2024-01-20T11:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:05:06.870 INFO:teuthology.orchestra.run.smithi134.stdout:(28/29): librados2-19.0.0-763.ga00c9940.el9.x86 5.0 MB/s | 3.4 MB 00:00 2024-01-20T11:05:07.171 INFO:teuthology.orchestra.run.smithi134.stdout:(29/29): ceph-common-19.0.0-763.ga00c9940.el9.x 6.3 MB/s | 19 MB 00:03 2024-01-20T11:05:07.174 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:07.174 INFO:teuthology.orchestra.run.smithi134.stdout:Total 16 MB/s | 55 MB 00:03 2024-01-20T11:05:07.175 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:05:07.206 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:05:07.206 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:05:07.678 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:05:07.678 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:05:07.690 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-19.0.0-763.ga00c9940.el9.x86_64.rpm 3.2 MB/s | 3.1 MB 00:00 2024-01-20T11:05:07.691 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:07.692 INFO:teuthology.orchestra.run.smithi110.stdout:Total 3.2 MB/s | 3.1 MB 00:00 2024-01-20T11:05:07.692 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:05:07.701 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:05:07.701 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:05:07.775 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:05:07.775 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:05:08.199 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:05:08.277 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:05:08.301 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:05:08.301 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:05:08.301 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-20T11:05:08.301 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-20T11:05:08.301 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-20T11:05:08.301 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:09.099 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:05:09.189 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-20T11:05:09.225 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-01-20T11:05:09.275 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-01-20T11:05:09.399 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:05:09.399 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:09.399 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:05:09.399 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:09.399 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:09.400 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:05:09.593 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-01-20T11:05:09.641 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-763.ga00c9940.el9.x86_64 5/31 2024-01-20T11:05:09.671 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd 2024-01-20T11:05:09.732 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-763.ga00c9940.el9.x86_64 5/31 2024-01-20T11:05:09.820 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-763.ga00c9940.el9.x86_64 6/31 2024-01-20T11:05:09.851 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 7/31 2024-01-20T11:05:09.904 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 7/31 2024-01-20T11:05:09.925 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-01-20T11:05:09.960 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-763.ga00c9940.el9.x 9/31 2024-01-20T11:05:10.029 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-763.ga00c9940.el9.x86_64 10/31 2024-01-20T11:05:10.053 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-763.ga00c9940.el9.x86_64 11/31 2024-01-20T11:05:10.110 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-763.ga00c9940.el9.x86_64 11/31 2024-01-20T11:05:10.215 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-20T11:05:10.236 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:50 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:10.252 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-20T11:05:10.292 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-20T11:05:10.333 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-20T11:05:10.364 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-20T11:05:10.388 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:10.388 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:10.389 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-01-20T11:05:10.389 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:10.389 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:05:10.389 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:19.0.0-763.ga00c9940.el9 ceph 170 k 2024-01-20T11:05:10.389 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:10.389 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:05:10.389 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:10.389 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:05:10.389 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:10.390 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 170 k 2024-01-20T11:05:10.390 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 490 k 2024-01-20T11:05:10.390 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:05:10.397 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-20T11:05:10.706 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-20T11:05:10.735 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 19/31 2024-01-20T11:05:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 19/31 2024-01-20T11:05:10.839 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-763.ga00c9940.el9.x86_64 20/31 2024-01-20T11:05:10.869 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-19.0.0-763.ga00c9940.el9.x86_64.rpm 356 kB/s | 170 kB 00:00 2024-01-20T11:05:10.869 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:10.870 INFO:teuthology.orchestra.run.smithi110.stdout:Total 355 kB/s | 170 kB 00:00 2024-01-20T11:05:10.870 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:05:10.877 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:05:10.878 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:05:10.881 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-20T11:05:10.932 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:05:10.933 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:05:11.081 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:05:11.178 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:05:11.338 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-20T11:05:11.363 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-763.ga00c9940.el9.x86_64 23/31 2024-01-20T11:05:11.418 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-763.ga00c9940.el9.x86_64 23/31 2024-01-20T11:05:11.494 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-763.ga00c9940.el9.x86_64 24/31 2024-01-20T11:05:11.528 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-763.ga00c9940.el9.x86 25/31 2024-01-20T11:05:11.651 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:05:11.974 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:05:11.974 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:11.975 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:05:11.975 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:11.975 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:11.975 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:05:12.198 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base-debuginfo 2024-01-20T11:05:12.778 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:52 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:12.930 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 67 M 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 22 M 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-01-20T11:05:12.931 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:12.932 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 89 M 2024-01-20T11:05:12.932 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 490 M 2024-01-20T11:05:12.932 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:05:13.724 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 26/31 2024-01-20T11:05:13.753 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 26/31 2024-01-20T11:05:14.288 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 26/31 2024-01-20T11:05:14.332 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 27/31 2024-01-20T11:05:14.422 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 27/31 2024-01-20T11:05:14.423 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-20T11:05:14.423 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-20T11:05:14.423 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:14.467 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-763.ga00c9940.el9.x86_64 28/31 2024-01-20T11:05:14.625 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): ceph-debuginfo-19.0.0-763.ga00c9940.el9. 13 MB/s | 22 MB 00:01 2024-01-20T11:05:17.767 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-base-debuginfo-19.0.0-763.ga00c9940 14 MB/s | 67 MB 00:04 2024-01-20T11:05:17.769 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:17.769 INFO:teuthology.orchestra.run.smithi110.stdout:Total 18 MB/s | 89 MB 00:04 2024-01-20T11:05:17.769 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:05:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:05:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:05:18.350 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:05:18.350 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:05:21.434 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:05:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-763.ga00c9940.el9.x86_64 28/31 2024-01-20T11:05:24.082 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-01-20T11:05:24.082 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-01-20T11:05:24.082 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-01-20T11:05:24.082 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-01-20T11:05:24.082 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-01-20T11:05:24.082 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-01-20T11:05:24.082 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-01-20T11:05:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:24.874 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-763.ga00c9940.el9.x86_64 29/31 2024-01-20T11:05:24.908 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-763.ga00c9940.el9.x86_64 29/31 2024-01-20T11:05:24.908 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:05:24.908 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-20T11:05:24.908 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-20T11:05:24.908 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-20T11:05:24.908 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:24.910 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-20T11:05:25.010 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-20T11:05:25.010 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-20T11:05:26.118 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 1/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 2/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-763.ga00c9940.el9.x86_64 3/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-763.ga00c9940.el9.x86_64 4/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 5/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-763.ga00c9940.el9.x86_64 6/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-763.ga00c9940.el9.x86_64 7/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-763.ga00c9940.el9.x 8/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-763.ga00c9940.el9.x86 9/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-763.ga00c9940.el9.x86_64 10/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-763.ga00c9940.el9.x86_64 11/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-763.ga00c9940.el9.x86_64 12/31 2024-01-20T11:05:26.119 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-763.ga00c9940.el9.x86_64 13/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-20T11:05:26.120 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-20T11:05:26.121 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-20T11:05:26.121 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-20T11:05:26.121 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-20T11:05:26.121 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-763.ga00c9940.el9.x86_64 28/31 2024-01-20T11:05:26.122 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-20T11:05:26.122 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 30/31 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.211 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-20T11:05:27.212 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:27.213 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:05:27.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-01-20T11:05:28.026 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:27 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:05:28.179 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:05:28.180 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:28.180 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:05:28.180 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:28.180 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:05:28.180 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-763.ga00c9940.el9 ceph 46 M 2024-01-20T11:05:28.180 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-763.ga00c9940.el9 ceph 163 k 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-01-20T11:05:28.181 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:28.182 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 47 M 2024-01-20T11:05:28.182 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 201 M 2024-01-20T11:05:28.182 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:05:28.341 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 300 kB 00:00 2024-01-20T11:05:28.366 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-01-20T11:05:28.387 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/2 2024-01-20T11:05:28.408 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-01-20T11:05:28.433 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): libcephsqlite-19.0.0-763.ga00c9940.el9.x 652 kB/s | 163 kB 00:00 2024-01-20T11:05:28.467 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-01-20T11:05:28.468 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 2/2 2024-01-20T11:05:30.418 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-test-19.0.0-763.ga00c9940.el9.x86_6 20 MB/s | 46 MB 00:02 2024-01-20T11:05:30.421 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:30.421 INFO:teuthology.orchestra.run.smithi134.stdout:Total 21 MB/s | 47 MB 00:02 2024-01-20T11:05:30.421 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:05:30.440 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:05:30.441 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:05:30.813 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:05:30.813 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:05:31.390 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:05:31.457 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-20T11:05:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-20T11:05:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-20T11:05:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-20T11:05:31.621 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-763.ga00c9940.el9.x86_64 5/6 2024-01-20T11:05:31.815 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 2/2 2024-01-20T11:05:31.815 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/2 2024-01-20T11:05:32.080 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2/2 2024-01-20T11:05:32.081 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:32.081 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:05:32.081 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:32.081 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:32.081 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:32.081 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:05:32.285 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-common-debuginfo 2024-01-20T11:05:32.859 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:12 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 411 M 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:05:33.022 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 411 M 2024-01-20T11:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.7 G 2024-01-20T11:05:33.023 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:05:35.975 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-763.ga00c9940.el9.x86_64 5/6 2024-01-20T11:05:36.044 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-763.ga00c9940.el9.x86_64 6/6 2024-01-20T11:05:38.403 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:19.0.0-763.ga00c9940.el9.x86_64 6/6 2024-01-20T11:05:38.404 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-763.ga00c9940.el9.x86_64 1/6 2024-01-20T11:05:38.404 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-763.ga00c9940.el9.x86_64 2/6 2024-01-20T11:05:38.404 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-20T11:05:38.404 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-20T11:05:38.404 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-20T11:05:38.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-16.el9.x86_64 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:38.829 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:05:39.141 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-01-20T11:05:39.707 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:38 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:05:39.860 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:05:39.862 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-01-20T11:05:39.862 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-763.ga00c9940.el9 ceph 6.6 k 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-763.ga00c9940.el9 ceph 2.1 M 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-763.ga00c9940.el9 ceph 1.5 M 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 245 k 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-763.ga00c9940.el9 ceph 4.7 M 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-763.ga00c9940.el9 ceph 17 M 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-20T11:05:39.863 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-20T11:05:39.864 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:39.865 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:05:39.866 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-01-20T11:05:39.866 INFO:teuthology.orchestra.run.smithi134.stdout:Install 36 Packages 2024-01-20T11:05:39.866 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:39.866 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 30 M 2024-01-20T11:05:39.866 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 106 M 2024-01-20T11:05:39.866 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:05:40.453 INFO:teuthology.orchestra.run.smithi134.stdout:(1/36): ceph-19.0.0-763.ga00c9940.el9.x86_64.rp 18 kB/s | 6.6 kB 00:00 2024-01-20T11:05:41.220 INFO:teuthology.orchestra.run.smithi134.stdout:(2/36): ceph-mgr-19.0.0-763.ga00c9940.el9.x86_6 1.3 MB/s | 1.5 MB 00:01 2024-01-20T11:05:41.312 INFO:teuthology.orchestra.run.smithi134.stdout:(3/36): ceph-mds-19.0.0-763.ga00c9940.el9.x86_6 1.7 MB/s | 2.1 MB 00:01 2024-01-20T11:05:41.437 INFO:teuthology.orchestra.run.smithi134.stdout:(4/36): ceph-mgr-modules-core-19.0.0-763.ga00c9 1.9 MB/s | 245 kB 00:00 2024-01-20T11:05:41.630 INFO:teuthology.orchestra.run.smithi134.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 974 kB/s | 187 kB 00:00 2024-01-20T11:05:41.680 INFO:teuthology.orchestra.run.smithi134.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-01-20T11:05:41.713 INFO:teuthology.orchestra.run.smithi134.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-01-20T11:05:41.747 INFO:teuthology.orchestra.run.smithi134.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-01-20T11:05:41.789 INFO:teuthology.orchestra.run.smithi134.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 918 kB/s | 39 kB 00:00 2024-01-20T11:05:41.839 INFO:teuthology.orchestra.run.smithi134.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-01-20T11:05:41.906 INFO:teuthology.orchestra.run.smithi134.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-01-20T11:05:41.940 INFO:teuthology.orchestra.run.smithi134.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-01-20T11:05:41.974 INFO:teuthology.orchestra.run.smithi134.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-01-20T11:05:42.007 INFO:teuthology.orchestra.run.smithi134.stdout:(14/36): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-01-20T11:05:42.041 INFO:teuthology.orchestra.run.smithi134.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-01-20T11:05:42.166 INFO:teuthology.orchestra.run.smithi134.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 179 kB/s | 22 kB 00:00 2024-01-20T11:05:42.550 INFO:teuthology.orchestra.run.smithi134.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-01-20T11:05:42.633 INFO:teuthology.orchestra.run.smithi134.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 524 kB/s | 43 kB 00:00 2024-01-20T11:05:42.767 INFO:teuthology.orchestra.run.smithi134.stdout:(19/36): ceph-mon-19.0.0-763.ga00c9940.el9.x86_ 2.0 MB/s | 4.7 MB 00:02 2024-01-20T11:05:42.792 INFO:teuthology.orchestra.run.smithi134.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-01-20T11:05:42.868 INFO:teuthology.orchestra.run.smithi134.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-01-20T11:05:42.960 INFO:teuthology.orchestra.run.smithi134.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-01-20T11:05:43.043 INFO:teuthology.orchestra.run.smithi134.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 279 kB/s | 23 kB 00:00 2024-01-20T11:05:43.119 INFO:teuthology.orchestra.run.smithi134.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-01-20T11:05:43.185 INFO:teuthology.orchestra.run.smithi134.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 858 kB/s | 358 kB 00:00 2024-01-20T11:05:43.211 INFO:teuthology.orchestra.run.smithi134.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-01-20T11:05:43.261 INFO:teuthology.orchestra.run.smithi134.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 616 kB/s | 46 kB 00:00 2024-01-20T11:05:43.303 INFO:teuthology.orchestra.run.smithi134.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 859 kB/s | 79 kB 00:00 2024-01-20T11:05:43.353 INFO:teuthology.orchestra.run.smithi134.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-01-20T11:05:43.379 INFO:teuthology.orchestra.run.smithi134.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-01-20T11:05:43.437 INFO:teuthology.orchestra.run.smithi134.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-01-20T11:05:43.454 INFO:teuthology.orchestra.run.smithi134.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-01-20T11:05:43.530 INFO:teuthology.orchestra.run.smithi134.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-01-20T11:05:43.588 INFO:teuthology.orchestra.run.smithi134.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-01-20T11:05:43.613 INFO:teuthology.orchestra.run.smithi134.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-01-20T11:05:44.439 INFO:teuthology.orchestra.run.smithi134.stdout:(36/36): ceph-osd-19.0.0-763.ga00c9940.el9.x86_ 5.2 MB/s | 17 MB 00:03 2024-01-20T11:05:44.445 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:44.445 INFO:teuthology.orchestra.run.smithi134.stdout:Total 6.5 MB/s | 30 MB 00:04 2024-01-20T11:05:44.467 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:05:44.511 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:05:44.511 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:05:44.922 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:05:44.923 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:05:46.259 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:05:46.333 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-20T11:05:46.366 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-20T11:05:46.394 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-20T11:05:46.465 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-20T11:05:46.501 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-20T11:05:46.534 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-20T11:05:46.627 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-20T11:05:46.705 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-20T11:05:46.745 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-20T11:05:46.826 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-20T11:05:46.879 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-20T11:05:46.918 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-20T11:05:46.967 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-20T11:05:47.036 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-20T11:05:47.224 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-20T11:05:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-20T11:05:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-20T11:05:47.339 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-20T11:05:47.369 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-20T11:05:47.446 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-20T11:05:47.488 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-20T11:05:47.521 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-20T11:05:47.562 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-20T11:05:47.596 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-20T11:05:47.631 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-20T11:05:47.722 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-20T11:05:47.762 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-20T11:05:47.835 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-20T11:05:47.970 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-20T11:05:48.099 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-20T11:05:48.262 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-763.ga00c9940.el9.n 31/36 2024-01-20T11:05:48.292 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 32/36 2024-01-20T11:05:48.317 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 32/36 2024-01-20T11:05:48.317 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:05:48.317 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-20T11:05:48.317 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-20T11:05:48.318 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-20T11:05:48.318 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:49.999 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-763.ga00c9940.el9.x86_64 33/36 2024-01-20T11:05:50.027 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-763.ga00c9940.el9.x86_64 33/36 2024-01-20T11:05:50.028 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:05:50.028 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-20T11:05:50.028 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-20T11:05:50.028 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-20T11:05:50.028 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:50.466 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-763.ga00c9940.el9.x86_64 34/36 2024-01-20T11:05:50.489 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-763.ga00c9940.el9.x86_64 34/36 2024-01-20T11:05:50.489 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:05:50.489 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-20T11:05:50.489 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-20T11:05:50.490 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-20T11:05:50.490 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:50.700 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-763.ga00c9940.el9.x86_64 35/36 2024-01-20T11:05:50.724 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-763.ga00c9940.el9.x86_64 35/36 2024-01-20T11:05:50.724 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:05:50.724 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-20T11:05:50.724 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-20T11:05:50.724 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-20T11:05:50.724 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:50.791 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-763.ga00c9940.el9.x86_64 36/36 2024-01-20T11:05:51.813 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-common-debuginfo-19.0.0-763.ga00c9940.el9. 22 MB/s | 411 MB 00:18 2024-01-20T11:05:51.814 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:51.814 INFO:teuthology.orchestra.run.smithi110.stdout:Total 22 MB/s | 411 MB 00:18 2024-01-20T11:05:51.814 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:05:51.819 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:05:51.819 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:19.0.0-763.ga00c9940.el9.x86_64 36/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-763.ga00c9940.el9.x86_64 1/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-763.ga00c9940.el9.x86_64 2/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 3/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-763.ga00c9940.el9.x86_64 4/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-763.ga00c9940.el9.x86_64 5/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-763.ga00c9940.el9.n 6/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-20T11:05:52.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-20T11:05:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-20T11:05:52.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-20T11:05:53.468 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-20T11:05:53.468 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:53.468 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:05:53.468 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:53.468 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:53.468 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:53.468 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:05:53.469 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:53.469 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:53.469 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-01-20T11:05:53.469 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-20T11:05:53.469 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-20T11:05:53.469 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-20T11:05:53.469 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-20T11:05:53.469 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-20T11:05:53.469 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-01-20T11:05:53.471 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:53.472 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:05:53.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-01-20T11:05:54.272 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:05:54.272 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:05:54.404 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:53 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:05:54.503 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:05:54.552 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:05:54.553 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:05:54.553 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:05:54.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-01-20T11:05:55.203 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:54 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:05:55.352 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:05:55.353 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:55.353 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:05:55.353 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:55.353 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:05:55.353 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 742 k 2024-01-20T11:05:55.353 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:55.353 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:05:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:05:55.354 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 742 k 2024-01-20T11:05:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 748 k 2024-01-20T11:05:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:05:55.824 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-19.0.0-763.ga00c9940.el9.noarch.rpm 1.5 MB/s | 742 kB 00:00 2024-01-20T11:05:55.825 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:55.825 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.5 MB/s | 742 kB 00:00 2024-01-20T11:05:55.825 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:05:55.829 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:05:55.829 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:05:55.835 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:05:55.835 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:05:55.911 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:05:56.322 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:05:56.410 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:05:56.781 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:05:57.072 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:05:57.072 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:57.072 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:05:57.072 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:05:57.072 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:57.073 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:05:57.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-01-20T11:05:57.852 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:56 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:05:58.008 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-763.ga00c9940.el9 ceph 145 k 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:58.009 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 145 k 2024-01-20T11:05:58.010 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 436 k 2024-01-20T11:05:58.010 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:05:58.398 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-19.0.0-763.ga00c994 374 kB/s | 145 kB 00:00 2024-01-20T11:05:58.398 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:05:58.399 INFO:teuthology.orchestra.run.smithi134.stdout:Total 372 kB/s | 145 kB 00:00 2024-01-20T11:05:58.399 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:05:58.406 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:05:58.407 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:05:58.457 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:05:58.458 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:05:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:05:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-763.ga00c9940.e 1/1 2024-01-20T11:05:58.695 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-763.ga00c9940.e 1/1 2024-01-20T11:05:58.695 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:05:58.695 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-20T11:05:58.695 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:59.699 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-763.ga00c9940.e 1/1 2024-01-20T11:05:59.699 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:59.700 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:05:59.700 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:05:59.700 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:05:59.700 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:05:59.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-01-20T11:06:00.531 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:59 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:00.632 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:06:00.683 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:00.684 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:06:00.684 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:00.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-01-20T11:06:01.323 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:00 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:01.476 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:01.476 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:01.476 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 1.7 M 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 28 k 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 15 k 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:01.477 INFO:teuthology.orchestra.run.smithi134.stdout:Install 5 Packages 2024-01-20T11:06:01.478 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:01.478 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-01-20T11:06:01.479 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 8.5 M 2024-01-20T11:06:01.479 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:02.090 INFO:teuthology.orchestra.run.smithi134.stdout:(1/5): ceph-prometheus-alerts-19.0.0-763.ga00c9 58 kB/s | 15 kB 00:00 2024-01-20T11:06:02.115 INFO:teuthology.orchestra.run.smithi134.stdout:(2/5): ceph-grafana-dashboards-19.0.0-763.ga00c 99 kB/s | 28 kB 00:00 2024-01-20T11:06:02.207 INFO:teuthology.orchestra.run.smithi134.stdout:(3/5): ceph-mgr-dashboard-19.0.0-763.ga00c9940. 4.6 MB/s | 1.7 MB 00:00 2024-01-20T11:06:02.399 INFO:teuthology.orchestra.run.smithi134.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 100 kB/s | 31 kB 00:00 2024-01-20T11:06:02.624 INFO:teuthology.orchestra.run.smithi134.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 371 kB/s | 188 kB 00:00 2024-01-20T11:06:02.625 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:02.626 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-01-20T11:06:02.626 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:02.633 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:02.633 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:02.697 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:02.697 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:02.950 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:03.030 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-20T11:06:03.070 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-20T11:06:03.100 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-763.ga00c9940.el9.no 3/5 2024-01-20T11:06:03.335 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-763.ga00c9940.el9.n 4/5 2024-01-20T11:06:03.376 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-763.ga00c9940.el9.noarch 5/5 2024-01-20T11:06:03.930 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-763.ga00c9940.el9.noarch 5/5 2024-01-20T11:06:03.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-763.ga00c9940.el9.n 1/5 2024-01-20T11:06:03.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-763.ga00c9940.el9.noarch 2/5 2024-01-20T11:06:03.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-763.ga00c9940.el9.no 3/5 2024-01-20T11:06:03.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:04.240 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:04.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-20T11:06:05.052 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:04 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:05.204 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:05.205 INFO:teuthology.orchestra.run.smithi134.stdout:=========================================================================================== 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout:=========================================================================================== 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 7.4 M 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-20T11:06:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-20T11:06:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-20T11:06:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-20T11:06:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:05.207 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:05.207 INFO:teuthology.orchestra.run.smithi134.stdout:=========================================================================================== 2024-01-20T11:06:05.207 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-01-20T11:06:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:05.207 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2024-01-20T11:06:05.208 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-01-20T11:06:05.208 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:05.480 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-01-20T11:06:05.513 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-01-20T11:06:05.547 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-01-20T11:06:05.697 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2024-01-20T11:06:05.756 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.2 MB/s | 3.0 MB 00:00 2024-01-20T11:06:05.839 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-01-20T11:06:05.865 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-01-20T11:06:05.990 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.4 MB/s | 802 kB 00:00 2024-01-20T11:06:06.082 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-76 9.9 MB/s | 7.4 MB 00:00 2024-01-20T11:06:06.240 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-01-20T11:06:06.265 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 711 kB/s | 194 kB 00:00 2024-01-20T11:06:06.482 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 617 kB/s | 245 kB 00:00 2024-01-20T11:06:06.485 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:06.485 INFO:teuthology.orchestra.run.smithi134.stdout:Total 29 MB/s | 37 MB 00:01 2024-01-20T11:06:06.498 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:06.522 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:06.522 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:06.901 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:06.902 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:07.624 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-20T11:06:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-20T11:06:07.821 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-20T11:06:07.856 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-20T11:06:08.053 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-20T11:06:08.080 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-20T11:06:08.156 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-20T11:06:08.801 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-20T11:06:08.908 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-20T11:06:10.119 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-20T11:06:11.141 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-20T11:06:11.179 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-763.ga00c99 12/12 2024-01-20T11:06:13.586 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-763.ga00c99 12/12 2024-01-20T11:06:13.586 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-763.ga00c99 1/12 2024-01-20T11:06:13.586 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-20T11:06:13.586 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-20T11:06:13.586 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-20T11:06:13.586 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-20T11:06:13.586 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-20T11:06:13.587 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-20T11:06:13.587 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-20T11:06:13.587 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-20T11:06:13.587 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-20T11:06:13.587 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-20T11:06:14.002 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-20T11:06:14.002 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:14.002 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:14.002 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:06:14.002 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-20T11:06:14.002 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-20T11:06:14.002 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-20T11:06:14.002 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-20T11:06:14.002 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-20T11:06:14.003 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-20T11:06:14.003 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-20T11:06:14.003 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-20T11:06:14.003 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-20T11:06:14.003 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-20T11:06:14.003 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-20T11:06:14.003 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:14.003 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:14.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-01-20T11:06:14.962 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:13 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:15.115 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:15.116 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-20T11:06:15.116 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:06:15.116 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-20T11:06:15.116 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:15.116 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 51 k 2024-01-20T11:06:15.116 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T11:06:15.116 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-20T11:06:15.116 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-20T11:06:15.116 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-01-20T11:06:15.117 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:15.118 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-01-20T11:06:15.118 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-01-20T11:06:15.118 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:15.624 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 160 kB/s | 24 kB 00:00 2024-01-20T11:06:15.649 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 102 kB/s | 18 kB 00:00 2024-01-20T11:06:15.708 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): ceph-mgr-rook-19.0.0-763.ga00c9940.el9. 214 kB/s | 51 kB 00:00 2024-01-20T11:06:15.750 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 174 kB 00:00 2024-01-20T11:06:15.775 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-01-20T11:06:15.825 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 211 kB 00:00 2024-01-20T11:06:15.842 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 464 kB/s | 43 kB 00:00 2024-01-20T11:06:16.026 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 128 kB/s | 32 kB 00:00 2024-01-20T11:06:16.042 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 69 kB/s | 15 kB 00:00 2024-01-20T11:06:16.101 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-01-20T11:06:16.126 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 773 kB/s | 202 kB 00:00 2024-01-20T11:06:16.168 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-01-20T11:06:16.377 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.9 MB/s | 1.0 MB 00:00 2024-01-20T11:06:16.381 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:16.381 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-01-20T11:06:16.381 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:16.394 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:16.394 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:16.489 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:16.491 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:17.008 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:17.173 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-20T11:06:17.236 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-20T11:06:17.300 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-20T11:06:17.338 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-20T11:06:17.379 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-20T11:06:17.472 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-20T11:06:17.580 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-20T11:06:17.638 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-20T11:06:18.354 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-20T11:06:18.402 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-20T11:06:18.435 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-20T11:06:18.483 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-20T11:06:18.506 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-763.ga00c9940.el9.noarch 13/13 2024-01-20T11:06:19.377 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-763.ga00c9940.el9.noarch 13/13 2024-01-20T11:06:19.377 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-763.ga00c9940.el9.noarch 1/13 2024-01-20T11:06:19.377 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-20T11:06:19.377 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-20T11:06:19.377 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-20T11:06:19.377 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-20T11:06:19.377 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-20T11:06:19.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-20T11:06:19.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-20T11:06:19.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-20T11:06:19.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-20T11:06:19.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-20T11:06:19.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-20T11:06:19.760 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-20T11:06:19.761 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-20T11:06:19.761 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-20T11:06:19.761 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-20T11:06:19.761 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-20T11:06:19.761 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-20T11:06:19.761 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:19.761 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:20.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-01-20T11:06:20.573 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:19 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:20.726 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:20.726 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-20T11:06:20.726 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:06:20.726 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 133 k 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-01-20T11:06:20.727 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:20.728 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.8 M 2024-01-20T11:06:20.728 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-01-20T11:06:20.728 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:20.986 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-20T11:06:21.061 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-mgr-cephadm-19.0.0-763.ga00c9940.el 530 kB/s | 133 kB 00:00 2024-01-20T11:06:21.137 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-20T11:06:21.437 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 154 kB/s | 58 kB 00:00 2024-01-20T11:06:21.463 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 157 kB/s | 51 kB 00:00 2024-01-20T11:06:21.696 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 774 kB/s | 548 kB 00:00 2024-01-20T11:06:21.698 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:21.698 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.0 MB/s | 6.8 MB 00:00 2024-01-20T11:06:21.698 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:21.707 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:21.708 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:21.796 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:21.798 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:22.269 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-20T11:06:22.314 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-20T11:06:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-20T11:06:22.909 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-20T11:06:22.969 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-20T11:06:22.993 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-763.ga00c9940.el9.noarch 6/6 2024-01-20T11:06:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-763.ga00c9940.el9.noarch 6/6 2024-01-20T11:06:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-763.ga00c9940.el9.noarch 1/6 2024-01-20T11:06:23.798 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-20T11:06:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-20T11:06:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-20T11:06:23.799 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-20T11:06:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:24.112 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:24.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-01-20T11:06:24.909 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:23 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:25.061 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:25.061 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:25.061 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:06:25.061 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:25.061 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:25.061 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-763.ga00c9940.el9 ceph 886 k 2024-01-20T11:06:25.061 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T11:06:25.062 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-20T11:06:25.062 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:25.062 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:25.062 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:25.062 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-01-20T11:06:25.062 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:25.062 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 964 k 2024-01-20T11:06:25.063 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.8 M 2024-01-20T11:06:25.063 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:25.197 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 587 kB/s | 79 kB 00:00 2024-01-20T11:06:25.522 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-19.0.0-763.ga00c9940.el9.x86_6 1.9 MB/s | 886 kB 00:00 2024-01-20T11:06:25.523 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:25.524 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.0 MB/s | 964 kB 00:00 2024-01-20T11:06:25.524 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:25.544 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:25.544 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:25.620 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:25.620 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:25.803 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-20T11:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 2/2 2024-01-20T11:06:26.672 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 2/2 2024-01-20T11:06:26.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 1/2 2024-01-20T11:06:27.011 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-20T11:06:27.011 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:27.011 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:27.011 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-20T11:06:27.011 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:27.011 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:27.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-01-20T11:06:27.858 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:26 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:28.011 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:28.011 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:28.011 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-20T11:06:28.011 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-763.ga00c9940.el9 ceph-noarch 262 k 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 262 k 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.2 M 2024-01-20T11:06:28.012 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:28.259 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-volume-19.0.0-763.ga00c9940.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2024-01-20T11:06:28.260 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:28.260 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.0 MB/s | 262 kB 00:00 2024-01-20T11:06:28.260 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:28.267 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:28.267 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:28.304 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:28.380 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:28.521 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:28.594 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:06:28.609 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:06:28.609 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:06:28.609 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-20T11:06:28.610 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:29.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-763.ga00c9940.el9.noarch 1/1 2024-01-20T11:06:29.635 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:29.635 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:29.635 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-763.ga00c9940.el9.noarch 2024-01-20T11:06:29.636 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:29.636 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:29.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-01-20T11:06:30.544 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:29 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:30.697 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:30.698 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:06:30.698 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:30.698 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-763.ga00c9940.el9 ceph 127 k 2024-01-20T11:06:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:30.698 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:30.698 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:30.699 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:06:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:30.699 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-01-20T11:06:30.699 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 456 k 2024-01-20T11:06:30.699 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:31.046 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-19.0.0-763.ga00c9940.el9.x86_64. 365 kB/s | 127 kB 00:00 2024-01-20T11:06:31.047 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:31.047 INFO:teuthology.orchestra.run.smithi134.stdout:Total 363 kB/s | 127 kB 00:00 2024-01-20T11:06:31.048 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:31.053 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:31.053 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:31.108 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:31.108 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:31.249 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:31.344 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:31.777 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:32.088 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:32.088 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:32.088 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:32.088 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:06:32.088 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:32.088 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:32.346 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-01-20T11:06:32.916 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:31 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:33.017 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:06:33.068 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:33.069 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:06:33.069 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:33.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-01-20T11:06:33.715 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:32 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:33.868 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-763.ga00c9940.el9 ceph 31 k 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 k 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 140 k 2024-01-20T11:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:34.117 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-19.0.0-763.ga00c9940.el9.x86_64 126 kB/s | 31 kB 00:00 2024-01-20T11:06:34.117 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:34.117 INFO:teuthology.orchestra.run.smithi134.stdout:Total 126 kB/s | 31 kB 00:00 2024-01-20T11:06:34.117 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:34.120 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:34.121 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:34.150 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:34.150 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:34.242 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:34.355 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:34.738 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:35.029 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:35.029 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:35.029 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:35.029 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:06:35.030 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:35.030 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:35.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-01-20T11:06:35.766 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:06:35.787 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:34 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:35.884 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:06:35.889 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:06:35.940 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:35.941 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:06:35.941 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:36.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-01-20T11:06:36.573 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:35 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:36.675 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:06:36.727 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:36.728 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:06:36.728 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:36.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-01-20T11:06:37.363 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:36 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:37.464 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:06:37.515 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:37.516 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:06:37.516 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:37.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-01-20T11:06:38.156 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:37 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:38.257 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:06:38.307 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:38.308 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:06:38.309 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:38.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-01-20T11:06:38.948 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:37 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:39.049 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:06:39.101 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:39.102 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:06:39.102 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:39.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-01-20T11:06:39.736 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:38 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:39.836 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:06:39.888 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:39.889 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:06:39.889 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:39.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-01-20T11:06:40.525 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:39 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:40.678 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:40.679 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:40.679 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-20T11:06:40.679 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:40.679 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:40.679 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-763.ga00c9940.el9 ceph 85 k 2024-01-20T11:06:40.679 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:40.680 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:40.680 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:40.680 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:06:40.680 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:40.680 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 85 k 2024-01-20T11:06:40.680 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 231 k 2024-01-20T11:06:40.680 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:40.994 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-19.0.0-763.ga00c9940.el9.x86_64.rpm 271 kB/s | 85 kB 00:00 2024-01-20T11:06:40.994 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:40.995 INFO:teuthology.orchestra.run.smithi134.stdout:Total 270 kB/s | 85 kB 00:00 2024-01-20T11:06:40.995 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:41.002 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:41.002 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:41.057 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:41.057 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:41.192 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:41.284 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:41.658 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:41.976 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:41.977 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:41.977 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:41.977 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:06:41.977 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:41.977 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:42.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-01-20T11:06:42.802 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:41 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:42.957 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:42.957 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:42.957 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:06:42.957 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:42.957 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:42.957 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-763.ga00c9940.el9 ceph 3.1 M 2024-01-20T11:06:42.958 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:42.958 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:42.958 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:42.958 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:06:42.958 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:42.958 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.1 M 2024-01-20T11:06:42.958 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 13 M 2024-01-20T11:06:42.959 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:43.505 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-19.0.0-763.ga00c9940.el9.x86_64.rpm 5.7 MB/s | 3.1 MB 00:00 2024-01-20T11:06:43.505 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:43.506 INFO:teuthology.orchestra.run.smithi134.stdout:Total 5.6 MB/s | 3.1 MB 00:00 2024-01-20T11:06:43.506 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:43.515 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:43.515 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:43.586 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:43.587 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:43.755 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:06:44.008 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:44.031 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:06:44.031 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:06:44.031 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:06:44.031 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:06:44.031 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:06:44.031 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:06:44.081 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:44.105 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:44.105 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-20T11:06:44.105 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-20T11:06:44.105 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-20T11:06:44.105 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-20T11:06:44.105 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:44.274 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-01-20T11:06:44.826 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:24 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:06:44.980 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:06:44.981 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-20T11:06:44.981 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:06:44.981 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-20T11:06:44.981 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:06:44.982 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 1.4 M 2024-01-20T11:06:44.982 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:06:44.982 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:06:44.982 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-20T11:06:44.982 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:06:44.982 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:06:44.982 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.4 M 2024-01-20T11:06:44.982 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 6.4 M 2024-01-20T11:06:44.982 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:06:45.200 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:45.200 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:45.200 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:45.200 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:06:45.200 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:45.200 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:45.298 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-76 4.5 MB/s | 1.4 MB 00:00 2024-01-20T11:06:45.298 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:45.298 INFO:teuthology.orchestra.run.smithi110.stdout:Total 4.5 MB/s | 1.4 MB 00:00 2024-01-20T11:06:45.298 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:06:45.301 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:06:45.301 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:06:45.341 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:06:45.341 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:06:45.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-01-20T11:06:45.593 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:06:45.695 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-763.g 1/1 2024-01-20T11:06:46.001 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:44 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:46.095 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-763.g 1/1 2024-01-20T11:06:46.155 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:46.155 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:46.155 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-20T11:06:46.155 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:46.156 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:46.156 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-763.ga00c9940.el9 ceph 170 k 2024-01-20T11:06:46.156 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:46.156 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:46.156 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:46.156 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:06:46.156 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:46.157 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 170 k 2024-01-20T11:06:46.157 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 490 k 2024-01-20T11:06:46.157 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:46.342 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-763.g 1/1 2024-01-20T11:06:46.342 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:06:46.342 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:06:46.343 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:06:46.343 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:06:46.343 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:06:46.507 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-19.0.0-763.ga00c9940.el9.x86_64.rpm 486 kB/s | 170 kB 00:00 2024-01-20T11:06:46.508 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:46.508 INFO:teuthology.orchestra.run.smithi134.stdout:Total 484 kB/s | 170 kB 00:00 2024-01-20T11:06:46.508 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:06:46.516 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:06:46.516 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:06:46.552 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw-debuginfo 2024-01-20T11:06:46.571 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:06:46.571 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:06:46.727 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:06:46.831 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:47.106 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:27 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:06:47.260 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:06:47.261 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:06:47.261 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:06:47.261 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:06:47.261 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:06:47.261 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 170 M 2024-01-20T11:06:47.261 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:06:47.261 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:06:47.261 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:06:47.261 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:06:47.262 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:06:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:47.262 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 170 M 2024-01-20T11:06:47.262 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 721 M 2024-01-20T11:06:47.263 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:06:47.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:06:47.567 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:47.568 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:06:47.568 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:06:47.568 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:47.568 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:06:47.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base-debuginfo 2024-01-20T11:06:48.422 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:47 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:06:48.584 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 67 M 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 22 M 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-01-20T11:06:48.585 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:06:48.586 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 89 M 2024-01-20T11:06:48.586 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 490 M 2024-01-20T11:06:48.586 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:06:53.968 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): ceph-debuginfo-19.0.0-763.ga00c9940.el9. 4.1 MB/s | 22 MB 00:05 2024-01-20T11:06:56.225 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-radosgw-debuginfo-19.0.0-763.ga00c9940.el9 19 MB/s | 170 MB 00:08 2024-01-20T11:06:56.226 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:06:56.226 INFO:teuthology.orchestra.run.smithi110.stdout:Total 19 MB/s | 170 MB 00:08 2024-01-20T11:06:56.226 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:06:56.230 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:06:56.231 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:06:57.270 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:06:57.270 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:07:01.061 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-base-debuginfo-19.0.0-763.ga00c9940 5.3 MB/s | 67 MB 00:12 2024-01-20T11:07:01.063 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:07:01.063 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.1 MB/s | 89 MB 00:12 2024-01-20T11:07:01.064 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:07:01.068 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:07:01.068 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:07:01.649 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:07:01.649 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:07:04.710 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:07:11.626 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/2 2024-01-20T11:07:11.695 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 2/2 2024-01-20T11:07:14.962 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 2/2 2024-01-20T11:07:14.962 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/2 2024-01-20T11:07:14.990 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:07:15.179 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x8 1/1 2024-01-20T11:07:15.254 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2/2 2024-01-20T11:07:15.254 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:07:15.255 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:07:15.255 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:07:15.255 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:07:15.255 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:07:15.255 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:07:15.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-common-debuginfo 2024-01-20T11:07:16.062 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:15 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:07:16.215 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:07:16.216 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:07:16.216 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:07:16.216 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:07:16.216 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:07:16.216 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 411 M 2024-01-20T11:07:16.216 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:07:16.216 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:07:16.216 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:07:16.217 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:07:16.217 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:07:16.217 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 411 M 2024-01-20T11:07:16.217 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.7 G 2024-01-20T11:07:16.217 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:07:19.782 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x8 1/1 2024-01-20T11:07:20.055 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x8 1/1 2024-01-20T11:07:20.056 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:07:20.056 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:07:20.056 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:07:20.056 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:07:20.056 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:07:20.255 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test-debuginfo 2024-01-20T11:07:20.823 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:00 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:07:20.972 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:07:20.973 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:07:20.973 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:07:20.973 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:07:20.973 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:07:20.974 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 664 M 2024-01-20T11:07:20.974 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:07:20.974 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:07:20.974 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:07:20.974 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:07:20.974 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:07:20.974 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 664 M 2024-01-20T11:07:20.974 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 3.0 G 2024-01-20T11:07:20.974 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:07:42.820 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-common-debuginfo-19.0.0-763.ga00c9940.el9. 15 MB/s | 411 MB 00:26 2024-01-20T11:07:42.821 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:07:42.821 INFO:teuthology.orchestra.run.smithi134.stdout:Total 15 MB/s | 411 MB 00:26 2024-01-20T11:07:42.822 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:07:42.826 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:07:42.826 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:07:45.280 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:07:45.280 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:08:00.640 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-test-debuginfo-19.0.0-763.ga00c9940.el9.x8 17 MB/s | 664 MB 00:39 2024-01-20T11:08:00.641 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:08:00.641 INFO:teuthology.orchestra.run.smithi110.stdout:Total 17 MB/s | 664 MB 00:39 2024-01-20T11:08:00.641 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:08:00.645 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:08:00.645 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:08:04.557 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:08:04.557 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:08:26.806 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:08:26.906 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:08:34.597 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:08:34.881 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:08:34.881 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:08:34.881 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:08:34.881 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:08:34.881 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:08:34.881 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:08:35.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-01-20T11:08:35.726 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:03:34 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:08:35.890 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 1.4 M 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:08:35.891 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.4 M 2024-01-20T11:08:35.892 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 6.4 M 2024-01-20T11:08:35.892 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:08:36.427 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-76 2.7 MB/s | 1.4 MB 00:00 2024-01-20T11:08:36.428 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:08:36.428 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.7 MB/s | 1.4 MB 00:00 2024-01-20T11:08:36.428 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:08:36.432 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:08:36.432 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:08:36.473 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:08:36.473 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:08:36.726 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:08:36.817 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-763.g 1/1 2024-01-20T11:08:37.233 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-763.g 1/1 2024-01-20T11:08:37.516 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-763.g 1/1 2024-01-20T11:08:37.516 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:08:37.516 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:08:37.516 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:08:37.516 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:08:37.516 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:08:37.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw-debuginfo 2024-01-20T11:08:38.292 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:03:37 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:08:38.446 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:08:38.446 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:08:38.446 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:08:38.446 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:08:38.446 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:08:38.446 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 170 M 2024-01-20T11:08:38.447 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:08:38.447 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:08:38.447 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:08:38.447 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:08:38.447 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:08:38.447 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 170 M 2024-01-20T11:08:38.447 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 721 M 2024-01-20T11:08:38.448 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:08:46.104 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-radosgw-debuginfo-19.0.0-763.ga00c9940.el9 22 MB/s | 170 MB 00:07 2024-01-20T11:08:46.105 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:08:46.106 INFO:teuthology.orchestra.run.smithi134.stdout:Total 22 MB/s | 170 MB 00:07 2024-01-20T11:08:46.106 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:08:46.110 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:08:46.110 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:08:47.141 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:08:47.141 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:09:04.469 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:09:04.567 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x8 1/1 2024-01-20T11:09:09.140 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x8 1/1 2024-01-20T11:09:09.420 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x8 1/1 2024-01-20T11:09:09.420 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:09:09.420 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:09:09.420 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:09:09.420 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:09:09.421 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:09:09.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test-debuginfo 2024-01-20T11:09:10.204 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:04:09 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:09:10.356 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:09:10.357 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:09:10.357 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:09:10.357 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:09:10.357 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:09:10.357 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 664 M 2024-01-20T11:09:10.357 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:09:10.357 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:09:10.357 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:09:10.357 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:09:10.358 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:09:10.358 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 664 M 2024-01-20T11:09:10.358 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 3.0 G 2024-01-20T11:09:10.358 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:09:14.376 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:09:14.534 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:09:23.471 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:09:23.750 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:09:23.750 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:23.750 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:09:23.750 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:09:23.750 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:23.751 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:09:24.031 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base-debuginfo 2024-01-20T11:09:24.597 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:06:04 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:09:24.697 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:09:24.748 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:09:24.749 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-20T11:09:24.749 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:09:24.829 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-debuginfo 2024-01-20T11:09:25.386 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:06:05 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:09:25.538 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 20 M 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 20 M 2024-01-20T11:09:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 102 M 2024-01-20T11:09:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:09:31.080 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mgr-debuginfo-19.0.0-763.ga00c9940.el9.x86 3.5 MB/s | 20 MB 00:05 2024-01-20T11:09:31.080 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:09:31.081 INFO:teuthology.orchestra.run.smithi110.stdout:Total 3.5 MB/s | 20 MB 00:05 2024-01-20T11:09:31.081 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:09:31.085 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:09:31.085 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:09:31.233 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:09:31.233 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:09:33.504 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:09:33.668 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:09:34.755 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:09:35.025 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:09:35.025 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:35.026 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:09:35.026 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:09:35.026 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:35.026 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:09:35.167 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mds-debuginfo 2024-01-20T11:09:35.728 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:06:15 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:09:35.878 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:09:35.879 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:09:35.879 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:09:35.879 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:09:35.879 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:09:35.879 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 48 M 2024-01-20T11:09:35.880 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:35.880 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:09:35.880 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:09:35.880 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:09:35.880 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:35.880 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 48 M 2024-01-20T11:09:35.880 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 173 M 2024-01-20T11:09:35.880 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:09:39.191 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mds-debuginfo-19.0.0-763.ga00c9940.el9.x86 14 MB/s | 48 MB 00:03 2024-01-20T11:09:39.192 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:09:39.192 INFO:teuthology.orchestra.run.smithi110.stdout:Total 14 MB/s | 48 MB 00:03 2024-01-20T11:09:39.192 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:09:39.197 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:09:39.197 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:09:39.511 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:09:39.511 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:09:39.876 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-test-debuginfo-19.0.0-763.ga00c9940.el9.x8 22 MB/s | 664 MB 00:29 2024-01-20T11:09:39.877 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:09:39.877 INFO:teuthology.orchestra.run.smithi134.stdout:Total 22 MB/s | 664 MB 00:29 2024-01-20T11:09:39.877 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:09:39.883 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:09:39.883 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:09:43.789 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:09:43.789 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:09:44.272 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:09:44.390 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:09:45.763 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:09:46.048 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:09:46.048 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:46.048 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:09:46.048 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:09:46.048 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:46.048 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:09:46.234 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mon-debuginfo 2024-01-20T11:09:46.796 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:06:26 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:09:46.946 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:09:46.947 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:09:46.947 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:09:46.947 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:09:46.947 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:09:46.948 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 99 M 2024-01-20T11:09:46.948 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:46.948 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:09:46.948 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:09:46.948 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:09:46.948 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:09:46.948 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 99 M 2024-01-20T11:09:46.948 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 449 M 2024-01-20T11:09:46.948 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:09:52.110 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mon-debuginfo-19.0.0-763.ga00c9940.el9.x86 19 MB/s | 99 MB 00:05 2024-01-20T11:09:52.111 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:09:52.111 INFO:teuthology.orchestra.run.smithi110.stdout:Total 19 MB/s | 99 MB 00:05 2024-01-20T11:09:52.112 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:09:52.115 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:09:52.116 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:09:52.724 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:09:52.724 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:10:02.992 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:10:03.084 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:10:06.095 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:10:06.365 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:10:06.365 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:10:06.365 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:10:06.365 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:10:06.365 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:10:06.365 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:10:06.555 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-osd-debuginfo 2024-01-20T11:10:07.120 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:06:47 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:10:07.271 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:10:07.271 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:10:07.271 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:10:07.272 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:10:07.272 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:10:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 286 M 2024-01-20T11:10:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:10:07.272 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:10:07.272 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:10:07.272 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:10:07.272 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:10:07.273 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 286 M 2024-01-20T11:10:07.273 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.2 G 2024-01-20T11:10:07.273 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:10:19.886 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-osd-debuginfo-19.0.0-763.ga00c9940.el9.x86 23 MB/s | 286 MB 00:12 2024-01-20T11:10:19.887 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:10:19.888 INFO:teuthology.orchestra.run.smithi110.stdout:Total 23 MB/s | 286 MB 00:12 2024-01-20T11:10:19.888 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:10:19.892 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:10:19.892 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:10:21.611 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:10:21.612 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:10:51.013 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:10:51.109 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:10:53.404 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:10:53.791 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:10:58.722 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:10:58.978 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:10:58.979 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:10:58.979 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:10:58.979 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:10:58.979 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:10:58.979 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:10:59.188 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse-debuginfo 2024-01-20T11:10:59.755 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:07:39 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:10:59.907 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:10:59.908 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:10:59.908 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 5.9 M 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 5.9 M 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 28 M 2024-01-20T11:10:59.909 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:00.570 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-fuse-debuginfo-19.0.0-763.ga00c9940.el9.x8 9.0 MB/s | 5.9 MB 00:00 2024-01-20T11:11:00.571 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:00.571 INFO:teuthology.orchestra.run.smithi110.stdout:Total 9.0 MB/s | 5.9 MB 00:00 2024-01-20T11:11:00.571 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:00.575 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:00.575 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:00.643 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:00.644 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:01.386 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:01.477 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:11:02.036 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:11:02.322 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:11:02.322 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:02.322 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:02.322 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:02.322 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:02.322 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:02.448 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:11:02.502 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel-debuginfo 2024-01-20T11:11:02.733 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:11:02.733 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:02.733 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:11:02.733 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:02.733 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:02.733 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:11:03.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base-debuginfo 2024-01-20T11:11:03.054 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:07:43 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:03.207 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:03.208 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:03.208 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:03.208 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:03.208 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:03.209 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 108 k 2024-01-20T11:11:03.209 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:03.209 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:03.209 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:03.209 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:03.209 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:03.209 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 108 k 2024-01-20T11:11:03.209 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 439 k 2024-01-20T11:11:03.209 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:03.448 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-debuginfo-19.0.0-763.ga00c9940.e 453 kB/s | 108 kB 00:00 2024-01-20T11:11:03.449 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:03.449 INFO:teuthology.orchestra.run.smithi110.stdout:Total 451 kB/s | 108 kB 00:00 2024-01-20T11:11:03.449 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:03.452 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:03.452 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:03.488 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:03.488 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:03.602 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:06:02 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:11:03.603 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:03.696 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:11:03.703 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 is already installed. 2024-01-20T11:11:03.755 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:11:03.756 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-20T11:11:03.756 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:11:03.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-debuginfo 2024-01-20T11:11:04.041 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:11:04.292 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:11:04.292 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:04.292 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:04.292 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:04.292 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:04.292 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:04.399 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:06:03 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:11:04.477 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2-debuginfo 2024-01-20T11:11:04.551 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:11:04.551 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:04.551 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:11:04.551 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:04.551 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:11:04.552 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 20 M 2024-01-20T11:11:04.552 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:04.552 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:11:04.552 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:04.552 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:11:04.552 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:04.552 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 20 M 2024-01-20T11:11:04.553 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 102 M 2024-01-20T11:11:04.553 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:11:05.039 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:07:45 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:05.191 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:05.192 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:05.192 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:05.192 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 5.3 M 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 5.3 M 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 25 M 2024-01-20T11:11:05.193 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:05.653 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-mgr-debuginfo-19.0.0-763.ga00c9940.el9.x86 18 MB/s | 20 MB 00:01 2024-01-20T11:11:05.654 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:05.655 INFO:teuthology.orchestra.run.smithi134.stdout:Total 18 MB/s | 20 MB 00:01 2024-01-20T11:11:05.655 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:11:05.659 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:11:05.659 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:11:05.806 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:11:05.806 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:11:05.968 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs2-debuginfo-19.0.0-763.ga00c9940.el9.x 6.9 MB/s | 5.3 MB 00:00 2024-01-20T11:11:05.969 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:05.969 INFO:teuthology.orchestra.run.smithi110.stdout:Total 6.9 MB/s | 5.3 MB 00:00 2024-01-20T11:11:05.970 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:05.973 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:05.973 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:06.036 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:06.036 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:06.707 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:06.816 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:11:07.345 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:11:07.637 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:11:07.637 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:07.637 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:07.637 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:07.637 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:07.638 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:07.819 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2-debuginfo 2024-01-20T11:11:08.103 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:11:08.252 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:08.381 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:07:48 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:08.533 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 44 M 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:08.534 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:08.535 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 44 M 2024-01-20T11:11:08.535 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 252 M 2024-01-20T11:11:08.535 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:09.199 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:09.482 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:09.482 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:09.482 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:11:09.482 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:09.482 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:09.483 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:11:09.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mds-debuginfo 2024-01-20T11:11:10.242 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:06:09 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:11:10.392 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:11:10.392 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 48 M 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 48 M 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 173 M 2024-01-20T11:11:10.393 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:11:10.672 INFO:teuthology.orchestra.run.smithi110.stdout:librados2-debuginfo-19.0.0-763.ga00c9940.el9.x8 21 MB/s | 44 MB 00:02 2024-01-20T11:11:10.672 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:10.673 INFO:teuthology.orchestra.run.smithi110.stdout:Total 21 MB/s | 44 MB 00:02 2024-01-20T11:11:10.673 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:10.677 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:10.678 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:10.978 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:10.979 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:12.864 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-mds-debuginfo-19.0.0-763.ga00c9940.el9.x86 19 MB/s | 48 MB 00:02 2024-01-20T11:11:12.865 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:12.865 INFO:teuthology.orchestra.run.smithi134.stdout:Total 19 MB/s | 48 MB 00:02 2024-01-20T11:11:12.866 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:11:12.870 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:11:12.870 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:11:13.181 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:11:13.181 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:11:16.018 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:16.104 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:11:17.919 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:11:17.964 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:11:18.012 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:18.234 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:11:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:18.235 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:18.235 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:18.235 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:18.434 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1-debuginfo 2024-01-20T11:11:18.994 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:07:58 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:19.148 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:19.149 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:19.149 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:19.149 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:19.149 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:19.150 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 52 M 2024-01-20T11:11:19.150 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:19.150 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:19.150 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:19.150 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:19.150 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:19.151 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 52 M 2024-01-20T11:11:19.151 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 308 M 2024-01-20T11:11:19.151 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:19.384 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:19.665 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:19.665 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:19.665 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:11:19.666 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:19.666 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:19.666 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:11:19.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mon-debuginfo 2024-01-20T11:11:20.423 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:06:19 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:11:20.572 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:11:20.573 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:20.573 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 99 M 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 99 M 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 449 M 2024-01-20T11:11:20.574 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:11:21.608 INFO:teuthology.orchestra.run.smithi110.stdout:librbd1-debuginfo-19.0.0-763.ga00c9940.el9.x86_ 21 MB/s | 52 MB 00:02 2024-01-20T11:11:21.610 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:21.610 INFO:teuthology.orchestra.run.smithi110.stdout:Total 21 MB/s | 52 MB 00:02 2024-01-20T11:11:21.610 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:21.614 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:21.614 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:21.963 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:21.963 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:25.507 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-mon-debuginfo-19.0.0-763.ga00c9940.el9.x86 20 MB/s | 99 MB 00:04 2024-01-20T11:11:25.508 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:25.509 INFO:teuthology.orchestra.run.smithi134.stdout:Total 20 MB/s | 99 MB 00:04 2024-01-20T11:11:25.509 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:11:25.513 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:11:25.513 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:11:26.123 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:11:26.123 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:11:27.961 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:28.066 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librbd1-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:30.252 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:30.516 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:30.517 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:30.517 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:30.517 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:30.517 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:30.517 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:30.708 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs-debuginfo 2024-01-20T11:11:31.266 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:08:11 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:31.418 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:31.418 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:31.418 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 615 k 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 615 k 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.1 M 2024-01-20T11:11:31.419 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout:python3-cephfs-debuginfo-19.0.0-763.ga00c9940.e 2.2 MB/s | 615 kB 00:00 2024-01-20T11:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.2 MB/s | 615 kB 00:00 2024-01-20T11:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:31.735 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:31.735 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:31.902 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:31.993 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:11:32.383 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:11:32.649 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:11:32.649 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:32.649 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:32.650 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:32.650 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:32.650 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:32.836 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados-debuginfo 2024-01-20T11:11:33.392 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:08:13 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:33.545 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:33.546 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:33.546 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:33.546 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:33.546 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:33.546 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 770 k 2024-01-20T11:11:33.546 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:33.547 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:33.547 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:33.547 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:33.547 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:33.547 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 770 k 2024-01-20T11:11:33.547 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.7 M 2024-01-20T11:11:33.547 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:33.823 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rados-debuginfo-19.0.0-763.ga00c9940.el 2.7 MB/s | 770 kB 00:00 2024-01-20T11:11:33.824 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:33.824 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.7 MB/s | 770 kB 00:00 2024-01-20T11:11:33.824 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:33.827 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:33.827 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:33.864 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:33.865 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:34.042 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:34.148 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-debuginfo-2:19.0.0-763.ga00c9940.el9.x 1/1 2024-01-20T11:11:34.538 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-763.ga00c9940.el9.x 1/1 2024-01-20T11:11:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-763.ga00c9940.el9.x 1/1 2024-01-20T11:11:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:34.800 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:34.800 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:34.801 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:34.990 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd-debuginfo 2024-01-20T11:11:35.556 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:08:15 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:35.706 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 1.1 M 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.1 M 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 4.3 M 2024-01-20T11:11:35.707 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:36.019 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rbd-debuginfo-19.0.0-763.ga00c9940.el9. 3.6 MB/s | 1.1 MB 00:00 2024-01-20T11:11:36.019 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:36.019 INFO:teuthology.orchestra.run.smithi110.stdout:Total 3.6 MB/s | 1.1 MB 00:00 2024-01-20T11:11:36.019 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:36.022 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:36.022 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:36.061 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:36.062 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:36.279 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:36.379 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:11:36.392 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:11:36.471 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:36.793 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:11:37.077 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:11:37.077 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:37.077 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:37.077 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:37.077 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:37.077 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:37.291 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw-debuginfo 2024-01-20T11:11:37.848 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:08:17 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:37.999 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:38.001 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:38.001 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:38.001 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:38.001 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:38.001 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 337 k 2024-01-20T11:11:38.001 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:38.002 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:38.002 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:38.002 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:38.002 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:38.002 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 337 k 2024-01-20T11:11:38.002 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.0 M 2024-01-20T11:11:38.003 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:38.247 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rgw-debuginfo-19.0.0-763.ga00c9940.el9. 1.3 MB/s | 337 kB 00:00 2024-01-20T11:11:38.249 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:38.249 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-01-20T11:11:38.249 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:38.251 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:38.251 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:38.287 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:38.288 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:38.423 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:38.521 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:11:38.900 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:11:39.171 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:11:39.171 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:39.172 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:39.172 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:39.172 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:39.172 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:39.345 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:39.363 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse-debuginfo 2024-01-20T11:11:39.627 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:39.627 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:39.627 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:11:39.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:39.627 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:39.627 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:11:39.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-osd-debuginfo 2024-01-20T11:11:39.922 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:08:19 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:40.073 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 509 k 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:40.074 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:40.075 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 509 k 2024-01-20T11:11:40.075 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.5 M 2024-01-20T11:11:40.075 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:40.416 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:06:39 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:11:40.467 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-debuginfo-19.0.0-763.ga00c9940.el9.x86 1.3 MB/s | 509 kB 00:00 2024-01-20T11:11:40.468 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:40.468 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.3 MB/s | 509 kB 00:00 2024-01-20T11:11:40.468 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:40.472 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:40.472 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:40.507 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:40.507 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:40.566 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:11:40.566 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:40.566 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:11:40.566 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:40.566 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:11:40.567 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 286 M 2024-01-20T11:11:40.567 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:40.567 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:11:40.567 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:11:40.567 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:11:40.567 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:11:40.567 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 286 M 2024-01-20T11:11:40.568 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.2 G 2024-01-20T11:11:40.568 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:11:40.662 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:40.765 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:41.138 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:41.393 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:11:41.393 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:41.393 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:11:41.393 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:11:41.393 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:41.394 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:11:41.611 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror-debuginfo 2024-01-20T11:11:42.174 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:08:22 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:11:42.324 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:11:42.324 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:42.324 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 52 M 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 52 M 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 298 M 2024-01-20T11:11:42.325 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:11:52.578 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-debuginfo-19.0.0-763.ga00c9940.el9.x 5.0 MB/s | 52 MB 00:10 2024-01-20T11:11:52.578 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:52.579 INFO:teuthology.orchestra.run.smithi110.stdout:Total 5.0 MB/s | 52 MB 00:10 2024-01-20T11:11:52.579 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:11:52.583 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:11:52.583 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:11:52.920 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:11:52.920 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:11:55.032 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-osd-debuginfo-19.0.0-763.ga00c9940.el9.x86 20 MB/s | 286 MB 00:14 2024-01-20T11:11:55.033 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:11:55.033 INFO:teuthology.orchestra.run.smithi134.stdout:Total 20 MB/s | 286 MB 00:14 2024-01-20T11:11:55.034 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:11:55.037 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:11:55.038 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:11:56.749 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:11:56.749 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:11:58.816 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:11:58.901 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:12:01.019 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:12:01.287 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:12:01.287 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:12:01.287 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:12:01.287 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:12:01.287 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:12:01.288 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:12:01.466 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd-debuginfo 2024-01-20T11:12:02.026 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:08:42 ago on Sat 20 Jan 2024 11:03:20 AM UTC. 2024-01-20T11:12:02.179 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-20T11:12:02.179 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:12:02.179 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-20T11:12:02.179 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:12:02.179 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-20T11:12:02.179 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 1.2 M 2024-01-20T11:12:02.179 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:12:02.180 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-20T11:12:02.180 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-20T11:12:02.180 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-20T11:12:02.180 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:12:02.180 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.2 M 2024-01-20T11:12:02.180 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 5.5 M 2024-01-20T11:12:02.180 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-20T11:12:02.656 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-debuginfo-19.0.0-763.ga00c9940.el9.x86_ 2.6 MB/s | 1.2 MB 00:00 2024-01-20T11:12:02.656 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:12:02.656 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.6 MB/s | 1.2 MB 00:00 2024-01-20T11:12:02.657 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-20T11:12:02.660 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-20T11:12:02.660 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-20T11:12:02.701 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-20T11:12:02.702 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-20T11:12:02.936 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-20T11:12:03.038 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:12:03.462 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:12:03.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:12:03.729 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:12:03.729 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-20T11:12:03.729 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:12:03.730 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:12:03.730 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-20T11:12:03.902 DEBUG:teuthology.parallel:result is None 2024-01-20T11:12:26.137 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:12:26.244 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:12:33.543 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:12:33.827 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:12:33.827 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:33.827 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:12:33.827 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:12:33.827 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:33.827 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:12:34.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse-debuginfo 2024-01-20T11:12:34.603 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:07:33 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:12:34.754 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:12:34.755 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:34.755 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:12:34.755 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 5.9 M 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 5.9 M 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 28 M 2024-01-20T11:12:34.756 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:12:35.564 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-fuse-debuginfo-19.0.0-763.ga00c9940.el9.x8 7.4 MB/s | 5.9 MB 00:00 2024-01-20T11:12:35.564 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:12:35.565 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.3 MB/s | 5.9 MB 00:00 2024-01-20T11:12:35.565 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:12:35.569 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:12:35.569 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:12:35.637 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:12:35.638 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:12:36.374 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:12:36.471 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:12:37.041 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:12:37.330 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:12:37.330 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:37.330 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:12:37.330 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:12:37.330 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:37.330 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:12:37.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel-debuginfo 2024-01-20T11:12:38.102 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:07:37 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:12:38.254 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:12:38.255 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:38.255 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:12:38.255 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:38.255 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:12:38.255 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 108 k 2024-01-20T11:12:38.255 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:38.256 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:12:38.256 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:38.256 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:12:38.256 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:38.256 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 108 k 2024-01-20T11:12:38.256 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 439 k 2024-01-20T11:12:38.256 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:12:38.495 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-debuginfo-19.0.0-763.ga00c9940.e 460 kB/s | 108 kB 00:00 2024-01-20T11:12:38.496 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:12:38.496 INFO:teuthology.orchestra.run.smithi134.stdout:Total 457 kB/s | 108 kB 00:00 2024-01-20T11:12:38.496 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:12:38.496 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:12:38.496 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:12:38.529 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:12:38.530 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:12:38.641 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:12:38.731 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:12:39.089 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:12:39.368 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:12:39.368 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:39.368 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:12:39.368 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:12:39.368 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:39.368 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:12:39.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2-debuginfo 2024-01-20T11:12:40.118 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:07:39 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:12:40.267 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:12:40.267 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 5.3 M 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 5.3 M 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 25 M 2024-01-20T11:12:40.268 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:12:40.778 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs2-debuginfo-19.0.0-763.ga00c9940.el9.x 11 MB/s | 5.3 MB 00:00 2024-01-20T11:12:40.778 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:12:40.779 INFO:teuthology.orchestra.run.smithi134.stdout:Total 10 MB/s | 5.3 MB 00:00 2024-01-20T11:12:40.779 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:12:40.782 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:12:40.782 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:12:40.846 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:12:40.847 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:12:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:12:41.600 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:12:42.118 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:12:42.405 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:12:42.405 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:42.405 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:12:42.405 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:12:42.405 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:42.405 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:12:42.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2-debuginfo 2024-01-20T11:12:43.191 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:07:42 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:12:43.340 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:12:43.341 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:43.341 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:12:43.341 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:43.341 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:12:43.341 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 44 M 2024-01-20T11:12:43.341 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:43.341 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:12:43.341 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:43.341 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:12:43.342 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:43.342 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 44 M 2024-01-20T11:12:43.342 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 252 M 2024-01-20T11:12:43.342 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:12:45.689 INFO:teuthology.orchestra.run.smithi134.stdout:librados2-debuginfo-19.0.0-763.ga00c9940.el9.x8 19 MB/s | 44 MB 00:02 2024-01-20T11:12:45.690 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:12:45.690 INFO:teuthology.orchestra.run.smithi134.stdout:Total 19 MB/s | 44 MB 00:02 2024-01-20T11:12:45.690 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:12:45.694 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:12:45.694 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:12:45.996 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:12:45.997 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:12:51.017 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:12:51.123 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:12:52.923 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:12:53.197 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_6 1/1 2024-01-20T11:12:53.197 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:53.197 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:12:53.198 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:12:53.198 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:53.198 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:12:53.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1-debuginfo 2024-01-20T11:12:53.971 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:07:52 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:12:54.123 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:12:54.123 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:54.123 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:12:54.123 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 52 M 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 52 M 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 308 M 2024-01-20T11:12:54.124 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:12:57.516 INFO:teuthology.orchestra.run.smithi134.stdout:librbd1-debuginfo-19.0.0-763.ga00c9940.el9.x86_ 15 MB/s | 52 MB 00:03 2024-01-20T11:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:12:57.520 INFO:teuthology.orchestra.run.smithi134.stdout:Total 15 MB/s | 52 MB 00:03 2024-01-20T11:12:57.520 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:12:57.521 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:12:57.521 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:12:57.868 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:12:57.868 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:13:03.858 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:13:03.947 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librbd1-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:13:06.169 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:13:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:13:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:06.437 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:13:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:13:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:06.437 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:13:06.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs-debuginfo 2024-01-20T11:13:07.239 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:08:06 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:13:07.400 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 615 k 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:13:07.401 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:07.402 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 615 k 2024-01-20T11:13:07.402 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.1 M 2024-01-20T11:13:07.402 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:13:07.851 INFO:teuthology.orchestra.run.smithi134.stdout:python3-cephfs-debuginfo-19.0.0-763.ga00c9940.e 1.3 MB/s | 615 kB 00:00 2024-01-20T11:13:07.852 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:13:07.852 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.3 MB/s | 615 kB 00:00 2024-01-20T11:13:07.852 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:13:07.856 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:13:07.856 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:13:07.893 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:13:07.893 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:13:08.065 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:13:08.167 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:13:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:13:08.870 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-763.ga00c9940.el9. 1/1 2024-01-20T11:13:08.870 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:08.870 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:13:08.870 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:13:08.870 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:08.870 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:13:09.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados-debuginfo 2024-01-20T11:13:09.626 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:08:08 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:13:09.778 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:13:09.778 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:09.779 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:13:09.779 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:09.779 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:13:09.779 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 770 k 2024-01-20T11:13:09.779 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:09.779 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:13:09.780 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:09.780 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:13:09.780 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:09.780 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 770 k 2024-01-20T11:13:09.780 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.7 M 2024-01-20T11:13:09.781 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:13:10.079 INFO:teuthology.orchestra.run.smithi134.stdout:python3-rados-debuginfo-19.0.0-763.ga00c9940.el 2.5 MB/s | 770 kB 00:00 2024-01-20T11:13:10.080 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:13:10.080 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.5 MB/s | 770 kB 00:00 2024-01-20T11:13:10.080 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:13:10.084 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:13:10.084 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:13:10.120 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:13:10.121 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:13:10.300 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:13:10.394 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-debuginfo-2:19.0.0-763.ga00c9940.el9.x 1/1 2024-01-20T11:13:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-763.ga00c9940.el9.x 1/1 2024-01-20T11:13:11.053 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-763.ga00c9940.el9.x 1/1 2024-01-20T11:13:11.053 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:11.053 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:13:11.053 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:13:11.053 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:11.053 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:13:11.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd-debuginfo 2024-01-20T11:13:11.820 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:08:10 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:13:11.969 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:13:11.970 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:11.970 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:13:11.970 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:11.970 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:13:11.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 1.1 M 2024-01-20T11:13:11.970 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:11.970 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:13:11.971 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:11.971 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:13:11.971 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:11.971 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.1 M 2024-01-20T11:13:11.971 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 4.3 M 2024-01-20T11:13:11.971 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:13:12.857 INFO:teuthology.orchestra.run.smithi134.stdout:python3-rbd-debuginfo-19.0.0-763.ga00c9940.el9. 1.3 MB/s | 1.1 MB 00:00 2024-01-20T11:13:12.858 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:13:12.858 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.3 MB/s | 1.1 MB 00:00 2024-01-20T11:13:12.858 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:13:12.862 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:13:12.862 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:13:12.903 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:13:12.904 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:13:13.123 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:13:13.231 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:13:13.629 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:13:13.934 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:13:13.934 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:13.934 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:13:13.935 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:13:13.935 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:13.935 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:13:14.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw-debuginfo 2024-01-20T11:13:14.723 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:08:13 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:13:14.875 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 337 k 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 337 k 2024-01-20T11:13:14.876 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.0 M 2024-01-20T11:13:14.877 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:13:15.294 INFO:teuthology.orchestra.run.smithi134.stdout:python3-rgw-debuginfo-19.0.0-763.ga00c9940.el9. 809 kB/s | 337 kB 00:00 2024-01-20T11:13:15.294 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:13:15.295 INFO:teuthology.orchestra.run.smithi134.stdout:Total 806 kB/s | 337 kB 00:00 2024-01-20T11:13:15.295 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:13:15.298 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:13:15.298 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:13:15.333 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:13:15.333 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:13:15.468 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:13:15.558 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:13:15.970 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:13:16.262 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86 1/1 2024-01-20T11:13:16.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:16.262 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:13:16.263 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:13:16.263 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:16.263 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:13:16.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse-debuginfo 2024-01-20T11:13:17.049 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:08:16 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:13:17.199 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:13:17.200 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:13:17.200 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:17.200 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:13:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 509 k 2024-01-20T11:13:17.200 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:17.200 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:13:17.200 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:17.200 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:13:17.201 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:17.201 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 509 k 2024-01-20T11:13:17.201 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.5 M 2024-01-20T11:13:17.201 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:13:17.747 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-debuginfo-19.0.0-763.ga00c9940.el9.x86 932 kB/s | 509 kB 00:00 2024-01-20T11:13:17.747 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:13:17.748 INFO:teuthology.orchestra.run.smithi134.stdout:Total 929 kB/s | 509 kB 00:00 2024-01-20T11:13:17.748 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:13:17.751 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:13:17.751 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:13:17.789 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:13:17.789 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:13:17.950 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:13:18.054 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:13:18.475 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:13:18.760 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:13:18.761 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:18.761 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:13:18.761 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:13:18.761 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:18.761 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:13:18.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror-debuginfo 2024-01-20T11:13:19.542 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:08:18 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:13:19.692 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:13:19.693 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:19.693 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:13:19.693 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 52 M 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 52 M 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 298 M 2024-01-20T11:13:19.694 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:13:23.566 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-debuginfo-19.0.0-763.ga00c9940.el9.x 13 MB/s | 52 MB 00:03 2024-01-20T11:13:23.587 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:13:23.587 INFO:teuthology.orchestra.run.smithi134.stdout:Total 13 MB/s | 52 MB 00:03 2024-01-20T11:13:23.587 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:13:23.587 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:13:23.587 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:13:23.908 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:13:23.908 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:13:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:13:29.898 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:13:31.927 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:13:32.210 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_ 1/1 2024-01-20T11:13:32.210 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:32.210 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:13:32.210 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:13:32.210 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:32.210 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:13:32.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd-debuginfo 2024-01-20T11:13:32.985 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:08:31 ago on Sat 20 Jan 2024 11:05:01 AM UTC. 2024-01-20T11:13:33.137 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-20T11:13:33.138 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:33.138 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-763.ga00c9940.el9 ceph 1.2 M 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.2 M 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 5.5 M 2024-01-20T11:13:33.139 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-20T11:13:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-debuginfo-19.0.0-763.ga00c9940.el9.x86_ 4.0 MB/s | 1.2 MB 00:00 2024-01-20T11:13:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-20T11:13:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:Total 4.0 MB/s | 1.2 MB 00:00 2024-01-20T11:13:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-20T11:13:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-20T11:13:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-20T11:13:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-20T11:13:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-20T11:13:33.730 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-20T11:13:33.834 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:13:34.242 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:13:34.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 1/1 2024-01-20T11:13:34.533 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:34.533 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-20T11:13:34.533 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-debuginfo-2:19.0.0-763.ga00c9940.el9.x86_64 2024-01-20T11:13:34.533 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:34.533 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-20T11:13:34.741 DEBUG:teuthology.parallel:result is None 2024-01-20T11:13:34.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a00c9940991c5e1e9ab595faa3f6e1281d8ae167 2024-01-20T11:13:34.888 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-20T11:13:34.914 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-763.ga00c9940.el9 2024-01-20T11:13:34.915 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-763.ga00c9940.el9 2024-01-20T11:13:34.915 INFO:teuthology.task.install:The correct ceph version 19.0.0-763.ga00c9940 is installed. 2024-01-20T11:13:34.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a00c9940991c5e1e9ab595faa3f6e1281d8ae167 2024-01-20T11:13:35.142 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-20T11:13:35.162 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-763.ga00c9940.el9 2024-01-20T11:13:35.163 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-763.ga00c9940.el9 2024-01-20T11:13:35.163 INFO:teuthology.task.install:The correct ceph version 19.0.0-763.ga00c9940 is installed. 2024-01-20T11:13:35.165 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-20T11:13:35.165 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:35.165 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-20T11:13:35.199 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:13:35.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-20T11:13:35.233 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-20T11:13:35.233 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:35.233 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-01-20T11:13:35.270 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-20T11:13:35.339 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:13:35.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-01-20T11:13:35.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-20T11:13:35.435 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-20T11:13:35.435 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:35.435 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-20T11:13:35.463 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-20T11:13:35.531 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:13:35.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-20T11:13:35.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-20T11:13:35.625 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-20T11:13:35.626 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:35.626 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-01-20T11:13:35.653 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-20T11:13:35.720 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:13:35.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-01-20T11:13:35.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-20T11:13:35.814 INFO:teuthology.run_tasks:Running task ceph... 2024-01-20T11:13:35.902 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-20T11:13:35.902 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2024-01-20T11:13:35.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-01-20T11:13:35.931 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-20T11:13:35.932 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-20T11:13:35.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-20T11:13:35.998 INFO:tasks.ceph:Creating extra log directories... 2024-01-20T11:13:35.998 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-20T11:13:36.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-20T11:13:36.077 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-20T11:13:36.077 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-20T11:13:36.077 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621', 'branch': 'main', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7522621', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 587, 'sha1': 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'a00c9940991c5e1e9ab595faa3f6e1281d8ae167', 'targets': {'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtYQuegvVyO6UZG27OP9qpqVkjKdMu9D5GDJXb4WYgvaW0gqlywdzyv+IURvYFDwwphGS9JaMvWDYVc0Logf+c=', 'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNf/dmXNPIdSY8E4I0n7Bmhwc9mJyxV0TODuUn9bQpvSl5ohdkYiIji26ylANr2L59U4qZ8itxKut7Q+UOPlBjU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-19_15:25:09', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-20T11:13:36.078 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-20T11:13:36.106 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-20T11:13:36.121 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2024-01-20T11:13:36.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-01-20T11:13:36.188 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:36.189 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-01-20T11:13:36.241 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-20T11:13:36.241 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-01-20T11:13:36.295 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-20T11:13:36.296 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-20T11:13:36.296 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-01-20T11:13:36.296 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-20T11:13:36.296 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-01-20T11:13:36.296 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-01-20 11:05:08.541851099 +0000 2024-01-20T11:13:36.296 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-01-20 11:01:42.799149482 +0000 2024-01-20T11:13:36.296 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-01-20 11:01:42.799149482 +0000 2024-01-20T11:13:36.296 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-01-20 11:01:42.799149482 +0000 2024-01-20T11:13:36.296 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-20T11:13:36.360 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-01-20T11:13:36.361 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-01-20T11:13:36.361 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 7.673e-05 s, 6.7 MB/s 2024-01-20T11:13:36.362 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-20T11:13:36.417 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-01-20T11:13:36.472 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-20T11:13:36.472 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-20T11:13:36.472 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-01-20T11:13:36.472 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-20T11:13:36.472 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-01-20T11:13:36.472 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-01-20 11:05:08.542851083 +0000 2024-01-20T11:13:36.473 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-01-20 11:01:43.122144391 +0000 2024-01-20T11:13:36.473 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-01-20 11:01:43.122144391 +0000 2024-01-20T11:13:36.473 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-01-20 11:01:43.122144391 +0000 2024-01-20T11:13:36.473 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-20T11:13:36.537 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-01-20T11:13:36.537 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-01-20T11:13:36.538 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 7.8547e-05 s, 6.5 MB/s 2024-01-20T11:13:36.539 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-20T11:13:36.594 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-01-20T11:13:36.649 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-20T11:13:36.649 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-20T11:13:36.649 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-01-20T11:13:36.649 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-20T11:13:36.650 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-01-20T11:13:36.650 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-01-20 11:05:08.542851083 +0000 2024-01-20T11:13:36.650 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-01-20 11:01:43.434139473 +0000 2024-01-20T11:13:36.650 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-01-20 11:01:43.434139473 +0000 2024-01-20T11:13:36.650 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-01-20 11:01:43.434139473 +0000 2024-01-20T11:13:36.650 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-20T11:13:36.715 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-01-20T11:13:36.715 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-01-20T11:13:36.715 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 7.9879e-05 s, 6.4 MB/s 2024-01-20T11:13:36.716 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-20T11:13:36.772 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-01-20T11:13:36.826 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-20T11:13:36.826 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-20T11:13:36.826 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-01-20T11:13:36.826 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-20T11:13:36.826 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-01-20T11:13:36.827 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-01-20 11:05:08.542851083 +0000 2024-01-20T11:13:36.827 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-01-20 11:01:43.750134492 +0000 2024-01-20T11:13:36.827 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-01-20 11:01:43.750134492 +0000 2024-01-20T11:13:36.827 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-01-20 11:01:43.750134492 +0000 2024-01-20T11:13:36.827 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-20T11:13:36.891 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-01-20T11:13:36.892 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-01-20T11:13:36.892 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 7.8517e-05 s, 6.5 MB/s 2024-01-20T11:13:36.893 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-20T11:13:36.948 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:36.948 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:13:36.948 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-01-20T11:13:36.965 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-20T11:13:36.965 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-01-20T11:13:37.022 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-20T11:13:37.022 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-20T11:13:37.022 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-01-20T11:13:37.022 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-20T11:13:37.022 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-20T11:13:37.022 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-20 11:06:44.370019304 +0000 2024-01-20T11:13:37.022 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-20 11:01:42.805558388 +0000 2024-01-20T11:13:37.022 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-20 11:01:42.805558388 +0000 2024-01-20T11:13:37.023 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-20 11:01:42.805558388 +0000 2024-01-20T11:13:37.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-20T11:13:37.090 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-20T11:13:37.090 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-20T11:13:37.091 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000196549 s, 2.6 MB/s 2024-01-20T11:13:37.092 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-20T11:13:37.148 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-01-20T11:13:37.205 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-20T11:13:37.205 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-20T11:13:37.205 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-01-20T11:13:37.205 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-20T11:13:37.205 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-20T11:13:37.205 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-20 11:06:44.371019281 +0000 2024-01-20T11:13:37.206 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-20 11:01:43.152550325 +0000 2024-01-20T11:13:37.206 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-20 11:01:43.152550325 +0000 2024-01-20T11:13:37.206 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-20 11:01:43.152550325 +0000 2024-01-20T11:13:37.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-20T11:13:37.274 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-20T11:13:37.274 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-20T11:13:37.274 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000211869 s, 2.4 MB/s 2024-01-20T11:13:37.275 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-20T11:13:37.331 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-01-20T11:13:37.386 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-20T11:13:37.387 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-20T11:13:37.387 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-01-20T11:13:37.387 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-20T11:13:37.387 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-20T11:13:37.387 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-20 11:06:44.371019281 +0000 2024-01-20T11:13:37.387 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-20 11:01:43.502542194 +0000 2024-01-20T11:13:37.387 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-20 11:01:43.502542194 +0000 2024-01-20T11:13:37.387 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-20 11:01:43.502542194 +0000 2024-01-20T11:13:37.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-20T11:13:37.452 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-20T11:13:37.452 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-20T11:13:37.452 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000256203 s, 2.0 MB/s 2024-01-20T11:13:37.453 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-20T11:13:37.509 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-01-20T11:13:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-20T11:13:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-20T11:13:37.565 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-01-20T11:13:37.565 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-20T11:13:37.565 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-20T11:13:37.566 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-20 11:06:44.371019281 +0000 2024-01-20T11:13:37.566 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-20 11:01:43.840534341 +0000 2024-01-20T11:13:37.566 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-20 11:01:43.840534341 +0000 2024-01-20T11:13:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-20 11:01:43.840534341 +0000 2024-01-20T11:13:37.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-20T11:13:37.631 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-20T11:13:37.631 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-20T11:13:37.632 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000208366 s, 2.5 MB/s 2024-01-20T11:13:37.633 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-20T11:13:37.689 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:37.689 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-20T11:13:37.689 INFO:tasks.ceph:Generating config... 2024-01-20T11:13:37.690 INFO:tasks.ceph:[client] debug monc = 20 2024-01-20T11:13:37.691 INFO:tasks.ceph:[client] debug ms = 1 2024-01-20T11:13:37.691 INFO:tasks.ceph:[client] debug objecter = 20 2024-01-20T11:13:37.691 INFO:tasks.ceph:[client] debug rados = 20 2024-01-20T11:13:37.691 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-01-20T11:13:37.691 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-01-20T11:13:37.691 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-01-20T11:13:37.691 INFO:tasks.ceph:[global] ms type = async 2024-01-20T11:13:37.691 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-01-20T11:13:37.691 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-20T11:13:37.691 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-20T11:13:37.691 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-20T11:13:37.691 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-20T11:13:37.691 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-20T11:13:37.691 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-01-20T11:13:37.691 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-01-20T11:13:37.692 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-01-20T11:13:37.692 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-01-20T11:13:37.692 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-01-20T11:13:37.692 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-01-20T11:13:37.692 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-01-20T11:13:37.692 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-01-20T11:13:37.692 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-20T11:13:37.692 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-20T11:13:37.692 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-20T11:13:37.692 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] osd class default list = * 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] osd class load list = * 2024-01-20T11:13:37.696 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-01-20T11:13:37.697 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-20T11:13:37.697 INFO:tasks.ceph:Setting up mon.a... 2024-01-20T11:13:37.698 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-20T11:13:37.753 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.keyring 2024-01-20T11:13:37.755 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-20T11:13:37.800 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-20T11:13:37.866 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.110'), ('mon.c', '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]'), ('mon.b', '172.21.15.134')] 2024-01-20T11:13:37.866 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.134', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-01-20T11:13:37.867 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:37.867 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-20T11:13:37.921 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.110 --addv c '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]' --add b 172.21.15.134 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:setting min_mon_release = quincy 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 17 (quincy) 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:election_strategy: 1 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:13:38.005 INFO:teuthology.orchestra.run.smithi110.stdout:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:13:38.006 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-20T11:13:38.007 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-20T11:13:38.021 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 16ed1bef-072b-4816-8684-5d140f76194c... 2024-01-20T11:13:38.022 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-20T11:13:38.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-20T11:13:38.111 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2024-01-20T11:13:38.111 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2024-01-20T11:13:38.111 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-20T11:13:38.111 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2024-01-20T11:13:38.111 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.111 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2024-01-20T11:13:38.111 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.111 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2024-01-20T11:13:38.111 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2024-01-20T11:13:38.112 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2024-01-20T11:13:38.113 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.134" 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: mon election default strategy = 1 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 5000 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: ms type = async 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: osd heartbeat grace = 80 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.114 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug trim objects = true 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.115 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2024-01-20T11:13:38.116 INFO:teuthology.orchestra.run.smithi110.stdout: bdev async discard = True 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: bdev enable discard = True 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore allocator = bitmap 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore zero block detection = True 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 1 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 1 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 10 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2024-01-20T11:13:38.117 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd class default list = * 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd class load list = * 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd client watch timeout = 120 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify cached snaps = True 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify missing on start = True 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd fast shutdown = False 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock override recovery settings = True 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue = debug_random 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue cut off = debug_random 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: osd shutdown pgref assert = True 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2024-01-20T11:13:38.118 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2024-01-20T11:13:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2024-01-20T11:13:38.120 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: mon min osdmap epochs = 50 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd crush smoke test = False 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune interval = 2 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune min = 15 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune txsize = 2 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub interval = 300 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on pool no app = False 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: paxos service trim min = 10 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2024-01-20T11:13:38.121 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2024-01-20T11:13:38.122 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-20T11:13:38.122 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-20T11:13:38.122 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-01-20T11:13:38.122 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-01-20T11:13:38.122 INFO:teuthology.orchestra.run.smithi110.stdout: debug objecter = 20 2024-01-20T11:13:38.122 INFO:teuthology.orchestra.run.smithi110.stdout: debug rados = 20 2024-01-20T11:13:38.122 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2024-01-20T11:13:38.122 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2024-01-20T11:13:38.122 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.123 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-01-20T11:13:38.124 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-01-20T11:13:38.125 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = "172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.134" 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: mon client directed command retry = 5 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: mon election default strategy = 1 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: ms inject socket failures = 5000 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: ms type = async 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: osd heartbeat grace = 80 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-01-20T11:13:38.126 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug trim objects = true 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.127 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: bdev async discard = True 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: bdev enable discard = True 2024-01-20T11:13:38.128 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore allocator = bitmap 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore zero block detection = True 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 1 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 1 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: debug monc = 20 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 10 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: osd class default list = * 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: osd class load list = * 2024-01-20T11:13:38.129 INFO:teuthology.orchestra.run.smithi134.stdout: osd client watch timeout = 120 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify cached snaps = True 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify missing on start = True 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: osd fast shutdown = False 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock override recovery settings = True 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: osd op queue = debug_random 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: osd op queue cut off = debug_random 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: osd shutdown pgref assert = True 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-01-20T11:13:38.130 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-01-20T11:13:38.131 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.132 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: mon min osdmap epochs = 50 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd crush smoke test = False 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune interval = 2 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune min = 15 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune txsize = 2 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: mon scrub interval = 300 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on pool no app = False 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: paxos service trim min = 10 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-01-20T11:13:38.133 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-20T11:13:38.134 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-20T11:13:38.134 INFO:teuthology.orchestra.run.smithi134.stdout: debug monc = 20 2024-01-20T11:13:38.134 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-01-20T11:13:38.134 INFO:teuthology.orchestra.run.smithi134.stdout: debug objecter = 20 2024-01-20T11:13:38.134 INFO:teuthology.orchestra.run.smithi134.stdout: debug rados = 20 2024-01-20T11:13:38.134 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-01-20T11:13:38.134 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.c] 2024-01-20T11:13:38.134 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.b] 2024-01-20T11:13:38.136 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-20T11:13:38.136 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-20T11:13:38.231 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-20T11:13:38.231 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:38.231 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-20T11:13:38.246 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:38.246 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-20T11:13:38.302 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:13:38.302 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:38.302 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-20T11:13:38.302 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-20T11:13:38.381 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:13:38.381 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:38.435 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:13:38.436 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:13:38.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-20T11:13:38.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-20T11:13:38.476 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:13:38.476 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:38.531 INFO:tasks.ceph:Setting up mon nodes... 2024-01-20T11:13:38.532 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-20T11:13:38.532 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-20T11:13:38.586 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-20T11:13:38.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-20T11:13:38.651 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-20T11:13:38.653 INFO:tasks.ceph:Setting up mds nodes... 2024-01-20T11:13:38.654 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-20T11:13:38.654 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-20T11:13:38.696 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-20T11:13:38.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-20T11:13:38.753 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-20T11:13:38.768 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-20T11:13:38.768 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-20T11:13:38.769 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-20T11:13:38.795 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:38.795 INFO:tasks.ceph:role: osd.0 2024-01-20T11:13:38.795 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:13:38.795 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-20T11:13:38.861 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-20T11:13:38.861 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-20T11:13:38.861 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-20T11:13:38.861 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-20T11:13:38.861 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-20T11:13:38.861 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-01-20T11:13:38.861 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-20T11:13:38.861 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-20T11:13:38.862 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-20T11:13:38.862 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-20T11:13:38.894 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-01-20T11:13:38.895 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-01-20T11:13:38.896 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-20T11:13:38.966 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-20T11:13:39.038 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-20T11:13:39.104 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:39.104 INFO:tasks.ceph:role: osd.1 2024-01-20T11:13:39.104 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:13:39.104 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-20T11:13:39.170 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-20T11:13:39.171 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-20T11:13:39.171 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-20T11:13:39.171 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-20T11:13:39.171 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-20T11:13:39.171 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-01-20T11:13:39.171 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-20T11:13:39.171 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-20T11:13:39.171 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-20T11:13:39.171 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-20T11:13:39.203 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-01-20T11:13:39.205 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-01-20T11:13:39.205 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-20T11:13:39.275 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-20T11:13:39.346 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-20T11:13:39.413 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:39.413 INFO:tasks.ceph:role: osd.2 2024-01-20T11:13:39.413 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:13:39.413 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-20T11:13:39.480 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-20T11:13:39.480 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-20T11:13:39.480 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-20T11:13:39.480 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-20T11:13:39.480 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-20T11:13:39.480 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-01-20T11:13:39.480 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-20T11:13:39.480 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-20T11:13:39.480 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-20T11:13:39.481 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-20T11:13:39.512 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-01-20T11:13:39.514 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-01-20T11:13:39.514 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-20T11:13:39.588 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-20T11:13:39.659 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-20T11:13:39.726 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:39.726 INFO:tasks.ceph:role: osd.3 2024-01-20T11:13:39.726 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:13:39.727 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-20T11:13:39.793 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-20T11:13:39.793 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-20T11:13:39.793 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-20T11:13:39.793 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-20T11:13:39.793 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-20T11:13:39.794 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-01-20T11:13:39.794 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-20T11:13:39.794 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-20T11:13:39.794 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-20T11:13:39.794 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-20T11:13:39.826 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-01-20T11:13:39.828 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-01-20T11:13:39.828 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-20T11:13:39.898 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-20T11:13:39.969 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:40.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:40.066+0000 7f34de40e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-20T11:13:40.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:40.066+0000 7f34de40e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-20T11:13:40.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:40.066+0000 7f34de40e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-20T11:13:40.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:40.066+0000 7f34de40e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-20T11:13:40.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:40.066+0000 7f34de40e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-20T11:13:40.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:40.066+0000 7f34de40e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-20T11:13:42.904 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-20T11:13:42.936 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:43.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:43.026+0000 7f7ab880c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-20T11:13:43.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:43.026+0000 7f7ab880c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-20T11:13:43.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:43.026+0000 7f7ab880c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-20T11:13:43.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:43.027+0000 7f7ab880c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-20T11:13:43.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:43.027+0000 7f7ab880c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-20T11:13:43.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:43.027+0000 7f7ab880c640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-20T11:13:45.863 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-20T11:13:45.894 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:45.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:45.985+0000 7f886cc13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-20T11:13:45.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:45.985+0000 7f886cc13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-20T11:13:45.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:45.985+0000 7f886cc13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-20T11:13:45.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:45.985+0000 7f886cc13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-20T11:13:45.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:45.985+0000 7f886cc13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-20T11:13:45.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:45.985+0000 7f886cc13640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-20T11:13:48.821 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-20T11:13:48.852 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:48.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:48.942+0000 7f38af5bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-20T11:13:48.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:48.942+0000 7f38af5bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-20T11:13:48.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:48.942+0000 7f38af5bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-20T11:13:48.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:48.942+0000 7f38af5bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-20T11:13:48.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:48.942+0000 7f38af5bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-20T11:13:48.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:13:48.942+0000 7f38af5bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-20T11:13:51.778 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-20T11:13:51.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-20T11:13:51.842 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:51.842 INFO:tasks.ceph:role: osd.4 2024-01-20T11:13:51.842 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:13:51.842 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-20T11:13:51.920 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-20T11:13:51.920 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-20T11:13:51.920 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-20T11:13:51.921 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-20T11:13:51.921 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-20T11:13:51.921 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-01-20T11:13:51.921 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-20T11:13:51.921 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-20T11:13:51.921 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-20T11:13:51.921 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-20T11:13:52.437 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-01-20T11:13:52.439 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-01-20T11:13:52.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-20T11:13:52.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-20T11:13:52.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-20T11:13:52.654 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:52.655 INFO:tasks.ceph:role: osd.5 2024-01-20T11:13:52.655 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:13:52.655 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-20T11:13:52.729 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-20T11:13:52.730 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-20T11:13:52.730 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-20T11:13:52.730 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-20T11:13:52.730 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-20T11:13:52.730 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-01-20T11:13:52.730 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-20T11:13:52.730 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-20T11:13:52.730 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-20T11:13:52.730 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-20T11:13:53.180 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-01-20T11:13:53.182 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-01-20T11:13:53.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-20T11:13:53.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-20T11:13:53.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-20T11:13:53.384 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:53.385 INFO:tasks.ceph:role: osd.6 2024-01-20T11:13:53.385 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:13:53.385 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-20T11:13:53.460 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-20T11:13:53.946 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-01-20T11:13:53.948 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-01-20T11:13:53.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-20T11:13:54.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-20T11:13:54.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-20T11:13:54.189 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-20T11:13:54.189 INFO:tasks.ceph:role: osd.7 2024-01-20T11:13:54.189 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:13:54.189 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-20T11:13:54.263 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-20T11:13:54.263 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-20T11:13:54.263 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-20T11:13:54.263 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-20T11:13:54.264 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-20T11:13:54.264 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-01-20T11:13:54.264 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-20T11:13:54.264 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-20T11:13:54.264 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-20T11:13:54.264 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-20T11:13:54.776 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-01-20T11:13:54.778 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-01-20T11:13:54.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-20T11:13:54.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-20T11:13:54.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:55.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:55.067+0000 7ffa89e17640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-20T11:13:55.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:55.067+0000 7ffa89e17640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-20T11:13:55.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:55.067+0000 7ffa89e17640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-20T11:13:55.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:55.068+0000 7ffa89e17640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-20T11:13:55.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:55.068+0000 7ffa89e17640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-20T11:13:55.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:55.068+0000 7ffa89e17640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-20T11:13:57.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-20T11:13:57.950 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:13:58.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:58.043+0000 7f889be11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-20T11:13:58.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:58.043+0000 7f889be11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-20T11:13:58.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:58.043+0000 7f889be11640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-20T11:13:58.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:58.043+0000 7f889be11640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-20T11:13:58.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:58.043+0000 7f889be11640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-20T11:13:58.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:13:58.044+0000 7f889be11640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-20T11:14:00.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-20T11:14:00.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:14:01.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:01.014+0000 7f96f0a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-20T11:14:01.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:01.014+0000 7f96f0a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-20T11:14:01.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:01.014+0000 7f96f0a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-20T11:14:01.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:01.014+0000 7f96f0a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-20T11:14:01.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:01.014+0000 7f96f0a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-20T11:14:01.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:01.015+0000 7f96f0a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-20T11:14:03.858 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-20T11:14:03.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:14:03.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:03.981+0000 7f060f138640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-20T11:14:03.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:03.981+0000 7f060f138640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-20T11:14:03.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:03.981+0000 7f060f138640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-20T11:14:03.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:03.981+0000 7f060f138640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-20T11:14:03.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:03.981+0000 7f060f138640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-20T11:14:03.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:03.982+0000 7f060f138640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-20T11:14:06.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-20T11:14:06.864 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-20T11:14:06.865 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:06.865 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-20T11:14:06.896 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:06.897 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-20T11:14:06.968 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:06.968 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-20T11:14:07.038 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:07.038 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-20T11:14:07.106 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:07.106 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-20T11:14:07.175 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:07.175 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-20T11:14:07.207 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:07.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-20T11:14:07.277 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:07.277 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-20T11:14:07.347 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:07.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-20T11:14:07.416 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:07.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-20T11:14:07.485 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:07.485 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-20T11:14:07.502 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:07.502 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-20T11:14:07.541 INFO:tasks.ceph:Adding keys to all mons... 2024-01-20T11:14:07.542 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-20T11:14:07.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-20T11:14:07.584 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2024-01-20T11:14:07.584 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDiqqtlLJb6IhAA6f6jzLVFRfYwgZAmlBDf6w== 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDkqqtl5Qv+AxAAa3lB2KnDguEfMtXTB0m3rw== 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnqqtlddywARAAzW/4rj5yts3q/PcgRLL8lg== 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDpqqtlrGfROhAAGU8uI5WuIhcwP15WriNwzA== 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDsqqtl/R4/OBAABN29JR1xjKHcVhdOv1tBsw== 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDiqqtlwJPfJhAAc2F4FRmXo8LUnVB2tlf+Sg== 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDzqqtl/g4NBBAA7KGF5PBGcH7x4jpz9VXfJQ== 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQD2qqtlJH+eAhAArcGT6y/JvND64iK7IMZtKQ== 2024-01-20T11:14:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2024-01-20T11:14:07.586 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQD5qqtldI3fABAAeLmapbs16X42yBtjikCSFQ== 2024-01-20T11:14:07.586 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2024-01-20T11:14:07.586 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQD7qqtlJM+FOhAA5XGEhCyMITmcDaKLEs8Slg== 2024-01-20T11:14:07.586 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2024-01-20T11:14:07.586 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDiqqtlpL+JKRAAJu/H8SzFcV+4IRBmj6P+7Q== 2024-01-20T11:14:07.586 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2024-01-20T11:14:07.586 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDiqqtlIsf4LBAA/9OH31T24ggXYJd2EIohLw== 2024-01-20T11:14:07.607 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.y] 2024-01-20T11:14:07.607 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDiqqtlLJb6IhAA6f6jzLVFRfYwgZAmlBDf6w== 2024-01-20T11:14:07.607 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDkqqtl5Qv+AxAAa3lB2KnDguEfMtXTB0m3rw== 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDnqqtlddywARAAzW/4rj5yts3q/PcgRLL8lg== 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDpqqtlrGfROhAAGU8uI5WuIhcwP15WriNwzA== 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.3] 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDsqqtl/R4/OBAABN29JR1xjKHcVhdOv1tBsw== 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.x] 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDiqqtlwJPfJhAAc2F4FRmXo8LUnVB2tlf+Sg== 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.4] 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDzqqtl/g4NBBAA7KGF5PBGcH7x4jpz9VXfJQ== 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.5] 2024-01-20T11:14:07.608 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD2qqtlJH+eAhAArcGT6y/JvND64iK7IMZtKQ== 2024-01-20T11:14:07.609 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.6] 2024-01-20T11:14:07.609 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD5qqtldI3fABAAeLmapbs16X42yBtjikCSFQ== 2024-01-20T11:14:07.609 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.7] 2024-01-20T11:14:07.609 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD7qqtlJM+FOhAA5XGEhCyMITmcDaKLEs8Slg== 2024-01-20T11:14:07.609 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-01-20T11:14:07.609 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDiqqtlpL+JKRAAJu/H8SzFcV+4IRBmj6P+7Q== 2024-01-20T11:14:07.609 INFO:teuthology.orchestra.run.smithi134.stdout:[client.1] 2024-01-20T11:14:07.609 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDiqqtlIsf4LBAA/9OH31T24ggXYJd2EIohLw== 2024-01-20T11:14:07.610 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-20T11:14:07.629 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-20T11:14:07.703 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:07.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:07.757 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:07.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:07.811 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:07.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:07.865 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:07.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:07.919 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-20T11:14:07.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-20T11:14:07.973 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:07.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:08.027 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:08.029 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:08.080 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:08.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:08.134 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:08.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-20T11:14:08.188 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-20T11:14:08.190 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-20T11:14:08.245 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-20T11:14:08.247 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-20T11:14:08.301 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-20T11:14:08.302 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-20T11:14:08.328 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-20T11:14:08.423 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-20T11:14:08.450 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-20T11:14:08.517 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-20T11:14:08.613 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-20T11:14:08.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-20T11:14:08.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-20T11:14:08.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-20T11:14:08.795 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:14:08.797 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-20T11:14:08.850 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-20T11:14:08.850 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-20T11:14:08.851 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-20T11:14:08.851 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-01-20T11:14:08.853 INFO:tasks.ceph.mon.a:Started 2024-01-20T11:14:08.853 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-20T11:14:08.854 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-20T11:14:08.854 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-01-20T11:14:08.856 INFO:tasks.ceph.mon.c:Started 2024-01-20T11:14:08.856 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-20T11:14:08.856 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-20T11:14:08.856 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-01-20T11:14:08.893 INFO:tasks.ceph.mon.b:Started 2024-01-20T11:14:08.893 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-20T11:14:08.893 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-20T11:14:08.894 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-01-20T11:14:08.896 INFO:tasks.ceph.mgr.y:Started 2024-01-20T11:14:08.896 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-20T11:14:08.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-01-20T11:14:08.899 INFO:tasks.ceph.mgr.x:Started 2024-01-20T11:14:08.899 INFO:tasks.ceph:Setting crush tunables to default 2024-01-20T11:14:08.899 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd crush tunables default 2024-01-20T11:14:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.000+0000 7f899eb0f640 10 monclient: get_monmap_and_config 2024-01-20T11:14:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.000+0000 7f899eb0f640 10 monclient: build_initial_monmap 2024-01-20T11:14:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.000+0000 7f899eb0f640 10 monclient: monmap: 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:14:09.001180+0000 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:14:09.001180+0000 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:09.001 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.001+0000 7f899eb0f640 1 Processor -- start 2024-01-20T11:14:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.001+0000 7f899eb0f640 1 -- start start 2024-01-20T11:14:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.001+0000 7f899eb0f640 10 monclient: init 2024-01-20T11:14:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.001+0000 7f899eb0f640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.001+0000 7f899eb0f640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-01-20T11:14:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.001+0000 7f899eb0f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 0x7f8998142660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.001+0000 7f899eb0f640 10 monclient(hunting): picked mon.noname-c con 0x7f8998142280 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.001+0000 7f899eb0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f899813f1a0 addr v1:172.21.15.110:6789/0 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.001+0000 7f899eb0f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 0x7f89980b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899eb0f640 10 monclient(hunting): picked mon.noname-a con 0x7f89980b26a0 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899eb0f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8998142e00 con 0x7f899813f1a0 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899eb0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899eb0f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89980aaa90 con 0x7f89980b26a0 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899eb0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899eb0f640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89980aad70 con 0x7f8998142280 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899eb0f640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899eb0f640 10 monclient(hunting): authenticate will time out at 1292.352220s 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f8997fff640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f899813f1a0 legacy=0x7f899813f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899d0ac640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 msgr2=0x7f89980b2ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899d0ac640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 0x7f89980b2ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899c8ab640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 msgr2=0x7f8998142660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:14:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.002+0000 7f899c8ab640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 0x7f8998142660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:09.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.203+0000 7f8997fff640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f899813f1a0 legacy=0x7f899813f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:14:09.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.203+0000 7f899d0ac640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 msgr2=0x7f89980b2ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:14:09.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.203+0000 7f899c8ab640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 msgr2=0x7f8998142660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:14:09.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.203+0000 7f899d0ac640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 0x7f89980b2ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:14:09.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.203+0000 7f899c8ab640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 0x7f8998142660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:14:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.604+0000 7f899d0ac640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 msgr2=0x7f89980b2ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:14:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.604+0000 7f8997fff640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f899813f1a0 legacy=0x7f899813f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:14:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.604+0000 7f899d0ac640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 0x7f89980b2ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:14:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.604+0000 7f899c8ab640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 msgr2=0x7f8998142660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:14:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:09.604+0000 7f899c8ab640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 0x7f8998142660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:14:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:10.405+0000 7f899d0ac640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 msgr2=0x7f89980b2ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:14:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:10.405+0000 7f8997fff640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f899813f1a0 legacy=0x7f899813f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:14:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:10.405+0000 7f899d0ac640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 0x7f89980b2ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:14:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:10.405+0000 7f899c8ab640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 msgr2=0x7f8998142660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:14:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:10.405+0000 7f899c8ab640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 0x7f8998142660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:14:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 10 monclient(hunting): tick 2024-01-20T11:14:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:14:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 1 monclient(hunting): continuing hunt 2024-01-20T11:14:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 msgr2=0x7f8998142660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8998142280 0x7f8998142660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 msgr2=0x7f89980b2ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f89980b26a0 0x7f89980b2ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f899813f1a0 legacy=0x7f899813f600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 10 monclient: _add_conns ranks=[2,1] 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 0x7f897c005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-b con 0x7f897c003690 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f897c006120 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89980ab050 con 0x7f897c006120 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89980aaa90 con 0x7f897c003690 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f89967fc640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f899c8ab640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:14:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f899c8ab640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.002+0000 7f8997fff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c006120 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:14:12.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.203+0000 7f899c8ab640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:14:12.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.203+0000 7f899c8ab640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:14:12.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.203+0000 7f8997fff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c006120 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:14:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.604+0000 7f899c8ab640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:14:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.604+0000 7f899c8ab640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:14:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:12.604+0000 7f8997fff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c006120 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:14:13.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:13.405+0000 7f899c8ab640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:14:13.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:13.405+0000 7f8997fff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c006120 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:14:13.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:13.405+0000 7f899c8ab640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:14:15.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient(hunting): tick 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 1 monclient(hunting): continuing hunt 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c003690 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c006120 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f899d0ac640 1 -- reap_dead start 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 0x7f897c005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-a con 0x7f897c009ea0 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f897c00a350 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 0x7f897c00d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-c con 0x7f897c00a8a0 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8998142e00 con 0x7f897c00a350 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89980aaa90 con 0x7f897c009ea0 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89980aad70 con 0x7f897c00a8a0 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f89967fc640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f899c8ab640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f899d0ac640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f8997fff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c00a350 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f899c8ab640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:15.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.002+0000 7f899d0ac640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:15.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.203+0000 7f899c8ab640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:14:15.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.203+0000 7f899d0ac640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:14:15.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.203+0000 7f899c8ab640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:14:15.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.203+0000 7f899d0ac640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:14:15.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.203+0000 7f8997fff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c00a350 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:14:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.604+0000 7f899c8ab640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:14:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.604+0000 7f8997fff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c00a350 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:14:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.604+0000 7f899d0ac640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:14:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.604+0000 7f899c8ab640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:14:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:15.604+0000 7f899d0ac640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:14:15.776 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:00:06.782 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:14:16.025 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:00:07.050 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:14:16.032 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:00:07.057 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:14:16.083 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:00:07.089 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:14:16.345 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:00:07.369 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:14:16.353 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:00:07.377 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:14:16.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:16.405+0000 7f899c8ab640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:14:16.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:16.405+0000 7f8997fff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c00a350 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:14:16.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:16.405+0000 7f899d0ac640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:14:16.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:16.405+0000 7f899c8ab640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:14:16.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:16.405+0000 7f899d0ac640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:14:18.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 10 monclient(hunting): tick 2024-01-20T11:14:18.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 1 monclient(hunting): continuing hunt 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f897c00a8a0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f897c009ea0 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f897c00a350 legacy=0x7f897c0085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 10 monclient: _add_conns ranks=[2,0] 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-b con 0x7f897c00d6b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:18.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f897c00e100 addr v1:172.21.15.110:6789/0 2024-01-20T11:14:18.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89980ab050 con 0x7f897c00e100 2024-01-20T11:14:18.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:18.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89980aaa90 con 0x7f897c00d6b0 2024-01-20T11:14:18.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.002+0000 7f89967fc640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:18.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.022+0000 7f8997fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f897c00e100 0x7f897c012830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56116/0 (socket says 172.21.15.110:56116) 2024-01-20T11:14:18.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.022+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 learned_addr learned my addr 172.21.15.110:0/1553923214 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:14:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.325+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:18.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.330+0000 7f899c8ab640 10 monclient(hunting): get_auth_request con 0x7f897c00d6b0 auth_method 0 2024-01-20T11:14:18.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.330+0000 7f899c8ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:18.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.330+0000 7f899c8ab640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:18.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.330+0000 7f899c8ab640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:18.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.334+0000 7f899c8ab640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 msgr2=0x7f897c00da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-20T11:14:18.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.334+0000 7f899c8ab640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 msgr2=0x7f897c00da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:18.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.334+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:18.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.334+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:18.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.535+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:18.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.535+0000 7f899c8ab640 10 monclient(hunting): get_auth_request con 0x7f897c00d6b0 auth_method 0 2024-01-20T11:14:18.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.535+0000 7f899c8ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:18.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.535+0000 7f899c8ab640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:18.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.535+0000 7f899c8ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:18.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.536+0000 7f899c8ab640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 msgr2=0x7f897c00da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-20T11:14:18.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.536+0000 7f899c8ab640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 msgr2=0x7f897c00da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:18.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.536+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:18.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.536+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:14:18.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.946+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:18.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.946+0000 7f899c8ab640 10 monclient(hunting): get_auth_request con 0x7f897c00d6b0 auth_method 0 2024-01-20T11:14:18.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.946+0000 7f899c8ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:18.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.946+0000 7f899c8ab640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:18.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.946+0000 7f899c8ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:18.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.962+0000 7f899c8ab640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 msgr2=0x7f897c00da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-20T11:14:18.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.962+0000 7f899c8ab640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 msgr2=0x7f897c00da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:18.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.962+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:18.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:18.962+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:14:19.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:19.763+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:19.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:19.764+0000 7f899c8ab640 10 monclient(hunting): get_auth_request con 0x7f897c00d6b0 auth_method 0 2024-01-20T11:14:19.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:19.764+0000 7f899c8ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:19.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:19.764+0000 7f899c8ab640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:19.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:19.764+0000 7f899c8ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:19.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:19.764+0000 7f899c8ab640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 msgr2=0x7f897c00da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-20T11:14:19.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:19.764+0000 7f899c8ab640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 msgr2=0x7f897c00da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:19.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:19.764+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:19.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:19.765+0000 7f899c8ab640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:14:21.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient(hunting): tick 2024-01-20T11:14:21.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:14:21.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 1 monclient(hunting): continuing hunt 2024-01-20T11:14:21.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:14:21.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 msgr2=0x7f897c00da90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:21.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c00d6b0 0x7f897c00da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 >> v1:172.21.15.110:6789/0 conn(0x7f897c00e100 legacy=0x7f897c012830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 reap_dead start 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f897c009ea0 addr v1:172.21.15.110:6789/0 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-c con 0x7f897c00a350 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-a con 0x7f897c0146e0 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8998142e00 con 0x7f897c009ea0 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89980aaa90 con 0x7f897c0146e0 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:21.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89980aad70 con 0x7f897c00a350 2024-01-20T11:14:21.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.002+0000 7f89967fc640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:21.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.003+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:21.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.003+0000 7f8997fff640 10 monclient(hunting): get_auth_request con 0x7f897c00a350 auth_method 0 2024-01-20T11:14:21.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.003+0000 7f8997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:21.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.003+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:21.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.003+0000 7f8997fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:21.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.004+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:21.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.004+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:21.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.004+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:21.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.004+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:21.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.004+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:21.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.005+0000 7f899d0ac640 10 monclient(hunting): get_auth_request con 0x7f897c0146e0 auth_method 0 2024-01-20T11:14:21.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.005+0000 7f899d0ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:21.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.005+0000 7f899d0ac640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:21.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.005+0000 7f899d0ac640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:21.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.012+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-01-20T11:14:21.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.012+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:21.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.012+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:21.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.012+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.205+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.206+0000 7f8997fff640 10 monclient(hunting): get_auth_request con 0x7f897c00a350 auth_method 0 2024-01-20T11:14:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.206+0000 7f8997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.206+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.206+0000 7f8997fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.206+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.206+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.206+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.207+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:14:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.213+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.213+0000 7f899d0ac640 10 monclient(hunting): get_auth_request con 0x7f897c0146e0 auth_method 0 2024-01-20T11:14:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.213+0000 7f899d0ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.213+0000 7f899d0ac640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.213+0000 7f899d0ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.214+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.214+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.214+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.214+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:14:21.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.608+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:21.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.609+0000 7f8997fff640 10 monclient(hunting): get_auth_request con 0x7f897c00a350 auth_method 0 2024-01-20T11:14:21.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.609+0000 7f8997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:21.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.609+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:21.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.609+0000 7f8997fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:21.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.610+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:21.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.610+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:21.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.610+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:21.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.610+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:14:21.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.615+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:21.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.616+0000 7f899d0ac640 10 monclient(hunting): get_auth_request con 0x7f897c0146e0 auth_method 0 2024-01-20T11:14:21.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.616+0000 7f899d0ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:21.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.616+0000 7f899d0ac640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:21.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.616+0000 7f899d0ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:21.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.617+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:21.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.617+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:21.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.617+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:21.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:21.617+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:14:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.411+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.412+0000 7f8997fff640 10 monclient(hunting): get_auth_request con 0x7f897c00a350 auth_method 0 2024-01-20T11:14:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.412+0000 7f8997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.412+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.412+0000 7f8997fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:22.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.413+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:22.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.413+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:22.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.413+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:22.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.413+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:14:22.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.418+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:22.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.419+0000 7f899d0ac640 10 monclient(hunting): get_auth_request con 0x7f897c0146e0 auth_method 0 2024-01-20T11:14:22.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.419+0000 7f899d0ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:22.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.419+0000 7f899d0ac640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.419+0000 7f899d0ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.419+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.419+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.419+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:22.419+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 10 monclient(hunting): tick 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 1 monclient(hunting): continuing hunt 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 msgr2=0x7f897c005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.134:3300/0 conn(0x7f897c00a350 0x7f897c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 msgr2=0x7f897c00d0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 1 --2- 172.21.15.110:0/1553923214 >> v2:172.21.15.110:3300/0 conn(0x7f897c0146e0 0x7f897c00d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 >> v1:172.21.15.110:6789/0 conn(0x7f897c009ea0 legacy=0x7f897c013fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 10 monclient: _add_conns ranks=[1,2] 2024-01-20T11:14:24.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f897c014b30 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 10 monclient(hunting): picked mon.noname-b con 0x7f897c011830 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897c00dd20 con 0x7f897c014b30 2024-01-20T11:14:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 1 -- 172.21.15.110:0/1553923214 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89980aaa90 con 0x7f897c011830 2024-01-20T11:14:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f89967fc640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.003+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:24.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.004+0000 7f8997fff640 10 monclient(hunting): get_auth_request con 0x7f897c011830 auth_method 0 2024-01-20T11:14:24.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.004+0000 7f8997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:24.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.004+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:24.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.004+0000 7f8997fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:24.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.005+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 msgr2=0x7f897c014fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:24.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.005+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 msgr2=0x7f897c014fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:24.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.005+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:24.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.005+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:24.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.208+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:24.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.209+0000 7f8997fff640 10 monclient(hunting): get_auth_request con 0x7f897c011830 auth_method 0 2024-01-20T11:14:24.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.209+0000 7f8997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:24.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.209+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:24.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.209+0000 7f8997fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:24.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.209+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 msgr2=0x7f897c014fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:24.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.209+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 msgr2=0x7f897c014fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:24.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.209+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:24.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.209+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:14:24.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.611+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:24.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.611+0000 7f8997fff640 10 monclient(hunting): get_auth_request con 0x7f897c011830 auth_method 0 2024-01-20T11:14:24.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.611+0000 7f8997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:24.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.611+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:24.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.611+0000 7f8997fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.612+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 msgr2=0x7f897c014fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.612+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 msgr2=0x7f897c014fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.612+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:24.612+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:14:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:25.413+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:25.414+0000 7f8997fff640 10 monclient(hunting): get_auth_request con 0x7f897c011830 auth_method 0 2024-01-20T11:14:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:25.414+0000 7f8997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:25.414+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:25.414+0000 7f8997fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:25.414+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 msgr2=0x7f897c014fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-20T11:14:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:25.414+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 msgr2=0x7f897c014fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:25.414+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:25.414+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:14:26.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.364+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981022038 0 0) 0x7f897c00dd20 con 0x7f897c014b30 2024-01-20T11:14:26.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.364+0000 7f89977fe640 10 monclient(hunting): _negotiate 2024-01-20T11:14:26.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.364+0000 7f89977fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:26.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.364+0000 7f89977fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:26.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.364+0000 7f89977fe640 10 monclient(hunting): my global_id is 4099 2024-01-20T11:14:26.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.364+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8988003520 con 0x7f897c014b30 2024-01-20T11:14:26.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3226353605 0 0) 0x7f8988003520 con 0x7f897c014b30 2024-01-20T11:14:26.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 msgr2=0x7f897c014fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:26.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f897c011830 0x7f897c014fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient: _finish_hunting 0 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 monclient: found mon.noname-c-legacy 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.134:6789/0 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c016530 con 0x7f897c014b30 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient: _finish_auth 0 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient: _check_auth_tickets 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2748620985 0 0) 0x7f898c009950 con 0x7f897c014b30 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f899eb0f640 5 monclient: authenticate success, global_id 4099 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f899eb0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:26.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient: dump: 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] but i'm connected to v1:172.21.15.134:6789/0, reconnecting 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:26.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 >> v1:172.21.15.134:6789/0 conn(0x7f897c014b30 legacy=0x7f897c008580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f899d0ac640 1 -- 172.21.15.110:0/1553923214 reap_dead start 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8988003970 0x7f8988003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient(hunting): picked mon.c con 0x7f8988003970 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 0x7f8988004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient(hunting): picked mon.b con 0x7f89880047c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89880051d0 0x7f898800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient(hunting): picked mon.a con 0x7f89880051d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980aaa90 con 0x7f89880051d0 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898800c100 con 0x7f89880047c0 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:26.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898800c2c0 con 0x7f8988003970 2024-01-20T11:14:26.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:26.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f89977fe640 10 monclient(hunting): _finish_auth 0 2024-01-20T11:14:26.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.410+0000 7f899eb0f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-01-20T11:14:26.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.429+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89880051d0 0x7f898800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:26.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.440+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 0x7f8988004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:26.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.442+0000 7f8997fff640 10 monclient(hunting): get_auth_request con 0x7f89880047c0 auth_method 0 2024-01-20T11:14:26.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.442+0000 7f8997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:26.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.442+0000 7f8997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:26.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.442+0000 7f8997fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:14:26.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.443+0000 7f8997fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:26.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.443+0000 7f8997fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:26.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.443+0000 7f8997fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:14:26.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 10 monclient(hunting): handle_auth_done global_id 4099 payload 293 2024-01-20T11:14:26.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8988003970 msgr2=0x7f8988003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:26.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8988003970 0x7f8988003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89880051d0 msgr2=0x7f898800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89880051d0 0x7f898800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 10 monclient: _finish_hunting 0 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 1 monclient: found mon.b 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 1 -- 172.21.15.110:0/1553923214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988003520 con 0x7f89880047c0 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 10 monclient: _finish_auth 0 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 10 monclient: _check_auth_tickets 2024-01-20T11:14:26.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.445+0000 7f8997fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:26.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.486+0000 7f8997fff640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 0x7f8988004c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8984003ad0 tx=0x7f89880057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd40c6939a5a9639 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984034600 con 0x7f89880047c0 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 10 monclient: dump: 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:26.494 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8984030050 con 0x7f89880047c0 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 1 -- 172.21.15.110:0/1553923214 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89840349c0 con 0x7f89880047c0 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 4 set_mon_vals no callback set 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f89977fe640 10 monclient: dump: 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.495 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.496 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:26.496 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:26.496 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:26.496 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:26.496 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:26.496 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:26.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 10 monclient: shutdown 2024-01-20T11:14:26.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:26.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 1 -- 172.21.15.110:0/1553923214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 msgr2=0x7f8988004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:26.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 0x7f8988004c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8984003ad0 tx=0x7f89880057f0 comp rx=0 tx=0).stop 2024-01-20T11:14:26.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 1 -- 172.21.15.110:0/1553923214 shutdown_connections 2024-01-20T11:14:26.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89880051d0 0x7f898800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 0x7f8988004c40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.494+0000 7f899eb0f640 1 --2- 172.21.15.110:0/1553923214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8988003970 0x7f8988003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.495+0000 7f899eb0f640 1 -- 172.21.15.110:0/1553923214 >> 172.21.15.110:0/1553923214 conn(0x7f89980663b0 msgr2=0x7f89980667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.495+0000 7f899eb0f640 1 -- 172.21.15.110:0/1553923214 shutdown_connections 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.495+0000 7f899eb0f640 1 -- 172.21.15.110:0/1553923214 wait complete. 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.496+0000 7f899eb0f640 10 monclient: build_initial_monmap 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.496+0000 7f899eb0f640 10 monclient: monmap: 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:26.499 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.496+0000 7f899eb0f640 1 librados: starting msgr at 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.496+0000 7f899eb0f640 1 librados: starting objecter 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.496+0000 7f899eb0f640 1 Processor -- start 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.496+0000 7f899eb0f640 1 -- start start 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.496+0000 7f899eb0f640 1 librados: setting wanted keys 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.496+0000 7f899eb0f640 1 librados: calling monclient init 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.496+0000 7f899eb0f640 10 monclient: init 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.497+0000 7f899eb0f640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8988003970 0x7f89980ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 10 monclient(hunting): picked mon.noname-a con 0x7f8988003970 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:26.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 0x7f8998139c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 10 monclient(hunting): picked mon.noname-b con 0x7f89880047c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89880051d0 0x7f899813a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 10 monclient(hunting): picked mon.noname-c con 0x7f89880051d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980acfe0 con 0x7f8988003970 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980ae9e0 con 0x7f89880047c0 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899813a6f0 con 0x7f89880051d0 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:26.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.498+0000 7f899eb0f640 10 monclient(hunting): authenticate will time out at 1309.848296s 2024-01-20T11:14:26.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.505+0000 7f899c8ab640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8988003970 0x7f89980ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:26.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.506+0000 7f899d0ac640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89880051d0 0x7f899813a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:26.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.507+0000 7f899d0ac640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89880051d0 0x7f899813a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:44710/0 (socket says 172.21.15.110:44710) 2024-01-20T11:14:26.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.507+0000 7f899d0ac640 1 -- 172.21.15.110:0/1007498451 learned_addr learned my addr 172.21.15.110:0/1007498451 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:14:26.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.507+0000 7f899d0ac640 10 monclient(hunting): get_auth_request con 0x7f89880051d0 auth_method 0 2024-01-20T11:14:26.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.507+0000 7f899d0ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:26.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.507+0000 7f899d0ac640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:26.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.507+0000 7f899d0ac640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:26.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.508+0000 7f899d0ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:26.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.508+0000 7f899d0ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:26.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.508+0000 7f899d0ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:26.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2024-01-20T11:14:26.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 1 -- 172.21.15.110:0/1007498451 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 msgr2=0x7f8998139c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:26.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1007498451 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 0x7f8998139c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:26.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 1 -- 172.21.15.110:0/1007498451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8988003970 msgr2=0x7f89980ad750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:26.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1007498451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8988003970 0x7f89980ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:26.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 10 monclient: _finish_hunting 0 2024-01-20T11:14:26.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 1 monclient: found mon.noname-c 2024-01-20T11:14:26.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:26.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:14:26.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.510+0000 7f899d0ac640 1 -- 172.21.15.110:0/1007498451 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89980b1830 con 0x7f89880051d0 2024-01-20T11:14:26.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.513+0000 7f899d0ac640 1 --2- 172.21.15.110:0/1007498451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89880051d0 0x7f899813a1b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f897c009e20 tx=0x7f8990011f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:26.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.513+0000 7f89957fa640 10 client.?.objecter ms_handle_connect 0x7f89880051d0 2024-01-20T11:14:26.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.513+0000 7f89957fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.562+0000 7f89957fa640 1 -- 172.21.15.110:0/1007498451 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8990011030 con 0x7f89880051d0 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 10 monclient: dump: 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:26.563 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 10 monclient: _finish_auth 0 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 10 monclient: _check_auth_tickets 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 1 -- 172.21.15.110:0/1007498451 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89900111b0 con 0x7f89880051d0 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 1 -- 172.21.15.110:0/1007498451 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899000f690 con 0x7f89880051d0 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89957fa640 10 monclient: dump: 2024-01-20T11:14:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f899eb0f640 5 monclient: authenticate success, global_id 4103 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f899eb0f640 10 monclient: _renew_subs 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f899eb0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f89967fc640 4 set_mon_vals no callback set 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f899eb0f640 1 -- 172.21.15.110:0/1007498451 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89980acad0 con 0x7f89880051d0 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f899eb0f640 10 client.4103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f899eb0f640 10 monclient: _renew_subs 2024-01-20T11:14:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f899eb0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f899eb0f640 1 -- 172.21.15.110:0/1007498451 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89980b1550 con 0x7f89880051d0 2024-01-20T11:14:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.563+0000 7f899eb0f640 1 librados: init done 2024-01-20T11:14:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.564+0000 7f899eb0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.564+0000 7f899eb0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.564+0000 7f899eb0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.564+0000 7f899eb0f640 1 -- 172.21.15.110:0/1007498451 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89980ad120 con 0x7f89880051d0 2024-01-20T11:14:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.590+0000 7f89957fa640 1 -- 172.21.15.110:0/1007498451 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f899000f7f0 con 0x7f89880051d0 2024-01-20T11:14:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.590+0000 7f89957fa640 1 -- 172.21.15.110:0/1007498451 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8990022a10 con 0x7f89880051d0 2024-01-20T11:14:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.590+0000 7f89957fa640 10 client.4103.objecter ms_dispatch 0x7f8998000b90 osd_map(1..1 src has 1..1) v4 2024-01-20T11:14:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.590+0000 7f89957fa640 3 client.4103.objecter handle_osd_map got epochs [1,1] > 0 2024-01-20T11:14:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.590+0000 7f89957fa640 3 client.4103.objecter handle_osd_map decoding full epoch 1 2024-01-20T11:14:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.590+0000 7f89957fa640 20 client.4103.objecter dump_active .. 0 homeless 2024-01-20T11:14:26.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.767+0000 7f89957fa640 1 -- 172.21.15.110:0/1007498451 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f89980ad120 con 0x7f89880051d0 2024-01-20T11:14:26.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.767+0000 7f89957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:26.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.767+0000 7f89957fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:26.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.864+0000 7f899eb0f640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-20T11:14:26.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.864+0000 7f899eb0f640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-20T11:14:26.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.864+0000 7f899eb0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:26.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:26.864+0000 7f899eb0f640 1 -- 172.21.15.110:0/1007498451 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f89980ad300 con 0x7f89880051d0 2024-01-20T11:14:27.008 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:27.007+0000 7fe56084c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-20T11:14:27.061 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:27.061+0000 7fe56084c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-20T11:14:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.124+0000 7f89957fa640 1 -- 172.21.15.110:0/1007498451 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f89900223f0 con 0x7f89880051d0 2024-01-20T11:14:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.124+0000 7f89957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-20T11:14:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.124+0000 7f89957fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-01-20T11:14:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2024-01-20T11:14:27.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:27.124+0000 7f32fbc41040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 10 librados: watch_flush enter 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 10 librados: watch_flush exit 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 20 client.4103.objecter shutdown clearing up homeless session... 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 10 client.4103.objecter successfully canceled tick 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 10 monclient: shutdown 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 -- 172.21.15.110:0/1007498451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89880051d0 msgr2=0x7f899813a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 --2- 172.21.15.110:0/1007498451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89880051d0 0x7f899813a1b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f897c009e20 tx=0x7f8990011f60 comp rx=0 tx=0).stop 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 -- 172.21.15.110:0/1007498451 shutdown_connections 2024-01-20T11:14:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 --2- 172.21.15.110:0/1007498451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89880051d0 0x7f899813a1b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:27.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 --2- 172.21.15.110:0/1007498451 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89880047c0 0x7f8998139c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:27.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 --2- 172.21.15.110:0/1007498451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8988003970 0x7f89980ad750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:27.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 -- 172.21.15.110:0/1007498451 >> 172.21.15.110:0/1007498451 conn(0x7f89980663b0 msgr2=0x7f89980b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:27.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 -- 172.21.15.110:0/1007498451 shutdown_connections 2024-01-20T11:14:27.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 -- 172.21.15.110:0/1007498451 wait complete. 2024-01-20T11:14:27.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:27.125+0000 7f899eb0f640 1 librados: shutdown 2024-01-20T11:14:27.136 INFO:tasks.ceph:check_enable_crimson: False 2024-01-20T11:14:27.136 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-20T11:14:27.137 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:27.137 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-20T11:14:27.165 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:27.165 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-20T11:14:27.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:27.186+0000 7f32fbc41040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-20T11:14:27.215 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:27.215 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-20T11:14:27.281 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-20T11:14:27.281 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-20T11:14:27.348 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:27.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-20T11:14:27.375 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:27.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-20T11:14:27.442 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:27.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-20T11:14:27.509 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-20T11:14:27.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-20T11:14:27.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 34615fe4-d784-4baa-93fe-23f986a84457 0 2024-01-20T11:14:27.598 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:27.598+0000 7fe56084c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-20T11:14:27.654 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:27.653+0000 7fe56084c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-20T11:14:27.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.698+0000 7f20b4634640 10 monclient: get_monmap_and_config 2024-01-20T11:14:27.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.698+0000 7f20b4634640 10 monclient: build_initial_monmap 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.698+0000 7f20b4634640 10 monclient: monmap: 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:14:27.699324+0000 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:14:27.699324+0000 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:27.699 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:27.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.698+0000 7f20b4634640 1 Processor -- start 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.698+0000 7f20b4634640 1 -- start start 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.698+0000 7f20b4634640 10 monclient: init 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac13d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 10 monclient(hunting): picked mon.noname-b con 0x7f20ac06f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20ac06fd00 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f20ac140930 0x7f20ac13d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 10 monclient(hunting): picked mon.noname-c con 0x7f20ac140930 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac0b03c0 con 0x7f20ac06fd00 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ac13fe00 con 0x7f20ac140930 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:27.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac140de0 con 0x7f20ac06f920 2024-01-20T11:14:27.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:27.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.699+0000 7f20b4634640 10 monclient(hunting): authenticate will time out at 1312.085550s 2024-01-20T11:14:27.702 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:27.700+0000 7fe56084c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-20T11:14:27.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.712+0000 7f20b2bd1640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f20ac140930 0x7f20ac13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:27.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.712+0000 7f20b1bcf640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f20ac06fd00 0x7f20ac140220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:42108/0 (socket says 172.21.15.134:42108) 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.712+0000 7f20b1bcf640 1 -- 172.21.15.134:0/1621019153 learned_addr learned my addr 172.21.15.134:0/1621019153 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.712+0000 7f20b23d0640 1 --2- 172.21.15.134:0/1621019153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac13d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.713+0000 7f20b23d0640 10 monclient(hunting): get_auth_request con 0x7f20ac06f920 auth_method 0 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.713+0000 7f20b23d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.713+0000 7f20b23d0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.713+0000 7f20b23d0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.713+0000 7f20b2bd1640 10 monclient(hunting): get_auth_request con 0x7f20ac140930 auth_method 0 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.713+0000 7f20b2bd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.713+0000 7f20b2bd1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:27.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.713+0000 7f20b2bd1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:27.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.714+0000 7f20b2bd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:27.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.714+0000 7f20b2bd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:27.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.714+0000 7f20b2bd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:27.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.714+0000 7f20b23d0640 1 -- 172.21.15.134:0/1621019153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 msgr2=0x7f20ac13d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-20T11:14:27.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.714+0000 7f20b23d0640 1 -- 172.21.15.134:0/1621019153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 msgr2=0x7f20ac13d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:14:27.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.714+0000 7f20b23d0640 1 --2- 172.21.15.134:0/1621019153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac13d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:14:27.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.714+0000 7f20b23d0640 1 --2- 172.21.15.134:0/1621019153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac13d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:14:27.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.715+0000 7f20b2bd1640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2024-01-20T11:14:27.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.716+0000 7f20b2bd1640 1 -- 172.21.15.134:0/1621019153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 msgr2=0x7f20ac13d0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:14:27.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.716+0000 7f20b2bd1640 1 --2- 172.21.15.134:0/1621019153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac13d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:27.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.716+0000 7f20b2bd1640 1 -- 172.21.15.134:0/1621019153 >> v1:172.21.15.134:6789/0 conn(0x7f20ac06fd00 legacy=0x7f20ac140220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:27.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.716+0000 7f20b2bd1640 10 monclient: _finish_hunting 0 2024-01-20T11:14:27.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.716+0000 7f20b2bd1640 1 monclient: found mon.noname-c 2024-01-20T11:14:27.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.716+0000 7f20b2bd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:27.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.716+0000 7f20b2bd1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:14:27.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.716+0000 7f20b2bd1640 1 -- 172.21.15.134:0/1621019153 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac071600 con 0x7f20ac140930 2024-01-20T11:14:27.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.722+0000 7f20b2bd1640 1 --2- 172.21.15.134:0/1621019153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 0x7f20ac13d600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009d90 tx=0x7f20a402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70ca28180cf59ae0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:27.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 1 -- 172.21.15.134:0/1621019153 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a402cbc0 con 0x7f20ac140930 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 10 monclient: dump: 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:27.728 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 10 monclient: _finish_auth 0 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 10 monclient: _check_auth_tickets 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 1 -- 172.21.15.134:0/1621019153 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a402cd20 con 0x7f20ac140930 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 1 -- 172.21.15.134:0/1621019153 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4034660 con 0x7f20ac140930 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b4634640 5 monclient: authenticate success, global_id 4108 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b4634640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b4634640 4 set_mon_vals no callback set 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:27.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b13ce640 10 monclient: dump: 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:27.730 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:27.731 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:27.729+0000 7f32fbc41040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-20T11:14:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b4634640 10 monclient: shutdown 2024-01-20T11:14:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b4634640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b4634640 1 -- 172.21.15.134:0/1621019153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 msgr2=0x7f20ac13d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.727+0000 7f20b4634640 1 --2- 172.21.15.134:0/1621019153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 0x7f20ac13d600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009d90 tx=0x7f20a402be10 comp rx=0 tx=0).stop 2024-01-20T11:14:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.728+0000 7f20b4634640 1 -- 172.21.15.134:0/1621019153 shutdown_connections 2024-01-20T11:14:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.728+0000 7f20b4634640 1 --2- 172.21.15.134:0/1621019153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 0x7f20ac13d600 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.728+0000 7f20b4634640 1 --2- 172.21.15.134:0/1621019153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac13d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:27.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.728+0000 7f20b4634640 1 -- 172.21.15.134:0/1621019153 >> 172.21.15.134:0/1621019153 conn(0x7f20ac066dc0 msgr2=0x7f20ac0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.728+0000 7f20b4634640 1 -- 172.21.15.134:0/1621019153 shutdown_connections 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.728+0000 7f20b4634640 1 -- 172.21.15.134:0/1621019153 wait complete. 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.728+0000 7f20b4634640 10 monclient: build_initial_monmap 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.728+0000 7f20b4634640 10 monclient: monmap: 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:27.736 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:27.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 librados: starting msgr at 2024-01-20T11:14:27.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 librados: starting objecter 2024-01-20T11:14:27.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 Processor -- start 2024-01-20T11:14:27.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 -- start start 2024-01-20T11:14:27.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 librados: setting wanted keys 2024-01-20T11:14:27.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 librados: calling monclient init 2024-01-20T11:14:27.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient: init 2024-01-20T11:14:27.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:27.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:14:27.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac06c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:27.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient(hunting): picked mon.noname-c con 0x7f20ac06f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:27.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20ac06fd00 0x7f20ac06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:27.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient(hunting): picked mon.noname-a con 0x7f20ac06fd00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:27.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 0x7f20ac1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:27.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient(hunting): picked mon.noname-b con 0x7f20ac140930 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac141b80 con 0x7f20ac06fd00 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac06c040 con 0x7f20ac140930 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac148880 con 0x7f20ac06f920 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.729+0000 7f20b4634640 10 monclient(hunting): authenticate will time out at 1312.115879s 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b2bd1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 0x7f20ac1435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b1bcf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20ac06fd00 0x7f20ac06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b23d0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac06c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b2bd1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 0x7f20ac1435c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40942/0 (socket says 172.21.15.134:40942) 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b2bd1640 1 -- 172.21.15.134:0/1736048025 learned_addr learned my addr 172.21.15.134:0/1736048025 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:27.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b2bd1640 10 monclient(hunting): get_auth_request con 0x7f20ac140930 auth_method 0 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b2bd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b2bd1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b2bd1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b23d0640 10 monclient(hunting): get_auth_request con 0x7f20ac06f920 auth_method 0 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b23d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b23d0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b23d0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b1bcf640 10 monclient(hunting): get_auth_request con 0x7f20ac06fd00 auth_method 0 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b1bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b1bcf640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.730+0000 7f20b1bcf640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.731+0000 7f20b23d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:27.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.731+0000 7f20b23d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.731+0000 7f20b23d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.731+0000 7f20b2bd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.731+0000 7f20b2bd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.731+0000 7f20b2bd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.731+0000 7f20b1bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.731+0000 7f20b1bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.731+0000 7f20b1bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.733+0000 7f20b2bd1640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.733+0000 7f20b2bd1640 1 -- 172.21.15.134:0/1736048025 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 msgr2=0x7f20ac06c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.734+0000 7f20b2bd1640 1 --2- 172.21.15.134:0/1736048025 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac06c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.734+0000 7f20b2bd1640 1 -- 172.21.15.134:0/1736048025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20ac06fd00 msgr2=0x7f20ac06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.734+0000 7f20b2bd1640 1 --2- 172.21.15.134:0/1736048025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20ac06fd00 0x7f20ac06caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:27.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.734+0000 7f20b2bd1640 10 monclient: _finish_hunting 0 2024-01-20T11:14:27.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.734+0000 7f20b2bd1640 1 monclient: found mon.noname-b 2024-01-20T11:14:27.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.734+0000 7f20b2bd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:27.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.734+0000 7f20b2bd1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:14:27.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.734+0000 7f20b2bd1640 1 -- 172.21.15.134:0/1736048025 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a4009a70 con 0x7f20ac140930 2024-01-20T11:14:27.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.736+0000 7f20b2bd1640 1 --2- 172.21.15.134:0/1736048025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 0x7f20ac1435c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009d90 tx=0x7f20a4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:27.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.736+0000 7f20a2ffd640 10 client.?.objecter ms_handle_connect 0x7f20ac140930 2024-01-20T11:14:27.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.736+0000 7f20a2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:27.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 1 -- 172.21.15.134:0/1736048025 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4040650 con 0x7f20ac140930 2024-01-20T11:14:27.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:27.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:27.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 10 monclient: dump: 2024-01-20T11:14:27.745 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:27.745 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:27.745 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:27.745 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 1 -- 172.21.15.134:0/1736048025 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a402ce90 con 0x7f20ac140930 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b4634640 5 monclient: authenticate success, global_id 4114 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 1 -- 172.21.15.134:0/1736048025 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4040a10 con 0x7f20ac140930 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b4634640 10 monclient: _renew_subs 2024-01-20T11:14:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b4634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b4634640 1 -- 172.21.15.134:0/1736048025 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac148a40 con 0x7f20ac140930 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b0bcd640 4 set_mon_vals no callback set 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20a2ffd640 10 monclient: dump: 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:27.747 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b4634640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b4634640 10 monclient: _renew_subs 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b4634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b4634640 1 -- 172.21.15.134:0/1736048025 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac143ce0 con 0x7f20ac140930 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.739+0000 7f20b4634640 1 librados: init done 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.740+0000 7f20b4634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.740+0000 7f20b4634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.740+0000 7f20b4634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.740+0000 7f20b4634640 1 -- 172.21.15.134:0/1736048025 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac06c260 con 0x7f20ac140930 2024-01-20T11:14:27.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.743+0000 7f20a2ffd640 1 -- 172.21.15.134:0/1736048025 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f20a4052430 con 0x7f20ac140930 2024-01-20T11:14:27.760 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:27.760+0000 7fe56084c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-20T11:14:27.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:27.782+0000 7f32fbc41040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-20T11:14:27.803 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:27.803+0000 7fe56084c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-20T11:14:27.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:27.826+0000 7f32fbc41040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-20T11:14:27.879 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:27.879+0000 7f32fbc41040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-20T11:14:27.907 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:27.906+0000 7fe56084c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-20T11:14:27.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.915+0000 7f20a2ffd640 1 -- 172.21.15.134:0/1736048025 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f20a4049570 con 0x7f20ac140930 2024-01-20T11:14:27.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.915+0000 7f20a2ffd640 10 client.4114.objecter ms_dispatch 0x7f20ac000b90 osd_map(2..2 src has 1..2) v4 2024-01-20T11:14:27.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.915+0000 7f20a2ffd640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-01-20T11:14:27.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.915+0000 7f20a2ffd640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-01-20T11:14:27.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.915+0000 7f20a2ffd640 20 client.4114.objecter dump_active .. 0 homeless 2024-01-20T11:14:27.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.915+0000 7f20a2ffd640 1 -- 172.21.15.134:0/1736048025 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f20a4044020 con 0x7f20ac140930 2024-01-20T11:14:27.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.915+0000 7f20a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:27.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.915+0000 7f20a2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:27.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:27.921+0000 7f32fbc41040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-20T11:14:27.979 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:27.978+0000 7fe56084c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-20T11:14:27.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.987+0000 7f20b4634640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "34615fe4-d784-4baa-93fe-23f986a84457", "id": 0}] 2024-01-20T11:14:27.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.987+0000 7f20b4634640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "34615fe4-d784-4baa-93fe-23f986a84457", "id": 0}] 2024-01-20T11:14:27.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.987+0000 7f20b4634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:27.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:27.987+0000 7f20b4634640 1 -- 172.21.15.134:0/1736048025 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "34615fe4-d784-4baa-93fe-23f986a84457", "id": 0} v 0) v1 -- 0x7f20ac075500 con 0x7f20ac140930 2024-01-20T11:14:28.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:28.020+0000 7f32fbc41040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-20T11:14:28.026 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:28.025+0000 7fe56084c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-20T11:14:28.077 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:28.077+0000 7fe56084c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-20T11:14:28.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:28.091+0000 7f32fbc41040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-20T11:14:28.125 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:28.124+0000 7fe56084c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-20T11:14:28.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:28.144+0000 7f32fbc41040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-20T11:14:28.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.170+0000 7f20a2ffd640 1 -- 172.21.15.134:0/1736048025 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "34615fe4-d784-4baa-93fe-23f986a84457", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f20a4056020 con 0x7f20ac140930 2024-01-20T11:14:28.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.170+0000 7f20a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "34615fe4-d784-4baa-93fe-23f986a84457", "id": 0}] 2024-01-20T11:14:28.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.170+0000 7f20a2ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:28.171 INFO:teuthology.orchestra.run.smithi134.stdout:0 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 10 librados: watch_flush enter 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 10 librados: watch_flush exit 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 20 client.4114.objecter shutdown clearing up homeless session... 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 10 client.4114.objecter successfully canceled tick 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 10 monclient: shutdown 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 -- 172.21.15.134:0/1736048025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 msgr2=0x7f20ac1435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 --2- 172.21.15.134:0/1736048025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 0x7f20ac1435c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009d90 tx=0x7f20a4004a40 comp rx=0 tx=0).stop 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 -- 172.21.15.134:0/1736048025 shutdown_connections 2024-01-20T11:14:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 --2- 172.21.15.134:0/1736048025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20ac140930 0x7f20ac1435c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 --2- 172.21.15.134:0/1736048025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20ac06fd00 0x7f20ac06caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 --2- 172.21.15.134:0/1736048025 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20ac06f920 0x7f20ac06c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 -- 172.21.15.134:0/1736048025 >> 172.21.15.134:0/1736048025 conn(0x7f20ac066dc0 msgr2=0x7f20ac0b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:28.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 -- 172.21.15.134:0/1736048025 shutdown_connections 2024-01-20T11:14:28.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 -- 172.21.15.134:0/1736048025 wait complete. 2024-01-20T11:14:28.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.171+0000 7f20b4634640 1 librados: shutdown 2024-01-20T11:14:28.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 09fd7abf-c068-48f2-8166-35df5fa549b8 1 2024-01-20T11:14:28.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:28.200+0000 7f32fbc41040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-20T11:14:28.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:28.248+0000 7f32fbc41040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-20T11:14:28.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.282+0000 7ff39144f640 10 monclient: get_monmap_and_config 2024-01-20T11:14:28.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.282+0000 7ff39144f640 10 monclient: build_initial_monmap 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.282+0000 7ff39144f640 10 monclient: monmap: 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:14:28.283329+0000 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:14:28.283329+0000 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:28.283 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.282+0000 7ff39144f640 1 Processor -- start 2024-01-20T11:14:28.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.282+0000 7ff39144f640 1 -- start start 2024-01-20T11:14:28.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.282+0000 7ff39144f640 10 monclient: init 2024-01-20T11:14:28.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:28.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-01-20T11:14:28.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff38c0b8550 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:28.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff38c0b4190 0x7ff38c1459a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 10 monclient(hunting): picked mon.noname-a con 0x7ff38c0b4190 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c145ee0 0x7ff38c14c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 10 monclient(hunting): picked mon.noname-b con 0x7ff38c145ee0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c146370 con 0x7ff38c0b8550 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38c0aadc0 con 0x7ff38c0b4190 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0b8930 con 0x7ff38c145ee0 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.283+0000 7ff39144f640 10 monclient(hunting): authenticate will time out at 1312.669506s 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38affd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff38c0b8550 0x7ff38c0b1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:42114/0 (socket says 172.21.15.134:42114) 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38b7fe640 1 --2- 172.21.15.134:0/2497606583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c145ee0 0x7ff38c14c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38affd640 1 -- 172.21.15.134:0/2497606583 learned_addr learned my addr 172.21.15.134:0/2497606583 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-01-20T11:14:28.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38a7fc640 1 --2- 172.21.15.134:0/2497606583 >> v2:172.21.15.110:3300/0 conn(0x7ff38c0b4190 0x7ff38c1459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38b7fe640 10 monclient(hunting): get_auth_request con 0x7ff38c145ee0 auth_method 0 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38a7fc640 10 monclient(hunting): get_auth_request con 0x7ff38c0b4190 auth_method 0 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.284+0000 7ff38a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:28.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.285+0000 7ff38a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:28.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.286+0000 7ff38b7fe640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-01-20T11:14:28.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.286+0000 7ff38b7fe640 1 -- 172.21.15.134:0/2497606583 >> v2:172.21.15.110:3300/0 conn(0x7ff38c0b4190 msgr2=0x7ff38c1459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:28.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.286+0000 7ff38b7fe640 1 --2- 172.21.15.134:0/2497606583 >> v2:172.21.15.110:3300/0 conn(0x7ff38c0b4190 0x7ff38c1459a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.286+0000 7ff38b7fe640 1 -- 172.21.15.134:0/2497606583 >> v1:172.21.15.134:6789/0 conn(0x7ff38c0b8550 legacy=0x7ff38c0b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:28.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.286+0000 7ff38b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:14:28.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.286+0000 7ff38b7fe640 1 monclient: found mon.noname-b 2024-01-20T11:14:28.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.286+0000 7ff38b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:28.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.286+0000 7ff38b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.286+0000 7ff38b7fe640 1 -- 172.21.15.134:0/2497606583 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c072990 con 0x7ff38c145ee0 2024-01-20T11:14:28.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.288+0000 7ff38b7fe640 1 --2- 172.21.15.134:0/2497606583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c145ee0 0x7ff38c14c770 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff37c0097a0 tx=0x7ff37c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=639d0eb329ed872a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:28.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 1 -- 172.21.15.134:0/2497606583 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c031830 con 0x7ff38c145ee0 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 10 monclient: dump: 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:28.293 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 10 monclient: _finish_auth 0 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 1 -- 172.21.15.134:0/2497606583 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c031990 con 0x7ff38c145ee0 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 1 -- 172.21.15.134:0/2497606583 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c031d20 con 0x7ff38c145ee0 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 5 monclient: authenticate success, global_id 4112 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 4 set_mon_vals no callback set 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff389ffb640 10 monclient: dump: 2024-01-20T11:14:28.294 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:28.295 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.295 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.295 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.295 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:28.295 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.295 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:28.295 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:28.295 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:28.295 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 10 monclient: shutdown 2024-01-20T11:14:28.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:28.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 1 -- 172.21.15.134:0/2497606583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c145ee0 msgr2=0x7ff38c14c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:28.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 1 --2- 172.21.15.134:0/2497606583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c145ee0 0x7ff38c14c770 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff37c0097a0 tx=0x7ff37c02b8d0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 1 -- 172.21.15.134:0/2497606583 shutdown_connections 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 1 --2- 172.21.15.134:0/2497606583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c145ee0 0x7ff38c14c770 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 1 --2- 172.21.15.134:0/2497606583 >> v2:172.21.15.110:3300/0 conn(0x7ff38c0b4190 0x7ff38c1459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.292+0000 7ff39144f640 1 -- 172.21.15.134:0/2497606583 >> 172.21.15.134:0/2497606583 conn(0x7ff38c0669e0 msgr2=0x7ff38c0b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.293+0000 7ff39144f640 1 -- 172.21.15.134:0/2497606583 shutdown_connections 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.293+0000 7ff39144f640 1 -- 172.21.15.134:0/2497606583 wait complete. 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.293+0000 7ff39144f640 10 monclient: build_initial_monmap 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.293+0000 7ff39144f640 10 monclient: monmap: 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-01-20T11:14:28.300 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.293+0000 7ff39144f640 1 librados: starting msgr at 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 librados: starting objecter 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 Processor -- start 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 -- start start 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 librados: setting wanted keys 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 librados: calling monclient init 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient: init 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:28.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff38c0b4190 0x7ff38c0b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient(hunting): picked mon.noname-a con 0x7ff38c0b4190 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c0b8550 0x7ff38c0afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient(hunting): picked mon.noname-c con 0x7ff38c0b8550 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff38c145ee0 0x7ff38c1d77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient(hunting): picked mon.noname-b con 0x7ff38c145ee0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c120370 con 0x7ff38c0b4190 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c1d7280 con 0x7ff38c145ee0 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0b1100 con 0x7ff38c0b8550 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:28.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.294+0000 7ff39144f640 10 monclient(hunting): authenticate will time out at 1312.681032s 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff38c145ee0 0x7ff38c1d77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38a7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c0b8550 0x7ff38c0afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff38c0b4190 0x7ff38c0b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff38c145ee0 0x7ff38c1d77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40958/0 (socket says 172.21.15.134:40958) 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38b7fe640 1 -- 172.21.15.134:0/2760969236 learned_addr learned my addr 172.21.15.134:0/2760969236 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38b7fe640 10 monclient(hunting): get_auth_request con 0x7ff38c145ee0 auth_method 0 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38a7fc640 10 monclient(hunting): get_auth_request con 0x7ff38c0b8550 auth_method 0 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:28.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38affd640 10 monclient(hunting): get_auth_request con 0x7ff38c0b4190 auth_method 0 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.295+0000 7ff38affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.296+0000 7ff38a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.296+0000 7ff38a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.296+0000 7ff38a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.296+0000 7ff38b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.296+0000 7ff38b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.296+0000 7ff38b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.296+0000 7ff38affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.296+0000 7ff38affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.296+0000 7ff38affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:28.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.298+0000 7ff38a7fc640 10 monclient(hunting): handle_auth_done global_id 4115 payload 824 2024-01-20T11:14:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.299+0000 7ff38a7fc640 1 -- 172.21.15.134:0/2760969236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff38c145ee0 msgr2=0x7ff38c1d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.299+0000 7ff38a7fc640 1 --2- 172.21.15.134:0/2760969236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff38c145ee0 0x7ff38c1d77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.299+0000 7ff38a7fc640 1 -- 172.21.15.134:0/2760969236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff38c0b4190 msgr2=0x7ff38c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.299+0000 7ff38a7fc640 1 --2- 172.21.15.134:0/2760969236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff38c0b4190 0x7ff38c0b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.299+0000 7ff38a7fc640 10 monclient: _finish_hunting 0 2024-01-20T11:14:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.299+0000 7ff38a7fc640 1 monclient: found mon.noname-c 2024-01-20T11:14:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.299+0000 7ff38a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.299+0000 7ff38a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.299+0000 7ff38a7fc640 1 -- 172.21.15.134:0/2760969236 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c009400 con 0x7ff38c0b8550 2024-01-20T11:14:28.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.301+0000 7ff38a7fc640 1 --2- 172.21.15.134:0/2760969236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c0b8550 0x7ff38c0afe40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e850 tx=0x7ff38000ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.301+0000 7ff3737fe640 10 client.?.objecter ms_handle_connect 0x7ff38c0b8550 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.301+0000 7ff3737fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 1 -- 172.21.15.134:0/2760969236 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3800095b0 con 0x7ff38c0b8550 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 10 monclient: dump: 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:28.307 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 10 monclient: _finish_auth 0 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 10 monclient: _check_auth_tickets 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 1 -- 172.21.15.134:0/2760969236 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38000c940 con 0x7ff38c0b8550 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff39144f640 5 monclient: authenticate success, global_id 4115 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff39144f640 10 monclient: _renew_subs 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff39144f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff39144f640 1 -- 172.21.15.134:0/2760969236 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c0b0380 con 0x7ff38c0b8550 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 1 -- 172.21.15.134:0/2760969236 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38000ccd0 con 0x7ff38c0b8550 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:28.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3737fe640 10 monclient: dump: 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff39144f640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff39144f640 10 monclient: _renew_subs 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff39144f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff3897fa640 4 set_mon_vals no callback set 2024-01-20T11:14:28.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff39144f640 1 -- 172.21.15.134:0/2760969236 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c0ad3d0 con 0x7ff38c0b8550 2024-01-20T11:14:28.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.304+0000 7ff39144f640 1 librados: init done 2024-01-20T11:14:28.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.305+0000 7ff39144f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:28.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.305+0000 7ff39144f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:28.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.305+0000 7ff39144f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.305+0000 7ff39144f640 1 -- 172.21.15.134:0/2760969236 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38c075500 con 0x7ff38c0b8550 2024-01-20T11:14:28.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.307+0000 7ff3737fe640 1 -- 172.21.15.134:0/2760969236 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7ff380005990 con 0x7ff38c0b8550 2024-01-20T11:14:28.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.307+0000 7ff3737fe640 1 -- 172.21.15.134:0/2760969236 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7ff380013470 con 0x7ff38c0b8550 2024-01-20T11:14:28.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.307+0000 7ff3737fe640 10 client.4115.objecter ms_dispatch 0x7ff38c000b90 osd_map(3..3 src has 1..3) v4 2024-01-20T11:14:28.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.307+0000 7ff3737fe640 3 client.4115.objecter handle_osd_map got epochs [3,3] > 0 2024-01-20T11:14:28.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.307+0000 7ff3737fe640 3 client.4115.objecter handle_osd_map decoding full epoch 3 2024-01-20T11:14:28.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.307+0000 7ff3737fe640 20 client.4115.objecter dump_active .. 0 homeless 2024-01-20T11:14:28.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.407+0000 7ff3737fe640 1 -- 172.21.15.134:0/2760969236 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7ff380021060 con 0x7ff38c0b8550 2024-01-20T11:14:28.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.407+0000 7ff3737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:28.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.407+0000 7ff3737fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:28.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.479+0000 7ff39144f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "09fd7abf-c068-48f2-8166-35df5fa549b8", "id": 1}] 2024-01-20T11:14:28.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.479+0000 7ff39144f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "09fd7abf-c068-48f2-8166-35df5fa549b8", "id": 1}] 2024-01-20T11:14:28.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.479+0000 7ff39144f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.479+0000 7ff39144f640 1 -- 172.21.15.134:0/2760969236 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd new", "uuid": "09fd7abf-c068-48f2-8166-35df5fa549b8", "id": 1} v 0) v1 -- 0x7ff38c141400 con 0x7ff38c0b8550 2024-01-20T11:14:28.518 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:28.518+0000 7fe56084c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-20T11:14:28.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.587+0000 7ff3737fe640 1 -- 172.21.15.134:0/2760969236 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "09fd7abf-c068-48f2-8166-35df5fa549b8", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff380017070 con 0x7ff38c0b8550 2024-01-20T11:14:28.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.587+0000 7ff3737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "09fd7abf-c068-48f2-8166-35df5fa549b8", "id": 1}] 2024-01-20T11:14:28.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.587+0000 7ff3737fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:28.588 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-01-20T11:14:28.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 10 librados: watch_flush enter 2024-01-20T11:14:28.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 10 librados: watch_flush exit 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 20 client.4115.objecter shutdown clearing up homeless session... 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 10 client.4115.objecter successfully canceled tick 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 10 monclient: shutdown 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 -- 172.21.15.134:0/2760969236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c0b8550 msgr2=0x7ff38c0afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 --2- 172.21.15.134:0/2760969236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c0b8550 0x7ff38c0afe40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e850 tx=0x7ff38000ed10 comp rx=0 tx=0).stop 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 -- 172.21.15.134:0/2760969236 shutdown_connections 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 --2- 172.21.15.134:0/2760969236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff38c145ee0 0x7ff38c1d77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 --2- 172.21.15.134:0/2760969236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff38c0b8550 0x7ff38c0afe40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 --2- 172.21.15.134:0/2760969236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff38c0b4190 0x7ff38c0b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 -- 172.21.15.134:0/2760969236 >> 172.21.15.134:0/2760969236 conn(0x7ff38c0669e0 msgr2=0x7ff38c13db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 -- 172.21.15.134:0/2760969236 shutdown_connections 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 -- 172.21.15.134:0/2760969236 wait complete. 2024-01-20T11:14:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.588+0000 7ff39144f640 1 librados: shutdown 2024-01-20T11:14:28.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 90618c4b-85ed-4ae8-8ae4-c8627385de33 2 2024-01-20T11:14:28.645 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:28.645+0000 7f32fbc41040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-20T11:14:28.662 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:28.662+0000 7fe56084c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-20T11:14:28.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.698+0000 7f3671e4f640 10 monclient: get_monmap_and_config 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.698+0000 7f3671e4f640 10 monclient: build_initial_monmap 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.698+0000 7f3671e4f640 10 monclient: monmap: 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:14:28.699390+0000 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:14:28.699390+0000 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:28.699 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.698+0000 7f3671e4f640 1 Processor -- start 2024-01-20T11:14:28.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.698+0000 7f3671e4f640 1 -- start start 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.698+0000 7f3671e4f640 10 monclient: init 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f366c0b8550 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f366c0b4190 0x7f366c1459a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 10 monclient(hunting): picked mon.noname-c con 0x7f366c0b4190 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f366c145ee0 0x7f366c14c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 10 monclient(hunting): picked mon.noname-a con 0x7f366c145ee0 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c0aadc0 con 0x7f366c0b8550 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c146330 con 0x7f366c145ee0 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c0b8930 con 0x7f366c0b4190 2024-01-20T11:14:28.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.699+0000 7f3671e4f640 10 monclient(hunting): authenticate will time out at 1313.085725s 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.700+0000 7f3662ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f366c0b4190 0x7f366c1459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.700+0000 7f3662ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f366c0b4190 0x7f366c1459a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48662/0 (socket says 172.21.15.134:48662) 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.700+0000 7f3662ffd640 1 -- 172.21.15.134:0/3846923810 learned_addr learned my addr 172.21.15.134:0/3846923810 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.700+0000 7f3662ffd640 10 monclient(hunting): get_auth_request con 0x7f366c0b4190 auth_method 0 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.700+0000 7f3662ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.700+0000 7f3662ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.700+0000 7f3662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.701+0000 7f3662ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.701+0000 7f3662ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:28.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.701+0000 7f3662ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:28.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.703+0000 7f3662ffd640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-01-20T11:14:28.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.703+0000 7f3662ffd640 1 -- 172.21.15.134:0/3846923810 >> v2:172.21.15.110:3300/0 conn(0x7f366c145ee0 msgr2=0x7f366c14c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:28.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.703+0000 7f3662ffd640 1 --2- 172.21.15.134:0/3846923810 >> v2:172.21.15.110:3300/0 conn(0x7f366c145ee0 0x7f366c14c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.703+0000 7f3662ffd640 1 -- 172.21.15.134:0/3846923810 >> v1:172.21.15.134:6789/0 conn(0x7f366c0b8550 legacy=0x7f366c0b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:28.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.703+0000 7f3662ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:14:28.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.703+0000 7f3662ffd640 1 monclient: found mon.noname-c 2024-01-20T11:14:28.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.703+0000 7f3662ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:28.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.703+0000 7f3662ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:14:28.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.703+0000 7f3662ffd640 1 -- 172.21.15.134:0/3846923810 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c072990 con 0x7f366c0b4190 2024-01-20T11:14:28.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.705+0000 7f3662ffd640 1 --2- 172.21.15.134:0/3846923810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f366c0b4190 0x7f366c1459a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f366c06a660 tx=0x7f365402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35a6fad974a289b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.717+0000 7f366affd640 1 -- 172.21.15.134:0/3846923810 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654031870 con 0x7f366c0b4190 2024-01-20T11:14:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.717+0000 7f366affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.717+0000 7f366affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.717+0000 7f366affd640 10 monclient: dump: 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.717+0000 7f366affd640 10 monclient: _finish_auth 0 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.717+0000 7f366affd640 10 monclient: _check_auth_tickets 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.717+0000 7f366affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.717+0000 7f3671e4f640 5 monclient: authenticate success, global_id 4123 2024-01-20T11:14:28.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.717+0000 7f3671e4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.740+0000 7f366affd640 1 -- 172.21.15.134:0/3846923810 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36540319d0 con 0x7f366c0b4190 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.740+0000 7f366affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.740+0000 7f366affd640 1 -- 172.21.15.134:0/3846923810 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654031d60 con 0x7f366c0b4190 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.740+0000 7f3671e4f640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.740+0000 7f3671e4f640 4 set_mon_vals no callback set 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.740+0000 7f366affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.740+0000 7f366affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.740+0000 7f366affd640 10 monclient: dump: 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.742 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:28.742 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.742 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:28.742 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:28.742 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:28.742 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 10 monclient: shutdown 2024-01-20T11:14:28.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:28.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 1 -- 172.21.15.134:0/3846923810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f366c0b4190 msgr2=0x7f366c1459a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:28.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 1 --2- 172.21.15.134:0/3846923810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f366c0b4190 0x7f366c1459a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f366c06a660 tx=0x7f365402b910 comp rx=0 tx=0).stop 2024-01-20T11:14:28.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 1 -- 172.21.15.134:0/3846923810 shutdown_connections 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 1 --2- 172.21.15.134:0/3846923810 >> v2:172.21.15.110:3300/0 conn(0x7f366c145ee0 0x7f366c14c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 1 --2- 172.21.15.134:0/3846923810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f366c0b4190 0x7f366c1459a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 1 -- 172.21.15.134:0/3846923810 >> 172.21.15.134:0/3846923810 conn(0x7f366c0669e0 msgr2=0x7f366c0b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 1 -- 172.21.15.134:0/3846923810 shutdown_connections 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 1 -- 172.21.15.134:0/3846923810 wait complete. 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 10 monclient: build_initial_monmap 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.741+0000 7f3671e4f640 10 monclient: monmap: 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:28.745 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 librados: starting msgr at 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 librados: starting objecter 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 Processor -- start 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 -- start start 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 librados: setting wanted keys 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 librados: calling monclient init 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient: init 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f366c0b4190 0x7f366c0b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient(hunting): picked mon.noname-b con 0x7f366c0b4190 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f366c0b8550 0x7f366c0adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient(hunting): picked mon.noname-c con 0x7f366c0b8550 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:28.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366c145ee0 0x7f366c1d7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient(hunting): picked mon.noname-a con 0x7f366c145ee0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0b0090 con 0x7f366c145ee0 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0adc10 con 0x7f366c0b4190 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c1d7530 con 0x7f366c0b8550 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.742+0000 7f3671e4f640 10 monclient(hunting): authenticate will time out at 1313.129078s 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.743+0000 7f3662ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f366c0b8550 0x7f366c0adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.743+0000 7f3662ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f366c0b8550 0x7f366c0adec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.134:52592/0 (socket says 172.21.15.134:52592) 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.743+0000 7f3662ffd640 1 -- 172.21.15.134:0/1820058799 learned_addr learned my addr 172.21.15.134:0/1820058799 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.743+0000 7f3662ffd640 10 monclient(hunting): get_auth_request con 0x7f366c0b8550 auth_method 0 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.743+0000 7f3662ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:28.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.743+0000 7f3662ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.743+0000 7f3662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.744+0000 7f3662ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.744+0000 7f3662ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.744+0000 7f3662ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.744+0000 7f366b7fe640 1 --2- 172.21.15.134:0/1820058799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f366c0b4190 0x7f366c0b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366b7fe640 10 monclient(hunting): get_auth_request con 0x7f366c0b4190 auth_method 0 2024-01-20T11:14:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366bfff640 1 --2- 172.21.15.134:0/1820058799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366c145ee0 0x7f366c1d7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366bfff640 10 monclient(hunting): get_auth_request con 0x7f366c145ee0 auth_method 0 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.745+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.746+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.746+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.746+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 10 monclient(hunting): handle_auth_done global_id 4118 payload 824 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 1 -- 172.21.15.134:0/1820058799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f366c0b4190 msgr2=0x7f366c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 1 --2- 172.21.15.134:0/1820058799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f366c0b4190 0x7f366c0b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 1 -- 172.21.15.134:0/1820058799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366c145ee0 msgr2=0x7f366c1d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 1 --2- 172.21.15.134:0/1820058799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366c145ee0 0x7f366c1d7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:14:28.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 1 monclient: found mon.noname-c 2024-01-20T11:14:28.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:28.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.747+0000 7f3662ffd640 1 -- 172.21.15.134:0/1820058799 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36540094d0 con 0x7f366c0b8550 2024-01-20T11:14:28.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.750+0000 7f3662ffd640 1 --2- 172.21.15.134:0/1820058799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f366c0b8550 0x7f366c0adec0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3654000c00 tx=0x7f36540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:28.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.750+0000 7f3668ff9640 10 client.?.objecter ms_handle_connect 0x7f366c0b8550 2024-01-20T11:14:28.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.750+0000 7f3668ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 1 -- 172.21.15.134:0/1820058799 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365403ad10 con 0x7f366c0b8550 2024-01-20T11:14:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 10 monclient: dump: 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 10 monclient: _finish_auth 0 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:14:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 1 -- 172.21.15.134:0/1820058799 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365403ae70 con 0x7f366c0b8550 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 1 -- 172.21.15.134:0/1820058799 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654044610 con 0x7f366c0b8550 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3668ff9640 10 monclient: dump: 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3671e4f640 5 monclient: authenticate success, global_id 4118 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3671e4f640 10 monclient: _renew_subs 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f366a7fc640 4 set_mon_vals no callback set 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3671e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3671e4f640 1 -- 172.21.15.134:0/1820058799 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c0b14b0 con 0x7f366c0b8550 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3671e4f640 10 client.4118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3671e4f640 10 monclient: _renew_subs 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3671e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3671e4f640 1 -- 172.21.15.134:0/1820058799 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c1d7f90 con 0x7f366c0b8550 2024-01-20T11:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.753+0000 7f3671e4f640 1 librados: init done 2024-01-20T11:14:28.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.754+0000 7f3671e4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:28.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.754+0000 7f3671e4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:28.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.754+0000 7f3671e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.754+0000 7f3671e4f640 1 -- 172.21.15.134:0/1820058799 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366c1d73a0 con 0x7f366c0b8550 2024-01-20T11:14:28.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.757+0000 7f3668ff9640 1 -- 172.21.15.134:0/1820058799 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f3654044770 con 0x7f366c0b8550 2024-01-20T11:14:28.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.757+0000 7f3668ff9640 1 -- 172.21.15.134:0/1820058799 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f365402d040 con 0x7f366c0b8550 2024-01-20T11:14:28.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.757+0000 7f3668ff9640 10 client.4118.objecter ms_dispatch 0x7f366c000b90 osd_map(4..4 src has 1..4) v4 2024-01-20T11:14:28.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.757+0000 7f3668ff9640 3 client.4118.objecter handle_osd_map got epochs [4,4] > 0 2024-01-20T11:14:28.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.757+0000 7f3668ff9640 3 client.4118.objecter handle_osd_map decoding full epoch 4 2024-01-20T11:14:28.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.757+0000 7f3668ff9640 20 client.4118.objecter dump_active .. 0 homeless 2024-01-20T11:14:28.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:28.790+0000 7f32fbc41040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-20T11:14:28.798 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:28.798+0000 7fe56084c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-20T11:14:28.842 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:28.842+0000 7fe56084c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-20T11:14:28.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.853+0000 7f3668ff9640 1 -- 172.21.15.134:0/1820058799 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f3654035020 con 0x7f366c0b8550 2024-01-20T11:14:28.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.853+0000 7f3668ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:28.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.853+0000 7f3668ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:28.906 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:28.906+0000 7f32fbc41040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-20T11:14:28.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.925+0000 7f3671e4f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "90618c4b-85ed-4ae8-8ae4-c8627385de33", "id": 2}] 2024-01-20T11:14:28.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.925+0000 7f3671e4f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "90618c4b-85ed-4ae8-8ae4-c8627385de33", "id": 2}] 2024-01-20T11:14:28.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.925+0000 7f3671e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:28.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:28.925+0000 7f3671e4f640 1 -- 172.21.15.134:0/1820058799 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd new", "uuid": "90618c4b-85ed-4ae8-8ae4-c8627385de33", "id": 2} v 0) v1 -- 0x7f366c075500 con 0x7f366c0b8550 2024-01-20T11:14:28.942 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:28.941+0000 7fe56084c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-20T11:14:28.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:28.954+0000 7f32fbc41040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-20T11:14:29.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.002+0000 7f3668ff9640 1 -- 172.21.15.134:0/1820058799 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "90618c4b-85ed-4ae8-8ae4-c8627385de33", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f365404d690 con 0x7f366c0b8550 2024-01-20T11:14:29.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.002+0000 7f3668ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "90618c4b-85ed-4ae8-8ae4-c8627385de33", "id": 2}] 2024-01-20T11:14:29.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.002+0000 7f3668ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:29.003 INFO:teuthology.orchestra.run.smithi134.stdout:2 2024-01-20T11:14:29.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.002+0000 7f3671e4f640 10 librados: watch_flush enter 2024-01-20T11:14:29.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.002+0000 7f3671e4f640 10 librados: watch_flush exit 2024-01-20T11:14:29.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.002+0000 7f3671e4f640 20 client.4118.objecter shutdown clearing up homeless session... 2024-01-20T11:14:29.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.002+0000 7f3671e4f640 10 client.4118.objecter successfully canceled tick 2024-01-20T11:14:29.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 10 monclient: shutdown 2024-01-20T11:14:29.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:29.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 -- 172.21.15.134:0/1820058799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f366c0b8550 msgr2=0x7f366c0adec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 --2- 172.21.15.134:0/1820058799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f366c0b8550 0x7f366c0adec0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3654000c00 tx=0x7f36540046c0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 -- 172.21.15.134:0/1820058799 shutdown_connections 2024-01-20T11:14:29.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 --2- 172.21.15.134:0/1820058799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366c145ee0 0x7f366c1d7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 --2- 172.21.15.134:0/1820058799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f366c0b8550 0x7f366c0adec0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 --2- 172.21.15.134:0/1820058799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f366c0b4190 0x7f366c0b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 -- 172.21.15.134:0/1820058799 >> 172.21.15.134:0/1820058799 conn(0x7f366c0669e0 msgr2=0x7f366c0b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:29.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 -- 172.21.15.134:0/1820058799 shutdown_connections 2024-01-20T11:14:29.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 -- 172.21.15.134:0/1820058799 wait complete. 2024-01-20T11:14:29.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.003+0000 7f3671e4f640 1 librados: shutdown 2024-01-20T11:14:29.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new f83babc8-dca0-4c13-9bdd-b643ece19536 3 2024-01-20T11:14:29.042 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:29.041+0000 7fe56084c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-20T11:14:29.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:29.061+0000 7f32fbc41040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-20T11:14:29.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.112+0000 7f2fb3275640 10 monclient: get_monmap_and_config 2024-01-20T11:14:29.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.112+0000 7f2fb3275640 10 monclient: build_initial_monmap 2024-01-20T11:14:29.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.112+0000 7f2fb3275640 10 monclient: monmap: 2024-01-20T11:14:29.113 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:29.113 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.113 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:14:29.113872+0000 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:14:29.113872+0000 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 1 Processor -- start 2024-01-20T11:14:29.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 1 -- start start 2024-01-20T11:14:29.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient: init 2024-01-20T11:14:29.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:29.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-01-20T11:14:29.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2fac0b8550 0x7f2fac0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient(hunting): picked mon.noname-a con 0x7f2fac0b8550 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:29.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2fac0b1b10 0x7f2fac0b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient(hunting): picked mon.noname-c con 0x7f2fac0b1b10 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient(hunting): picked mon.noname-b con 0x7f2fac1459a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac146370 con 0x7f2fac0b8550 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac0aadc0 con 0x7f2fac0b1b10 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0b8930 con 0x7f2fac1459a0 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.113+0000 7f2fb3275640 10 monclient(hunting): authenticate will time out at 1313.500013s 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.114+0000 7f2fb0810640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2fac0b1b10 0x7f2fac0b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.114+0000 7f2fb1011640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2fac0b8550 0x7f2fac0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.114+0000 7f2fb1812640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.114+0000 7f2fb0810640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2fac0b1b10 0x7f2fac0b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48674/0 (socket says 172.21.15.134:48674) 2024-01-20T11:14:29.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.114+0000 7f2fb1011640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2fac0b8550 0x7f2fac0b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.134:58520/0 (socket says 172.21.15.134:58520) 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.114+0000 7f2fb1812640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.134:52606/0 (socket says 172.21.15.134:52606) 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.114+0000 7f2fb1011640 1 -- 172.21.15.134:0/2059344931 learned_addr learned my addr 172.21.15.134:0/2059344931 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1011640 10 monclient(hunting): get_auth_request con 0x7f2fac0b8550 auth_method 0 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1011640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1011640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb0810640 10 monclient(hunting): get_auth_request con 0x7f2fac0b1b10 auth_method 0 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb0810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb0810640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb0810640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1812640 10 monclient(hunting): get_auth_request con 0x7f2fac1459a0 auth_method 0 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1812640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1812640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1812640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1011640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1011640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb1011640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb0810640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb0810640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.115+0000 7f2fb0810640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-01-20T11:14:29.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 1 -- 172.21.15.134:0/2059344931 >> v2:172.21.15.134:3300/0 conn(0x7f2fac0b1b10 msgr2=0x7f2fac0b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:29.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 1 --2- 172.21.15.134:0/2059344931 >> v2:172.21.15.134:3300/0 conn(0x7f2fac0b1b10 0x7f2fac0b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 1 -- 172.21.15.134:0/2059344931 >> v2:172.21.15.110:3300/0 conn(0x7f2fac0b8550 msgr2=0x7f2fac0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:29.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 1 --2- 172.21.15.134:0/2059344931 >> v2:172.21.15.110:3300/0 conn(0x7f2fac0b8550 0x7f2fac0b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 10 monclient: _finish_hunting 0 2024-01-20T11:14:29.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 1 monclient: found mon.noname-b 2024-01-20T11:14:29.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:29.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:14:29.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb1812640 1 -- 172.21.15.134:0/2059344931 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac072990 con 0x7f2fac1459a0 2024-01-20T11:14:29.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.117+0000 7f2fb0810640 1 --2- 172.21.15.134:0/2059344931 >> v2:172.21.15.134:3300/0 conn(0x7f2fac0b1b10 0x7f2fac0b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:14:29.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.118+0000 7f2fb1812640 1 --2- 172.21.15.134:0/2059344931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac145e30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fac06a660 tx=0x7f2fa402be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a87aa7b1774707d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 1 -- 172.21.15.134:0/2059344931 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa402cbd0 con 0x7f2fac1459a0 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 10 monclient: dump: 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.123 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 10 monclient: _finish_auth 0 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 10 monclient: _check_auth_tickets 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 1 -- 172.21.15.134:0/2059344931 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa402cd30 con 0x7f2fac1459a0 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 1 -- 172.21.15.134:0/2059344931 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa402cea0 con 0x7f2fac1459a0 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fb3275640 5 monclient: authenticate success, global_id 4121 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fb3275640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fb3275640 4 set_mon_vals no callback set 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.122+0000 7f2fa3fff640 10 monclient: dump: 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 10 monclient: shutdown 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 1 -- 172.21.15.134:0/2059344931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 msgr2=0x7f2fac145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 1 --2- 172.21.15.134:0/2059344931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac145e30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fac06a660 tx=0x7f2fa402be20 comp rx=0 tx=0).stop 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 1 -- 172.21.15.134:0/2059344931 shutdown_connections 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 1 --2- 172.21.15.134:0/2059344931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac145e30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 1 --2- 172.21.15.134:0/2059344931 >> v2:172.21.15.134:3300/0 conn(0x7f2fac0b1b10 0x7f2fac0b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 1 --2- 172.21.15.134:0/2059344931 >> v2:172.21.15.110:3300/0 conn(0x7f2fac0b8550 0x7f2fac0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 1 -- 172.21.15.134:0/2059344931 >> 172.21.15.134:0/2059344931 conn(0x7f2fac0669e0 msgr2=0x7f2fac0b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 1 -- 172.21.15.134:0/2059344931 shutdown_connections 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.123+0000 7f2fb3275640 1 -- 172.21.15.134:0/2059344931 wait complete. 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 10 monclient: build_initial_monmap 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 10 monclient: monmap: 2024-01-20T11:14:29.129 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 1 librados: starting msgr at 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 1 librados: starting objecter 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 1 Processor -- start 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 1 -- start start 2024-01-20T11:14:29.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 1 librados: setting wanted keys 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 1 librados: calling monclient init 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 10 monclient: init 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0b1b10 0x7f2fac1da9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.124+0000 7f2fb3275640 10 monclient(hunting): picked mon.noname-a con 0x7f2fac0b1b10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac0b8550 0x7f2fac13c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 10 monclient(hunting): picked mon.noname-b con 0x7f2fac0b8550 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac13c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 10 monclient(hunting): picked mon.noname-c con 0x7f2fac1459a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0b1400 con 0x7f2fac0b1b10 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0af530 con 0x7f2fac0b8550 2024-01-20T11:14:29.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac13cb40 con 0x7f2fac1459a0 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb3275640 10 monclient(hunting): authenticate will time out at 1313.511148s 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb0810640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac0b8550 0x7f2fac13c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb1812640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac13c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb1011640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0b1b10 0x7f2fac1da9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb1812640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac13c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.134:52618/0 (socket says 172.21.15.134:52618) 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb1812640 1 -- 172.21.15.134:0/1262189587 learned_addr learned my addr 172.21.15.134:0/1262189587 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb1812640 10 monclient(hunting): get_auth_request con 0x7f2fac1459a0 auth_method 0 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb1812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb1812640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb1812640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb0810640 10 monclient(hunting): get_auth_request con 0x7f2fac0b8550 auth_method 0 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb0810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb0810640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.125+0000 7f2fb0810640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1011640 10 monclient(hunting): get_auth_request con 0x7f2fac0b1b10 auth_method 0 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1011640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1011640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1812640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb0810640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb0810640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb0810640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1011640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1011640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.126+0000 7f2fb1011640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-01-20T11:14:29.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 1 -- 172.21.15.134:0/1262189587 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 msgr2=0x7f2fac13c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 1 --2- 172.21.15.134:0/1262189587 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac13c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 1 -- 172.21.15.134:0/1262189587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0b1b10 msgr2=0x7f2fac1da9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 1 --2- 172.21.15.134:0/1262189587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0b1b10 0x7f2fac1da9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 10 monclient: _finish_hunting 0 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 1 monclient: found mon.noname-b 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.129+0000 7f2fb0810640 1 -- 172.21.15.134:0/1262189587 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c009400 con 0x7f2fac0b8550 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.131+0000 7f2fb0810640 1 --2- 172.21.15.134:0/1262189587 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac0b8550 0x7f2fac13c0c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00e870 tx=0x7f2f9c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.131+0000 7f2fa1ffb640 10 client.?.objecter ms_handle_connect 0x7f2fac0b8550 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.131+0000 7f2fa1ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 1 -- 172.21.15.134:0/1262189587 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c00c940 con 0x7f2fac0b8550 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 10 monclient: dump: 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.136 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 10 monclient: _finish_auth 0 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 1 -- 172.21.15.134:0/1262189587 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c00caa0 con 0x7f2fac0b8550 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 1 -- 172.21.15.134:0/1262189587 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c00ce30 con 0x7f2fac0b8550 2024-01-20T11:14:29.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa1ffb640 10 monclient: dump: 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fb3275640 5 monclient: authenticate success, global_id 4132 2024-01-20T11:14:29.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fb3275640 10 monclient: _renew_subs 2024-01-20T11:14:29.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fa37fe640 4 set_mon_vals no callback set 2024-01-20T11:14:29.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fb3275640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fb3275640 1 -- 172.21.15.134:0/1262189587 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa4009a70 con 0x7f2fac0b8550 2024-01-20T11:14:29.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fb3275640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:29.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fb3275640 10 monclient: _renew_subs 2024-01-20T11:14:29.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fb3275640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fb3275640 1 -- 172.21.15.134:0/1262189587 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac0af1f0 con 0x7f2fac0b8550 2024-01-20T11:14:29.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.134+0000 7f2fb3275640 1 librados: init done 2024-01-20T11:14:29.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.135+0000 7f2fb3275640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:29.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.135+0000 7f2fb3275640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:29.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.135+0000 7f2fb3275640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.135+0000 7f2fb3275640 1 -- 172.21.15.134:0/1262189587 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac075500 con 0x7f2fac0b8550 2024-01-20T11:14:29.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.138+0000 7f2fa1ffb640 1 -- 172.21.15.134:0/1262189587 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f2f9c005990 con 0x7f2fac0b8550 2024-01-20T11:14:29.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.138+0000 7f2fa1ffb640 1 -- 172.21.15.134:0/1262189587 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f2f9c013900 con 0x7f2fac0b8550 2024-01-20T11:14:29.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.138+0000 7f2fa1ffb640 10 client.4132.objecter ms_dispatch 0x7f2fac000b90 osd_map(5..5 src has 1..5) v4 2024-01-20T11:14:29.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.138+0000 7f2fa1ffb640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-01-20T11:14:29.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.138+0000 7f2fa1ffb640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-01-20T11:14:29.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.138+0000 7f2fa1ffb640 20 client.4132.objecter dump_active .. 0 homeless 2024-01-20T11:14:29.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:29.158+0000 7f32fbc41040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-20T11:14:29.189 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:29.188+0000 7fe56084c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-20T11:14:29.233 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:29.232+0000 7fe56084c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-20T11:14:29.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.235+0000 7f2fa1ffb640 1 -- 172.21.15.134:0/1262189587 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f2f9c021060 con 0x7f2fac0b8550 2024-01-20T11:14:29.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.235+0000 7f2fa1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:29.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.235+0000 7f2fa1ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:29.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:29.302+0000 7f32fbc41040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-20T11:14:29.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.310+0000 7f2fb3275640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f83babc8-dca0-4c13-9bdd-b643ece19536", "id": 3}] 2024-01-20T11:14:29.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.310+0000 7f2fb3275640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f83babc8-dca0-4c13-9bdd-b643ece19536", "id": 3}] 2024-01-20T11:14:29.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.310+0000 7f2fb3275640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.310+0000 7f2fb3275640 1 -- 172.21.15.134:0/1262189587 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f83babc8-dca0-4c13-9bdd-b643ece19536", "id": 3} v 0) v1 -- 0x7f2fac13f890 con 0x7f2fac0b8550 2024-01-20T11:14:29.347 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:29.347+0000 7f32fbc41040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-20T11:14:29.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.374+0000 7f2fa1ffb640 1 -- 172.21.15.134:0/1262189587 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f83babc8-dca0-4c13-9bdd-b643ece19536", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2f9c017070 con 0x7f2fac0b8550 2024-01-20T11:14:29.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.374+0000 7f2fa1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f83babc8-dca0-4c13-9bdd-b643ece19536", "id": 3}] 2024-01-20T11:14:29.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.374+0000 7f2fa1ffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:29.376 INFO:teuthology.orchestra.run.smithi134.stdout:3 2024-01-20T11:14:29.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 10 librados: watch_flush enter 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 10 librados: watch_flush exit 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 20 client.4132.objecter shutdown clearing up homeless session... 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 10 client.4132.objecter successfully canceled tick 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 10 monclient: shutdown 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 -- 172.21.15.134:0/1262189587 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac0b8550 msgr2=0x7f2fac13c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 --2- 172.21.15.134:0/1262189587 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac0b8550 0x7f2fac13c0c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00e870 tx=0x7f2f9c00ed30 comp rx=0 tx=0).stop 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 -- 172.21.15.134:0/1262189587 shutdown_connections 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 --2- 172.21.15.134:0/1262189587 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac1459a0 0x7f2fac13c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 --2- 172.21.15.134:0/1262189587 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac0b8550 0x7f2fac13c0c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 --2- 172.21.15.134:0/1262189587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0b1b10 0x7f2fac1da9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 -- 172.21.15.134:0/1262189587 >> 172.21.15.134:0/1262189587 conn(0x7f2fac0669e0 msgr2=0x7f2fac1435c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 -- 172.21.15.134:0/1262189587 shutdown_connections 2024-01-20T11:14:29.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 -- 172.21.15.134:0/1262189587 wait complete. 2024-01-20T11:14:29.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.375+0000 7f2fb3275640 1 librados: shutdown 2024-01-20T11:14:29.379 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:29.379+0000 7fe56084c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-20T11:14:29.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 27eba313-a00b-4455-bd52-efe2b7792812 4 2024-01-20T11:14:29.500 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:29.500+0000 7f32fbc41040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-20T11:14:29.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.526+0000 7f7383525640 10 monclient: get_monmap_and_config 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.526+0000 7f7383525640 10 monclient: build_initial_monmap 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.526+0000 7f7383525640 10 monclient: monmap: 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:14:29.527368+0000 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:14:29.527368+0000 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:29.527 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.526+0000 7f7383525640 1 Processor -- start 2024-01-20T11:14:29.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.526+0000 7f7383525640 1 -- start start 2024-01-20T11:14:29.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.526+0000 7f7383525640 10 monclient: init 2024-01-20T11:14:29.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:29.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-01-20T11:14:29.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f737c0b8550 0x7f737c0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 10 monclient(hunting): picked mon.noname-a con 0x7f737c0b8550 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:29.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f737c0b1b10 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:29.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f737c1460b0 addr v1:172.21.15.110:6789/0 2024-01-20T11:14:29.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737c0aadc0 con 0x7f737c1460b0 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737c0b8930 con 0x7f737c0b1b10 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f737c072990 con 0x7f737c0b8550 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.527+0000 7f7383525640 10 monclient(hunting): authenticate will time out at 1313.913535s 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.528+0000 7f7380ac0640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f737c0b1b10 0x7f737c1459a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49872/0 (socket says 172.21.15.134:49872) 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.528+0000 7f7380ac0640 1 -- 172.21.15.134:0/2588443956 learned_addr learned my addr 172.21.15.134:0/2588443956 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.528+0000 7f73812c1640 1 --2- 172.21.15.134:0/2588443956 >> v2:172.21.15.110:3300/0 conn(0x7f737c0b8550 0x7f737c0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.528+0000 7f73812c1640 10 monclient(hunting): get_auth_request con 0x7f737c0b8550 auth_method 0 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.528+0000 7f73812c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.528+0000 7f73812c1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.528+0000 7f73812c1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.529+0000 7f73812c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.529+0000 7f73812c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.529+0000 7f73812c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.531+0000 7f73812c1640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2024-01-20T11:14:29.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.531+0000 7f73812c1640 1 -- 172.21.15.134:0/2588443956 >> v1:172.21.15.134:6789/0 conn(0x7f737c0b1b10 legacy=0x7f737c1459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.531+0000 7f73812c1640 1 -- 172.21.15.134:0/2588443956 >> v1:172.21.15.110:6789/0 conn(0x7f737c1460b0 legacy=0x7f737c14c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.531+0000 7f73812c1640 10 monclient: _finish_hunting 0 2024-01-20T11:14:29.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.531+0000 7f73812c1640 1 monclient: found mon.noname-a 2024-01-20T11:14:29.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.531+0000 7f73812c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:29.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.531+0000 7f73812c1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:14:29.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.531+0000 7f73812c1640 1 -- 172.21.15.134:0/2588443956 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c0a9be0 con 0x7f737c0b8550 2024-01-20T11:14:29.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.537+0000 7f73812c1640 1 --2- 172.21.15.134:0/2588443956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f737c0b8550 0x7f737c0b4190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f736c0097a0 tx=0x7f736c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32b4edfb3e4d5d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:29.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.541+0000 7f7373fff640 1 -- 172.21.15.134:0/2588443956 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c030610 con 0x7f737c0b8550 2024-01-20T11:14:29.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:14:29.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 10 monclient: dump: 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 10 monclient: _finish_auth 0 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 10 monclient: _check_auth_tickets 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 1 -- 172.21.15.134:0/2588443956 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736c030770 con 0x7f737c0b8550 2024-01-20T11:14:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 1 -- 172.21.15.134:0/2588443956 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c030b00 con 0x7f737c0b8550 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 5 monclient: authenticate success, global_id 4125 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 4 set_mon_vals no callback set 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7373fff640 10 monclient: dump: 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.544 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.545 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.545 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.545 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 10 monclient: shutdown 2024-01-20T11:14:29.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:29.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 1 -- 172.21.15.134:0/2588443956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f737c0b8550 msgr2=0x7f737c0b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 1 --2- 172.21.15.134:0/2588443956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f737c0b8550 0x7f737c0b4190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f736c0097a0 tx=0x7f736c02b960 comp rx=0 tx=0).stop 2024-01-20T11:14:29.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 1 -- 172.21.15.134:0/2588443956 shutdown_connections 2024-01-20T11:14:29.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 1 --2- 172.21.15.134:0/2588443956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f737c0b8550 0x7f737c0b4190 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 1 -- 172.21.15.134:0/2588443956 >> 172.21.15.134:0/2588443956 conn(0x7f737c0669e0 msgr2=0x7f737c0b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 1 -- 172.21.15.134:0/2588443956 shutdown_connections 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.542+0000 7f7383525640 1 -- 172.21.15.134:0/2588443956 wait complete. 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.543+0000 7f7383525640 10 monclient: build_initial_monmap 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.543+0000 7f7383525640 10 monclient: monmap: 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.543+0000 7f7383525640 1 librados: starting msgr at 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.543+0000 7f7383525640 1 librados: starting objecter 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.543+0000 7f7383525640 1 Processor -- start 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 1 -- start start 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 1 librados: setting wanted keys 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 1 librados: calling monclient init 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient: init 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f737c0b1b10 0x7f737c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient(hunting): picked mon.noname-a con 0x7f737c0b1b10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f737c0b8550 0x7f737c144750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient(hunting): picked mon.noname-b con 0x7f737c0b8550 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f737c1460b0 0x7f737c1d78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient(hunting): picked mon.noname-c con 0x7f737c1460b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c1d7e00 con 0x7f737c0b1b10 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c1d8040 con 0x7f737c0b8550 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c1d8560 con 0x7f737c1460b0 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.544+0000 7f7383525640 10 monclient(hunting): authenticate will time out at 1313.930713s 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7380ac0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f737c0b8550 0x7f737c144750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7381ac2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f737c1460b0 0x7f737c1d78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7380ac0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f737c0b8550 0x7f737c144750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48694/0 (socket says 172.21.15.134:48694) 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f73812c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f737c0b1b10 0x7f737c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7380ac0640 1 -- 172.21.15.134:0/1380483962 learned_addr learned my addr 172.21.15.134:0/1380483962 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7380ac0640 10 monclient(hunting): get_auth_request con 0x7f737c0b8550 auth_method 0 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7380ac0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7380ac0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7380ac0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f73812c1640 10 monclient(hunting): get_auth_request con 0x7f737c0b1b10 auth_method 0 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f73812c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f73812c1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f73812c1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7381ac2640 10 monclient(hunting): get_auth_request con 0x7f737c1460b0 auth_method 0 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7381ac2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7381ac2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.545+0000 7f7381ac2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.546+0000 7f7380ac0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.546+0000 7f7380ac0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.546+0000 7f7380ac0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.546+0000 7f73812c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.546+0000 7f73812c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.546+0000 7f73812c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.546+0000 7f7381ac2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.546+0000 7f7381ac2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.546+0000 7f7381ac2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-01-20T11:14:29.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 1 -- 172.21.15.134:0/1380483962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f737c1460b0 msgr2=0x7f737c1d78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 1 --2- 172.21.15.134:0/1380483962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f737c1460b0 0x7f737c1d78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 1 -- 172.21.15.134:0/1380483962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f737c0b1b10 msgr2=0x7f737c0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 1 --2- 172.21.15.134:0/1380483962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f737c0b1b10 0x7f737c0b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 10 monclient: _finish_hunting 0 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 1 monclient: found mon.noname-b 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.548+0000 7f7380ac0640 1 -- 172.21.15.134:0/1380483962 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c009400 con 0x7f737c0b8550 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.550+0000 7f7380ac0640 1 --2- 172.21.15.134:0/1380483962 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f737c0b8550 0x7f737c144750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f737800c400 tx=0x7f737800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.550+0000 7f7371ffb640 10 client.?.objecter ms_handle_connect 0x7f737c0b8550 2024-01-20T11:14:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.550+0000 7f7371ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:29.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 1 -- 172.21.15.134:0/1380483962 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7378005ce0 con 0x7f737c0b8550 2024-01-20T11:14:29.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:29.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 10 monclient: dump: 2024-01-20T11:14:29.555 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 10 monclient: _finish_auth 0 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 1 -- 172.21.15.134:0/1380483962 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7378005e40 con 0x7f737c0b8550 2024-01-20T11:14:29.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 1 -- 172.21.15.134:0/1380483962 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73780072d0 con 0x7f737c0b8550 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7383525640 5 monclient: authenticate success, global_id 4138 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7383525640 10 monclient: _renew_subs 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7383525640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7383525640 1 -- 172.21.15.134:0/1380483962 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737c1d87e0 con 0x7f737c0b8550 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f73737fe640 4 set_mon_vals no callback set 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7371ffb640 10 monclient: dump: 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.557 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7383525640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7383525640 10 monclient: _renew_subs 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7383525640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7383525640 1 -- 172.21.15.134:0/1380483962 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737c1cb450 con 0x7f737c0b8550 2024-01-20T11:14:29.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.554+0000 7f7383525640 1 librados: init done 2024-01-20T11:14:29.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.555+0000 7f7383525640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:29.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.555+0000 7f7383525640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:29.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.555+0000 7f7383525640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.555+0000 7f7383525640 1 -- 172.21.15.134:0/1380483962 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f737c1d8280 con 0x7f737c0b8550 2024-01-20T11:14:29.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.656+0000 7f7371ffb640 1 -- 172.21.15.134:0/1380483962 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f737801c440 con 0x7f737c0b8550 2024-01-20T11:14:29.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.656+0000 7f7371ffb640 1 -- 172.21.15.134:0/1380483962 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f737801b8c0 con 0x7f737c0b8550 2024-01-20T11:14:29.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.656+0000 7f7371ffb640 10 client.4138.objecter ms_dispatch 0x7f737c000b90 osd_map(6..6 src has 1..6) v4 2024-01-20T11:14:29.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.656+0000 7f7371ffb640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-01-20T11:14:29.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.656+0000 7f7371ffb640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-01-20T11:14:29.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.656+0000 7f7371ffb640 20 client.4138.objecter dump_active .. 0 homeless 2024-01-20T11:14:29.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.656+0000 7f7371ffb640 1 -- 172.21.15.134:0/1380483962 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f737c1d8280 con 0x7f737c0b8550 2024-01-20T11:14:29.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.656+0000 7f7371ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:29.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.656+0000 7f7371ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:29.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.728+0000 7f7383525640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "27eba313-a00b-4455-bd52-efe2b7792812", "id": 4}] 2024-01-20T11:14:29.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.728+0000 7f7383525640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "27eba313-a00b-4455-bd52-efe2b7792812", "id": 4}] 2024-01-20T11:14:29.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.728+0000 7f7383525640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.728+0000 7f7383525640 1 -- 172.21.15.134:0/1380483962 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "27eba313-a00b-4455-bd52-efe2b7792812", "id": 4} v 0) v1 -- 0x7f737c075500 con 0x7f737c0b8550 2024-01-20T11:14:29.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.811+0000 7f7371ffb640 1 -- 172.21.15.134:0/1380483962 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "27eba313-a00b-4455-bd52-efe2b7792812", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f737c1d87e0 con 0x7f737c0b8550 2024-01-20T11:14:29.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.811+0000 7f7371ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "27eba313-a00b-4455-bd52-efe2b7792812", "id": 4}] 2024-01-20T11:14:29.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.811+0000 7f7371ffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:29.813 INFO:teuthology.orchestra.run.smithi134.stdout:4 2024-01-20T11:14:29.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 10 librados: watch_flush enter 2024-01-20T11:14:29.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 10 librados: watch_flush exit 2024-01-20T11:14:29.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 20 client.4138.objecter shutdown clearing up homeless session... 2024-01-20T11:14:29.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 10 client.4138.objecter successfully canceled tick 2024-01-20T11:14:29.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 10 monclient: shutdown 2024-01-20T11:14:29.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:29.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 -- 172.21.15.134:0/1380483962 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f737c0b8550 msgr2=0x7f737c144750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 --2- 172.21.15.134:0/1380483962 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f737c0b8550 0x7f737c144750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f737800c400 tx=0x7f737800c8c0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 -- 172.21.15.134:0/1380483962 shutdown_connections 2024-01-20T11:14:29.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 --2- 172.21.15.134:0/1380483962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f737c1460b0 0x7f737c1d78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 --2- 172.21.15.134:0/1380483962 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f737c0b8550 0x7f737c144750 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 --2- 172.21.15.134:0/1380483962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f737c0b1b10 0x7f737c0b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 -- 172.21.15.134:0/1380483962 >> 172.21.15.134:0/1380483962 conn(0x7f737c0669e0 msgr2=0x7f737c073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:29.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 -- 172.21.15.134:0/1380483962 shutdown_connections 2024-01-20T11:14:29.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 -- 172.21.15.134:0/1380483962 wait complete. 2024-01-20T11:14:29.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.812+0000 7f7383525640 1 librados: shutdown 2024-01-20T11:14:29.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1 5 2024-01-20T11:14:29.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.922+0000 7f880afd7640 10 monclient: get_monmap_and_config 2024-01-20T11:14:29.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.922+0000 7f880afd7640 10 monclient: build_initial_monmap 2024-01-20T11:14:29.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.922+0000 7f880afd7640 10 monclient: monmap: 2024-01-20T11:14:29.923 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:29.923 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.923 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:14:29.923723+0000 2024-01-20T11:14:29.923 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:14:29.923723+0000 2024-01-20T11:14:29.923 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:29.923 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.924 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:29.924 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:29.924 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:29.924 INFO:teuthology.orchestra.run.smithi134.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:29.924 INFO:teuthology.orchestra.run.smithi134.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:29.924 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 1 Processor -- start 2024-01-20T11:14:29.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 1 -- start start 2024-01-20T11:14:29.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient: init 2024-01-20T11:14:29.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:29.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88040b8550 0x7f88040b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient(hunting): picked mon.noname-b con 0x7f88040b8550 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f88040b1b10 0x7f88040b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient(hunting): picked mon.noname-c con 0x7f88040b1b10 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88041459a0 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88040aadc0 con 0x7f88041459a0 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88040b8930 con 0x7f88040b1b10 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8804072990 con 0x7f88040b8550 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.923+0000 7f880afd7640 10 monclient(hunting): authenticate will time out at 1314.309885s 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8803fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f88040b1b10 0x7f88040b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8808d73640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88040b8550 0x7f88040b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8803fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f88040b1b10 0x7f88040b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48706/0 (socket says 172.21.15.134:48706) 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8803fff640 1 -- 172.21.15.134:0/1112060164 learned_addr learned my addr 172.21.15.134:0/1112060164 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8803fff640 10 monclient(hunting): get_auth_request con 0x7f88040b1b10 auth_method 0 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8803fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8803fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8803fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8808d73640 10 monclient(hunting): get_auth_request con 0x7f88040b8550 auth_method 0 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8808d73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8808d73640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.924+0000 7f8808d73640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.925+0000 7f8803fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.925+0000 7f8803fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.925+0000 7f8803fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.925+0000 7f8808d73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.925+0000 7f8808d73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.925+0000 7f8808d73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.927+0000 7f8803fff640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2024-01-20T11:14:29.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.927+0000 7f8803fff640 1 -- 172.21.15.134:0/1112060164 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88040b8550 msgr2=0x7f88040b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:29.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.927+0000 7f8803fff640 1 --2- 172.21.15.134:0/1112060164 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88040b8550 0x7f88040b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.927+0000 7f8803fff640 1 -- 172.21.15.134:0/1112060164 >> v1:172.21.15.134:6789/0 conn(0x7f88041459a0 legacy=0x7f88040b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.927+0000 7f8803fff640 10 monclient: _finish_hunting 0 2024-01-20T11:14:29.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.927+0000 7f8803fff640 1 monclient: found mon.noname-c 2024-01-20T11:14:29.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.927+0000 7f8803fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:29.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.927+0000 7f8803fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.927+0000 7f8803fff640 1 -- 172.21.15.134:0/1112060164 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88040a9be0 con 0x7f88040b1b10 2024-01-20T11:14:29.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.931+0000 7f8803fff640 1 --2- 172.21.15.134:0/1112060164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 0x7f88040b1f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009870 tx=0x7f87f002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc4d62d707f5657d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 1 -- 172.21.15.134:0/1112060164 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0030840 con 0x7f88040b1b10 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 10 monclient: dump: 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.936 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 10 monclient: _finish_auth 0 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 10 monclient: _check_auth_tickets 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 1 -- 172.21.15.134:0/1112060164 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f00309a0 con 0x7f88040b1b10 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f880afd7640 5 monclient: authenticate success, global_id 4141 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f880afd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f880afd7640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f880afd7640 4 set_mon_vals no callback set 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 1 -- 172.21.15.134:0/1112060164 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0030d30 con 0x7f88040b1b10 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.935+0000 7f88037fe640 10 monclient: dump: 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.938 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 10 monclient: shutdown 2024-01-20T11:14:29.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:29.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 1 -- 172.21.15.134:0/1112060164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 msgr2=0x7f88040b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 1 --2- 172.21.15.134:0/1112060164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 0x7f88040b1f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009870 tx=0x7f87f002b980 comp rx=0 tx=0).stop 2024-01-20T11:14:29.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 1 -- 172.21.15.134:0/1112060164 shutdown_connections 2024-01-20T11:14:29.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 1 --2- 172.21.15.134:0/1112060164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 0x7f88040b1f30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 1 --2- 172.21.15.134:0/1112060164 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88040b8550 0x7f88040b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 1 -- 172.21.15.134:0/1112060164 >> 172.21.15.134:0/1112060164 conn(0x7f88040669e0 msgr2=0x7f88040b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:29.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 1 -- 172.21.15.134:0/1112060164 shutdown_connections 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 1 -- 172.21.15.134:0/1112060164 wait complete. 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 10 monclient: build_initial_monmap 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.936+0000 7f880afd7640 10 monclient: monmap: 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.937+0000 7f880afd7640 1 librados: starting msgr at 2024-01-20T11:14:29.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.937+0000 7f880afd7640 1 librados: starting objecter 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.937+0000 7f880afd7640 1 Processor -- start 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.937+0000 7f880afd7640 1 -- start start 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.937+0000 7f880afd7640 1 librados: setting wanted keys 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.937+0000 7f880afd7640 1 librados: calling monclient init 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.937+0000 7f880afd7640 10 monclient: init 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 0x7f880413bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient(hunting): picked mon.noname-b con 0x7f88040b1b10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88040b8550 0x7f88040b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient(hunting): picked mon.noname-a con 0x7f88040b8550 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88041459a0 0x7f88041da840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient(hunting): picked mon.noname-c con 0x7f88041459a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:29.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88040afec0 con 0x7f88040b8550 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88040b0850 con 0x7f88040b1b10 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f880413ca50 con 0x7f88041459a0 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f880afd7640 10 monclient(hunting): authenticate will time out at 1314.324253s 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f8808d73640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 0x7f880413bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f8803fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88040b8550 0x7f88040b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.938+0000 7f8809574640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88041459a0 0x7f88041da840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8808d73640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 0x7f880413bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48708/0 (socket says 172.21.15.134:48708) 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8808d73640 1 -- 172.21.15.134:0/1495963533 learned_addr learned my addr 172.21.15.134:0/1495963533 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8808d73640 10 monclient(hunting): get_auth_request con 0x7f88040b1b10 auth_method 0 2024-01-20T11:14:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8808d73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8808d73640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8808d73640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8803fff640 10 monclient(hunting): get_auth_request con 0x7f88040b8550 auth_method 0 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8803fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8803fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8803fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8809574640 10 monclient(hunting): get_auth_request con 0x7f88041459a0 auth_method 0 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8809574640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8809574640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8809574640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8808d73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8808d73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8808d73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8803fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8803fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8803fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8809574640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:29.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8809574640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:29.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.939+0000 7f8809574640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 1 -- 172.21.15.134:0/1495963533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88041459a0 msgr2=0x7f88041da840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 1 --2- 172.21.15.134:0/1495963533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88041459a0 0x7f88041da840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 1 -- 172.21.15.134:0/1495963533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88040b8550 msgr2=0x7f88040b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 1 --2- 172.21.15.134:0/1495963533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88040b8550 0x7f88040b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 10 monclient: _finish_hunting 0 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 1 monclient: found mon.noname-b 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.942+0000 7f8808d73640 1 -- 172.21.15.134:0/1495963533 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f00094d0 con 0x7f88040b1b10 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.944+0000 7f8808d73640 1 --2- 172.21.15.134:0/1495963533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 0x7f880413bfb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87f800e9b0 tx=0x7f87f800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.944+0000 7f88017fa640 10 client.?.objecter ms_handle_connect 0x7f88040b1b10 2024-01-20T11:14:29.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.944+0000 7f88017fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:29.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 1 -- 172.21.15.134:0/1495963533 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8009670 con 0x7f88040b1b10 2024-01-20T11:14:29.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 10 monclient: dump: 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 10 monclient: _finish_auth 0 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 10 monclient: _check_auth_tickets 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:29.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 1 -- 172.21.15.134:0/1495963533 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f800c9a0 con 0x7f88040b1b10 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f880afd7640 5 monclient: authenticate success, global_id 4144 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 1 -- 172.21.15.134:0/1495963533 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f800cd50 con 0x7f88040b1b10 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f88017fa640 10 monclient: dump: 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:29.951 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f880afd7640 10 monclient: _renew_subs 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f8802ffd640 4 set_mon_vals no callback set 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f880afd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f880afd7640 1 -- 172.21.15.134:0/1495963533 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88041dad80 con 0x7f88040b1b10 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f880afd7640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f880afd7640 10 monclient: _renew_subs 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f880afd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f880afd7640 1 -- 172.21.15.134:0/1495963533 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88040af620 con 0x7f88040b1b10 2024-01-20T11:14:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.948+0000 7f880afd7640 1 librados: init done 2024-01-20T11:14:29.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.949+0000 7f880afd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:29.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.949+0000 7f880afd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:29.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.949+0000 7f880afd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:29.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:29.949+0000 7f880afd7640 1 -- 172.21.15.134:0/1495963533 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8804075500 con 0x7f88040b1b10 2024-01-20T11:14:30.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.051+0000 7f88017fa640 1 -- 172.21.15.134:0/1495963533 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f87f8005990 con 0x7f88040b1b10 2024-01-20T11:14:30.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.051+0000 7f88017fa640 1 -- 172.21.15.134:0/1495963533 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f87f80139f0 con 0x7f88040b1b10 2024-01-20T11:14:30.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.051+0000 7f88017fa640 10 client.4144.objecter ms_dispatch 0x7f8804000b90 osd_map(7..7 src has 1..7) v4 2024-01-20T11:14:30.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.051+0000 7f88017fa640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-01-20T11:14:30.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.051+0000 7f88017fa640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-01-20T11:14:30.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.051+0000 7f88017fa640 20 client.4144.objecter dump_active .. 0 homeless 2024-01-20T11:14:30.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.051+0000 7f88017fa640 1 -- 172.21.15.134:0/1495963533 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f87f8021060 con 0x7f88040b1b10 2024-01-20T11:14:30.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.051+0000 7f88017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:30.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.051+0000 7f88017fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:30.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.139+0000 7f880afd7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1", "id": 5}] 2024-01-20T11:14:30.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.139+0000 7f880afd7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1", "id": 5}] 2024-01-20T11:14:30.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.139+0000 7f880afd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.139+0000 7f880afd7640 1 -- 172.21.15.134:0/1495963533 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1", "id": 5} v 0) v1 -- 0x7f880413cb90 con 0x7f88040b1b10 2024-01-20T11:14:30.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.201+0000 7f88017fa640 1 -- 172.21.15.134:0/1495963533 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f87f8017070 con 0x7f88040b1b10 2024-01-20T11:14:30.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.201+0000 7f88017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1", "id": 5}] 2024-01-20T11:14:30.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.201+0000 7f88017fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:30.203 INFO:teuthology.orchestra.run.smithi134.stdout:5 2024-01-20T11:14:30.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.202+0000 7f880afd7640 10 librados: watch_flush enter 2024-01-20T11:14:30.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.202+0000 7f880afd7640 10 librados: watch_flush exit 2024-01-20T11:14:30.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.202+0000 7f880afd7640 20 client.4144.objecter shutdown clearing up homeless session... 2024-01-20T11:14:30.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.202+0000 7f880afd7640 10 client.4144.objecter successfully canceled tick 2024-01-20T11:14:30.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.202+0000 7f880afd7640 10 monclient: shutdown 2024-01-20T11:14:30.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.202+0000 7f880afd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.202+0000 7f880afd7640 1 -- 172.21.15.134:0/1495963533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 msgr2=0x7f880413bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.202+0000 7f880afd7640 1 --2- 172.21.15.134:0/1495963533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 0x7f880413bfb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87f800e9b0 tx=0x7f87f800ee70 comp rx=0 tx=0).stop 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.203+0000 7f880afd7640 1 -- 172.21.15.134:0/1495963533 shutdown_connections 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.203+0000 7f880afd7640 1 --2- 172.21.15.134:0/1495963533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88041459a0 0x7f88041da840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.203+0000 7f880afd7640 1 --2- 172.21.15.134:0/1495963533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88040b8550 0x7f88040b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.203+0000 7f880afd7640 1 --2- 172.21.15.134:0/1495963533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88040b1b10 0x7f880413bfb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.203+0000 7f880afd7640 1 -- 172.21.15.134:0/1495963533 >> 172.21.15.134:0/1495963533 conn(0x7f88040669e0 msgr2=0x7f88040b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.203+0000 7f880afd7640 1 -- 172.21.15.134:0/1495963533 shutdown_connections 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.203+0000 7f880afd7640 1 -- 172.21.15.134:0/1495963533 wait complete. 2024-01-20T11:14:30.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.203+0000 7f880afd7640 1 librados: shutdown 2024-01-20T11:14:30.207 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:30.207+0000 7fe56084c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-20T11:14:30.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 7fdee016-2887-40d6-ad61-4e4eb6df9fcc 6 2024-01-20T11:14:30.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.314+0000 7f2689856640 10 monclient: get_monmap_and_config 2024-01-20T11:14:30.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.314+0000 7f2689856640 10 monclient: build_initial_monmap 2024-01-20T11:14:30.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.314+0000 7f2689856640 10 monclient: monmap: 2024-01-20T11:14:30.315 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:30.315 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.315 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:14:30.315374+0000 2024-01-20T11:14:30.315 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:14:30.315374+0000 2024-01-20T11:14:30.315 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:30.315 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.316 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:30.316 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:30.316 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:30.316 INFO:teuthology.orchestra.run.smithi134.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:30.316 INFO:teuthology.orchestra.run.smithi134.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:30.316 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.314+0000 7f2689856640 1 Processor -- start 2024-01-20T11:14:30.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.314+0000 7f2689856640 1 -- start start 2024-01-20T11:14:30.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.314+0000 7f2689856640 10 monclient: init 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26840b8550 addr v1:172.21.15.110:6789/0 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f26840b4190 0x7f26841459a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 10 monclient(hunting): picked mon.noname-c con 0x7f26840b4190 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2684145ee0 0x7f268414c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 10 monclient(hunting): picked mon.noname-b con 0x7f2684145ee0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2684146370 con 0x7f26840b8550 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26840aadc0 con 0x7f26840b4190 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26840b8930 con 0x7f2684145ee0 2024-01-20T11:14:30.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2689856640 10 monclient(hunting): authenticate will time out at 1314.701463s 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f26827fc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f26840b4190 0x7f26841459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f26837fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2684145ee0 0x7f268414c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2682ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f26840b8550 0x7f26840b1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.134:51276/0 (socket says 172.21.15.134:51276) 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.315+0000 7f2682ffd640 1 -- 172.21.15.134:0/2945109791 learned_addr learned my addr 172.21.15.134:0/2945109791 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26837fe640 10 monclient(hunting): get_auth_request con 0x7f2684145ee0 auth_method 0 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26837fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26837fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26827fc640 10 monclient(hunting): get_auth_request con 0x7f26840b4190 auth_method 0 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26827fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26827fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:30.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.316+0000 7f26827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.318+0000 7f26827fc640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.318+0000 7f26827fc640 1 -- 172.21.15.134:0/2945109791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2684145ee0 msgr2=0x7f268414c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.318+0000 7f26827fc640 1 --2- 172.21.15.134:0/2945109791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2684145ee0 0x7f268414c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.318+0000 7f26827fc640 1 -- 172.21.15.134:0/2945109791 >> v1:172.21.15.110:6789/0 conn(0x7f26840b8550 legacy=0x7f26840b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.318+0000 7f26827fc640 10 monclient: _finish_hunting 0 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.318+0000 7f26827fc640 1 monclient: found mon.noname-c 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.318+0000 7f26827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.318+0000 7f26827fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.318+0000 7f26827fc640 1 -- 172.21.15.134:0/2945109791 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684072990 con 0x7f26840b4190 2024-01-20T11:14:30.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.320+0000 7f26827fc640 1 --2- 172.21.15.134:0/2945109791 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26840b4190 0x7f26841459a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f26740097a0 tx=0x7f267402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=939d84d4c683c788 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:30.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 1 -- 172.21.15.134:0/2945109791 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674031830 con 0x7f26840b4190 2024-01-20T11:14:30.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:30.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:30.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 10 monclient: dump: 2024-01-20T11:14:30.324 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 10 monclient: _finish_auth 0 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 1 -- 172.21.15.134:0/2945109791 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2674031990 con 0x7f26840b4190 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:30.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 1 -- 172.21.15.134:0/2945109791 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674031d20 con 0x7f26840b4190 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2689856640 5 monclient: authenticate success, global_id 4147 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2689856640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2689856640 4 set_mon_vals no callback set 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.323+0000 7f2681ffb640 10 monclient: dump: 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:30.326 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:30.327 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:30.327 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 10 monclient: shutdown 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 1 -- 172.21.15.134:0/2945109791 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26840b4190 msgr2=0x7f26841459a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 1 --2- 172.21.15.134:0/2945109791 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26840b4190 0x7f26841459a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f26740097a0 tx=0x7f267402b8d0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 1 -- 172.21.15.134:0/2945109791 shutdown_connections 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 1 --2- 172.21.15.134:0/2945109791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2684145ee0 0x7f268414c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 1 --2- 172.21.15.134:0/2945109791 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26840b4190 0x7f26841459a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 1 -- 172.21.15.134:0/2945109791 >> 172.21.15.134:0/2945109791 conn(0x7f26840669e0 msgr2=0x7f26840b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 1 -- 172.21.15.134:0/2945109791 shutdown_connections 2024-01-20T11:14:30.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.324+0000 7f2689856640 1 -- 172.21.15.134:0/2945109791 wait complete. 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.325+0000 7f2689856640 10 monclient: build_initial_monmap 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.325+0000 7f2689856640 10 monclient: monmap: 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.325+0000 7f2689856640 1 librados: starting msgr at 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.325+0000 7f2689856640 1 librados: starting objecter 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.325+0000 7f2689856640 1 Processor -- start 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 1 -- start start 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 1 librados: setting wanted keys 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 1 librados: calling monclient init 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient: init 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26840b4190 0x7f26840adb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient(hunting): picked mon.noname-a con 0x7f26840b4190 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26840b8550 0x7f26840ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient(hunting): picked mon.noname-c con 0x7f26840b8550 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2684145ee0 0x7f26841daa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient(hunting): picked mon.noname-b con 0x7f2684145ee0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26840aff60 con 0x7f26840b4190 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26841da4e0 con 0x7f2684145ee0 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26840aee40 con 0x7f26840b8550 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.326+0000 7f2689856640 10 monclient(hunting): authenticate will time out at 1314.712688s 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2684145ee0 0x7f26841daa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f2682ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26840b4190 0x7f26840adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26827fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26840b8550 0x7f26840ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2684145ee0 0x7f26841daa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48728/0 (socket says 172.21.15.134:48728) 2024-01-20T11:14:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 1 -- 172.21.15.134:0/2660871224 learned_addr learned my addr 172.21.15.134:0/2660871224 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 10 monclient(hunting): get_auth_request con 0x7f2684145ee0 auth_method 0 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26827fc640 10 monclient(hunting): get_auth_request con 0x7f26840b8550 auth_method 0 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26827fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26827fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f2682ffd640 10 monclient(hunting): get_auth_request con 0x7f26840b4190 auth_method 0 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f2682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f2682ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f2682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:30.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.327+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:30.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.328+0000 7f26827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:30.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.328+0000 7f26827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:30.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.328+0000 7f26827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:30.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.328+0000 7f2682ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:30.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.328+0000 7f2682ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:30.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.328+0000 7f2682ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:30.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2024-01-20T11:14:30.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 1 -- 172.21.15.134:0/2660871224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26840b8550 msgr2=0x7f26840ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:30.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 1 --2- 172.21.15.134:0/2660871224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26840b8550 0x7f26840ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 1 -- 172.21.15.134:0/2660871224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26840b4190 msgr2=0x7f26840adb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:30.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 1 --2- 172.21.15.134:0/2660871224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26840b4190 0x7f26840adb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 10 monclient: _finish_hunting 0 2024-01-20T11:14:30.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 1 monclient: found mon.noname-b 2024-01-20T11:14:30.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:30.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.330+0000 7f26837fe640 1 -- 172.21.15.134:0/2660871224 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2674009400 con 0x7f2684145ee0 2024-01-20T11:14:30.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.332+0000 7f26837fe640 1 --2- 172.21.15.134:0/2660871224 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2684145ee0 0x7f26841daa00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f267c00efc0 tx=0x7f267c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:30.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.332+0000 7f266bfff640 10 client.?.objecter ms_handle_connect 0x7f2684145ee0 2024-01-20T11:14:30.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.332+0000 7f266bfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f266bfff640 1 -- 172.21.15.134:0/2660871224 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c009ba0 con 0x7f2684145ee0 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f266bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f266bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f266bfff640 10 monclient: dump: 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:30.339 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f266bfff640 10 monclient: _finish_auth 0 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f266bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f266bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f2689856640 5 monclient: authenticate success, global_id 4150 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f2689856640 10 monclient: _renew_subs 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f2689856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f2689856640 1 -- 172.21.15.134:0/2660871224 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26840b0380 con 0x7f2684145ee0 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f2689856640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f2689856640 10 monclient: _renew_subs 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f2689856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f2689856640 1 -- 172.21.15.134:0/2660871224 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26841db120 con 0x7f2684145ee0 2024-01-20T11:14:30.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.334+0000 7f2689856640 1 librados: init done 2024-01-20T11:14:30.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.335+0000 7f2689856640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:30.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.335+0000 7f2689856640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:30.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.335+0000 7f2689856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.335+0000 7f2689856640 1 -- 172.21.15.134:0/2660871224 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26840b0680 con 0x7f2684145ee0 2024-01-20T11:14:30.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 1 -- 172.21.15.134:0/2660871224 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c00f040 con 0x7f2684145ee0 2024-01-20T11:14:30.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:30.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 1 -- 172.21.15.134:0/2660871224 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c0146a0 con 0x7f2684145ee0 2024-01-20T11:14:30.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 10 monclient: dump: 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 1 -- 172.21.15.134:0/2660871224 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f267c014c20 con 0x7f2684145ee0 2024-01-20T11:14:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f26817fa640 4 set_mon_vals no callback set 2024-01-20T11:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 1 -- 172.21.15.134:0/2660871224 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f267c0139d0 con 0x7f2684145ee0 2024-01-20T11:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 10 client.4150.objecter ms_dispatch 0x7f2684000b90 osd_map(8..8 src has 1..8) v4 2024-01-20T11:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-01-20T11:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-01-20T11:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 20 client.4150.objecter dump_active .. 0 homeless 2024-01-20T11:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 1 -- 172.21.15.134:0/2660871224 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f26840b0680 con 0x7f2684145ee0 2024-01-20T11:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.433+0000 7f266bfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:30.488 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:30.487+0000 7fe56084c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-20T11:14:30.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.505+0000 7f2689856640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7fdee016-2887-40d6-ad61-4e4eb6df9fcc", "id": 6}] 2024-01-20T11:14:30.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.505+0000 7f2689856640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7fdee016-2887-40d6-ad61-4e4eb6df9fcc", "id": 6}] 2024-01-20T11:14:30.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.505+0000 7f2689856640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.505+0000 7f2689856640 1 -- 172.21.15.134:0/2660871224 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7fdee016-2887-40d6-ad61-4e4eb6df9fcc", "id": 6} v 0) v1 -- 0x7f26840afd90 con 0x7f2684145ee0 2024-01-20T11:14:30.541 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:30.540+0000 7fe56084c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-20T11:14:30.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.561+0000 7f266bfff640 1 -- 172.21.15.134:0/2660871224 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7fdee016-2887-40d6-ad61-4e4eb6df9fcc", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f267c018020 con 0x7f2684145ee0 2024-01-20T11:14:30.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.561+0000 7f266bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7fdee016-2887-40d6-ad61-4e4eb6df9fcc", "id": 6}] 2024-01-20T11:14:30.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.561+0000 7f266bfff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:30.562 INFO:teuthology.orchestra.run.smithi134.stdout:6 2024-01-20T11:14:30.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 10 librados: watch_flush enter 2024-01-20T11:14:30.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 10 librados: watch_flush exit 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 20 client.4150.objecter shutdown clearing up homeless session... 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 10 client.4150.objecter successfully canceled tick 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 10 monclient: shutdown 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 -- 172.21.15.134:0/2660871224 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2684145ee0 msgr2=0x7f26841daa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 --2- 172.21.15.134:0/2660871224 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2684145ee0 0x7f26841daa00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f267c00efc0 tx=0x7f267c00c490 comp rx=0 tx=0).stop 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 -- 172.21.15.134:0/2660871224 shutdown_connections 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 --2- 172.21.15.134:0/2660871224 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2684145ee0 0x7f26841daa00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 --2- 172.21.15.134:0/2660871224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26840b8550 0x7f26840ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 --2- 172.21.15.134:0/2660871224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26840b4190 0x7f26840adb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 -- 172.21.15.134:0/2660871224 >> 172.21.15.134:0/2660871224 conn(0x7f26840669e0 msgr2=0x7f268413dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 -- 172.21.15.134:0/2660871224 shutdown_connections 2024-01-20T11:14:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 -- 172.21.15.134:0/2660871224 wait complete. 2024-01-20T11:14:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.562+0000 7f2689856640 1 librados: shutdown 2024-01-20T11:14:30.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new a63e46f9-9dd8-41d1-afc6-c5c317345707 7 2024-01-20T11:14:30.616 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:30.616+0000 7f32fbc41040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-20T11:14:30.647 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:30.647+0000 7fe56084c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-20T11:14:30.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.671+0000 7f36184a9640 10 monclient: get_monmap_and_config 2024-01-20T11:14:30.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.671+0000 7f36184a9640 10 monclient: build_initial_monmap 2024-01-20T11:14:30.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.671+0000 7f36184a9640 10 monclient: monmap: 2024-01-20T11:14:30.672 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:30.672 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.672 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:14:30.673054+0000 2024-01-20T11:14:30.673 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:14:30.673054+0000 2024-01-20T11:14:30.673 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:30.673 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.673 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:30.673 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:30.673 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:30.673 INFO:teuthology.orchestra.run.smithi134.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:30.673 INFO:teuthology.orchestra.run.smithi134.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:30.673 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.672+0000 7f36184a9640 1 Processor -- start 2024-01-20T11:14:30.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.672+0000 7f36184a9640 1 -- start start 2024-01-20T11:14:30.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.672+0000 7f36184a9640 10 monclient: init 2024-01-20T11:14:30.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:30.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f361006ba90 0x7f36100b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient(hunting): picked mon.noname-c con 0x7f361006ba90 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36100b0900 0x7f3610145820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient(hunting): picked mon.noname-a con 0x7f36100b0900 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3610145d60 0x7f36101461f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient(hunting): picked mon.noname-b con 0x7f3610145d60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361006be70 con 0x7f36100b0900 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361006ccf0 con 0x7f361006ba90 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36100ad9c0 con 0x7f3610145d60 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:30.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f36184a9640 10 monclient(hunting): authenticate will time out at 1315.059240s 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f3616245640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f361006ba90 0x7f36100b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.673+0000 7f3616a46640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3610145d60 0x7f36101461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616245640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f361006ba90 0x7f36100b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48734/0 (socket says 172.21.15.134:48734) 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616245640 1 -- 172.21.15.134:0/1202668759 learned_addr learned my addr 172.21.15.134:0/1202668759 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616245640 10 monclient(hunting): get_auth_request con 0x7f361006ba90 auth_method 0 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616245640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616245640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616a46640 10 monclient(hunting): get_auth_request con 0x7f3610145d60 auth_method 0 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616a46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616a46640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616a46640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:30.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616a46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:30.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616a46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:30.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.674+0000 7f3616a46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 10 monclient(hunting): handle_auth_done global_id 4142 payload 274 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 1 -- 172.21.15.134:0/1202668759 >> v2:172.21.15.134:3300/0 conn(0x7f361006ba90 msgr2=0x7f36100b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 1 --2- 172.21.15.134:0/1202668759 >> v2:172.21.15.134:3300/0 conn(0x7f361006ba90 0x7f36100b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 1 -- 172.21.15.134:0/1202668759 >> v2:172.21.15.110:3300/0 conn(0x7f36100b0900 msgr2=0x7f3610145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 1 --2- 172.21.15.134:0/1202668759 >> v2:172.21.15.110:3300/0 conn(0x7f36100b0900 0x7f3610145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 10 monclient: _finish_hunting 0 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 1 monclient: found mon.noname-b 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616a46640 1 -- 172.21.15.134:0/1202668759 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36100adca0 con 0x7f3610145d60 2024-01-20T11:14:30.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.676+0000 7f3616245640 1 --2- 172.21.15.134:0/1202668759 >> v2:172.21.15.134:3300/0 conn(0x7f361006ba90 0x7f36100b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:14:30.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.678+0000 7f3616a46640 1 --2- 172.21.15.134:0/1202668759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3610145d60 0x7f36101461f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f36100b6590 tx=0x7f360802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddbb0e55d06b9aca server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 1 -- 172.21.15.134:0/1202668759 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360802cbd0 con 0x7f3610145d60 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 10 monclient: dump: 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:30.683 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 10 monclient: _finish_auth 0 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 10 monclient: _check_auth_tickets 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 1 -- 172.21.15.134:0/1202668759 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360802cd30 con 0x7f3610145d60 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 1 -- 172.21.15.134:0/1202668759 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360802cea0 con 0x7f3610145d60 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f3615243640 10 monclient: dump: 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.684 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.685 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:30.685 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.685 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:30.685 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:30.685 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:30.685 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f36184a9640 5 monclient: authenticate success, global_id 4142 2024-01-20T11:14:30.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f36184a9640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:30.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f36184a9640 4 set_mon_vals no callback set 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f36184a9640 10 monclient: shutdown 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f36184a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f36184a9640 1 -- 172.21.15.134:0/1202668759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3610145d60 msgr2=0x7f36101461f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.682+0000 7f36184a9640 1 --2- 172.21.15.134:0/1202668759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3610145d60 0x7f36101461f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f36100b6590 tx=0x7f360802be20 comp rx=0 tx=0).stop 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 1 -- 172.21.15.134:0/1202668759 shutdown_connections 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 1 --2- 172.21.15.134:0/1202668759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3610145d60 0x7f36101461f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 1 --2- 172.21.15.134:0/1202668759 >> v2:172.21.15.110:3300/0 conn(0x7f36100b0900 0x7f3610145820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 1 --2- 172.21.15.134:0/1202668759 >> v2:172.21.15.134:3300/0 conn(0x7f361006ba90 0x7f36100b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 1 -- 172.21.15.134:0/1202668759 >> 172.21.15.134:0/1202668759 conn(0x7f3610066cd0 msgr2=0x7f36100b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 1 -- 172.21.15.134:0/1202668759 shutdown_connections 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 1 -- 172.21.15.134:0/1202668759 wait complete. 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 10 monclient: build_initial_monmap 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 10 monclient: monmap: 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 1 librados: starting msgr at 2024-01-20T11:14:30.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.683+0000 7f36184a9640 1 librados: starting objecter 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 Processor -- start 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 -- start start 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 librados: setting wanted keys 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 librados: calling monclient init 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient: init 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361006ba90 0x7f3610124750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient(hunting): picked mon.noname-c con 0x7f361006ba90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36100b0900 0x7f36101429c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient(hunting): picked mon.noname-a con 0x7f36100b0900 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3610145d60 0x7f3610142f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient(hunting): picked mon.noname-b con 0x7f3610145d60 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36101252a0 con 0x7f36100b0900 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36101242f0 con 0x7f3610145d60 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3610143440 con 0x7f361006ba90 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.684+0000 7f36184a9640 10 monclient(hunting): authenticate will time out at 1315.070679s 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3610145d60 0x7f3610142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616245640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361006ba90 0x7f3610124750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3610145d60 0x7f3610142f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48744/0 (socket says 172.21.15.134:48744) 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 1 -- 172.21.15.134:0/3874094722 learned_addr learned my addr 172.21.15.134:0/3874094722 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 10 monclient(hunting): get_auth_request con 0x7f3610145d60 auth_method 0 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:30.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616245640 10 monclient(hunting): get_auth_request con 0x7f361006ba90 auth_method 0 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616245640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616245640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616a46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:30.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.685+0000 7f3616245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:30.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 10 monclient(hunting): handle_auth_done global_id 4156 payload 824 2024-01-20T11:14:30.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 1 -- 172.21.15.134:0/3874094722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361006ba90 msgr2=0x7f3610124750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:30.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 1 --2- 172.21.15.134:0/3874094722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361006ba90 0x7f3610124750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 1 -- 172.21.15.134:0/3874094722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36100b0900 msgr2=0x7f36101429c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:30.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 1 --2- 172.21.15.134:0/3874094722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36100b0900 0x7f36101429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:30.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 10 monclient: _finish_hunting 0 2024-01-20T11:14:30.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 1 monclient: found mon.noname-b 2024-01-20T11:14:30.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:30.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.688+0000 7f3616a46640 1 -- 172.21.15.134:0/3874094722 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c0094d0 con 0x7f3610145d60 2024-01-20T11:14:30.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.690+0000 7f3616a46640 1 --2- 172.21.15.134:0/3874094722 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3610145d60 0x7f3610142f00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3608037040 tx=0x7f3608004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:30.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.690+0000 7f3606ffd640 10 client.?.objecter ms_handle_connect 0x7f3610145d60 2024-01-20T11:14:30.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.690+0000 7f3606ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 1 -- 172.21.15.134:0/3874094722 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360802cea0 con 0x7f3610145d60 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 10 monclient: dump: 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:30.695 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 10 monclient: _finish_auth 0 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 1 -- 172.21.15.134:0/3874094722 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608040670 con 0x7f3610145d60 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 1 -- 172.21.15.134:0/3874094722 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608040a20 con 0x7f3610145d60 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3606ffd640 10 monclient: dump: 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:30.696 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f36184a9640 5 monclient: authenticate success, global_id 4156 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f36184a9640 10 monclient: _renew_subs 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f36184a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f3614a42640 4 set_mon_vals no callback set 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.693+0000 7f36184a9640 1 -- 172.21.15.134:0/3874094722 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3608009a70 con 0x7f3610145d60 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.694+0000 7f36184a9640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.694+0000 7f36184a9640 10 monclient: _renew_subs 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.694+0000 7f36184a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.694+0000 7f36184a9640 1 -- 172.21.15.134:0/3874094722 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36101436b0 con 0x7f3610145d60 2024-01-20T11:14:30.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.694+0000 7f36184a9640 1 librados: init done 2024-01-20T11:14:30.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.695+0000 7f36184a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:30.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.695+0000 7f36184a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:30.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.695+0000 7f36184a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.695+0000 7f36184a9640 1 -- 172.21.15.134:0/3874094722 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36101da540 con 0x7f3610145d60 2024-01-20T11:14:30.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.791+0000 7f3606ffd640 1 -- 172.21.15.134:0/3874094722 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f3608003990 con 0x7f3610145d60 2024-01-20T11:14:30.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.791+0000 7f3606ffd640 1 -- 172.21.15.134:0/3874094722 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f36080046a0 con 0x7f3610145d60 2024-01-20T11:14:30.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.791+0000 7f3606ffd640 10 client.4156.objecter ms_dispatch 0x7f3610000b90 osd_map(9..9 src has 1..9) v4 2024-01-20T11:14:30.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.791+0000 7f3606ffd640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-01-20T11:14:30.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.791+0000 7f3606ffd640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-01-20T11:14:30.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.791+0000 7f3606ffd640 20 client.4156.objecter dump_active .. 0 homeless 2024-01-20T11:14:30.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.791+0000 7f3606ffd640 1 -- 172.21.15.134:0/3874094722 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f36101da540 con 0x7f3610145d60 2024-01-20T11:14:30.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.791+0000 7f3606ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:30.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.791+0000 7f3606ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:30.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.890+0000 7f36184a9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a63e46f9-9dd8-41d1-afc6-c5c317345707", "id": 7}] 2024-01-20T11:14:30.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.890+0000 7f36184a9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a63e46f9-9dd8-41d1-afc6-c5c317345707", "id": 7}] 2024-01-20T11:14:30.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.890+0000 7f36184a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:30.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.890+0000 7f36184a9640 1 -- 172.21.15.134:0/3874094722 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a63e46f9-9dd8-41d1-afc6-c5c317345707", "id": 7} v 0) v1 -- 0x7f3610075500 con 0x7f3610145d60 2024-01-20T11:14:30.897 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:30.895+0000 7f32fbc41040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-20T11:14:30.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:30.937+0000 7f3606ffd640 1 -- 172.21.15.134:0/3874094722 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 2) v1 ==== 44620+0+0 (secure 0 0 0) 0x7f3608004030 con 0x7f3610145d60 2024-01-20T11:14:30.952 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:30.952+0000 7f32fbc41040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-20T11:14:31.052 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:31.052+0000 7fe55b04e640 -1 mgr handle_mgr_map I was active but no longer am 2024-01-20T11:14:31.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-20T11:14:31.068+0000 7f32fbc41040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-20T11:14:31.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.435+0000 7f3606ffd640 1 -- 172.21.15.134:0/3874094722 <== mon.1 v2:172.21.15.134:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a63e46f9-9dd8-41d1-afc6-c5c317345707", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3608044020 con 0x7f3610145d60 2024-01-20T11:14:31.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.435+0000 7f3606ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a63e46f9-9dd8-41d1-afc6-c5c317345707", "id": 7}] 2024-01-20T11:14:31.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.435+0000 7f3606ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:31.437 INFO:teuthology.orchestra.run.smithi134.stdout:7 2024-01-20T11:14:31.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 10 librados: watch_flush enter 2024-01-20T11:14:31.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 10 librados: watch_flush exit 2024-01-20T11:14:31.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 20 client.4156.objecter shutdown clearing up homeless session... 2024-01-20T11:14:31.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 10 client.4156.objecter successfully canceled tick 2024-01-20T11:14:31.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 10 monclient: shutdown 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 1 -- 172.21.15.134:0/3874094722 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3610145d60 msgr2=0x7f3610142f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 1 --2- 172.21.15.134:0/3874094722 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3610145d60 0x7f3610142f00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3608037040 tx=0x7f3608004a40 comp rx=0 tx=0).stop 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 1 -- 172.21.15.134:0/3874094722 shutdown_connections 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 1 --2- 172.21.15.134:0/3874094722 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3610145d60 0x7f3610142f00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 1 --2- 172.21.15.134:0/3874094722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36100b0900 0x7f36101429c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 1 --2- 172.21.15.134:0/3874094722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361006ba90 0x7f3610124750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 1 -- 172.21.15.134:0/3874094722 >> 172.21.15.134:0/3874094722 conn(0x7f3610066cd0 msgr2=0x7f3610144e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.436+0000 7f36184a9640 1 -- 172.21.15.134:0/3874094722 shutdown_connections 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.437+0000 7f36184a9640 1 -- 172.21.15.134:0/3874094722 wait complete. 2024-01-20T11:14:31.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-20T11:14:31.437+0000 7f36184a9640 1 librados: shutdown 2024-01-20T11:14:31.448 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-20T11:14:31.448 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-20T11:14:31.448 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-01-20T11:14:31.451 INFO:tasks.ceph.osd.0:Started 2024-01-20T11:14:31.451 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-20T11:14:31.451 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-20T11:14:31.451 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-01-20T11:14:31.453 INFO:tasks.ceph.osd.1:Started 2024-01-20T11:14:31.453 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-20T11:14:31.453 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-20T11:14:31.453 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-01-20T11:14:31.455 INFO:tasks.ceph.osd.2:Started 2024-01-20T11:14:31.455 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-20T11:14:31.455 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-20T11:14:31.456 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-01-20T11:14:31.458 INFO:tasks.ceph.osd.3:Started 2024-01-20T11:14:31.458 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-20T11:14:31.458 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-20T11:14:31.458 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-01-20T11:14:31.459 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:31.457+0000 7f18c7212040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-20T11:14:31.460 INFO:tasks.ceph.osd.4:Started 2024-01-20T11:14:31.460 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-20T11:14:31.460 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-20T11:14:31.461 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-01-20T11:14:31.462 INFO:tasks.ceph.osd.5:Started 2024-01-20T11:14:31.462 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-20T11:14:31.463 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-20T11:14:31.463 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-01-20T11:14:31.464 INFO:tasks.ceph.osd.6:Started 2024-01-20T11:14:31.465 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-20T11:14:31.465 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-20T11:14:31.465 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-01-20T11:14:31.467 INFO:tasks.ceph.osd.7:Started 2024-01-20T11:14:31.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-20T11:14:31.511 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:31.511+0000 7f18c7212040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.574+0000 7f888d95e640 10 monclient: get_monmap_and_config 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.574+0000 7f888d95e640 10 monclient: build_initial_monmap 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.574+0000 7f888d95e640 10 monclient: monmap: 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:14:31.575116+0000 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:14:31.575116+0000 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:31.576 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 1 Processor -- start 2024-01-20T11:14:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 1 -- start start 2024-01-20T11:14:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 10 monclient: init 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88880a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f88880a9ad0 0x7f88881423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 10 monclient(hunting): picked mon.noname-a con 0x7f88880a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8888142920 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88880b25c0 con 0x7f88880a96f0 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888813fbe0 con 0x7f8888142920 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88880aaea0 con 0x7f88880a9ad0 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.575+0000 7f888d95e640 10 monclient(hunting): authenticate will time out at 1314.926180s 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.576+0000 7f88867fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f88880a9ad0 0x7f88881423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.577+0000 7f88877fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8888142920 0x7f888814c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:43294/0 (socket says 172.21.15.110:43294) 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.577+0000 7f88877fe640 1 -- 172.21.15.110:0/4252088342 learned_addr learned my addr 172.21.15.110:0/4252088342 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.577+0000 7f88867fc640 10 monclient(hunting): get_auth_request con 0x7f88880a9ad0 auth_method 0 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.577+0000 7f88867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.577+0000 7f88867fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.577+0000 7f88867fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.578+0000 7f88867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.578+0000 7f88867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:31.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.578+0000 7f88867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.580+0000 7f88867fc640 10 monclient(hunting): handle_auth_done global_id 4149 payload 274 2024-01-20T11:14:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.580+0000 7f88867fc640 1 -- 172.21.15.110:0/4252088342 >> v1:172.21.15.134:6789/0 conn(0x7f8888142920 legacy=0x7f888814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.580+0000 7f88867fc640 1 -- 172.21.15.110:0/4252088342 >> v1:172.21.15.110:6789/0 conn(0x7f88880a96f0 legacy=0x7f888813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.580+0000 7f88867fc640 10 monclient: _finish_hunting 0 2024-01-20T11:14:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.580+0000 7f88867fc640 1 monclient: found mon.noname-a 2024-01-20T11:14:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.580+0000 7f88867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.580+0000 7f88867fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:14:31.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.580+0000 7f88867fc640 1 -- 172.21.15.110:0/4252088342 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88880ab0a0 con 0x7f88880a9ad0 2024-01-20T11:14:31.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.585+0000 7f88867fc640 1 --2- 172.21.15.110:0/4252088342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88880a9ad0 0x7f88881423e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f887c0097a0 tx=0x7f887c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c49955b94f2525 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:31.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.587+0000 7f8885ffb640 1 -- 172.21.15.110:0/4252088342 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c031830 con 0x7f88880a9ad0 2024-01-20T11:14:31.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.587+0000 7f8885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:31.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.587+0000 7f8885ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.587+0000 7f8885ffb640 10 monclient: dump: 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.587+0000 7f8885ffb640 10 monclient: _finish_auth 0 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.587+0000 7f8885ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:14:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.587+0000 7f8885ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:31.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.587+0000 7f888d95e640 5 monclient: authenticate success, global_id 4149 2024-01-20T11:14:31.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.587+0000 7f888d95e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:14:31.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.589+0000 7f8885ffb640 1 -- 172.21.15.110:0/4252088342 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c031990 con 0x7f88880a9ad0 2024-01-20T11:14:31.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.589+0000 7f8885ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.589+0000 7f8885ffb640 1 -- 172.21.15.110:0/4252088342 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c031d20 con 0x7f88880a9ad0 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.589+0000 7f888d95e640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.589+0000 7f888d95e640 4 set_mon_vals no callback set 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.589+0000 7f8885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.589+0000 7f8885ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.589+0000 7f8885ffb640 10 monclient: dump: 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:31.590 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:31.591 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:31.591 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:31.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.590+0000 7f888d95e640 10 monclient: shutdown 2024-01-20T11:14:31.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.590+0000 7f888d95e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:31.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.590+0000 7f888d95e640 1 -- 172.21.15.110:0/4252088342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88880a9ad0 msgr2=0x7f88881423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:31.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.590+0000 7f888d95e640 1 --2- 172.21.15.110:0/4252088342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88880a9ad0 0x7f88881423e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f887c0097a0 tx=0x7f887c02b8d0 comp rx=0 tx=0).stop 2024-01-20T11:14:31.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.590+0000 7f888d95e640 1 -- 172.21.15.110:0/4252088342 shutdown_connections 2024-01-20T11:14:31.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.590+0000 7f888d95e640 1 --2- 172.21.15.110:0/4252088342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88880a9ad0 0x7f88881423e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:31.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.590+0000 7f888d95e640 1 -- 172.21.15.110:0/4252088342 >> 172.21.15.110:0/4252088342 conn(0x7f88880b7bc0 msgr2=0x7f88880b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.591+0000 7f888d95e640 1 -- 172.21.15.110:0/4252088342 shutdown_connections 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.591+0000 7f888d95e640 1 -- 172.21.15.110:0/4252088342 wait complete. 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.592+0000 7f888d95e640 10 monclient: build_initial_monmap 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.592+0000 7f888d95e640 10 monclient: monmap: 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:31.593 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.592+0000 7f888d95e640 1 librados: starting msgr at 2024-01-20T11:14:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.592+0000 7f888d95e640 1 librados: starting objecter 2024-01-20T11:14:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.592+0000 7f888d95e640 1 Processor -- start 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.593+0000 7f888d95e640 1 -- start start 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.593+0000 7f888d95e640 1 librados: setting wanted keys 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.593+0000 7f888d95e640 1 librados: calling monclient init 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.593+0000 7f888d95e640 10 monclient: init 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.593+0000 7f888d95e640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.593+0000 7f888d95e640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.593+0000 7f888d95e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88880a96f0 0x7f88881df960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.593+0000 7f888d95e640 10 monclient(hunting): picked mon.noname-b con 0x7f88880a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.593+0000 7f888d95e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88880a9ad0 0x7f8888141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 10 monclient(hunting): picked mon.noname-c con 0x7f88880a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8888142920 0x7f88881dedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 10 monclient(hunting): picked mon.noname-a con 0x7f8888142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88880715d0 con 0x7f8888142920 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88881df4d0 con 0x7f88880a96f0 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8888071820 con 0x7f88880a9ad0 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f888d95e640 10 monclient(hunting): authenticate will time out at 1314.944651s 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.594+0000 7f8886ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88880a96f0 0x7f88881df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88877fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8888142920 0x7f88881dedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f8886ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88880a96f0 0x7f88881df960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:44964/0 (socket says 172.21.15.110:44964) 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f8886ffd640 1 -- 172.21.15.110:0/2892652082 learned_addr learned my addr 172.21.15.110:0/2892652082 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f8886ffd640 10 monclient(hunting): get_auth_request con 0x7f88880a96f0 auth_method 0 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f8886ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f8886ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f8886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88867fc640 1 --2- 172.21.15.110:0/2892652082 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88880a9ad0 0x7f8888141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88877fe640 10 monclient(hunting): get_auth_request con 0x7f8888142920 auth_method 0 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88877fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88877fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88867fc640 10 monclient(hunting): get_auth_request con 0x7f88880a9ad0 auth_method 0 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88867fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.595+0000 7f88867fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.596+0000 7f8886ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.596+0000 7f8886ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.596+0000 7f8886ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.596+0000 7f88877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.596+0000 7f88877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.596+0000 7f88877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.596+0000 7f88867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.596+0000 7f88867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.596+0000 7f88867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 10 monclient(hunting): handle_auth_done global_id 4152 payload 824 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 1 -- 172.21.15.110:0/2892652082 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88880a9ad0 msgr2=0x7f8888141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 1 --2- 172.21.15.110:0/2892652082 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88880a9ad0 0x7f8888141780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 1 -- 172.21.15.110:0/2892652082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88880a96f0 msgr2=0x7f88881df960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 1 --2- 172.21.15.110:0/2892652082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88880a96f0 0x7f88881df960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 10 monclient: _finish_hunting 0 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 1 monclient: found mon.noname-a 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.599+0000 7f88877fe640 1 -- 172.21.15.110:0/2892652082 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c009400 con 0x7f8888142920 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.601+0000 7f88877fe640 1 --2- 172.21.15.110:0/2892652082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8888142920 0x7f88881dedb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f888000cad0 tx=0x7f888000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.601+0000 7f886ffff640 10 client.?.objecter ms_handle_connect 0x7f8888142920 2024-01-20T11:14:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.601+0000 7f886ffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f886ffff640 1 -- 172.21.15.110:0/2892652082 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888000abd0 con 0x7f8888142920 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f886ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f886ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f886ffff640 10 monclient: dump: 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:31.606 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f886ffff640 10 monclient: _finish_auth 0 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f886ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f886ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f886ffff640 1 -- 172.21.15.110:0/2892652082 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888000ad30 con 0x7f8888142920 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f888d95e640 5 monclient: authenticate success, global_id 4152 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f888d95e640 10 monclient: _renew_subs 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f888d95e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f888d95e640 1 -- 172.21.15.110:0/2892652082 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88880ab6a0 con 0x7f8888142920 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.605+0000 7f886ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f886ffff640 1 -- 172.21.15.110:0/2892652082 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888000aea0 con 0x7f8888142920 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f886ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f886ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:14:31.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f886ffff640 10 monclient: dump: 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f88857fa640 4 set_mon_vals no callback set 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f888d95e640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f888d95e640 10 monclient: _renew_subs 2024-01-20T11:14:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f888d95e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:14:31.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f888d95e640 1 -- 172.21.15.110:0/2892652082 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888806f8f0 con 0x7f8888142920 2024-01-20T11:14:31.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.606+0000 7f888d95e640 1 librados: init done 2024-01-20T11:14:31.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.607+0000 7f888d95e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:31.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.607+0000 7f888d95e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:31.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.607+0000 7f888d95e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:14:31.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.607+0000 7f888d95e640 1 -- 172.21.15.110:0/2892652082 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8888075490 con 0x7f8888142920 2024-01-20T11:14:31.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.621+0000 7f886ffff640 1 -- 172.21.15.110:0/2892652082 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 2) v1 ==== 44620+0+0 (secure 0 0 0) 0x7f8880007070 con 0x7f8888142920 2024-01-20T11:14:31.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.622+0000 7f886ffff640 1 -- 172.21.15.110:0/2892652082 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8880011df0 con 0x7f8888142920 2024-01-20T11:14:31.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.622+0000 7f886ffff640 10 client.4152.objecter ms_dispatch 0x7f8888000b90 osd_map(10..10 src has 1..10) v4 2024-01-20T11:14:31.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.622+0000 7f886ffff640 3 client.4152.objecter handle_osd_map got epochs [10,10] > 0 2024-01-20T11:14:31.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.622+0000 7f886ffff640 3 client.4152.objecter handle_osd_map decoding full epoch 10 2024-01-20T11:14:31.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.622+0000 7f886ffff640 20 client.4152.objecter dump_active .. 0 homeless 2024-01-20T11:14:31.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.817+0000 7f886ffff640 1 -- 172.21.15.110:0/2892652082 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f88800196d0 con 0x7f8888142920 2024-01-20T11:14:31.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.817+0000 7f886ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:31.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.817+0000 7f886ffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:31.932 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:31.931+0000 7f18c7212040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-20T11:14:31.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.942+0000 7f888d95e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:31.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.942+0000 7f888d95e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:31.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.942+0000 7f888d95e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:14:31.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:31.942+0000 7f888d95e640 1 -- 172.21.15.110:0/2892652082 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88881df610 con 0x7f8888142920 2024-01-20T11:14:31.980 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:31.979+0000 7f18c7212040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-20T11:14:32.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.015+0000 7f886ffff640 1 -- 172.21.15.110:0/2892652082 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f8880017030 con 0x7f8888142920 2024-01-20T11:14:32.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.015+0000 7f886ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:32.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.015+0000 7f886ffff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:32.016 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:14:32.016 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":10,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:14:30.958130+0000","last_up_change":"0.000000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:14:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.016+0000 7f888d95e640 10 librados: watch_flush enter 2024-01-20T11:14:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.016+0000 7f888d95e640 10 librados: watch_flush exit 2024-01-20T11:14:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.016+0000 7f888d95e640 20 client.4152.objecter shutdown clearing up homeless session... 2024-01-20T11:14:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.016+0000 7f888d95e640 10 client.4152.objecter successfully canceled tick 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.016+0000 7f888d95e640 10 monclient: shutdown 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.016+0000 7f888d95e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.016+0000 7f888d95e640 1 -- 172.21.15.110:0/2892652082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8888142920 msgr2=0x7f88881dedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.016+0000 7f888d95e640 1 --2- 172.21.15.110:0/2892652082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8888142920 0x7f88881dedb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f888000cad0 tx=0x7f888000cf90 comp rx=0 tx=0).stop 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.017+0000 7f888d95e640 1 -- 172.21.15.110:0/2892652082 shutdown_connections 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.017+0000 7f888d95e640 1 --2- 172.21.15.110:0/2892652082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8888142920 0x7f88881dedb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.017+0000 7f888d95e640 1 --2- 172.21.15.110:0/2892652082 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88880a9ad0 0x7f8888141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.017+0000 7f888d95e640 1 --2- 172.21.15.110:0/2892652082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88880a96f0 0x7f88881df960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.017+0000 7f888d95e640 1 -- 172.21.15.110:0/2892652082 >> 172.21.15.110:0/2892652082 conn(0x7f88880b7bc0 msgr2=0x7f88880b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.017+0000 7f888d95e640 1 -- 172.21.15.110:0/2892652082 shutdown_connections 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.017+0000 7f888d95e640 1 -- 172.21.15.110:0/2892652082 wait complete. 2024-01-20T11:14:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:14:32.017+0000 7f888d95e640 1 librados: shutdown 2024-01-20T11:14:32.026 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.025+0000 7f18c7212040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-20T11:14:32.039 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-20T11:14:32.039 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-20T11:14:32.081 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.080+0000 7f18c7212040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-20T11:14:32.127 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.126+0000 7f18c7212040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-20T11:14:32.233 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.232+0000 7f18c7212040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-20T11:14:32.294 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.293+0000 7f18c7212040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-20T11:14:32.339 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.338+0000 7f18c7212040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-20T11:14:32.396 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.396+0000 7f18c7212040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-20T11:14:32.447 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.446+0000 7f18c7212040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-20T11:14:32.845 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-20T11:14:32.848 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.848+0000 7f18c7212040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-20T11:14:32.989 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:32.988+0000 7f18c7212040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:32.999+0000 7f98c35f0640 10 monclient: get_monmap_and_config 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:32.999+0000 7f98c35f0640 10 monclient: build_initial_monmap 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:32.999+0000 7f98c35f0640 10 monclient: monmap: 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:14:33.000507+0000 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:14:33.000507+0000 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:33.000 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:33.001 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:33.001 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:33.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.000+0000 7f98c35f0640 1 Processor -- start 2024-01-20T11:14:33.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 1 -- start start 2024-01-20T11:14:33.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 10 monclient: init 2024-01-20T11:14:33.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:33.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-01-20T11:14:33.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f98bc0a8a30 0x7f98bc13efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:33.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 10 monclient(hunting): picked mon.noname-c con 0x7f98bc0a8a30 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f98bc0a9c90 0x7f98bc13f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 10 monclient(hunting): picked mon.noname-a con 0x7f98bc0a9c90 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98bc142100 addr v1:172.21.15.110:6789/0 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.001+0000 7f98c35f0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc0a8e10 con 0x7f98bc142100 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.002+0000 7f98c35f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.002+0000 7f98c35f0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98bc0aacf0 con 0x7f98bc0a9c90 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.002+0000 7f98c35f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.002+0000 7f98c35f0640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98bc0aafd0 con 0x7f98bc0a8a30 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.002+0000 7f98c35f0640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.002+0000 7f98c35f0640 10 monclient(hunting): authenticate will time out at 1316.352206s 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.003+0000 7f98c138c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f98bc0a8a30 0x7f98bc13efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.003+0000 7f98c1b8d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f98bc142100 0x7f98bc142570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:38646/0 (socket says 172.21.15.110:38646) 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.003+0000 7f98c1b8d640 1 -- 172.21.15.110:0/227507107 learned_addr learned my addr 172.21.15.110:0/227507107 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:14:33.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.003+0000 7f98c138c640 10 monclient(hunting): get_auth_request con 0x7f98bc0a8a30 auth_method 0 2024-01-20T11:14:33.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.003+0000 7f98c138c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:33.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.003+0000 7f98c138c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:33.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.003+0000 7f98c138c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:33.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.004+0000 7f98c138c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:33.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.004+0000 7f98c138c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:33.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.004+0000 7f98c138c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:33.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.004+0000 7f98c0b8b640 1 --2- 172.21.15.110:0/227507107 >> v2:172.21.15.110:3300/0 conn(0x7f98bc0a9c90 0x7f98bc13f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:33.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.006+0000 7f98c138c640 10 monclient(hunting): handle_auth_done global_id 4165 payload 274 2024-01-20T11:14:33.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.006+0000 7f98c138c640 1 -- 172.21.15.110:0/227507107 >> v2:172.21.15.110:3300/0 conn(0x7f98bc0a9c90 msgr2=0x7f98bc13f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:33.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.006+0000 7f98c138c640 1 --2- 172.21.15.110:0/227507107 >> v2:172.21.15.110:3300/0 conn(0x7f98bc0a9c90 0x7f98bc13f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.006+0000 7f98c138c640 1 -- 172.21.15.110:0/227507107 >> v1:172.21.15.110:6789/0 conn(0x7f98bc142100 legacy=0x7f98bc142570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:33.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.006+0000 7f98c138c640 10 monclient: _finish_hunting 0 2024-01-20T11:14:33.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.006+0000 7f98c138c640 1 monclient: found mon.noname-c 2024-01-20T11:14:33.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.006+0000 7f98c138c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:33.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.007+0000 7f98c138c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:14:33.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.007+0000 7f98c138c640 1 -- 172.21.15.110:0/227507107 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc0a90c0 con 0x7f98bc0a8a30 2024-01-20T11:14:33.009 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.009+0000 7f98c138c640 1 --2- 172.21.15.110:0/227507107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98bc0a8a30 0x7f98bc13efa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f98bc072430 tx=0x7f98b802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4510cbd18f52bc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:33.022 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 1 -- 172.21.15.110:0/227507107 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8031830 con 0x7f98bc0a8a30 2024-01-20T11:14:33.022 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:33.022 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:33.022 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 10 monclient: dump: 2024-01-20T11:14:33.022 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:33.022 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 10 monclient: _finish_auth 0 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 10 monclient: _check_auth_tickets 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 1 -- 172.21.15.110:0/227507107 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b8031990 con 0x7f98bc0a8a30 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98c35f0640 5 monclient: authenticate success, global_id 4165 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98c35f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:14:33.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98c35f0640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98c35f0640 4 set_mon_vals no callback set 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 1 -- 172.21.15.110:0/227507107 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8031d20 con 0x7f98bc0a8a30 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.015+0000 7f98b3fff640 10 monclient: dump: 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:33.024 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 10 monclient: shutdown 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 1 -- 172.21.15.110:0/227507107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98bc0a8a30 msgr2=0x7f98bc13efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 1 --2- 172.21.15.110:0/227507107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98bc0a8a30 0x7f98bc13efa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f98bc072430 tx=0x7f98b802b8d0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 1 -- 172.21.15.110:0/227507107 shutdown_connections 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 1 --2- 172.21.15.110:0/227507107 >> v2:172.21.15.110:3300/0 conn(0x7f98bc0a9c90 0x7f98bc13f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 1 --2- 172.21.15.110:0/227507107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98bc0a8a30 0x7f98bc13efa0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 1 -- 172.21.15.110:0/227507107 >> 172.21.15.110:0/227507107 conn(0x7f98bc0b4320 msgr2=0x7f98bc0b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 1 -- 172.21.15.110:0/227507107 shutdown_connections 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.016+0000 7f98c35f0640 1 -- 172.21.15.110:0/227507107 wait complete. 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.017+0000 7f98c35f0640 10 monclient: build_initial_monmap 2024-01-20T11:14:33.025 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.017+0000 7f98c35f0640 10 monclient: monmap: 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.018+0000 7f98c35f0640 1 librados: starting msgr at 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.018+0000 7f98c35f0640 1 librados: starting objecter 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.018+0000 7f98c35f0640 1 Processor -- start 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.018+0000 7f98c35f0640 1 -- start start 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.018+0000 7f98c35f0640 1 librados: setting wanted keys 2024-01-20T11:14:33.026 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.018+0000 7f98c35f0640 1 librados: calling monclient init 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.018+0000 7f98c35f0640 10 monclient: init 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98bc0a8a30 0x7f98bc0b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient(hunting): picked mon.noname-c con 0x7f98bc0a8a30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98bc142100 0x7f98bc139300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient(hunting): picked mon.noname-a con 0x7f98bc142100 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98bc139ed0 0x7f98bc13a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient(hunting): picked mon.noname-b con 0x7f98bc139ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc143a80 con 0x7f98bc142100 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc143c80 con 0x7f98bc139ed0 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:33.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc143f60 con 0x7f98bc0a8a30 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c35f0640 10 monclient(hunting): authenticate will time out at 1316.369613s 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.019+0000 7f98c138c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98bc0a8a30 0x7f98bc0b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c1b8d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98bc139ed0 0x7f98bc13a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c138c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98bc0a8a30 0x7f98bc0b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56976/0 (socket says 172.21.15.110:56976) 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c138c640 1 -- 172.21.15.110:0/2808737653 learned_addr learned my addr 172.21.15.110:0/2808737653 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c138c640 10 monclient(hunting): get_auth_request con 0x7f98bc0a8a30 auth_method 0 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c138c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c138c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c138c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c0b8b640 1 --2- 172.21.15.110:0/2808737653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98bc142100 0x7f98bc139300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c0b8b640 10 monclient(hunting): get_auth_request con 0x7f98bc142100 auth_method 0 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c0b8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:33.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c0b8b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.020+0000 7f98c0b8b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c1b8d640 10 monclient(hunting): get_auth_request con 0x7f98bc139ed0 auth_method 0 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c1b8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c1b8d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c1b8d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c138c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c138c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c138c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c0b8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c0b8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c0b8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c1b8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c1b8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:33.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.021+0000 7f98c1b8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:33.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 10 monclient(hunting): handle_auth_done global_id 4151 payload 824 2024-01-20T11:14:33.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 1 -- 172.21.15.110:0/2808737653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98bc139ed0 msgr2=0x7f98bc13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:33.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 1 --2- 172.21.15.110:0/2808737653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98bc139ed0 0x7f98bc13a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 1 -- 172.21.15.110:0/2808737653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98bc142100 msgr2=0x7f98bc139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:33.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 1 --2- 172.21.15.110:0/2808737653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98bc142100 0x7f98bc139300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 10 monclient: _finish_hunting 0 2024-01-20T11:14:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 1 monclient: found mon.noname-c 2024-01-20T11:14:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:14:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.025+0000 7f98c138c640 1 -- 172.21.15.110:0/2808737653 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8009400 con 0x7f98bc0a8a30 2024-01-20T11:14:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.029+0000 7f98c138c640 1 --2- 172.21.15.110:0/2808737653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98bc0a8a30 0x7f98bc0b77b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009890 tx=0x7f98b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.029+0000 7f98b1ffb640 10 client.?.objecter ms_handle_connect 0x7f98bc0a8a30 2024-01-20T11:14:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.029+0000 7f98b1ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98b1ffb640 1 -- 172.21.15.110:0/2808737653 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b803ba70 con 0x7f98bc0a8a30 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98b1ffb640 10 monclient: dump: 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:33.033 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98b1ffb640 10 monclient: _finish_auth 0 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98b1ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98c35f0640 5 monclient: authenticate success, global_id 4151 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98c35f0640 10 monclient: _renew_subs 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98c35f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98c35f0640 1 -- 172.21.15.110:0/2808737653 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98bc139990 con 0x7f98bc0a8a30 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.032+0000 7f98c35f0640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.033+0000 7f98c35f0640 10 monclient: _renew_subs 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.033+0000 7f98c35f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:33.034 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.033+0000 7f98c35f0640 1 -- 172.21.15.110:0/2808737653 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc13a870 con 0x7f98bc0a8a30 2024-01-20T11:14:33.035 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.033+0000 7f98c35f0640 1 librados: init done 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98c35f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98c35f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98c35f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98c35f0640 1 -- 172.21.15.110:0/2808737653 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98bc139c30 con 0x7f98bc0a8a30 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98b1ffb640 1 -- 172.21.15.110:0/2808737653 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b803bbd0 con 0x7f98bc0a8a30 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98b1ffb640 1 -- 172.21.15.110:0/2808737653 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b804e440 con 0x7f98bc0a8a30 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98b37fe640 4 set_mon_vals no callback set 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.034+0000 7f98b1ffb640 10 monclient: dump: 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:33.036 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:33.037 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:33.037 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:33.037 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:33.037 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:33.037 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:33.037 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:33.037 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:33.037 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:33.078 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.078+0000 7f98b1ffb640 1 -- 172.21.15.110:0/2808737653 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 3) v1 ==== 88909+0+0 (secure 0 0 0) 0x7f98b804e5a0 con 0x7f98bc0a8a30 2024-01-20T11:14:33.091 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:33.090+0000 7f18c7212040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-20T11:14:33.149 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:33.149+0000 7f18c7212040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-20T11:14:33.256 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:33.255+0000 7f18c7212040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-20T11:14:33.259 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.259+0000 7f98b1ffb640 1 -- 172.21.15.110:0/2808737653 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f98b807a900 con 0x7f98bc0a8a30 2024-01-20T11:14:33.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.259+0000 7f98b1ffb640 10 client.4151.objecter ms_dispatch 0x7f98bc000b90 osd_map(10..10 src has 1..10) v4 2024-01-20T11:14:33.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.259+0000 7f98b1ffb640 3 client.4151.objecter handle_osd_map got epochs [10,10] > 0 2024-01-20T11:14:33.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.259+0000 7f98b1ffb640 3 client.4151.objecter handle_osd_map decoding full epoch 10 2024-01-20T11:14:33.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.259+0000 7f98b1ffb640 20 client.4151.objecter dump_active .. 0 homeless 2024-01-20T11:14:33.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.259+0000 7f98b1ffb640 1 -- 172.21.15.110:0/2808737653 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f98b8076c40 con 0x7f98bc0a8a30 2024-01-20T11:14:33.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.259+0000 7f98b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:33.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.259+0000 7f98b1ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:33.359 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:33.358+0000 7f18c7212040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-20T11:14:33.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.383+0000 7f98c35f0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:33.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.383+0000 7f98c35f0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:33.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.383+0000 7f98c35f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:33.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.383+0000 7f98c35f0640 1 -- 172.21.15.110:0/2808737653 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98bc0b3e50 con 0x7f98bc0a8a30 2024-01-20T11:14:33.454 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.454+0000 7f98b1ffb640 1 -- 172.21.15.110:0/2808737653 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f98b8035090 con 0x7f98bc0a8a30 2024-01-20T11:14:33.454 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.454+0000 7f98b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:33.454 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.454+0000 7f98b1ffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:33.454 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-20T11:14:33.455 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:14:30.958130+0000","last_up_change":"0.000000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 10 librados: watch_flush enter 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 10 librados: watch_flush exit 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 20 client.4151.objecter shutdown clearing up homeless session... 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 10 client.4151.objecter successfully canceled tick 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 10 monclient: shutdown 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 1 -- 172.21.15.110:0/2808737653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98bc0a8a30 msgr2=0x7f98bc0b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 1 --2- 172.21.15.110:0/2808737653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98bc0a8a30 0x7f98bc0b77b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009890 tx=0x7f98b80046c0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 1 -- 172.21.15.110:0/2808737653 shutdown_connections 2024-01-20T11:14:33.456 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 1 --2- 172.21.15.110:0/2808737653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98bc139ed0 0x7f98bc13a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.457 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 1 --2- 172.21.15.110:0/2808737653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98bc142100 0x7f98bc139300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.457 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 1 --2- 172.21.15.110:0/2808737653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98bc0a8a30 0x7f98bc0b77b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:33.457 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 1 -- 172.21.15.110:0/2808737653 >> 172.21.15.110:0/2808737653 conn(0x7f98bc0b4320 msgr2=0x7f98bc137140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:33.457 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 1 -- 172.21.15.110:0/2808737653 shutdown_connections 2024-01-20T11:14:33.457 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.455+0000 7f98c35f0640 1 -- 172.21.15.110:0/2808737653 wait complete. 2024-01-20T11:14:33.457 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:33.456+0000 7f98c35f0640 1 librados: shutdown 2024-01-20T11:14:33.469 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-20T11:14:33.547 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:33.546+0000 7f18c7212040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-20T11:14:33.600 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:33.599+0000 7f18c7212040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-20T11:14:33.722 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:33.721+0000 7f18c7212040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-20T11:14:34.358 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:34.358+0000 7f18c7212040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-20T11:14:34.670 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:34.669+0000 7f18c7212040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-20T11:14:34.726 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:34.725+0000 7f18c7212040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-20T11:14:34.841 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:34.840+0000 7f18c7212040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-20T11:14:35.361 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:35.360+0000 7f18c15fe640 -1 mgr handle_mgr_map I was active but no longer am 2024-01-20T11:14:36.070 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:36.069+0000 7eff83e0d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-20T11:14:36.152 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:36.151+0000 7eff83e0d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-20T11:14:36.848 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:36.849+0000 7eff83e0d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-20T11:14:36.925 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:36.926+0000 7eff83e0d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-20T11:14:37.000 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:37.000+0000 7eff83e0d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-20T11:14:37.090 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:37.091+0000 7eff83e0d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-20T11:14:37.163 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:37.163+0000 7eff83e0d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-20T11:14:37.347 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:37.347+0000 7eff83e0d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-20T11:14:37.441 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:37.441+0000 7eff83e0d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-20T11:14:37.514 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:37.514+0000 7eff83e0d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-20T11:14:37.598 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:37.598+0000 7eff83e0d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-20T11:14:37.669 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:37.669+0000 7eff83e0d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-20T11:14:38.090 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:38.090+0000 7eff83e0d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-20T11:14:38.318 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:38.319+0000 7eff83e0d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-20T11:14:38.479 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:38.480+0000 7eff83e0d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-20T11:14:38.552 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:38.552+0000 7eff83e0d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-20T11:14:38.708 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:38.709+0000 7eff83e0d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-20T11:14:38.829 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:38.829+0000 7eff83e0d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-20T11:14:38.966 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:07.418 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:38.968 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:07.451 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:38.999 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:07.458 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.029 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:07.512 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.031 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:07.484 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.056 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:39.056+0000 7eff83e0d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-20T11:14:39.063 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:07.523 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.074 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:07.559 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.079 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:07.540 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.123 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:39.124+0000 7eff83e0d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-20T11:14:39.164 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:07.648 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.164 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:07.625 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.236 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:39.236+0000 7eff83e0d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-20T11:14:39.579 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-20T11:14:39.573+0000 5dc2480 -1 Falling back to public interface 2024-01-20T11:14:39.609 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-01-20T11:14:39.601+0000 5dc2480 -1 Falling back to public interface 2024-01-20T11:14:39.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-20T11:14:39.608+0000 5dc2480 -1 Falling back to public interface 2024-01-20T11:14:39.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-20T11:14:39.705+0000 5dc2480 -1 Falling back to public interface 2024-01-20T11:14:39.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:14:39.720+0000 5dc2480 -1 Falling back to public interface 2024-01-20T11:14:39.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:14:39.735+0000 8c7a640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6820/0: (98) Address already in use 2024-01-20T11:14:39.783 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:08.248 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.845 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:08.311 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:39.976 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:39.976+0000 7eff83e0d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-20T11:14:40.062 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:08.521 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.070 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:08.554 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.153 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:08.605 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.157 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:08.642 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.202 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:08.663 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.210 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:08.684 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.223 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:08.678 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.259 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:40.259+0000 7eff83e0d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-20T11:14:40.272 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:08.746 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.276 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-20T11:14:40.285 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:08.741 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.311 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:40.311+0000 7eff83e0d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-20T11:14:40.364 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.365+0000 7efdce17c640 10 monclient: get_monmap_and_config 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.365+0000 7efdce17c640 10 monclient: build_initial_monmap 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.365+0000 7efdce17c640 10 monclient: monmap: 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:14:40.365640+0000 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:14:40.365640+0000 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:40.365 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:40.366 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:40.366 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:40.367 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.365+0000 7efdce17c640 1 Processor -- start 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.365+0000 7efdce17c640 1 -- start start 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.365+0000 7efdce17c640 10 monclient: init 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efdc80a8a30 0x7efdc813efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 10 monclient(hunting): picked mon.noname-c con 0x7efdc80a8a30 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efdc80a9c90 0x7efdc813f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 10 monclient(hunting): picked mon.noname-a con 0x7efdc80a9c90 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdc8142100 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc80a8e10 con 0x7efdc8142100 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:40.368 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc80aacf0 con 0x7efdc80a9c90 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc80aafd0 con 0x7efdc80a8a30 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdce17c640 10 monclient(hunting): authenticate will time out at 1323.716607s 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.366+0000 7efdc6ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efdc80a9c90 0x7efdc813f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc77fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efdc80a8a30 0x7efdc813efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc6ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efdc80a9c90 0x7efdc813f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:59064/0 (socket says 172.21.15.110:59064) 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc6ffd640 1 -- 172.21.15.110:0/1629780177 learned_addr learned my addr 172.21.15.110:0/1629780177 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc6ffd640 10 monclient(hunting): get_auth_request con 0x7efdc80a9c90 auth_method 0 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:40.369 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc77fe640 10 monclient(hunting): get_auth_request con 0x7efdc80a8a30 auth_method 0 2024-01-20T11:14:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.367+0000 7efdc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.368+0000 7efdc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.368+0000 7efdc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.368+0000 7efdc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.369+0000 7efdc6ffd640 10 monclient(hunting): handle_auth_done global_id 4194 payload 274 2024-01-20T11:14:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.369+0000 7efdc6ffd640 1 -- 172.21.15.110:0/1629780177 >> v2:172.21.15.134:3300/0 conn(0x7efdc80a8a30 msgr2=0x7efdc813efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.369+0000 7efdc6ffd640 1 --2- 172.21.15.110:0/1629780177 >> v2:172.21.15.134:3300/0 conn(0x7efdc80a8a30 0x7efdc813efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.369+0000 7efdc6ffd640 1 -- 172.21.15.110:0/1629780177 >> v1:172.21.15.134:6789/0 conn(0x7efdc8142100 legacy=0x7efdc8142570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.369+0000 7efdc6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:14:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.369+0000 7efdc6ffd640 1 monclient: found mon.noname-a 2024-01-20T11:14:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.369+0000 7efdc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.369+0000 7efdc6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:14:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.369+0000 7efdc6ffd640 1 -- 172.21.15.110:0/1629780177 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc80a90c0 con 0x7efdc80a9c90 2024-01-20T11:14:40.373 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.373+0000 7efdc6ffd640 1 --2- 172.21.15.110:0/1629780177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdc80a9c90 0x7efdc813f4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efdc8072430 tx=0x7efdb002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b006506e1e51d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:40.377 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 1 -- 172.21.15.110:0/1629780177 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0030860 con 0x7efdc80a9c90 2024-01-20T11:14:40.377 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 10 monclient: dump: 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 10 monclient: _finish_auth 0 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:14:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 1 -- 172.21.15.110:0/1629780177 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb00309c0 con 0x7efdc80a9c90 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 1 -- 172.21.15.110:0/1629780177 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0030d50 con 0x7efdc80a9c90 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdce17c640 5 monclient: authenticate success, global_id 4194 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdce17c640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdce17c640 4 set_mon_vals no callback set 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.377+0000 7efdc67fc640 10 monclient: dump: 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:40.379 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:40.380 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:40.380 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:40.380 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:40.380 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:40.380 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 10 monclient: shutdown 2024-01-20T11:14:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 1 -- 172.21.15.110:0/1629780177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdc80a9c90 msgr2=0x7efdc813f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 1 --2- 172.21.15.110:0/1629780177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdc80a9c90 0x7efdc813f4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efdc8072430 tx=0x7efdb002b9a0 comp rx=0 tx=0).stop 2024-01-20T11:14:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 1 -- 172.21.15.110:0/1629780177 shutdown_connections 2024-01-20T11:14:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 1 --2- 172.21.15.110:0/1629780177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdc80a9c90 0x7efdc813f4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 1 --2- 172.21.15.110:0/1629780177 >> v2:172.21.15.134:3300/0 conn(0x7efdc80a8a30 0x7efdc813efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 1 -- 172.21.15.110:0/1629780177 >> 172.21.15.110:0/1629780177 conn(0x7efdc80b4320 msgr2=0x7efdc80b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 1 -- 172.21.15.110:0/1629780177 shutdown_connections 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.378+0000 7efdce17c640 1 -- 172.21.15.110:0/1629780177 wait complete. 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.379+0000 7efdce17c640 10 monclient: build_initial_monmap 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.379+0000 7efdce17c640 10 monclient: monmap: 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:40.384 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 librados: starting msgr at 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 librados: starting objecter 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 Processor -- start 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 -- start start 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 librados: setting wanted keys 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 librados: calling monclient init 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient: init 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdc80a8a30 0x7efdc813a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient(hunting): picked mon.noname-c con 0x7efdc80a8a30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdc8142100 0x7efdc8144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient(hunting): picked mon.noname-a con 0x7efdc8142100 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdc8143240 0x7efdc81457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient(hunting): picked mon.noname-b con 0x7efdc8143240 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:40.385 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc80b6150 con 0x7efdc8142100 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc80b6430 con 0x7efdc8143240 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc8145200 con 0x7efdc80a8a30 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.380+0000 7efdce17c640 10 monclient(hunting): authenticate will time out at 1323.731104s 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdc80a8a30 0x7efdc813a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdc8143240 0x7efdc81457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc6ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdc8142100 0x7efdc8144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdc8143240 0x7efdc81457a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:57582/0 (socket says 172.21.15.110:57582) 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc7fff640 1 -- 172.21.15.110:0/510999402 learned_addr learned my addr 172.21.15.110:0/510999402 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc7fff640 10 monclient(hunting): get_auth_request con 0x7efdc8143240 auth_method 0 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:40.386 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc77fe640 10 monclient(hunting): get_auth_request con 0x7efdc80a8a30 auth_method 0 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.381+0000 7efdc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc6ffd640 10 monclient(hunting): get_auth_request con 0x7efdc8142100 auth_method 0 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:40.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:40.388 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:40.388 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:40.388 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.382+0000 7efdc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 10 monclient(hunting): handle_auth_done global_id 4216 payload 824 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 1 -- 172.21.15.110:0/510999402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdc80a8a30 msgr2=0x7efdc813a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 1 --2- 172.21.15.110:0/510999402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdc80a8a30 0x7efdc813a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 1 -- 172.21.15.110:0/510999402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdc8142100 msgr2=0x7efdc8144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 1 --2- 172.21.15.110:0/510999402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdc8142100 0x7efdc8144b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 1 monclient: found mon.noname-b 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.384+0000 7efdc7fff640 1 -- 172.21.15.110:0/510999402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb00094d0 con 0x7efdc8143240 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.387+0000 7efdc7fff640 1 --2- 172.21.15.110:0/510999402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdc8143240 0x7efdc81457a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c6b0 tx=0x7efdc000cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:40.389 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.387+0000 7efdbffff640 10 client.?.objecter ms_handle_connect 0x7efdc8143240 2024-01-20T11:14:40.390 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.387+0000 7efdbffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 1 -- 172.21.15.110:0/510999402 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0003b70 con 0x7efdc8143240 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 10 monclient: dump: 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:40.392 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 10 monclient: _finish_auth 0 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 10 monclient: _check_auth_tickets 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 1 -- 172.21.15.110:0/510999402 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc0003cf0 con 0x7efdc8143240 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 1 -- 172.21.15.110:0/510999402 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0003ea0 con 0x7efdc8143240 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdbffff640 10 monclient: dump: 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:40.393 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdc5ffb640 4 set_mon_vals no callback set 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdce17c640 5 monclient: authenticate success, global_id 4216 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdce17c640 10 monclient: _renew_subs 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdce17c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdce17c640 1 -- 172.21.15.110:0/510999402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc8145ce0 con 0x7efdc8143240 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdce17c640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdce17c640 10 monclient: _renew_subs 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdce17c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:40.394 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdce17c640 1 -- 172.21.15.110:0/510999402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc81daa00 con 0x7efdc8143240 2024-01-20T11:14:40.395 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.391+0000 7efdce17c640 1 librados: init done 2024-01-20T11:14:40.395 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.392+0000 7efdce17c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:40.395 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.392+0000 7efdce17c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:40.395 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.392+0000 7efdce17c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:40.395 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.392+0000 7efdce17c640 1 -- 172.21.15.110:0/510999402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc8145070 con 0x7efdc8143240 2024-01-20T11:14:40.398 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:14:40.392+0000 5dc2480 -1 Falling back to public interface 2024-01-20T11:14:40.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.411+0000 7efdbffff640 1 -- 172.21.15.110:0/510999402 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 4) v1 ==== 88909+0+0 (secure 0 0 0) 0x7efdc0005310 con 0x7efdc8143240 2024-01-20T11:14:40.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.413+0000 7efdbffff640 1 -- 172.21.15.110:0/510999402 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (secure 0 0 0) 0x7efdc00588b0 con 0x7efdc8143240 2024-01-20T11:14:40.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.413+0000 7efdbffff640 10 client.4216.objecter ms_dispatch 0x7efdc8000b90 osd_map(11..11 src has 1..11) v4 2024-01-20T11:14:40.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.413+0000 7efdbffff640 3 client.4216.objecter handle_osd_map got epochs [11,11] > 0 2024-01-20T11:14:40.414 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.413+0000 7efdbffff640 3 client.4216.objecter handle_osd_map decoding full epoch 11 2024-01-20T11:14:40.414 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.414+0000 7efdbffff640 20 client.4216.objecter dump_active .. 0 homeless 2024-01-20T11:14:40.427 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:40.428+0000 7eff83e0d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-20T11:14:40.453 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:08.913 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.468 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:08.952 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.517 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.517+0000 7efdbffff640 1 -- 172.21.15.110:0/510999402 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7efdc0054bf0 con 0x7efdc8143240 2024-01-20T11:14:40.517 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.517+0000 7efdbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:40.517 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.517+0000 7efdbffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:40.527 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:08.980 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.528 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.013 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.581 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.042 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.587 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.587+0000 7efdce17c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:40.587 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.587+0000 7efdce17c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:40.587 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.587+0000 7efdce17c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:14:40.588 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.587+0000 7efdce17c640 1 -- 172.21.15.110:0/510999402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdc813df50 con 0x7efdc8143240 2024-01-20T11:14:40.675 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.675+0000 7efdbffff640 1 -- 172.21.15.110:0/510999402 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 5) v1 ==== 88909+0+0 (secure 0 0 0) 0x7efdc00546a0 con 0x7efdc8143240 2024-01-20T11:14:40.730 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.730+0000 7efdbffff640 1 -- 172.21.15.110:0/510999402 <== mon.1 v2:172.21.15.134:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7efdc0040df0 con 0x7efdc8143240 2024-01-20T11:14:40.730 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.730+0000 7efdbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:40.731 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.730+0000 7efdbffff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:40.731 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-20T11:14:40.731 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":12,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:14:40.464348+0000","last_up_change":"0.000000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:14:40.732 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 10 librados: watch_flush enter 2024-01-20T11:14:40.732 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 10 librados: watch_flush exit 2024-01-20T11:14:40.732 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 20 client.4216.objecter shutdown clearing up homeless session... 2024-01-20T11:14:40.732 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 10 client.4216.objecter successfully canceled tick 2024-01-20T11:14:40.732 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 10 monclient: shutdown 2024-01-20T11:14:40.732 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 -- 172.21.15.110:0/510999402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdc8143240 msgr2=0x7efdc81457a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 --2- 172.21.15.110:0/510999402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdc8143240 0x7efdc81457a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c6b0 tx=0x7efdc000cb70 comp rx=0 tx=0).stop 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 -- 172.21.15.110:0/510999402 shutdown_connections 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 --2- 172.21.15.110:0/510999402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdc8143240 0x7efdc81457a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 --2- 172.21.15.110:0/510999402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdc8142100 0x7efdc8144b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 --2- 172.21.15.110:0/510999402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdc80a8a30 0x7efdc813a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 -- 172.21.15.110:0/510999402 >> 172.21.15.110:0/510999402 conn(0x7efdc80b4320 msgr2=0x7efdc813d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 -- 172.21.15.110:0/510999402 shutdown_connections 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 -- 172.21.15.110:0/510999402 wait complete. 2024-01-20T11:14:40.733 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:40.731+0000 7efdce17c640 1 librados: shutdown 2024-01-20T11:14:40.744 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-20T11:14:40.784 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.244 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.793 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:09.276 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.832 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:09.298 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.843 INFO:tasks.ceph.osd.5.smithi134.stderr:2024-01-20T11:14:40.835+0000 5dc2480 -1 Falling back to public interface 2024-01-20T11:14:40.849 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.334 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.855 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:09.308 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:40.858 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:14:40.853+0000 5dc2480 -1 Falling back to public interface 2024-01-20T11:14:40.946 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.406 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.055 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.515 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.060 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:09.543 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.117 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.602 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.127 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:09.579 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.202 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:09.667 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.241 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.702 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.325 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:09.799 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.330 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.790 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.342 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:09.826 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.391 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:09.846 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.393 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.878 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.394 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:09.846 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.529 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.989 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.555 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:10.020 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.602 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.062 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.618 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.102 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.658 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.143 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.663 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:10.116 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.695 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:10.169 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.764 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:10.219 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.824 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.284 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.849 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:10.314 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.869 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.329 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.897 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.381 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.926 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.411 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:41.932 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:10.384 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.007 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:10.481 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.119 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.579 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.129 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:10.585 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.140 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.600 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.145 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:10.610 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.166 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.650 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.193 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.678 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.206 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:10.658 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.279 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:10.753 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.290 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:42.290+0000 7eff24ead640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-20T11:14:42.316 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:14:42.317+0000 7eff24ead640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-20T11:14:42.411 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.871 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.417 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.878 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.418 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:10.874 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.441 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.925 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.448 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:10.914 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.473 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.958 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.476 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:10.928 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.547 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:11.020 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.707 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:11.162 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.719 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:11.185 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.720 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:11.181 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.822 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:11.296 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.883 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:11.342 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.937 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:11.397 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.962 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:11.446 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.964 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:11.416 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.968 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:11.453 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:42.997 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:11.452 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.009 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:11.475 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.017 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:11.470 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.064 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:11.548 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.065 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:11.550 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.084 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:11.558 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.288 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:11.744 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.305 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:11.770 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.311 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:11.772 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.358 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:11.832 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.413 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:11.874 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.588 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:12.044 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.625 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:12.098 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.838 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:12.304 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.881 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:12.337 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:43.902 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:12.367 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:44.097 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:12.570 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:44.151 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:12.625 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:44.464 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:12.920 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:44.518 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:12.974 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:47.551 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.679+0000 7f689dd3c640 10 monclient: get_monmap_and_config 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.679+0000 7f689dd3c640 10 monclient: build_initial_monmap 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.679+0000 7f689dd3c640 10 monclient: monmap: 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:14:47.680712+0000 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:14:47.680712+0000 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:47.680 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:47.681 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:47.681 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:47.681 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:47.681 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:47.681 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:47.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.679+0000 7f689dd3c640 1 Processor -- start 2024-01-20T11:14:47.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.679+0000 7f689dd3c640 1 -- start start 2024-01-20T11:14:47.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.679+0000 7f689dd3c640 10 monclient: init 2024-01-20T11:14:47.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:47.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-20T11:14:47.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f68980a89c0 0x7f689813f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:47.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 10 monclient(hunting): picked mon.noname-a con 0x7f68980a89c0 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68980a8da0 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f689813f730 0x7f6898142a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 10 monclient(hunting): picked mon.noname-b con 0x7f689813f730 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68980b21b0 con 0x7f68980a8da0 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68980aa220 con 0x7f68980a89c0 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68980a9f20 con 0x7f689813f730 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.680+0000 7f689dd3c640 10 monclient(hunting): authenticate will time out at 1331.031947s 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f6897fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f689813f730 0x7f6898142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f6896ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f68980a8da0 0x7f68981422f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:36064/0 (socket says 172.21.15.110:36064) 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f6896ffd640 1 -- 172.21.15.110:0/1027989603 learned_addr learned my addr 172.21.15.110:0/1027989603 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f68977fe640 1 --2- 172.21.15.110:0/1027989603 >> v2:172.21.15.110:3300/0 conn(0x7f68980a89c0 0x7f689813f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:47.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f6897fff640 10 monclient(hunting): get_auth_request con 0x7f689813f730 auth_method 0 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f6897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f6897fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f6897fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f68977fe640 10 monclient(hunting): get_auth_request con 0x7f68980a89c0 auth_method 0 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.681+0000 7f68977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.682+0000 7f68977fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.682+0000 7f68977fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.682+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.682+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.682+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.682+0000 7f68977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.682+0000 7f68977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:47.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.682+0000 7f68977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.684+0000 7f6897fff640 10 monclient(hunting): handle_auth_done global_id 4169 payload 274 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.684+0000 7f6897fff640 1 -- 172.21.15.110:0/1027989603 >> v2:172.21.15.110:3300/0 conn(0x7f68980a89c0 msgr2=0x7f689813f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.684+0000 7f6897fff640 1 --2- 172.21.15.110:0/1027989603 >> v2:172.21.15.110:3300/0 conn(0x7f68980a89c0 0x7f689813f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.684+0000 7f6897fff640 1 -- 172.21.15.110:0/1027989603 >> v1:172.21.15.134:6789/0 conn(0x7f68980a8da0 legacy=0x7f68981422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.684+0000 7f6897fff640 10 monclient: _finish_hunting 0 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.684+0000 7f6897fff640 1 monclient: found mon.noname-b 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.684+0000 7f6897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.684+0000 7f6897fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.684+0000 7f6897fff640 1 -- 172.21.15.110:0/1027989603 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68980a9760 con 0x7f689813f730 2024-01-20T11:14:47.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.686+0000 7f6897fff640 1 --2- 172.21.15.110:0/1027989603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f689813f730 0x7f6898142a00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6890009d90 tx=0x7f689002bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a6c372bc3853c8a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.690+0000 7f68967fc640 1 -- 172.21.15.110:0/1027989603 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689002cb30 con 0x7f689813f730 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 10 monclient: dump: 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:47.692 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 10 monclient: _finish_auth 0 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 10 monclient: _check_auth_tickets 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 1 -- 172.21.15.110:0/1027989603 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689002cc90 con 0x7f689813f730 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f689dd3c640 5 monclient: authenticate success, global_id 4169 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f689dd3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 1 -- 172.21.15.110:0/1027989603 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6890034440 con 0x7f689813f730 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f68967fc640 10 monclient: dump: 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:47.693 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:47.694 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:47.694 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:47.694 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:47.694 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:47.694 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:47.694 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:47.694 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:47.694 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f689dd3c640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:47.694 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f689dd3c640 4 set_mon_vals no callback set 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f689dd3c640 10 monclient: shutdown 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f689dd3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f689dd3c640 1 -- 172.21.15.110:0/1027989603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f689813f730 msgr2=0x7f6898142a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.691+0000 7f689dd3c640 1 --2- 172.21.15.110:0/1027989603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f689813f730 0x7f6898142a00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6890009d90 tx=0x7f689002bd80 comp rx=0 tx=0).stop 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.692+0000 7f689dd3c640 1 -- 172.21.15.110:0/1027989603 shutdown_connections 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.692+0000 7f689dd3c640 1 --2- 172.21.15.110:0/1027989603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f689813f730 0x7f6898142a00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.692+0000 7f689dd3c640 1 --2- 172.21.15.110:0/1027989603 >> v2:172.21.15.110:3300/0 conn(0x7f68980a89c0 0x7f689813f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.692+0000 7f689dd3c640 1 -- 172.21.15.110:0/1027989603 >> 172.21.15.110:0/1027989603 conn(0x7f6898066f80 msgr2=0x7f68980b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.692+0000 7f689dd3c640 1 -- 172.21.15.110:0/1027989603 shutdown_connections 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.692+0000 7f689dd3c640 1 -- 172.21.15.110:0/1027989603 wait complete. 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.692+0000 7f689dd3c640 10 monclient: build_initial_monmap 2024-01-20T11:14:47.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.692+0000 7f689dd3c640 10 monclient: monmap: 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.693+0000 7f689dd3c640 1 librados: starting msgr at 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.693+0000 7f689dd3c640 1 librados: starting objecter 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.693+0000 7f689dd3c640 1 Processor -- start 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.693+0000 7f689dd3c640 1 -- start start 2024-01-20T11:14:47.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.693+0000 7f689dd3c640 1 librados: setting wanted keys 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.693+0000 7f689dd3c640 1 librados: calling monclient init 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.693+0000 7f689dd3c640 10 monclient: init 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68980a89c0 0x7f6898141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient(hunting): picked mon.noname-a con 0x7f68980a89c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68980a8da0 0x7f68981dec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient(hunting): picked mon.noname-c con 0x7f68980a8da0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f689813f730 0x7f68981df150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient(hunting): picked mon.noname-b con 0x7f689813f730 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689806da50 con 0x7f68980a89c0 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6898145850 con 0x7f689813f730 2024-01-20T11:14:47.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689806e160 con 0x7f68980a8da0 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.694+0000 7f689dd3c640 10 monclient(hunting): authenticate will time out at 1331.045658s 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6897fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f689813f730 0x7f68981df150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68980a8da0 0x7f68981dec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68980a8da0 0x7f68981dec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:49194/0 (socket says 172.21.15.110:49194) 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 1 -- 172.21.15.110:0/3012809733 learned_addr learned my addr 172.21.15.110:0/3012809733 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 10 monclient(hunting): get_auth_request con 0x7f68980a8da0 auth_method 0 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6897fff640 10 monclient(hunting): get_auth_request con 0x7f689813f730 auth_method 0 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6897fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:47.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6897fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f68977fe640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68980a89c0 0x7f6898141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.695+0000 7f6896ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f68977fe640 10 monclient(hunting): get_auth_request con 0x7f68980a89c0 auth_method 0 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f68977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f68977fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f68977fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f68977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f68977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:47.703 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.696+0000 7f68977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:47.707 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 10 monclient(hunting): handle_auth_done global_id 4172 payload 824 2024-01-20T11:14:47.707 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 1 -- 172.21.15.110:0/3012809733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f689813f730 msgr2=0x7f68981df150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:47.707 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f689813f730 0x7f68981df150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 1 -- 172.21.15.110:0/3012809733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68980a89c0 msgr2=0x7f6898141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68980a89c0 0x7f6898141530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 1 monclient: found mon.noname-c 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.698+0000 7f6896ffd640 1 -- 172.21.15.110:0/3012809733 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6890009a70 con 0x7f68980a8da0 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.701+0000 7f6896ffd640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68980a8da0 0x7f68981dec10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f688c00c4a0 tx=0x7f688c00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.701+0000 7f687ffff640 10 client.?.objecter ms_handle_connect 0x7f68980a8da0 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.701+0000 7f687ffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f687ffff640 1 -- 172.21.15.110:0/3012809733 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c005ce0 con 0x7f68980a8da0 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f687ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f687ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:47.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f687ffff640 10 monclient: dump: 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f687ffff640 10 monclient: _finish_auth 0 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f687ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f687ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f689dd3c640 5 monclient: authenticate success, global_id 4172 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f689dd3c640 10 monclient: _renew_subs 2024-01-20T11:14:47.709 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f689dd3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689806e440 con 0x7f68980a8da0 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f689dd3c640 10 client.4172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f689dd3c640 10 monclient: _renew_subs 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f689dd3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6898144c30 con 0x7f68980a8da0 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.703+0000 7f689dd3c640 1 librados: init done 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f689dd3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f689dd3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f689dd3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6898067cc0 con 0x7f68980a8da0 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f687ffff640 1 -- 172.21.15.110:0/3012809733 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688c005e40 con 0x7f68980a8da0 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f687ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f687ffff640 1 -- 172.21.15.110:0/3012809733 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c007270 con 0x7f68980a8da0 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f687ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:47.710 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f687ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f687ffff640 10 monclient: dump: 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:47.711 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.705+0000 7f6895ffb640 4 set_mon_vals no callback set 2024-01-20T11:14:47.897 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.896+0000 7f687ffff640 1 -- 172.21.15.110:0/3012809733 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 7) v1 ==== 89142+0+0 (secure 0 0 0) 0x7f688c007490 con 0x7f68980a8da0 2024-01-20T11:14:47.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.897+0000 7f687ffff640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f688406c7d0 0x7f688406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:47.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.897+0000 7f68977fe640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f688406c7d0 0x7f688406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:47.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.897+0000 7f68977fe640 10 monclient: get_auth_request con 0x7f688406c7d0 auth_method 0 2024-01-20T11:14:47.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.898+0000 7f687ffff640 10 client.4172.objecter ms_handle_connect 0x7f688406c7d0 2024-01-20T11:14:47.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.898+0000 7f68977fe640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f688406c7d0 0x7f688406ec90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6888009aa0 tx=0x7f6888009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:47.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.915+0000 7f687ffff640 1 -- 172.21.15.110:0/3012809733 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f688c057f50 con 0x7f68980a8da0 2024-01-20T11:14:47.916 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.915+0000 7f687ffff640 10 client.4172.objecter ms_dispatch 0x7f6898000b90 osd_map(12..12 src has 1..12) v4 2024-01-20T11:14:47.916 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.915+0000 7f687ffff640 3 client.4172.objecter handle_osd_map got epochs [12,12] > 0 2024-01-20T11:14:47.916 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.915+0000 7f687ffff640 3 client.4172.objecter handle_osd_map decoding full epoch 12 2024-01-20T11:14:47.916 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:47.915+0000 7f687ffff640 20 client.4172.objecter dump_active .. 0 homeless 2024-01-20T11:14:48.140 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.139+0000 7f687ffff640 1 -- 172.21.15.110:0/3012809733 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f688c054290 con 0x7f68980a8da0 2024-01-20T11:14:48.140 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.139+0000 7f687ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:48.140 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.139+0000 7f687ffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:48.360 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.359+0000 7f689dd3c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:48.360 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.359+0000 7f689dd3c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:48.360 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.359+0000 7f689dd3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:48.361 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.359+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68981456b0 con 0x7f68980a8da0 2024-01-20T11:14:48.376 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.375+0000 7f687ffff640 1 -- 172.21.15.110:0/3012809733 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f68981456b0 con 0x7f68980a8da0 2024-01-20T11:14:48.376 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.375+0000 7f687ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:48.376 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.375+0000 7f687ffff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:48.377 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-20T11:14:48.377 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":12,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:14:40.464348+0000","last_up_change":"0.000000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:14:48.378 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 10 librados: watch_flush enter 2024-01-20T11:14:48.378 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 10 librados: watch_flush exit 2024-01-20T11:14:48.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 20 client.4172.objecter shutdown clearing up homeless session... 2024-01-20T11:14:48.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 10 client.4172.objecter successfully canceled tick 2024-01-20T11:14:48.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f688406c7d0 msgr2=0x7f688406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:48.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f688406c7d0 0x7f688406ec90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6888009aa0 tx=0x7f6888009190 comp rx=0 tx=0).stop 2024-01-20T11:14:48.379 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 10 monclient: shutdown 2024-01-20T11:14:48.380 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:48.380 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68980a8da0 msgr2=0x7f68981dec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:48.380 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68980a8da0 0x7f68981dec10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f688c00c4a0 tx=0x7f688c00c960 comp rx=0 tx=0).stop 2024-01-20T11:14:48.380 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 shutdown_connections 2024-01-20T11:14:48.380 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f688406c7d0 0x7f688406ec90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:48.380 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f689813f730 0x7f68981df150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:48.381 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68980a8da0 0x7f68981dec10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:48.381 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 --2- 172.21.15.110:0/3012809733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68980a89c0 0x7f6898141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:48.381 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.377+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 >> 172.21.15.110:0/3012809733 conn(0x7f6898066f80 msgr2=0x7f6898133670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:48.382 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.378+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 shutdown_connections 2024-01-20T11:14:48.382 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.378+0000 7f689dd3c640 1 -- 172.21.15.110:0/3012809733 wait complete. 2024-01-20T11:14:48.382 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:48.378+0000 7f689dd3c640 1 librados: shutdown 2024-01-20T11:14:48.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-20T11:14:48.501 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:16.961 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:48.622 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:17.074 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:48.642 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:17.125 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:48.706 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:17.191 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:49.119 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:17.580 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:49.443 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:17.908 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:50.097 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:18.553 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:50.126 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:18.599 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:53.957 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:22.416 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:53.961 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:22.421 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.023 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:22.475 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.027 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:22.479 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.288 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:22.773 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.292 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:22.777 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.332 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:22.816 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.338 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:22.822 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.487 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:22.948 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.491 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:22.952 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.966 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:23.432 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:54.973 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:23.438 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:55.197 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-20T11:14:55.297 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:23.770 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:55.301 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:23.775 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:55.314 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.313+0000 7fc293cbb640 10 monclient: get_monmap_and_config 2024-01-20T11:14:55.314 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.313+0000 7fc293cbb640 10 monclient: build_initial_monmap 2024-01-20T11:14:55.314 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.313+0000 7fc293cbb640 10 monclient: monmap: 2024-01-20T11:14:55.314 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:14:55.314 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:55.314 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:14:55.315187+0000 2024-01-20T11:14:55.314 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:14:55.315187+0000 2024-01-20T11:14:55.315 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:55.315 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:55.315 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:14:55.315 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:14:55.315 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:14:55.315 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:14:55.315 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:14:55.315 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:55.317 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.314+0000 7fc293cbb640 1 Processor -- start 2024-01-20T11:14:55.317 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.314+0000 7fc293cbb640 1 -- start start 2024-01-20T11:14:55.317 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.314+0000 7fc293cbb640 10 monclient: init 2024-01-20T11:14:55.317 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.314+0000 7fc293cbb640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:55.317 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.314+0000 7fc293cbb640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-01-20T11:14:55.317 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.314+0000 7fc293cbb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc28c0a89c0 0x7fc28c13f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:55.317 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.314+0000 7fc293cbb640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c0a89c0 addr v2:172.21.15.134:3300/0 2024-01-20T11:14:55.317 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.314+0000 7fc293cbb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc28c0a8da0 addr v1:172.21.15.134:6789/0 2024-01-20T11:14:55.317 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc293cbb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc28c13f730 0x7fc28c142a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc293cbb640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c13f730 addr v2:172.21.15.110:3300/0 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc293cbb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0b21b0 con 0x7fc28c0a8da0 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc293cbb640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc293cbb640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c0aa220 con 0x7fc28c13f730 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc293cbb640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc293cbb640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c0a9f20 con 0x7fc28c0a89c0 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc293cbb640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc293cbb640 10 monclient(hunting): authenticate will time out at 1338.666406s 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.315+0000 7fc292258640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc28c13f730 0x7fc28c142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc291256640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc28c0a8da0 0x7fc28c1422f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:43246/0 (socket says 172.21.15.110:43246) 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc291256640 1 -- 172.21.15.110:0/2462399742 learned_addr learned my addr 172.21.15.110:0/2462399742 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc292258640 10 monclient(hunting): get_auth_request con 0x7fc28c13f730 auth_method 0 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc292258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc292258640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc292258640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:55.318 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc291a57640 1 --2- 172.21.15.110:0/2462399742 >> v2:172.21.15.134:3300/0 conn(0x7fc28c0a89c0 0x7fc28c13f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc292258640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc292258640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc292258640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc291a57640 10 monclient(hunting): get_auth_request con 0x7fc28c0a89c0 auth_method 0 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc291a57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc291a57640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.316+0000 7fc291a57640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.318+0000 7fc291a57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.318+0000 7fc291a57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.318+0000 7fc291a57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:55.320 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.319+0000 7fc292258640 10 monclient(hunting): handle_auth_done global_id 4212 payload 274 2024-01-20T11:14:55.321 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.319+0000 7fc292258640 1 -- 172.21.15.110:0/2462399742 >> v2:172.21.15.134:3300/0 conn(0x7fc28c0a89c0 msgr2=0x7fc28c13f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:14:55.321 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.319+0000 7fc292258640 1 --2- 172.21.15.110:0/2462399742 >> v2:172.21.15.134:3300/0 conn(0x7fc28c0a89c0 0x7fc28c13f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:55.321 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.319+0000 7fc292258640 1 -- 172.21.15.110:0/2462399742 >> v1:172.21.15.134:6789/0 conn(0x7fc28c0a8da0 legacy=0x7fc28c1422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:55.321 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.319+0000 7fc292258640 10 monclient: _finish_hunting 0 2024-01-20T11:14:55.321 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.319+0000 7fc292258640 1 monclient: found mon.noname-a 2024-01-20T11:14:55.321 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.319+0000 7fc292258640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:55.321 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.319+0000 7fc292258640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:14:55.321 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.319+0000 7fc292258640 1 -- 172.21.15.110:0/2462399742 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0a9760 con 0x7fc28c13f730 2024-01-20T11:14:55.322 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.322+0000 7fc292258640 1 --2- 172.21.15.110:0/2462399742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc28c13f730 0x7fc28c142a00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0b8800 tx=0x7fc28402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec70ebda21587fae server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 1 -- 172.21.15.110:0/2462399742 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28402cb60 con 0x7fc28c13f730 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 10 monclient: dump: 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:55.327 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 10 monclient: _finish_auth 0 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 10 monclient: _check_auth_tickets 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc293cbb640 5 monclient: authenticate success, global_id 4212 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 1 -- 172.21.15.110:0/2462399742 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28402ccc0 con 0x7fc28c13f730 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 1 -- 172.21.15.110:0/2462399742 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284034660 con 0x7fc28c13f730 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc293cbb640 10 monclient: get_monmap_and_config success 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc290a55640 10 monclient: dump: 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:55.328 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:55.329 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:55.329 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:55.329 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:55.329 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:55.329 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:55.329 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:55.329 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:55.329 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.326+0000 7fc293cbb640 4 set_mon_vals no callback set 2024-01-20T11:14:55.334 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 10 monclient: shutdown 2024-01-20T11:14:55.334 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:55.334 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 1 -- 172.21.15.110:0/2462399742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc28c13f730 msgr2=0x7fc28c142a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:55.334 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 1 --2- 172.21.15.110:0/2462399742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc28c13f730 0x7fc28c142a00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0b8800 tx=0x7fc28402bdb0 comp rx=0 tx=0).stop 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 1 -- 172.21.15.110:0/2462399742 shutdown_connections 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 1 --2- 172.21.15.110:0/2462399742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc28c13f730 0x7fc28c142a00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 1 --2- 172.21.15.110:0/2462399742 >> v2:172.21.15.134:3300/0 conn(0x7fc28c0a89c0 0x7fc28c13f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 1 -- 172.21.15.110:0/2462399742 >> 172.21.15.110:0/2462399742 conn(0x7fc28c066f80 msgr2=0x7fc28c0b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 1 -- 172.21.15.110:0/2462399742 shutdown_connections 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.327+0000 7fc293cbb640 1 -- 172.21.15.110:0/2462399742 wait complete. 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.328+0000 7fc293cbb640 10 monclient: build_initial_monmap 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.328+0000 7fc293cbb640 10 monclient: monmap: 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:55.335 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.328+0000 7fc293cbb640 1 librados: starting msgr at 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.328+0000 7fc293cbb640 1 librados: starting objecter 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.329+0000 7fc293cbb640 1 Processor -- start 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.329+0000 7fc293cbb640 1 -- start start 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.329+0000 7fc293cbb640 1 librados: setting wanted keys 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.329+0000 7fc293cbb640 1 librados: calling monclient init 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.329+0000 7fc293cbb640 10 monclient: init 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc293cbb640 10 monclient: _reopen_session rank -1 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc293cbb640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc293cbb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc28c0a89c0 0x7fc28c13a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc293cbb640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c0a89c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc293cbb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc28c0a8da0 0x7fc28c137d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:55.336 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc293cbb640 10 monclient(hunting): picked mon.noname-b con 0x7fc28c0a8da0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc293cbb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28c13f730 0x7fc28c1382c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc291256640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc28c0a8da0 0x7fc28c137d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc291a57640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc28c0a89c0 0x7fc28c13a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.330+0000 7fc293cbb640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c13f730 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc293cbb640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc292258640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28c13f730 0x7fc28c1382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc293cbb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0ac280 con 0x7fc28c0a89c0 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc293cbb640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291256640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc28c0a8da0 0x7fc28c137d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:35186/0 (socket says 172.21.15.110:35186) 2024-01-20T11:14:55.337 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291256640 1 -- 172.21.15.110:0/4017562817 learned_addr learned my addr 172.21.15.110:0/4017562817 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0ac560 con 0x7fc28c0a8da0 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc293cbb640 10 monclient(hunting): start opening mon connection 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0ab3d0 con 0x7fc28c13f730 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc293cbb640 10 monclient(hunting): _renew_subs 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc293cbb640 10 monclient(hunting): authenticate will time out at 1338.682681s 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291256640 10 monclient(hunting): get_auth_request con 0x7fc28c0a8da0 auth_method 0 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291256640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291256640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291a57640 10 monclient(hunting): get_auth_request con 0x7fc28c0a89c0 auth_method 0 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291a57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291a57640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc291a57640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc292258640 10 monclient(hunting): get_auth_request con 0x7fc28c13f730 auth_method 0 2024-01-20T11:14:55.338 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc292258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc292258640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.331+0000 7fc292258640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.332+0000 7fc291256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.332+0000 7fc291256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.332+0000 7fc291256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.332+0000 7fc292258640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.332+0000 7fc292258640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.332+0000 7fc292258640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.332+0000 7fc291a57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.332+0000 7fc291a57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:14:55.339 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.332+0000 7fc291a57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:14:55.340 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 10 monclient(hunting): handle_auth_done global_id 4175 payload 824 2024-01-20T11:14:55.340 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 1 -- 172.21.15.110:0/4017562817 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc28c0a8da0 msgr2=0x7fc28c137d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:55.340 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc28c0a8da0 0x7fc28c137d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 1 -- 172.21.15.110:0/4017562817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc28c0a89c0 msgr2=0x7fc28c13a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc28c0a89c0 0x7fc28c13a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 10 monclient: _finish_hunting 0 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 1 monclient: found mon.noname-c 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.334+0000 7fc292258640 1 -- 172.21.15.110:0/4017562817 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc284009a70 con 0x7fc28c13f730 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.337+0000 7fc292258640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28c13f730 0x7fc28c1382c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc284033000 tx=0x7fc284004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.337+0000 7fc2827fc640 10 client.?.objecter ms_handle_connect 0x7fc28c13f730 2024-01-20T11:14:55.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.337+0000 7fc2827fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:14:55.345 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 1 -- 172.21.15.110:0/4017562817 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284040450 con 0x7fc28c13f730 2024-01-20T11:14:55.345 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:55.345 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:14:55.345 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 10 monclient: dump: 2024-01-20T11:14:55.345 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:55.345 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:55.345 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:55.345 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:55.345 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 10 monclient: _finish_auth 0 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 10 monclient: _check_auth_tickets 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 1 -- 172.21.15.110:0/4017562817 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2840405b0 con 0x7fc28c13f730 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 1 -- 172.21.15.110:0/4017562817 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284040940 con 0x7fc28c13f730 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc2827fc640 10 monclient: dump: 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:14:55.346 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc283fff640 4 set_mon_vals no callback set 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc293cbb640 5 monclient: authenticate success, global_id 4175 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc293cbb640 10 monclient: _renew_subs 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc293cbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.341+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c13b290 con 0x7fc28c13f730 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.342+0000 7fc293cbb640 10 client.4175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:14:55.347 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.342+0000 7fc293cbb640 10 monclient: _renew_subs 2024-01-20T11:14:55.348 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.342+0000 7fc293cbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:55.348 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.342+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c13ae00 con 0x7fc28c13f730 2024-01-20T11:14:55.348 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.342+0000 7fc293cbb640 1 librados: init done 2024-01-20T11:14:55.348 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.343+0000 7fc293cbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:55.348 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.343+0000 7fc293cbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:55.348 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.343+0000 7fc293cbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:55.348 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.343+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c067cc0 con 0x7fc28c13f730 2024-01-20T11:14:55.404 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.403+0000 7fc2827fc640 1 -- 172.21.15.110:0/4017562817 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 7) v1 ==== 89142+0+0 (secure 0 0 0) 0x7fc284040aa0 con 0x7fc28c13f730 2024-01-20T11:14:55.404 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.403+0000 7fc2827fc640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc27006c820 0x7fc27006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:14:55.405 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.403+0000 7fc291a57640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc27006c820 0x7fc27006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:14:55.405 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.404+0000 7fc291a57640 10 monclient: get_auth_request con 0x7fc27006c820 auth_method 0 2024-01-20T11:14:55.405 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.404+0000 7fc291a57640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc27006c820 0x7fc27006ece0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc288009b40 tx=0x7fc288009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:14:55.405 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.404+0000 7fc2827fc640 10 client.4175.objecter ms_handle_connect 0x7fc27006c820 2024-01-20T11:14:55.515 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:23.971 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:55.521 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:23.977 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:55.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.701+0000 7fc2827fc640 1 -- 172.21.15.110:0/4017562817 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc28407b240 con 0x7fc28c13f730 2024-01-20T11:14:55.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.701+0000 7fc2827fc640 10 client.4175.objecter ms_dispatch 0x7fc28c000b90 osd_map(12..12 src has 1..12) v4 2024-01-20T11:14:55.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.701+0000 7fc2827fc640 3 client.4175.objecter handle_osd_map got epochs [12,12] > 0 2024-01-20T11:14:55.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.701+0000 7fc2827fc640 3 client.4175.objecter handle_osd_map decoding full epoch 12 2024-01-20T11:14:55.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.701+0000 7fc2827fc640 20 client.4175.objecter dump_active .. 0 homeless 2024-01-20T11:14:55.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.701+0000 7fc2827fc640 1 -- 172.21.15.110:0/4017562817 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc284077580 con 0x7fc28c13f730 2024-01-20T11:14:55.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.701+0000 7fc2827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:14:55.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.701+0000 7fc2827fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:14:55.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.996+0000 7fc293cbb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:55.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.996+0000 7fc293cbb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:55.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.996+0000 7fc293cbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:14:55.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:55.996+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc28c0ac6a0 con 0x7fc28c13f730 2024-01-20T11:14:56.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.022+0000 7fc2827fc640 1 -- 172.21.15.110:0/4017562817 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fc28c0ac6a0 con 0x7fc28c13f730 2024-01-20T11:14:56.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.022+0000 7fc2827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:14:56.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.022+0000 7fc2827fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:14:56.024 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-20T11:14:56.024 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":12,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:14:40.464348+0000","last_up_change":"0.000000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:14:56.026 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.024+0000 7fc293cbb640 10 librados: watch_flush enter 2024-01-20T11:14:56.026 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.024+0000 7fc293cbb640 10 librados: watch_flush exit 2024-01-20T11:14:56.026 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.024+0000 7fc293cbb640 20 client.4175.objecter shutdown clearing up homeless session... 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.024+0000 7fc293cbb640 10 client.4175.objecter successfully canceled tick 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.024+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc27006c820 msgr2=0x7fc27006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.024+0000 7fc293cbb640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc27006c820 0x7fc27006ece0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc288009b40 tx=0x7fc288009190 comp rx=0 tx=0).stop 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 10 monclient: shutdown 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28c13f730 msgr2=0x7fc28c1382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28c13f730 0x7fc28c1382c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc284033000 tx=0x7fc284004a40 comp rx=0 tx=0).stop 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 shutdown_connections 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc27006c820 0x7fc27006ece0 secure :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc288009b40 tx=0x7fc288009190 comp rx=0 tx=0).stop 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28c13f730 0x7fc28c1382c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc28c0a8da0 0x7fc28c137d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 1 --2- 172.21.15.110:0/4017562817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc28c0a89c0 0x7fc28c13a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:14:56.027 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.025+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 >> 172.21.15.110:0/4017562817 conn(0x7fc28c066f80 msgr2=0x7fc28c0b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:14:56.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.026+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 shutdown_connections 2024-01-20T11:14:56.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.026+0000 7fc293cbb640 1 -- 172.21.15.110:0/4017562817 wait complete. 2024-01-20T11:14:56.028 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:14:56.026+0000 7fc293cbb640 1 librados: shutdown 2024-01-20T11:14:56.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-20T11:14:58.164 INFO:tasks.ceph.osd.6.smithi134.stderr:==00:00:00:26.616 34422== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:58.358 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:26.818 34437== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:58.915 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:27.398 34436== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:59.104 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:27.565 34438== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:59.159 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:27.644 34435== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:59.273 INFO:tasks.ceph.osd.7.smithi134.stderr:==00:00:00:27.738 34420== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:59.584 INFO:tasks.ceph.osd.4.smithi134.stderr:==00:00:00:28.040 34421== Warning: unimplemented fcntl command: 1038 2024-01-20T11:14:59.831 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:00:28.304 34419== Warning: unimplemented fcntl command: 1038 2024-01-20T11:15:02.847 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-20T11:15:02.977 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.976+0000 7fe404ea2640 10 monclient: get_monmap_and_config 2024-01-20T11:15:02.977 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.976+0000 7fe404ea2640 10 monclient: build_initial_monmap 2024-01-20T11:15:02.977 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.976+0000 7fe404ea2640 10 monclient: monmap: 2024-01-20T11:15:02.977 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:15:02.977 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:02.977 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:15:02.977893+0000 2024-01-20T11:15:02.977 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:15:02.977893+0000 2024-01-20T11:15:02.977 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:02.977 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:02.978 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:02.978 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:02.978 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:02.978 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:02.978 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:02.978 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:02.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.977+0000 7fe404ea2640 1 Processor -- start 2024-01-20T11:15:02.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.977+0000 7fe404ea2640 1 -- start start 2024-01-20T11:15:02.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.977+0000 7fe404ea2640 10 monclient: init 2024-01-20T11:15:02.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe4000a8a30 0x7fe40013efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 10 monclient(hunting): picked mon.noname-c con 0x7fe4000a8a30 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4000a9c90 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe40013f4e0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4000a8e10 con 0x7fe40013f4e0 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4000aacf0 con 0x7fe4000a9c90 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4000aafd0 con 0x7fe4000a8a30 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:02.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:02.978+0000 7fe404ea2640 10 monclient(hunting): authenticate will time out at 1346.329889s 2024-01-20T11:15:03.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.006+0000 7fe3fed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe40013f4e0 0x7fe400142810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58480/0 (socket says 172.21.15.110:58480) 2024-01-20T11:15:03.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.006+0000 7fe3fed9d640 1 -- 172.21.15.110:0/4244947919 learned_addr learned my addr 172.21.15.110:0/4244947919 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:03.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.009+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627728703 0 0) 0x7fe4000a8e10 con 0x7fe40013f4e0 2024-01-20T11:15:03.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.009+0000 7fe3fd59a640 10 monclient(hunting): _negotiate 2024-01-20T11:15:03.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.009+0000 7fe3fd59a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:03.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.009+0000 7fe3fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:03.011 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.009+0000 7fe3fd59a640 10 monclient(hunting): my global_id is 4218 2024-01-20T11:15:03.011 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.009+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e8003520 con 0x7fe40013f4e0 2024-01-20T11:15:03.014 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3485939775 0 0) 0x7fe3e8003520 con 0x7fe40013f4e0 2024-01-20T11:15:03.014 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 >> v2:172.21.15.134:3300/0 conn(0x7fe4000a8a30 msgr2=0x7fe40013efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:03.014 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 --2- 172.21.15.110:0/4244947919 >> v2:172.21.15.134:3300/0 conn(0x7fe4000a8a30 0x7fe40013efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.014 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 >> v1:172.21.15.134:6789/0 conn(0x7fe4000a9c90 legacy=0x7fe400142100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient: _finish_hunting 0 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 monclient: found mon.noname-a-legacy 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.110:6789/0 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4000a90c0 con 0x7fe40013f4e0 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient: _finish_auth 0 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient: _check_auth_tickets 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2748620985 0 0) 0x7fe3f80019f0 con 0x7fe40013f4e0 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient: dump: 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.015 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 >> v1:172.21.15.110:6789/0 conn(0x7fe40013f4e0 legacy=0x7fe400142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 --2- 172.21.15.110:0/4244947919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3e80039c0 0x7fe3e8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient(hunting): picked mon.b con 0x7fe3e80039c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 --2- 172.21.15.110:0/4244947919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 0x7fe3e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient(hunting): picked mon.a con 0x7fe3e80047c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:03.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 --2- 172.21.15.110:0/4244947919 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3e80052a0 0x7fe3e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient(hunting): picked mon.c con 0x7fe3e80052a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4000aafd0 con 0x7fe3e80047c0 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e800c1f0 con 0x7fe3e80039c0 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e800c3b0 con 0x7fe3e80052a0 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.013+0000 7fe3fd59a640 10 monclient(hunting): _finish_auth 0 2024-01-20T11:15:03.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.016+0000 7fe3fdd9b640 1 --2- 172.21.15.110:0/4244947919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 0x7fe3e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:03.018 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.018+0000 7fe3fdd9b640 10 monclient(hunting): get_auth_request con 0x7fe3e80047c0 auth_method 0 2024-01-20T11:15:03.018 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.018+0000 7fe3fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:03.018 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.018+0000 7fe3fdd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:03.018 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.018+0000 7fe3fdd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:15:03.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.019+0000 7fe3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:03.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.019+0000 7fe3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:03.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.019+0000 7fe3fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 10 monclient(hunting): handle_auth_done global_id 4218 payload 293 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 1 -- 172.21.15.110:0/4244947919 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3e80052a0 msgr2=0x7fe3e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 1 --2- 172.21.15.110:0/4244947919 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3e80052a0 0x7fe3e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 1 -- 172.21.15.110:0/4244947919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3e80039c0 msgr2=0x7fe3e8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 1 --2- 172.21.15.110:0/4244947919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3e80039c0 0x7fe3e8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 10 monclient: _finish_hunting 0 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 1 monclient: found mon.a 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 1 -- 172.21.15.110:0/4244947919 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e8003520 con 0x7fe3e80047c0 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 10 monclient: _finish_auth 0 2024-01-20T11:15:03.023 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 10 monclient: _check_auth_tickets 2024-01-20T11:15:03.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:03.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.021+0000 7fe3fed9d640 1 -- 172.21.15.110:0/4244947919 reap_dead start 2024-01-20T11:15:03.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.022+0000 7fe404ea2640 5 monclient: authenticate success, global_id 4218 2024-01-20T11:15:03.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.022+0000 7fe404ea2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:03.024 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.023+0000 7fe3fdd9b640 1 --2- 172.21.15.110:0/4244947919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 0x7fe3e8004c40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4001620 tx=0x7fe3e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b225abbe45c8e47e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:03.029 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4005270 con 0x7fe3e80047c0 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 10 monclient: dump: 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f402e070 con 0x7fe3e80047c0 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:03.030 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 1 -- 172.21.15.110:0/4244947919 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4003610 con 0x7fe3e80047c0 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe3fd59a640 10 monclient: dump: 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe404ea2640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.027+0000 7fe404ea2640 4 set_mon_vals no callback set 2024-01-20T11:15:03.031 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.028+0000 7fe404ea2640 10 monclient: shutdown 2024-01-20T11:15:03.032 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.028+0000 7fe404ea2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:03.032 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.028+0000 7fe404ea2640 1 -- 172.21.15.110:0/4244947919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 msgr2=0x7fe3e8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:03.032 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.028+0000 7fe404ea2640 1 --2- 172.21.15.110:0/4244947919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 0x7fe3e8004c40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4001620 tx=0x7fe3e80057f0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.032 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.028+0000 7fe404ea2640 1 -- 172.21.15.110:0/4244947919 shutdown_connections 2024-01-20T11:15:03.032 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.028+0000 7fe404ea2640 1 --2- 172.21.15.110:0/4244947919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 0x7fe3e8004c40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.032 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.028+0000 7fe404ea2640 1 -- 172.21.15.110:0/4244947919 >> 172.21.15.110:0/4244947919 conn(0x7fe4000b4320 msgr2=0x7fe4000b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:03.032 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.028+0000 7fe404ea2640 1 -- 172.21.15.110:0/4244947919 shutdown_connections 2024-01-20T11:15:03.032 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.028+0000 7fe404ea2640 1 -- 172.21.15.110:0/4244947919 wait complete. 2024-01-20T11:15:03.035 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.029+0000 7fe404ea2640 10 monclient: build_initial_monmap 2024-01-20T11:15:03.035 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.029+0000 7fe404ea2640 10 monclient: monmap: 2024-01-20T11:15:03.035 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:15:03.035 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:03.035 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:03.035 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-20T11:15:03.035 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:03.035 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.030+0000 7fe404ea2640 1 librados: starting msgr at 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.030+0000 7fe404ea2640 1 librados: starting objecter 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.030+0000 7fe404ea2640 1 Processor -- start 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.030+0000 7fe404ea2640 1 -- start start 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.030+0000 7fe404ea2640 1 librados: setting wanted keys 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.030+0000 7fe404ea2640 1 librados: calling monclient init 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.030+0000 7fe404ea2640 10 monclient: init 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.031+0000 7fe404ea2640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.031+0000 7fe404ea2640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.031+0000 7fe404ea2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 0x7fe40006d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.031+0000 7fe404ea2640 10 monclient(hunting): picked mon.noname-a con 0x7fe3e80047c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:03.036 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.031+0000 7fe404ea2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe400142fb0 0x7fe4001deec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.031+0000 7fe404ea2640 10 monclient(hunting): picked mon.noname-c con 0x7fe400142fb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.031+0000 7fe404ea2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4001df400 0x7fe4001db750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe404ea2640 10 monclient(hunting): picked mon.noname-b con 0x7fe4001df400 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe404ea2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe404ea2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe400145f40 con 0x7fe3e80047c0 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe404ea2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe404ea2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40006eb40 con 0x7fe4001df400 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe404ea2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe404ea2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40006ee20 con 0x7fe400142fb0 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe404ea2640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe404ea2640 10 monclient(hunting): authenticate will time out at 1346.383222s 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fe59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 0x7fe40006d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fed9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4001df400 0x7fe4001db750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fed9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4001df400 0x7fe4001db750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:37648/0 (socket says 172.21.15.110:37648) 2024-01-20T11:15:03.037 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fed9d640 1 -- 172.21.15.110:0/2130922754 learned_addr learned my addr 172.21.15.110:0/2130922754 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fed9d640 10 monclient(hunting): get_auth_request con 0x7fe4001df400 auth_method 0 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fe59c640 10 monclient(hunting): get_auth_request con 0x7fe3e80047c0 auth_method 0 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fe59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.032+0000 7fe3fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.033+0000 7fe3fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.033+0000 7fe3fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.033+0000 7fe3fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.033+0000 7fe3fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.033+0000 7fe3fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:03.038 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.033+0000 7fe3fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:03.039 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 10 monclient(hunting): handle_auth_done global_id 4249 payload 824 2024-01-20T11:15:03.039 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 1 -- 172.21.15.110:0/2130922754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe400142fb0 msgr2=0x7fe4001deec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:03.039 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe400142fb0 0x7fe4001deec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.039 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 1 -- 172.21.15.110:0/2130922754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 msgr2=0x7fe40006d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:03.039 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 0x7fe40006d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.039 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:15:03.039 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 1 monclient: found mon.noname-b 2024-01-20T11:15:03.039 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:03.040 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:03.040 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.037+0000 7fe3fed9d640 1 -- 172.21.15.110:0/2130922754 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4001df880 con 0x7fe4001df400 2024-01-20T11:15:03.040 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.040+0000 7fe3fed9d640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4001df400 0x7fe4001db750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe40013f8c0 tx=0x7fe3f800ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:03.041 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.040+0000 7fe3eeffd640 10 client.?.objecter ms_handle_connect 0x7fe4001df400 2024-01-20T11:15:03.041 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.040+0000 7fe3eeffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 1 -- 172.21.15.110:0/2130922754 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800a610 con 0x7fe4001df400 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 10 monclient: dump: 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:03.046 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 1 -- 172.21.15.110:0/2130922754 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f800a770 con 0x7fe4001df400 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 1 -- 172.21.15.110:0/2130922754 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800ab20 con 0x7fe4001df400 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3eeffd640 10 monclient: dump: 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.047 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe404ea2640 5 monclient: authenticate success, global_id 4249 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe3fcd99640 4 set_mon_vals no callback set 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe404ea2640 10 monclient: _renew_subs 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe404ea2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe404ea2640 1 -- 172.21.15.110:0/2130922754 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4001dfa70 con 0x7fe4001df400 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe404ea2640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe404ea2640 10 monclient: _renew_subs 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe404ea2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe404ea2640 1 -- 172.21.15.110:0/2130922754 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4001dbe70 con 0x7fe4001df400 2024-01-20T11:15:03.049 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.043+0000 7fe404ea2640 1 librados: init done 2024-01-20T11:15:03.049 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.045+0000 7fe3ecff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:03.049 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.045+0000 7fe3ecff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:03.049 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.045+0000 7fe3ecff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:03.049 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.045+0000 7fe3ecff9640 1 -- 172.21.15.110:0/2130922754 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d8005080 con 0x7fe4001df400 2024-01-20T11:15:03.306 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.305+0000 7fe3eeffd640 1 -- 172.21.15.110:0/2130922754 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 7) v1 ==== 89142+0+0 (secure 0 0 0) 0x7fe3f800ac80 con 0x7fe4001df400 2024-01-20T11:15:03.307 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.305+0000 7fe3eeffd640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe3e006c820 0x7fe3e006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:03.307 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.306+0000 7fe3fe59c640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe3e006c820 0x7fe3e006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:03.307 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.306+0000 7fe3fe59c640 10 monclient: get_auth_request con 0x7fe3e006c820 auth_method 0 2024-01-20T11:15:03.307 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.306+0000 7fe3eeffd640 10 client.4249.objecter ms_handle_connect 0x7fe3e006c820 2024-01-20T11:15:03.307 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.306+0000 7fe3fe59c640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe3e006c820 0x7fe3e006ece0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0004810 tx=0x7fe3f0009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:03.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.340+0000 7fe3eeffd640 1 -- 172.21.15.110:0/2130922754 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe3f8058420 con 0x7fe4001df400 2024-01-20T11:15:03.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.340+0000 7fe3eeffd640 10 client.4249.objecter ms_dispatch 0x7fe400000b90 osd_map(12..12 src has 1..12) v4 2024-01-20T11:15:03.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.340+0000 7fe3eeffd640 3 client.4249.objecter handle_osd_map got epochs [12,12] > 0 2024-01-20T11:15:03.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.340+0000 7fe3eeffd640 3 client.4249.objecter handle_osd_map decoding full epoch 12 2024-01-20T11:15:03.341 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.340+0000 7fe3eeffd640 20 client.4249.objecter dump_active .. 0 homeless 2024-01-20T11:15:03.342 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.341+0000 7fe3eeffd640 1 -- 172.21.15.110:0/2130922754 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe3f8054060 con 0x7fe4001df400 2024-01-20T11:15:03.342 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.341+0000 7fe3eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:03.342 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.341+0000 7fe3eeffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:03.714 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.713+0000 7fe3ecff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:03.715 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.713+0000 7fe3ecff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:03.715 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.713+0000 7fe3ecff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:03.715 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.713+0000 7fe3ecff9640 1 -- 172.21.15.110:0/2130922754 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3d8005600 con 0x7fe4001df400 2024-01-20T11:15:03.741 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.740+0000 7fe3eeffd640 1 -- 172.21.15.110:0/2130922754 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fe3f8057680 con 0x7fe4001df400 2024-01-20T11:15:03.741 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.740+0000 7fe3eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:03.741 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.740+0000 7fe3eeffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:03.741 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-20T11:15:03.741 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":12,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:14:40.464348+0000","last_up_change":"0.000000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:15:03.745 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 10 librados: watch_flush enter 2024-01-20T11:15:03.745 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 10 librados: watch_flush exit 2024-01-20T11:15:03.745 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 20 client.4249.objecter shutdown clearing up homeless session... 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 10 client.4249.objecter successfully canceled tick 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 -- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe3e006c820 msgr2=0x7fe3e006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe3e006c820 0x7fe3e006ece0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0004810 tx=0x7fe3f0009190 comp rx=0 tx=0).stop 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 10 monclient: shutdown 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 -- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4001df400 msgr2=0x7fe4001db750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4001df400 0x7fe4001db750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe40013f8c0 tx=0x7fe3f800ccb0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 -- 172.21.15.110:0/2130922754 shutdown_connections 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe3e006c820 0x7fe3e006ece0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4001df400 0x7fe4001db750 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe400142fb0 0x7fe4001deec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 --2- 172.21.15.110:0/2130922754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3e80047c0 0x7fe40006d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.743+0000 7fe3ecff9640 1 -- 172.21.15.110:0/2130922754 >> 172.21.15.110:0/2130922754 conn(0x7fe4000b4320 msgr2=0x7fe400139f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.744+0000 7fe3ecff9640 1 -- 172.21.15.110:0/2130922754 shutdown_connections 2024-01-20T11:15:03.746 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.744+0000 7fe3ecff9640 1 -- 172.21.15.110:0/2130922754 wait complete. 2024-01-20T11:15:03.747 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:03.744+0000 7fe3ecff9640 1 librados: shutdown 2024-01-20T11:15:03.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-20T11:15:06.464 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-01-20T11:15:06.459+0000 5dc2480 -1 osd.6 0 log_to_monitors true 2024-01-20T11:15:06.893 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-20T11:15:06.889+0000 5dc2480 -1 osd.2 0 log_to_monitors true 2024-01-20T11:15:07.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-20T11:15:07.128+0000 5dc2480 -1 osd.1 0 log_to_monitors true 2024-01-20T11:15:07.385 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-20T11:15:07.381+0000 5dc2480 -1 osd.0 0 log_to_monitors true 2024-01-20T11:15:07.438 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:15:07.434+0000 5dc2480 -1 osd.3 0 log_to_monitors true 2024-01-20T11:15:07.504 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:15:07.500+0000 5dc2480 -1 osd.7 0 log_to_monitors true 2024-01-20T11:15:08.080 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:15:08.077+0000 5dc2480 -1 osd.4 0 log_to_monitors true 2024-01-20T11:15:08.430 INFO:tasks.ceph.osd.5.smithi134.stderr:2024-01-20T11:15:08.427+0000 5dc2480 -1 osd.5 0 log_to_monitors true 2024-01-20T11:15:09.659 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-01-20T11:15:09.648+0000 c494640 -1 osd.6 0 waiting for initial osdmap 2024-01-20T11:15:10.062 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-01-20T11:15:10.043+0000 10253640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-20T11:15:10.565 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-20T11:15:10.669 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.668+0000 7fcc7d24f640 10 monclient: get_monmap_and_config 2024-01-20T11:15:10.669 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.668+0000 7fcc7d24f640 10 monclient: build_initial_monmap 2024-01-20T11:15:10.669 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.668+0000 7fcc7d24f640 10 monclient: monmap: 2024-01-20T11:15:10.669 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:15:10.670153+0000 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:15:10.670153+0000 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:10.670 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:10.672 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.669+0000 7fcc7d24f640 1 Processor -- start 2024-01-20T11:15:10.672 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.669+0000 7fcc7d24f640 1 -- start start 2024-01-20T11:15:10.672 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.669+0000 7fcc7d24f640 10 monclient: init 2024-01-20T11:15:10.672 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:10.672 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-01-20T11:15:10.672 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc780a89c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:10.672 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcc780a8da0 0x7fcc781422f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 10 monclient(hunting): picked mon.noname-a con 0x7fcc780a8da0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc78142830 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc780b21b0 con 0x7fcc780a89c0 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc780a9760 con 0x7fcc78142830 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc780aa220 con 0x7fcc780a8da0 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.670+0000 7fcc7d24f640 10 monclient(hunting): authenticate will time out at 1354.022076s 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.671+0000 7fcc7759e640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fcc78142830 0x7fcc7814c7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:54796/0 (socket says 172.21.15.110:54796) 2024-01-20T11:15:10.673 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.671+0000 7fcc7759e640 1 -- 172.21.15.110:0/3209782833 learned_addr learned my addr 172.21.15.110:0/3209782833 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:10.679 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.677+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788327442 0 0) 0x7fcc780a9760 con 0x7fcc78142830 2024-01-20T11:15:10.679 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.677+0000 7fcc75d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:15:10.679 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.677+0000 7fcc75d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:10.679 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.677+0000 7fcc75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:10.679 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.677+0000 7fcc75d9b640 10 monclient(hunting): my global_id is 4276 2024-01-20T11:15:10.679 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.677+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc60003520 con 0x7fcc78142830 2024-01-20T11:15:10.679 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.678+0000 7fcc7659c640 1 --2- 172.21.15.110:0/3209782833 >> v2:172.21.15.110:3300/0 conn(0x7fcc780a8da0 0x7fcc781422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:10.680 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.679+0000 7fcc7659c640 10 monclient(hunting): get_auth_request con 0x7fcc780a8da0 auth_method 0 2024-01-20T11:15:10.680 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.679+0000 7fcc7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:10.681 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.679+0000 7fcc7659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:10.681 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.679+0000 7fcc7659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:10.682 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1851339983 0 0) 0x7fcc60003520 con 0x7fcc78142830 2024-01-20T11:15:10.682 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 >> v2:172.21.15.110:3300/0 conn(0x7fcc780a8da0 msgr2=0x7fcc781422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:10.682 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 --2- 172.21.15.110:0/3209782833 >> v2:172.21.15.110:3300/0 conn(0x7fcc780a8da0 0x7fcc781422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:10.682 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 >> v1:172.21.15.110:6789/0 conn(0x7fcc780a89c0 legacy=0x7fcc7813f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:10.682 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:15:10.682 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 monclient: found mon.noname-c-legacy 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.134:6789/0 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780b4040 con 0x7fcc78142830 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient: _finish_auth 0 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2748620985 0 0) 0x7fcc700019f0 con 0x7fcc78142830 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient: dump: 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:10.683 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] but i'm connected to v1:172.21.15.134:6789/0, reconnecting 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 >> v1:172.21.15.134:6789/0 conn(0x7fcc78142830 legacy=0x7fcc7814c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 --2- 172.21.15.110:0/3209782833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient(hunting): picked mon.b con 0x7fcc60003970 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 --2- 172.21.15.110:0/3209782833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc600047c0 0x7fcc60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient(hunting): picked mon.c con 0x7fcc600047c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 --2- 172.21.15.110:0/3209782833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc60005380 0x7fcc6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:10.684 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient(hunting): picked mon.a con 0x7fcc60005380 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:10.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:10.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6000c1e0 con 0x7fcc60005380 2024-01-20T11:15:10.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:10.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6000c350 con 0x7fcc60003970 2024-01-20T11:15:10.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:10.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6000c510 con 0x7fcc600047c0 2024-01-20T11:15:10.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:10.685 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.681+0000 7fcc75d9b640 10 monclient(hunting): _finish_auth 0 2024-01-20T11:15:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.683+0000 7fcc76d9d640 1 --2- 172.21.15.110:0/3209782833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.683+0000 7fcc76d9d640 10 monclient(hunting): get_auth_request con 0x7fcc60003970 auth_method 0 2024-01-20T11:15:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.683+0000 7fcc76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.683+0000 7fcc76d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.683+0000 7fcc76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:15:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.684+0000 7fcc76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.684+0000 7fcc76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.684+0000 7fcc76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:15:10.687 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 10 monclient(hunting): handle_auth_done global_id 4276 payload 293 2024-01-20T11:15:10.687 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 1 -- 172.21.15.110:0/3209782833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc600047c0 msgr2=0x7fcc60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:10.687 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 1 --2- 172.21.15.110:0/3209782833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc600047c0 0x7fcc60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:10.687 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 1 -- 172.21.15.110:0/3209782833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc60005380 msgr2=0x7fcc6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:10.687 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 1 --2- 172.21.15.110:0/3209782833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc60005380 0x7fcc6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:10.687 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:15:10.687 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 1 monclient: found mon.b 2024-01-20T11:15:10.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:10.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:10.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 1 -- 172.21.15.110:0/3209782833 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780b4040 con 0x7fcc60003970 2024-01-20T11:15:10.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 10 monclient: _finish_auth 0 2024-01-20T11:15:10.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 10 monclient: _check_auth_tickets 2024-01-20T11:15:10.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.685+0000 7fcc76d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:10.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.686+0000 7fcc7d24f640 5 monclient: authenticate success, global_id 4276 2024-01-20T11:15:10.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.686+0000 7fcc7d24f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:10.688 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.686+0000 7fcc7759e640 1 -- 172.21.15.110:0/3209782833 reap_dead start 2024-01-20T11:15:10.689 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.687+0000 7fcc76d9d640 1 --2- 172.21.15.110:0/3209782833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc60003d90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c001750 tx=0x7fcc600057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=257721c2cab6f373 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0052d0 con 0x7fcc60003970 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 10 monclient: dump: 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:10.692 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c02e050 con 0x7fcc60003970 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 1 -- 172.21.15.110:0/3209782833 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0035f0 con 0x7fcc60003970 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc75d9b640 10 monclient: dump: 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:10.693 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:10.694 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:10.694 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:10.694 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc7d24f640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:10.694 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.691+0000 7fcc7d24f640 4 set_mon_vals no callback set 2024-01-20T11:15:10.696 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.692+0000 7fcc7d24f640 10 monclient: shutdown 2024-01-20T11:15:10.696 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.692+0000 7fcc7d24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:10.696 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.692+0000 7fcc7d24f640 1 -- 172.21.15.110:0/3209782833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 msgr2=0x7fcc60003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:10.696 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.692+0000 7fcc7d24f640 1 --2- 172.21.15.110:0/3209782833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc60003d90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c001750 tx=0x7fcc600057f0 comp rx=0 tx=0).stop 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.692+0000 7fcc7d24f640 1 -- 172.21.15.110:0/3209782833 shutdown_connections 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.692+0000 7fcc7d24f640 1 --2- 172.21.15.110:0/3209782833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc60003d90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.692+0000 7fcc7d24f640 1 -- 172.21.15.110:0/3209782833 >> 172.21.15.110:0/3209782833 conn(0x7fcc78066f80 msgr2=0x7fcc780b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.692+0000 7fcc7d24f640 1 -- 172.21.15.110:0/3209782833 shutdown_connections 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.692+0000 7fcc7d24f640 1 -- 172.21.15.110:0/3209782833 wait complete. 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.693+0000 7fcc7d24f640 10 monclient: build_initial_monmap 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.693+0000 7fcc7d24f640 10 monclient: monmap: 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:10.697 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.693+0000 7fcc7d24f640 1 librados: starting msgr at 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.693+0000 7fcc7d24f640 1 librados: starting objecter 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.693+0000 7fcc7d24f640 1 Processor -- start 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.693+0000 7fcc7d24f640 1 -- start start 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.693+0000 7fcc7d24f640 1 librados: setting wanted keys 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.693+0000 7fcc7d24f640 1 librados: calling monclient init 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.693+0000 7fcc7d24f640 10 monclient: init 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc780b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient(hunting): picked mon.noname-b con 0x7fcc60003970 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc780b0850 0x7fcc78144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient(hunting): picked mon.noname-c con 0x7fcc780b0850 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc78145200 0x7fcc78136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:10.698 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient(hunting): picked mon.noname-a con 0x7fcc78145200 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc78145d30 con 0x7fcc78145200 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc78145680 con 0x7fcc60003970 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc781461c0 con 0x7fcc780b0850 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.694+0000 7fcc7d24f640 10 monclient(hunting): authenticate will time out at 1354.045730s 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc76d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc780b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc7759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc78145200 0x7fcc78136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc76d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc780b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:33648/0 (socket says 172.21.15.110:33648) 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc76d9d640 1 -- 172.21.15.110:0/892954366 learned_addr learned my addr 172.21.15.110:0/892954366 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc76d9d640 10 monclient(hunting): get_auth_request con 0x7fcc60003970 auth_method 0 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc76d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:10.699 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:10.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc7759e640 10 monclient(hunting): get_auth_request con 0x7fcc78145200 auth_method 0 2024-01-20T11:15:10.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:10.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc7759e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:10.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.695+0000 7fcc7759e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:10.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.696+0000 7fcc76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:10.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.696+0000 7fcc76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:10.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.696+0000 7fcc76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:10.700 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.696+0000 7fcc7759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.696+0000 7fcc7759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.696+0000 7fcc7759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 1 -- 172.21.15.110:0/892954366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc780b0850 msgr2=0x7fcc78144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc780b0850 0x7fcc78144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 1 -- 172.21.15.110:0/892954366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 msgr2=0x7fcc780b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc780b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 10 monclient: _finish_hunting 0 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 1 monclient: found mon.noname-a 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:10.701 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.699+0000 7fcc7759e640 1 -- 172.21.15.110:0/892954366 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780b12f0 con 0x7fcc78145200 2024-01-20T11:15:10.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.701+0000 7fcc7759e640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc78145200 0x7fcc78136940 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc780a86e0 tx=0x7fcc7000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:10.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.701+0000 7fcc5f7fe640 10 client.?.objecter ms_handle_connect 0x7fcc78145200 2024-01-20T11:15:10.702 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.701+0000 7fcc5f7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc5f7fe640 1 -- 172.21.15.110:0/892954366 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000ce50 con 0x7fcc78145200 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc5f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc5f7fe640 10 monclient: dump: 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:10.705 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc5f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc5f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc7d24f640 5 monclient: authenticate success, global_id 4251 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc7d24f640 10 monclient: _renew_subs 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc7d24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc78145890 con 0x7fcc78145200 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc7d24f640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc7d24f640 10 monclient: _renew_subs 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc7d24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc78137060 con 0x7fcc78145200 2024-01-20T11:15:10.706 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.704+0000 7fcc7d24f640 1 librados: init done 2024-01-20T11:15:10.707 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc7d24f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:10.707 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc7d24f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:10.707 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc7d24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:10.707 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc78067cc0 con 0x7fcc78145200 2024-01-20T11:15:10.707 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc5f7fe640 1 -- 172.21.15.110:0/892954366 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7000a610 con 0x7fcc78145200 2024-01-20T11:15:10.707 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc5f7fe640 1 -- 172.21.15.110:0/892954366 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000a9a0 con 0x7fcc78145200 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc5f7fe640 10 monclient: dump: 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:10.708 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:10.709 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:10.709 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.705+0000 7fcc7559a640 4 set_mon_vals no callback set 2024-01-20T11:15:10.782 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.781+0000 7fcc5f7fe640 1 -- 172.21.15.110:0/892954366 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 7) v1 ==== 89142+0+0 (secure 0 0 0) 0x7fcc7000ab00 con 0x7fcc78145200 2024-01-20T11:15:10.782 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.781+0000 7fcc5f7fe640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcc6406c7d0 0x7fcc6406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:10.783 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.782+0000 7fcc76d9d640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcc6406c7d0 0x7fcc6406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:10.783 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.782+0000 7fcc76d9d640 10 monclient: get_auth_request con 0x7fcc6406c7d0 auth_method 0 2024-01-20T11:15:10.783 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.782+0000 7fcc5f7fe640 10 client.4251.objecter ms_handle_connect 0x7fcc6406c7d0 2024-01-20T11:15:10.783 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.782+0000 7fcc76d9d640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcc6406c7d0 0x7fcc6406ec90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c001750 tx=0x7fcc6c009030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:10.791 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.790+0000 7fcc5f7fe640 1 -- 172.21.15.110:0/892954366 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2479+0+0 (secure 0 0 0) 0x7fcc700587b0 con 0x7fcc78145200 2024-01-20T11:15:10.791 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.790+0000 7fcc5f7fe640 10 client.4251.objecter ms_dispatch 0x7fcc78000b90 osd_map(15..15 src has 1..15) v4 2024-01-20T11:15:10.791 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.790+0000 7fcc5f7fe640 3 client.4251.objecter handle_osd_map got epochs [15,15] > 0 2024-01-20T11:15:10.792 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.790+0000 7fcc5f7fe640 3 client.4251.objecter handle_osd_map decoding full epoch 15 2024-01-20T11:15:10.792 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:10.790+0000 7fcc5f7fe640 20 client.4251.objecter dump_active .. 0 homeless 2024-01-20T11:15:10.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-20T11:15:10.788+0000 c494640 -1 osd.0 0 waiting for initial osdmap 2024-01-20T11:15:10.812 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-20T11:15:10.800+0000 c494640 -1 osd.2 0 waiting for initial osdmap 2024-01-20T11:15:10.841 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-20T11:15:10.836+0000 ac7e640 -1 osd.1 0 waiting for initial osdmap 2024-01-20T11:15:11.147 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.146+0000 7fcc5f7fe640 1 -- 172.21.15.110:0/892954366 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fcc700543f0 con 0x7fcc78145200 2024-01-20T11:15:11.147 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.146+0000 7fcc5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:11.147 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.146+0000 7fcc5f7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:11.214 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-20T11:15:11.212+0000 10253640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-20T11:15:11.310 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-20T11:15:11.288+0000 10253640 -1 osd.0 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-20T11:15:11.417 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-20T11:15:11.399+0000 10253640 -1 osd.1 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-20T11:15:11.476 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.475+0000 7fcc7d24f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:11.476 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.475+0000 7fcc7d24f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:11.476 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.475+0000 7fcc7d24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:11.476 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.475+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc7804e970 con 0x7fcc78145200 2024-01-20T11:15:11.549 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.548+0000 7fcc5f7fe640 1 -- 172.21.15.110:0/892954366 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+7567 (secure 0 0 0) 0x7fcc70057a10 con 0x7fcc78145200 2024-01-20T11:15:11.549 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.548+0000 7fcc5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:11.549 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.548+0000 7fcc5f7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:11.549 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-20T11:15:11.549 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":16,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:15:11.293704+0000","last_up_change":"2024-01-20T11:15:10.265610+0000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6801","nonce":2798305725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6803","nonce":2798305725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6807","nonce":2798305725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6805","nonce":2798305725}]},"public_addr":"172.21.15.134:6801/2798305725","cluster_addr":"172.21.15.134:6803/2798305725","heartbeat_back_addr":"172.21.15.134:6807/2798305725","heartbeat_front_addr":"172.21.15.134:6805/2798305725","state":["exists","up"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.550+0000 7fcc7d24f640 10 librados: watch_flush enter 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.550+0000 7fcc7d24f640 10 librados: watch_flush exit 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.550+0000 7fcc7d24f640 20 client.4251.objecter shutdown clearing up homeless session... 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.550+0000 7fcc7d24f640 10 client.4251.objecter successfully canceled tick 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.550+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcc6406c7d0 msgr2=0x7fcc6406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.550+0000 7fcc7d24f640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcc6406c7d0 0x7fcc6406ec90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c001750 tx=0x7fcc6c009030 comp rx=0 tx=0).stop 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.550+0000 7fcc7d24f640 10 monclient: shutdown 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.550+0000 7fcc7d24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc78145200 msgr2=0x7fcc78136940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc78145200 0x7fcc78136940 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc780a86e0 tx=0x7fcc7000cc00 comp rx=0 tx=0).stop 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 shutdown_connections 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcc6406c7d0 0x7fcc6406ec90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:11.552 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc78145200 0x7fcc78136940 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:11.553 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc780b0850 0x7fcc78144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:11.553 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 --2- 172.21.15.110:0/892954366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc60003970 0x7fcc780b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:11.553 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 >> 172.21.15.110:0/892954366 conn(0x7fcc78066f80 msgr2=0x7fcc7813cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:11.553 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 shutdown_connections 2024-01-20T11:15:11.556 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 -- 172.21.15.110:0/892954366 wait complete. 2024-01-20T11:15:11.556 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:11.551+0000 7fcc7d24f640 1 librados: shutdown 2024-01-20T11:15:11.566 DEBUG:teuthology.misc:1 of 8 OSDs are up 2024-01-20T11:15:11.668 INFO:tasks.ceph.osd.5.smithi134.stderr:2024-01-20T11:15:11.662+0000 ac7e640 -1 osd.5 0 waiting for initial osdmap 2024-01-20T11:15:12.188 INFO:tasks.ceph.osd.5.smithi134.stderr:2024-01-20T11:15:12.167+0000 10253640 -1 osd.5 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-20T11:15:17.678 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:15:17.633+0000 5dc2480 -1 bdev(0x164afaf0 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-01-20T11:15:17.679 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:15:17.678+0000 5dc2480 -1 bdev(0x164afaf0 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-01-20T11:15:17.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:15:17.738+0000 5dc2480 -1 bdev(0x18f1acd0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 3 2024-01-20T11:15:17.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:15:17.750+0000 5dc2480 -1 bdev(0x18f1acd0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-20T11:15:17.836 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:15:17.763+0000 5dc2480 -1 bdev(0x18f1acd0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-20T11:15:18.352 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:15:17.745+0000 5dc2480 -1 bdev(0x164afaf0 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-01-20T11:15:18.373 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-20T11:15:18.426 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:15:18.352+0000 5dc2480 -1 bdev(0x164afaf0 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-01-20T11:15:18.466 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.466+0000 7fbedade9640 10 monclient: get_monmap_and_config 2024-01-20T11:15:18.466 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.466+0000 7fbedade9640 10 monclient: build_initial_monmap 2024-01-20T11:15:18.466 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.466+0000 7fbedade9640 10 monclient: monmap: 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:15:18.467485+0000 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:15:18.467485+0000 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:18.467 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:18.470 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.467+0000 7fbedade9640 1 Processor -- start 2024-01-20T11:15:18.470 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.467+0000 7fbedade9640 1 -- start start 2024-01-20T11:15:18.470 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.467+0000 7fbedade9640 10 monclient: init 2024-01-20T11:15:18.470 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:18.470 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-01-20T11:15:18.470 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 0x7fbed413efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:18.470 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 10 monclient(hunting): picked mon.noname-b con 0x7fbed40a8a30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbed40a9c90 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbed413f4e0 0x7fbed4142810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 10 monclient(hunting): picked mon.noname-c con 0x7fbed413f4e0 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed40a8e10 con 0x7fbed40a9c90 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed40aacf0 con 0x7fbed413f4e0 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40aafd0 con 0x7fbed40a8a30 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbedade9640 10 monclient(hunting): authenticate will time out at 1361.818528s 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.468+0000 7fbed8b85640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 0x7fbed413efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed3fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbed40a9c90 0x7fbed4142100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37226/0 (socket says 172.21.15.110:37226) 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed3fff640 1 -- 172.21.15.110:0/2798840566 learned_addr learned my addr 172.21.15.110:0/2798840566 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:18.471 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed9386640 1 --2- 172.21.15.110:0/2798840566 >> v2:172.21.15.134:3300/0 conn(0x7fbed413f4e0 0x7fbed4142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed8b85640 10 monclient(hunting): get_auth_request con 0x7fbed40a8a30 auth_method 0 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed8b85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed8b85640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed8b85640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed9386640 10 monclient(hunting): get_auth_request con 0x7fbed413f4e0 auth_method 0 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed9386640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed9386640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.469+0000 7fbed9386640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.470+0000 7fbed8b85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.470+0000 7fbed8b85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.470+0000 7fbed8b85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.470+0000 7fbed9386640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:18.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.470+0000 7fbed9386640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:18.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.470+0000 7fbed9386640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.471+0000 7fbed37fe640 1 -- 172.21.15.110:0/2798840566 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266520870 0 0) 0x7fbed40a8e10 con 0x7fbed40a9c90 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.471+0000 7fbed37fe640 10 monclient(hunting): _negotiate 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.471+0000 7fbed37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.471+0000 7fbed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.471+0000 7fbed37fe640 10 monclient(hunting): my global_id is 4254 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.471+0000 7fbed37fe640 1 -- 172.21.15.110:0/2798840566 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbec4003520 con 0x7fbed40a9c90 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.472+0000 7fbed9386640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.472+0000 7fbed9386640 1 -- 172.21.15.110:0/2798840566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 msgr2=0x7fbed413efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.472+0000 7fbed9386640 1 --2- 172.21.15.110:0/2798840566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 0x7fbed413efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.472+0000 7fbed9386640 1 -- 172.21.15.110:0/2798840566 >> v1:172.21.15.110:6789/0 conn(0x7fbed40a9c90 legacy=0x7fbed4142100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:18.474 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.472+0000 7fbed9386640 10 monclient: _finish_hunting 0 2024-01-20T11:15:18.475 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.472+0000 7fbed9386640 1 monclient: found mon.noname-c 2024-01-20T11:15:18.475 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.472+0000 7fbed9386640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:18.475 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.472+0000 7fbed9386640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:15:18.475 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.472+0000 7fbed9386640 1 -- 172.21.15.110:0/2798840566 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed40a90c0 con 0x7fbed413f4e0 2024-01-20T11:15:18.475 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.474+0000 7fbed9386640 1 --2- 172.21.15.110:0/2798840566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed413f4e0 0x7fbed4142810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009d90 tx=0x7fbecc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14cb1b4d71aee3e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:18.478 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 1 -- 172.21.15.110:0/2798840566 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc02cc50 con 0x7fbed413f4e0 2024-01-20T11:15:18.478 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:18.478 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:18.478 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 10 monclient: dump: 2024-01-20T11:15:18.478 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:18.478 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:18.478 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:18.478 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 1 -- 172.21.15.110:0/2798840566 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc02cdb0 con 0x7fbed413f4e0 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbedade9640 5 monclient: authenticate success, global_id 4282 2024-01-20T11:15:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbedade9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbedade9640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 1 -- 172.21.15.110:0/2798840566 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc034660 con 0x7fbed413f4e0 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbedade9640 4 set_mon_vals no callback set 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.478+0000 7fbed37fe640 10 monclient: dump: 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:18.480 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:18.481 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:18.481 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:18.481 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:18.481 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:18.485 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 10 monclient: shutdown 2024-01-20T11:15:18.485 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:18.486 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 1 -- 172.21.15.110:0/2798840566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed413f4e0 msgr2=0x7fbed4142810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:18.486 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 1 --2- 172.21.15.110:0/2798840566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed413f4e0 0x7fbed4142810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009d90 tx=0x7fbecc02bea0 comp rx=0 tx=0).stop 2024-01-20T11:15:18.486 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 1 -- 172.21.15.110:0/2798840566 shutdown_connections 2024-01-20T11:15:18.486 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 1 --2- 172.21.15.110:0/2798840566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed413f4e0 0x7fbed4142810 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:18.486 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 1 --2- 172.21.15.110:0/2798840566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 0x7fbed413efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:18.486 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 1 -- 172.21.15.110:0/2798840566 >> 172.21.15.110:0/2798840566 conn(0x7fbed40b4320 msgr2=0x7fbed40b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:18.486 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 1 -- 172.21.15.110:0/2798840566 shutdown_connections 2024-01-20T11:15:18.486 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 1 -- 172.21.15.110:0/2798840566 wait complete. 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 10 monclient: build_initial_monmap 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.479+0000 7fbedade9640 10 monclient: monmap: 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:18.487 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.480+0000 7fbedade9640 1 librados: starting msgr at 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.480+0000 7fbedade9640 1 librados: starting objecter 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.480+0000 7fbedade9640 1 Processor -- start 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.480+0000 7fbedade9640 1 -- start start 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.480+0000 7fbedade9640 1 librados: setting wanted keys 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.480+0000 7fbedade9640 1 librados: calling monclient init 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.480+0000 7fbedade9640 10 monclient: init 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 0x7fbed4139c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient(hunting): picked mon.noname-c con 0x7fbed40a8a30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed413f4e0 0x7fbed40b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient(hunting): picked mon.noname-a con 0x7fbed413f4e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:18.488 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed41da950 0x7fbed40b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient(hunting): picked mon.noname-b con 0x7fbed41da950 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40b76f0 con 0x7fbed413f4e0 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed4143ca0 con 0x7fbed41da950 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed4143f80 con 0x7fbed40a8a30 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbedade9640 10 monclient(hunting): authenticate will time out at 1361.831346s 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbed3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed413f4e0 0x7fbed40b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbed8b85640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 0x7fbed4139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.481+0000 7fbed9386640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed41da950 0x7fbed40b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:18.489 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed413f4e0 0x7fbed40b6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:41406/0 (socket says 172.21.15.110:41406) 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed3fff640 1 -- 172.21.15.110:0/2863241280 learned_addr learned my addr 172.21.15.110:0/2863241280 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed3fff640 10 monclient(hunting): get_auth_request con 0x7fbed413f4e0 auth_method 0 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed9386640 10 monclient(hunting): get_auth_request con 0x7fbed41da950 auth_method 0 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed9386640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed9386640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed9386640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed8b85640 10 monclient(hunting): get_auth_request con 0x7fbed40a8a30 auth_method 0 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed8b85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed8b85640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed8b85640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed9386640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:18.490 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed9386640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed9386640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.482+0000 7fbed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.483+0000 7fbed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.483+0000 7fbed8b85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.483+0000 7fbed8b85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.483+0000 7fbed8b85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 10 monclient(hunting): handle_auth_done global_id 4285 payload 824 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 1 -- 172.21.15.110:0/2863241280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 msgr2=0x7fbed4139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 0x7fbed4139c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 1 -- 172.21.15.110:0/2863241280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed413f4e0 msgr2=0x7fbed40b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed413f4e0 0x7fbed40b6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 10 monclient: _finish_hunting 0 2024-01-20T11:15:18.491 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 1 monclient: found mon.noname-b 2024-01-20T11:15:18.492 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:18.492 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:18.492 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.485+0000 7fbed9386640 1 -- 172.21.15.110:0/2863241280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc009a70 con 0x7fbed41da950 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.487+0000 7fbed9386640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed41da950 0x7fbed40b7b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbecc033000 tx=0x7fbecc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.487+0000 7fbed17fa640 10 client.?.objecter ms_handle_connect 0x7fbed41da950 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.487+0000 7fbed17fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 1 -- 172.21.15.110:0/2863241280 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc040650 con 0x7fbed41da950 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 10 monclient: dump: 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:18.494 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 10 monclient: _finish_auth 0 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 10 monclient: _check_auth_tickets 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 1 -- 172.21.15.110:0/2863241280 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc0407b0 con 0x7fbed41da950 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbedade9640 5 monclient: authenticate success, global_id 4285 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbedade9640 10 monclient: _renew_subs 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbedade9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed40b7390 con 0x7fbed41da950 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:18.495 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 1 -- 172.21.15.110:0/2863241280 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc040b60 con 0x7fbed41da950 2024-01-20T11:15:18.496 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:18.496 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:18.496 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed17fa640 10 monclient: dump: 2024-01-20T11:15:18.496 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:18.496 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:18.496 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:18.496 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:18.496 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:18.496 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbed27fc640 4 set_mon_vals no callback set 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbedade9640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbedade9640 10 monclient: _renew_subs 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbedade9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed40b82a0 con 0x7fbed41da950 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.491+0000 7fbedade9640 1 librados: init done 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.492+0000 7fbedade9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.492+0000 7fbedade9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.492+0000 7fbedade9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:18.497 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.492+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed40b1ce0 con 0x7fbed41da950 2024-01-20T11:15:18.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:15:17.837+0000 5dc2480 -1 bdev(0x18f1acd0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-20T11:15:18.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:15:18.168+0000 5dc2480 -1 bdev(0x18f1acd0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-20T11:15:18.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:15:18.262+0000 5dc2480 -1 bdev(0x18f1acd0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-20T11:15:18.550 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:15:18.427+0000 5dc2480 -1 bdev(0x164afaf0 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-01-20T11:15:18.665 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:15:18.640+0000 5dc2480 -1 bdev(0x164c0c80 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-20T11:15:18.781 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.781+0000 7fbed17fa640 1 -- 172.21.15.110:0/2863241280 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 7) v1 ==== 89142+0+0 (secure 0 0 0) 0x7fbecc040cc0 con 0x7fbed41da950 2024-01-20T11:15:18.782 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.782+0000 7fbed17fa640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbeb806c850 0x7fbeb806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:18.782 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.782+0000 7fbed8b85640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbeb806c850 0x7fbeb806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:18.782 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.782+0000 7fbed8b85640 10 monclient: get_auth_request con 0x7fbeb806c850 auth_method 0 2024-01-20T11:15:18.782 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.783+0000 7fbed17fa640 10 client.4285.objecter ms_handle_connect 0x7fbeb806c850 2024-01-20T11:15:18.782 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.783+0000 7fbed8b85640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbeb806c850 0x7fbeb806ed10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009b40 tx=0x7fbec8009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:18.785 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.785+0000 7fbed17fa640 1 -- 172.21.15.110:0/2863241280 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4017+0+0 (secure 0 0 0) 0x7fbecc07b240 con 0x7fbed41da950 2024-01-20T11:15:18.785 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.785+0000 7fbed17fa640 10 client.4285.objecter ms_dispatch 0x7fbed4000b90 osd_map(20..20 src has 1..20) v4 2024-01-20T11:15:18.785 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.785+0000 7fbed17fa640 3 client.4285.objecter handle_osd_map got epochs [20,20] > 0 2024-01-20T11:15:18.785 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.785+0000 7fbed17fa640 3 client.4285.objecter handle_osd_map decoding full epoch 20 2024-01-20T11:15:18.785 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.785+0000 7fbed17fa640 20 client.4285.objecter dump_active .. 0 homeless 2024-01-20T11:15:18.796 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.796+0000 7fbed17fa640 1 -- 172.21.15.110:0/2863241280 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fbecc077580 con 0x7fbed41da950 2024-01-20T11:15:18.796 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.796+0000 7fbed17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:18.796 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.796+0000 7fbed17fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:18.963 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:15:18.666+0000 5dc2480 -1 bdev(0x164c0c80 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 3 2024-01-20T11:15:18.964 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:15:18.775+0000 5dc2480 -1 bdev(0x164c0c80 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-20T11:15:18.977 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.977+0000 7fbedade9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:18.977 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.977+0000 7fbedade9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:18.977 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.977+0000 7fbedade9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:18.977 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:18.977+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbed40b7860 con 0x7fbed41da950 2024-01-20T11:15:19.067 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.067+0000 7fbed17fa640 1 -- 172.21.15.110:0/2863241280 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+11372 (secure 0 0 0) 0x7fbecc009450 con 0x7fbed41da950 2024-01-20T11:15:19.067 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.067+0000 7fbed17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:19.067 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.067+0000 7fbed17fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:19.067 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-20T11:15:19.068 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":20,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:15:17.116062+0000","last_up_change":"2024-01-20T11:15:13.343599+0000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-20T11:15:13.023751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6817","nonce":3175427153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6819","nonce":3175427153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6825","nonce":3175427153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6822","nonce":3175427153}]},"public_addr":"172.21.15.110:6817/3175427153","cluster_addr":"172.21.15.110:6819/3175427153","heartbeat_back_addr":"172.21.15.110:6825/3175427153","heartbeat_front_addr":"172.21.15.110:6822/3175427153","state":["exists","up"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6801","nonce":3373744578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6803","nonce":3373744578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6807","nonce":3373744578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6805","nonce":3373744578}]},"public_addr":"172.21.15.110:6801/3373744578","cluster_addr":"172.21.15.110:6803/3373744578","heartbeat_back_addr":"172.21.15.110:6807/3373744578","heartbeat_front_addr":"172.21.15.110:6805/3373744578","state":["exists","up"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6809","nonce":470946803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6811","nonce":470946803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6815","nonce":470946803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6813","nonce":470946803}]},"public_addr":"172.21.15.110:6809/470946803","cluster_addr":"172.21.15.110:6811/470946803","heartbeat_back_addr":"172.21.15.110:6815/470946803","heartbeat_front_addr":"172.21.15.110:6813/470946803","state":["exists","up"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6819","nonce":1586889676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6821","nonce":1586889676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6828","nonce":1586889676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6823","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6825","nonce":1586889676}]},"public_addr":"172.21.15.134:6819/1586889676","cluster_addr":"172.21.15.134:6821/1586889676","heartbeat_back_addr":"172.21.15.134:6828/1586889676","heartbeat_front_addr":"172.21.15.134:6825/1586889676","state":["exists","up"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6801","nonce":2798305725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6803","nonce":2798305725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6807","nonce":2798305725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6805","nonce":2798305725}]},"public_addr":"172.21.15.134:6801/2798305725","cluster_addr":"172.21.15.134:6803/2798305725","heartbeat_back_addr":"172.21.15.134:6807/2798305725","heartbeat_front_addr":"172.21.15.134:6805/2798305725","state":["exists","up"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.482016+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.271832+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:08.974739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:10.303347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:08.463225+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:15:19.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 10 librados: watch_flush enter 2024-01-20T11:15:19.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 10 librados: watch_flush exit 2024-01-20T11:15:19.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 20 client.4285.objecter shutdown clearing up homeless session... 2024-01-20T11:15:19.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 10 client.4285.objecter successfully canceled tick 2024-01-20T11:15:19.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbeb806c850 msgr2=0x7fbeb806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:19.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbeb806c850 0x7fbeb806ed10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009b40 tx=0x7fbec8009190 comp rx=0 tx=0).stop 2024-01-20T11:15:19.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 10 monclient: shutdown 2024-01-20T11:15:19.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:19.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed41da950 msgr2=0x7fbed40b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed41da950 0x7fbed40b7b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbecc033000 tx=0x7fbecc004a40 comp rx=0 tx=0).stop 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 shutdown_connections 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbeb806c850 0x7fbeb806ed10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed41da950 0x7fbed40b7b80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed413f4e0 0x7fbed40b6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 --2- 172.21.15.110:0/2863241280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40a8a30 0x7fbed4139c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 >> 172.21.15.110:0/2863241280 conn(0x7fbed40b4320 msgr2=0x7fbed413af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 shutdown_connections 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 -- 172.21.15.110:0/2863241280 wait complete. 2024-01-20T11:15:19.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:19.070+0000 7fbedade9640 1 librados: shutdown 2024-01-20T11:15:19.083 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-01-20T11:15:19.391 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:15:18.963+0000 5dc2480 -1 bdev(0x164c0c80 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-20T11:15:19.616 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:15:19.612+0000 ac7e640 -1 osd.7 0 waiting for initial osdmap 2024-01-20T11:15:19.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:15:19.654+0000 c494640 -1 osd.3 0 waiting for initial osdmap 2024-01-20T11:15:20.248 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:15:20.235+0000 1425b640 -1 osd.3 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-20T11:15:20.283 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:15:20.270+0000 1425b640 -1 osd.7 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-20T11:15:20.575 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:15:20.572+0000 ac7e640 -1 osd.4 0 waiting for initial osdmap 2024-01-20T11:15:21.048 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:15:21.035+0000 1425b640 -1 osd.4 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-20T11:15:25.889 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-20T11:15:25.978 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.978+0000 7fbaf8489640 10 monclient: get_monmap_and_config 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.978+0000 7fbaf8489640 10 monclient: build_initial_monmap 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.978+0000 7fbaf8489640 10 monclient: monmap: 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:15:25.979124+0000 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:15:25.979124+0000 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:25.979 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:25.980 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:25.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 1 Processor -- start 2024-01-20T11:15:25.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 1 -- start start 2024-01-20T11:15:25.980 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient: init 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbaf00a89c0 0x7fbaf013f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient(hunting): picked mon.noname-a con 0x7fbaf00a89c0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbaf00a8da0 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbaf013f730 0x7fbaf0142a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient(hunting): picked mon.noname-c con 0x7fbaf013f730 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf00b21b0 con 0x7fbaf00a8da0 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf00aa220 con 0x7fbaf00a89c0 2024-01-20T11:15:25.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf00a9f20 con 0x7fbaf013f730 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.979+0000 7fbaf8489640 10 monclient(hunting): authenticate will time out at 1369.330135s 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.980+0000 7fbaf6225640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbaf00a89c0 0x7fbaf013f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.980+0000 7fbaf6a26640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbaf013f730 0x7fbaf0142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.980+0000 7fbaf6225640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbaf00a89c0 0x7fbaf013f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39072/0 (socket says 172.21.15.110:39072) 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.980+0000 7fbaf6225640 1 -- 172.21.15.110:0/211688295 learned_addr learned my addr 172.21.15.110:0/211688295 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.980+0000 7fbaf6225640 10 monclient(hunting): get_auth_request con 0x7fbaf00a89c0 auth_method 0 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.980+0000 7fbaf6225640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.980+0000 7fbaf6225640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.980+0000 7fbaf6225640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.981+0000 7fbaf6225640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.981+0000 7fbaf6225640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:25.983 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.981+0000 7fbaf6225640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:25.984 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.981+0000 7fbaf6a26640 10 monclient(hunting): get_auth_request con 0x7fbaf013f730 auth_method 0 2024-01-20T11:15:25.984 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.981+0000 7fbaf6a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:25.984 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.981+0000 7fbaf6a26640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:25.984 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.981+0000 7fbaf6a26640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:25.984 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.982+0000 7fbaf6a26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:25.984 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.982+0000 7fbaf6a26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:25.984 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.982+0000 7fbaf6a26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.983+0000 7fbaf6225640 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.983+0000 7fbaf6225640 1 -- 172.21.15.110:0/211688295 >> v2:172.21.15.134:3300/0 conn(0x7fbaf013f730 msgr2=0x7fbaf0142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.983+0000 7fbaf6225640 1 --2- 172.21.15.110:0/211688295 >> v2:172.21.15.134:3300/0 conn(0x7fbaf013f730 0x7fbaf0142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.983+0000 7fbaf6225640 1 -- 172.21.15.110:0/211688295 >> v1:172.21.15.134:6789/0 conn(0x7fbaf00a8da0 legacy=0x7fbaf01422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.983+0000 7fbaf6225640 10 monclient: _finish_hunting 0 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.983+0000 7fbaf6225640 1 monclient: found mon.noname-a 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.983+0000 7fbaf6225640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.983+0000 7fbaf6225640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.983+0000 7fbaf6225640 1 -- 172.21.15.110:0/211688295 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00a9760 con 0x7fbaf00a89c0 2024-01-20T11:15:25.985 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.985+0000 7fbaf6225640 1 --2- 172.21.15.110:0/211688295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 0x7fbaf013f1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbaf01464c0 tx=0x7fbaec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261037c84d3c2227 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:25.988 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 1 -- 172.21.15.110:0/211688295 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec0317e0 con 0x7fbaf00a89c0 2024-01-20T11:15:25.988 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:25.988 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 10 monclient: dump: 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:25.989 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 10 monclient: _finish_auth 0 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 10 monclient: _check_auth_tickets 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 1 -- 172.21.15.110:0/211688295 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec031940 con 0x7fbaf00a89c0 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 1 -- 172.21.15.110:0/211688295 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec031cd0 con 0x7fbaf00a89c0 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf8489640 5 monclient: authenticate success, global_id 4269 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf8489640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf8489640 4 set_mon_vals no callback set 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.988+0000 7fbaf5223640 10 monclient: dump: 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:25.990 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:25.991 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:25.991 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:25.991 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:25.991 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:25.991 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:25.992 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:25.996 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 10 monclient: shutdown 2024-01-20T11:15:25.996 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:25.996 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 1 -- 172.21.15.110:0/211688295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 msgr2=0x7fbaf013f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:25.996 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 1 --2- 172.21.15.110:0/211688295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 0x7fbaf013f1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbaf01464c0 tx=0x7fbaec02b880 comp rx=0 tx=0).stop 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 1 -- 172.21.15.110:0/211688295 shutdown_connections 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 1 --2- 172.21.15.110:0/211688295 >> v2:172.21.15.134:3300/0 conn(0x7fbaf013f730 0x7fbaf0142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 1 --2- 172.21.15.110:0/211688295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 0x7fbaf013f1f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 1 -- 172.21.15.110:0/211688295 >> 172.21.15.110:0/211688295 conn(0x7fbaf0066f80 msgr2=0x7fbaf00b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 1 -- 172.21.15.110:0/211688295 shutdown_connections 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.989+0000 7fbaf8489640 1 -- 172.21.15.110:0/211688295 wait complete. 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.990+0000 7fbaf8489640 10 monclient: build_initial_monmap 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.990+0000 7fbaf8489640 10 monclient: monmap: 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:25.997 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.990+0000 7fbaf8489640 1 librados: starting msgr at 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.990+0000 7fbaf8489640 1 librados: starting objecter 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.990+0000 7fbaf8489640 1 Processor -- start 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.990+0000 7fbaf8489640 1 -- start start 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.990+0000 7fbaf8489640 1 librados: setting wanted keys 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.990+0000 7fbaf8489640 1 librados: calling monclient init 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.990+0000 7fbaf8489640 10 monclient: init 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:25.998 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 0x7fbaf0139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient(hunting): picked mon.noname-a con 0x7fbaf00a89c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaf00a8da0 0x7fbaf0137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient(hunting): picked mon.noname-c con 0x7fbaf00a8da0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaf013f730 0x7fbaf01dabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient(hunting): picked mon.noname-b con 0x7fbaf013f730 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf00ab7a0 con 0x7fbaf00a89c0 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf013ab40 con 0x7fbaf013f730 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf013bdf0 con 0x7fbaf00a8da0 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf8489640 10 monclient(hunting): authenticate will time out at 1369.341627s 2024-01-20T11:15:25.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf6225640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 0x7fbaf0139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.991+0000 7fbaf5a24640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaf00a8da0 0x7fbaf0137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6a26640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaf013f730 0x7fbaf01dabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6225640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 0x7fbaf0139d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39080/0 (socket says 172.21.15.110:39080) 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6225640 1 -- 172.21.15.110:0/3276035824 learned_addr learned my addr 172.21.15.110:0/3276035824 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6225640 10 monclient(hunting): get_auth_request con 0x7fbaf00a89c0 auth_method 0 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6225640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6225640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6225640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf5a24640 10 monclient(hunting): get_auth_request con 0x7fbaf00a8da0 auth_method 0 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf5a24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf5a24640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf5a24640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6a26640 10 monclient(hunting): get_auth_request con 0x7fbaf013f730 auth_method 0 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:26.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6a26640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6a26640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6225640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6225640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6225640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6a26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6a26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf6a26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf5a24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf5a24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.992+0000 7fbaf5a24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 10 monclient(hunting): handle_auth_done global_id 4297 payload 824 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 1 -- 172.21.15.110:0/3276035824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaf00a8da0 msgr2=0x7fbaf0137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaf00a8da0 0x7fbaf0137430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 1 -- 172.21.15.110:0/3276035824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 msgr2=0x7fbaf0139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 0x7fbaf0139d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 10 monclient: _finish_hunting 0 2024-01-20T11:15:26.002 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 1 monclient: found mon.noname-b 2024-01-20T11:15:26.002 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:26.002 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:26.002 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.995+0000 7fbaf6a26640 1 -- 172.21.15.110:0/3276035824 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec0094d0 con 0x7fbaf013f730 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.997+0000 7fbaf6a26640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaf013f730 0x7fbaf01dabd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbae800eba0 tx=0x7fbae800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.997+0000 7fbae6ffd640 10 client.?.objecter ms_handle_connect 0x7fbaf013f730 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:25.997+0000 7fbae6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 1 -- 172.21.15.110:0/3276035824 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae800cd20 con 0x7fbaf013f730 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 10 monclient: dump: 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.003 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 1 -- 172.21.15.110:0/3276035824 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae800f040 con 0x7fbaf013f730 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 1 -- 172.21.15.110:0/3276035824 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae80146c0 con 0x7fbaf013f730 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:26.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbae6ffd640 10 monclient: dump: 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf8489640 5 monclient: authenticate success, global_id 4297 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf4a22640 4 set_mon_vals no callback set 2024-01-20T11:15:26.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf8489640 10 monclient: _renew_subs 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf8489640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf013a810 con 0x7fbaf013f730 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf8489640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf8489640 10 monclient: _renew_subs 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf8489640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf013b240 con 0x7fbaf013f730 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.001+0000 7fbaf8489640 1 librados: init done 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.002+0000 7fbaf8489640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.002+0000 7fbaf8489640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.002+0000 7fbaf8489640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:26.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.002+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf0067cc0 con 0x7fbaf013f730 2024-01-20T11:15:26.245 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.245+0000 7fbae6ffd640 1 -- 172.21.15.110:0/3276035824 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fbae8014820 con 0x7fbaf013f730 2024-01-20T11:15:26.246 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.245+0000 7fbae6ffd640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbad406c8b0 0x7fbad406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:26.246 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.246+0000 7fbaf6225640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbad406c8b0 0x7fbad406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.246 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.246+0000 7fbaf6225640 10 monclient: get_auth_request con 0x7fbad406c8b0 auth_method 0 2024-01-20T11:15:26.246 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.246+0000 7fbae6ffd640 10 client.4297.objecter ms_handle_connect 0x7fbad406c8b0 2024-01-20T11:15:26.246 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.246+0000 7fbaf6225640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbad406c8b0 0x7fbad406ed70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbaec02b8c0 tx=0x7fbaec03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:26.249 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.249+0000 7fbae6ffd640 1 -- 172.21.15.110:0/3276035824 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbae8059420 con 0x7fbaf013f730 2024-01-20T11:15:26.249 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.249+0000 7fbae6ffd640 10 client.4297.objecter ms_dispatch 0x7fbaf0000b90 osd_map(23..23 src has 1..23) v4 2024-01-20T11:15:26.250 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.249+0000 7fbae6ffd640 3 client.4297.objecter handle_osd_map got epochs [23,23] > 0 2024-01-20T11:15:26.250 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.249+0000 7fbae6ffd640 3 client.4297.objecter handle_osd_map decoding full epoch 23 2024-01-20T11:15:26.250 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.249+0000 7fbae6ffd640 20 client.4297.objecter dump_active .. 0 homeless 2024-01-20T11:15:26.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.260+0000 7fbae6ffd640 1 -- 172.21.15.110:0/3276035824 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fbae8055760 con 0x7fbaf013f730 2024-01-20T11:15:26.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.260+0000 7fbae6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:26.260 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.260+0000 7fbae6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:26.435 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.435+0000 7fbaf8489640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:26.469 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.435+0000 7fbaf8489640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:26.469 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.435+0000 7fbaf8489640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:26.469 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.435+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaf004e970 con 0x7fbaf013f730 2024-01-20T11:15:26.469 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.460+0000 7fbae6ffd640 1 -- 172.21.15.110:0/3276035824 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+12887 (secure 0 0 0) 0x7fbae8018020 con 0x7fbaf013f730 2024-01-20T11:15:26.470 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.460+0000 7fbae6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:26.470 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.460+0000 7fbae6ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:26.470 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-20T11:15:26.470 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":23,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:15:23.961408+0000","last_up_change":"2024-01-20T11:15:22.093857+0000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-20T11:15:13.023751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6817","nonce":3175427153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6819","nonce":3175427153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6825","nonce":3175427153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6822","nonce":3175427153}]},"public_addr":"172.21.15.110:6817/3175427153","cluster_addr":"172.21.15.110:6819/3175427153","heartbeat_back_addr":"172.21.15.110:6825/3175427153","heartbeat_front_addr":"172.21.15.110:6822/3175427153","state":["exists","up"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6801","nonce":3373744578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6803","nonce":3373744578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6807","nonce":3373744578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6805","nonce":3373744578}]},"public_addr":"172.21.15.110:6801/3373744578","cluster_addr":"172.21.15.110:6803/3373744578","heartbeat_back_addr":"172.21.15.110:6807/3373744578","heartbeat_front_addr":"172.21.15.110:6805/3373744578","state":["exists","up"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6809","nonce":470946803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6811","nonce":470946803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6815","nonce":470946803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6813","nonce":470946803}]},"public_addr":"172.21.15.110:6809/470946803","cluster_addr":"172.21.15.110:6811/470946803","heartbeat_back_addr":"172.21.15.110:6815/470946803","heartbeat_front_addr":"172.21.15.110:6813/470946803","state":["exists","up"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6821","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6823","nonce":1038822226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6827","nonce":1038822226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6831","nonce":1038822226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6829","nonce":1038822226}]},"public_addr":"172.21.15.110:6823/1038822226","cluster_addr":"172.21.15.110:6827/1038822226","heartbeat_back_addr":"172.21.15.110:6831/1038822226","heartbeat_front_addr":"172.21.15.110:6829/1038822226","state":["exists","up"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6824","nonce":2594530523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6827","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6829","nonce":2594530523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6833","nonce":2594530523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6831","nonce":2594530523}]},"public_addr":"172.21.15.134:6824/2594530523","cluster_addr":"172.21.15.134:6829/2594530523","heartbeat_back_addr":"172.21.15.134:6833/2594530523","heartbeat_front_addr":"172.21.15.134:6831/2594530523","state":["exists","up"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6819","nonce":1586889676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6821","nonce":1586889676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6828","nonce":1586889676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6823","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6825","nonce":1586889676}]},"public_addr":"172.21.15.134:6819/1586889676","cluster_addr":"172.21.15.134:6821/1586889676","heartbeat_back_addr":"172.21.15.134:6828/1586889676","heartbeat_front_addr":"172.21.15.134:6825/1586889676","state":["exists","up"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6801","nonce":2798305725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6803","nonce":2798305725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6807","nonce":2798305725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6805","nonce":2798305725}]},"public_addr":"172.21.15.134:6801/2798305725","cluster_addr":"172.21.15.134:6803/2798305725","heartbeat_back_addr":"172.21.15.134:6807/2798305725","heartbeat_front_addr":"172.21.15.134:6805/2798305725","state":["exists","up"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6809","nonce":604027796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6811","nonce":604027796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6815","nonce":604027796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6813","nonce":604027796}]},"public_addr":"172.21.15.134:6809/604027796","cluster_addr":"172.21.15.134:6811/604027796","heartbeat_back_addr":"172.21.15.134:6815/604027796","heartbeat_front_addr":"172.21.15.134:6813/604027796","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.482016+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.271832+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:08.974739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.596210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:10.058818+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:10.303347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:08.463225+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.463577+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:15:26.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 10 librados: watch_flush enter 2024-01-20T11:15:26.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 10 librados: watch_flush exit 2024-01-20T11:15:26.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 20 client.4297.objecter shutdown clearing up homeless session... 2024-01-20T11:15:26.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 10 client.4297.objecter successfully canceled tick 2024-01-20T11:15:26.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbad406c8b0 msgr2=0x7fbad406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:26.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbad406c8b0 0x7fbad406ed70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbaec02b8c0 tx=0x7fbaec03d040 comp rx=0 tx=0).stop 2024-01-20T11:15:26.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 10 monclient: shutdown 2024-01-20T11:15:26.472 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaf013f730 msgr2=0x7fbaf01dabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaf013f730 0x7fbaf01dabd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbae800eba0 tx=0x7fbae800c490 comp rx=0 tx=0).stop 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 shutdown_connections 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbad406c8b0 0x7fbad406ed70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaf013f730 0x7fbaf01dabd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaf00a8da0 0x7fbaf0137430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 --2- 172.21.15.110:0/3276035824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaf00a89c0 0x7fbaf0139d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 >> 172.21.15.110:0/3276035824 conn(0x7fbaf0066f80 msgr2=0x7fbaf01335e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 shutdown_connections 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 -- 172.21.15.110:0/3276035824 wait complete. 2024-01-20T11:15:26.473 INFO:teuthology.misc.health.smithi110.stderr:2024-01-20T11:15:26.462+0000 7fbaf8489640 1 librados: shutdown 2024-01-20T11:15:26.474 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-20T11:15:26.474 INFO:tasks.ceph:Creating RBD pool 2024-01-20T11:15:26.474 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-20T11:15:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.572+0000 7fbb46cf2640 10 monclient: get_monmap_and_config 2024-01-20T11:15:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.572+0000 7fbb46cf2640 10 monclient: build_initial_monmap 2024-01-20T11:15:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.572+0000 7fbb46cf2640 10 monclient: monmap: 2024-01-20T11:15:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:26.573004+0000 2024-01-20T11:15:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:26.573004+0000 2024-01-20T11:15:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:26.573 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 1 Processor -- start 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 1 -- start start 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient: init 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4006a860 0x7fbb40070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient(hunting): picked mon.noname-b con 0x7fbb4006a860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbb4013d1f0 0x7fbb4013d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient(hunting): picked mon.noname-c con 0x7fbb4013d1f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb40143310 0x7fbb401437a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient(hunting): picked mon.noname-a con 0x7fbb40143310 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4006ac40 con 0x7fbb40143310 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb40071000 con 0x7fbb4013d1f0 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb40072460 con 0x7fbb4006a860 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.573+0000 7fbb46cf2640 10 monclient(hunting): authenticate will time out at 1369.924050s 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb44a8e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4006a860 0x7fbb40070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb4528f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb40143310 0x7fbb401437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb3ffff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbb4013d1f0 0x7fbb4013d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb44a8e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4006a860 0x7fbb40070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43420/0 (socket says 172.21.15.110:43420) 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb44a8e640 1 -- 172.21.15.110:0/2942011522 learned_addr learned my addr 172.21.15.110:0/2942011522 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb4528f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb40143310 0x7fbb401437a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39084/0 (socket says 172.21.15.110:39084) 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb44a8e640 10 monclient(hunting): get_auth_request con 0x7fbb4006a860 auth_method 0 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb44a8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:26.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb44a8e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb44a8e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb4528f640 10 monclient(hunting): get_auth_request con 0x7fbb40143310 auth_method 0 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb4528f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb4528f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.574+0000 7fbb4528f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb3ffff640 10 monclient(hunting): get_auth_request con 0x7fbb4013d1f0 auth_method 0 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb3ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb44a8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb44a8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb44a8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:26.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb4528f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb4528f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb4528f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.575+0000 7fbb3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 1 -- 172.21.15.110:0/2942011522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4006a860 msgr2=0x7fbb40070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 1 --2- 172.21.15.110:0/2942011522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4006a860 0x7fbb40070ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 1 -- 172.21.15.110:0/2942011522 >> v2:172.21.15.134:3300/0 conn(0x7fbb4013d1f0 msgr2=0x7fbb4013d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 1 --2- 172.21.15.110:0/2942011522 >> v2:172.21.15.134:3300/0 conn(0x7fbb4013d1f0 0x7fbb4013d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 10 monclient: _finish_hunting 0 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 1 monclient: found mon.noname-a 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.577+0000 7fbb4528f640 1 -- 172.21.15.110:0/2942011522 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb40072740 con 0x7fbb40143310 2024-01-20T11:15:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.579+0000 7fbb4528f640 1 --2- 172.21.15.110:0/2942011522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 0x7fbb401437a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbb400b5890 tx=0x7fbb3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ca638bf5742e07 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 1 -- 172.21.15.110:0/2942011522 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30031830 con 0x7fbb40143310 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 10 monclient: dump: 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:26.583 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 5 monclient: authenticate success, global_id 4275 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 1 -- 172.21.15.110:0/2942011522 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb30031990 con 0x7fbb40143310 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 4 set_mon_vals no callback set 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 1 -- 172.21.15.110:0/2942011522 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30031d20 con 0x7fbb40143310 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:26.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb3f7fe640 10 monclient: dump: 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:26.585 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 10 monclient: shutdown 2024-01-20T11:15:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 1 -- 172.21.15.110:0/2942011522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 msgr2=0x7fbb401437a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/2942011522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 0x7fbb401437a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbb400b5890 tx=0x7fbb3002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 1 -- 172.21.15.110:0/2942011522 shutdown_connections 2024-01-20T11:15:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/2942011522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 0x7fbb401437a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/2942011522 >> v2:172.21.15.134:3300/0 conn(0x7fbb4013d1f0 0x7fbb4013d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/2942011522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4006a860 0x7fbb40070ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 1 -- 172.21.15.110:0/2942011522 >> 172.21.15.110:0/2942011522 conn(0x7fbb400b4c60 msgr2=0x7fbb400b5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 1 -- 172.21.15.110:0/2942011522 shutdown_connections 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.583+0000 7fbb46cf2640 1 -- 172.21.15.110:0/2942011522 wait complete. 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.584+0000 7fbb46cf2640 10 monclient: build_initial_monmap 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.584+0000 7fbb46cf2640 10 monclient: monmap: 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.584+0000 7fbb46cf2640 1 librados: starting msgr at 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.584+0000 7fbb46cf2640 1 librados: starting objecter 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.584+0000 7fbb46cf2640 1 Processor -- start 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 1 -- start start 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 1 librados: setting wanted keys 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 1 librados: calling monclient init 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient: init 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4006a860 0x7fbb400adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient(hunting): picked mon.noname-b con 0x7fbb4006a860 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4013d1f0 0x7fbb40138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient(hunting): picked mon.noname-c con 0x7fbb4013d1f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 0x7fbb40139000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient(hunting): picked mon.noname-a con 0x7fbb40143310 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb401ca8a0 con 0x7fbb40143310 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb401c9210 con 0x7fbb4006a860 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb401c8cf0 con 0x7fbb4013d1f0 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.585+0000 7fbb46cf2640 10 monclient(hunting): authenticate will time out at 1369.936119s 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.586+0000 7fbb4528f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 0x7fbb40139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.586+0000 7fbb3ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4013d1f0 0x7fbb40138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.586+0000 7fbb44a8e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4006a860 0x7fbb400adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.586+0000 7fbb4528f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 0x7fbb40139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39100/0 (socket says 172.21.15.110:39100) 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.586+0000 7fbb4528f640 1 -- 172.21.15.110:0/867230298 learned_addr learned my addr 172.21.15.110:0/867230298 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.586+0000 7fbb4528f640 10 monclient(hunting): get_auth_request con 0x7fbb40143310 auth_method 0 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.586+0000 7fbb4528f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.586+0000 7fbb4528f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.586+0000 7fbb4528f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb44a8e640 10 monclient(hunting): get_auth_request con 0x7fbb4006a860 auth_method 0 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb44a8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb44a8e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb44a8e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb3ffff640 10 monclient(hunting): get_auth_request con 0x7fbb4013d1f0 auth_method 0 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb3ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb44a8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb44a8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb44a8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb4528f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb4528f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb4528f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.587+0000 7fbb3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 10 monclient(hunting): handle_auth_done global_id 4303 payload 824 2024-01-20T11:15:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 1 -- 172.21.15.110:0/867230298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4013d1f0 msgr2=0x7fbb40138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4013d1f0 0x7fbb40138ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 1 -- 172.21.15.110:0/867230298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 msgr2=0x7fbb40139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 0x7fbb40139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 10 monclient: _finish_hunting 0 2024-01-20T11:15:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 1 monclient: found mon.noname-b 2024-01-20T11:15:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.590+0000 7fbb44a8e640 1 -- 172.21.15.110:0/867230298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb30009400 con 0x7fbb4006a860 2024-01-20T11:15:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.592+0000 7fbb44a8e640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4006a860 0x7fbb400adb00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800efc0 tx=0x7fbb3800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.592+0000 7fbb3d7fa640 10 client.?.objecter ms_handle_connect 0x7fbb4006a860 2024-01-20T11:15:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.592+0000 7fbb3d7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 1 -- 172.21.15.110:0/867230298 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3800cda0 con 0x7fbb4006a860 2024-01-20T11:15:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 10 monclient: dump: 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 10 monclient: _finish_auth 0 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 1 -- 172.21.15.110:0/867230298 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3800f040 con 0x7fbb4006a860 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 1 -- 172.21.15.110:0/867230298 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb38014680 con 0x7fbb4006a860 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3d7fa640 10 monclient: dump: 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:26.598 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb46cf2640 5 monclient: authenticate success, global_id 4303 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb46cf2640 10 monclient: _renew_subs 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb46cf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb401c8f10 con 0x7fbb4006a860 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb3effd640 4 set_mon_vals no callback set 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb46cf2640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb46cf2640 10 monclient: _renew_subs 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb46cf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb40070200 con 0x7fbb4006a860 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.595+0000 7fbb46cf2640 1 librados: init done 2024-01-20T11:15:26.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.596+0000 7fbb46cf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:26.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.596+0000 7fbb46cf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:26.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.596+0000 7fbb46cf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:26.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.596+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb401c93d0 con 0x7fbb4006a860 2024-01-20T11:15:26.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.617+0000 7fbb3d7fa640 1 -- 172.21.15.110:0/867230298 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fbb380147e0 con 0x7fbb4006a860 2024-01-20T11:15:26.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.617+0000 7fbb3d7fa640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c860 0x7fbb2c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:26.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.618+0000 7fbb3ffff640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c860 0x7fbb2c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:26.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.618+0000 7fbb3ffff640 10 monclient: get_auth_request con 0x7fbb2c06c860 auth_method 0 2024-01-20T11:15:26.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.618+0000 7fbb3ffff640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c860 0x7fbb2c06ed20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb400ae790 tx=0x7fbb34009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:26.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.618+0000 7fbb3d7fa640 10 client.4303.objecter ms_handle_connect 0x7fbb2c06c860 2024-01-20T11:15:26.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.620+0000 7fbb3d7fa640 1 -- 172.21.15.110:0/867230298 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbb38059410 con 0x7fbb4006a860 2024-01-20T11:15:26.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.620+0000 7fbb3d7fa640 10 client.4303.objecter ms_dispatch 0x7fbb40000b90 osd_map(23..23 src has 1..23) v4 2024-01-20T11:15:26.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.620+0000 7fbb3d7fa640 3 client.4303.objecter handle_osd_map got epochs [23,23] > 0 2024-01-20T11:15:26.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.620+0000 7fbb3d7fa640 3 client.4303.objecter handle_osd_map decoding full epoch 23 2024-01-20T11:15:26.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.620+0000 7fbb3d7fa640 20 client.4303.objecter dump_active .. 0 homeless 2024-01-20T11:15:26.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.855+0000 7fbb3d7fa640 1 -- 172.21.15.110:0/867230298 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fbb38055750 con 0x7fbb4006a860 2024-01-20T11:15:26.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.855+0000 7fbb3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:26.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:26.855+0000 7fbb3d7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:27.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.024+0000 7fbb46cf2640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-20T11:15:27.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.024+0000 7fbb46cf2640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-20T11:15:27.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.024+0000 7fbb46cf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:27.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.024+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fbb40075500 con 0x7fbb4006a860 2024-01-20T11:15:27.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.487+0000 7fbb3d7fa640 1 -- 172.21.15.110:0/867230298 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v24) v1 ==== 109+0+0 (secure 0 0 0) 0x7fbb38058d70 con 0x7fbb4006a860 2024-01-20T11:15:27.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.487+0000 7fbb3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-20T11:15:27.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.487+0000 7fbb3d7fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-01-20T11:15:27.487 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'rbd' created 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 10 librados: watch_flush enter 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 10 librados: watch_flush exit 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 20 client.4303.objecter shutdown clearing up homeless session... 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 10 client.4303.objecter successfully canceled tick 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c860 msgr2=0x7fbb2c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c860 0x7fbb2c06ed20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb400ae790 tx=0x7fbb34009190 comp rx=0 tx=0).stop 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 10 monclient: shutdown 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4006a860 msgr2=0x7fbb400adb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4006a860 0x7fbb400adb00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800efc0 tx=0x7fbb3800c490 comp rx=0 tx=0).stop 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 shutdown_connections 2024-01-20T11:15:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c860 0x7fbb2c06ed20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb40143310 0x7fbb40139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4013d1f0 0x7fbb40138ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 --2- 172.21.15.110:0/867230298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4006a860 0x7fbb400adb00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 >> 172.21.15.110:0/867230298 conn(0x7fbb400b4c60 msgr2=0x7fbb400b07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 shutdown_connections 2024-01-20T11:15:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 -- 172.21.15.110:0/867230298 wait complete. 2024-01-20T11:15:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.489+0000 7fbb46cf2640 1 librados: shutdown 2024-01-20T11:15:27.500 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-20T11:15:27.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.603+0000 7f68fbde9640 10 monclient: get_monmap_and_config 2024-01-20T11:15:27.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.603+0000 7f68fbde9640 10 monclient: build_initial_monmap 2024-01-20T11:15:27.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.603+0000 7f68fbde9640 10 monclient: monmap: 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:27.604365+0000 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:27.604365+0000 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:27.604 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:27.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.604+0000 7f68fbde9640 1 Processor -- start 2024-01-20T11:15:27.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.604+0000 7f68fbde9640 1 -- start start 2024-01-20T11:15:27.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.604+0000 7f68fbde9640 10 monclient: init 2024-01-20T11:15:27.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:27.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-01-20T11:15:27.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f68f40b4ca0 0x7f68f413f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 10 monclient(hunting): picked mon.noname-a con 0x7f68f40b4ca0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68f40b3a30 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f68f4142710 0x7f68f414c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 10 monclient(hunting): picked mon.noname-c con 0x7f68f4142710 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f40b4470 con 0x7f68f40b3a30 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f40aa160 con 0x7f68f40b4ca0 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f40b5080 con 0x7f68f4142710 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fbde9640 10 monclient(hunting): authenticate will time out at 1370.955423s 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68f9b85640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f68f40b4ca0 0x7f68f413f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.605+0000 7f68fa386640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f68f4142710 0x7f68f414c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68f9b85640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f68f40b4ca0 0x7f68f413f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39108/0 (socket says 172.21.15.110:39108) 2024-01-20T11:15:27.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68f9b85640 1 -- 172.21.15.110:0/1185783439 learned_addr learned my addr 172.21.15.110:0/1185783439 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:27.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68f9b85640 10 monclient(hunting): get_auth_request con 0x7f68f40b4ca0 auth_method 0 2024-01-20T11:15:27.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68f9b85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:27.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68f9b85640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:27.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68f9b85640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:27.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68fa386640 10 monclient(hunting): get_auth_request con 0x7f68f4142710 auth_method 0 2024-01-20T11:15:27.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68fa386640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:27.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68fa386640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:27.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68fa386640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:27.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68f9b85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:27.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68f9b85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:27.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.606+0000 7f68f9b85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.607+0000 7f68fa386640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.607+0000 7f68fa386640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.607+0000 7f68fa386640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.609+0000 7f68f9b85640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.609+0000 7f68f9b85640 1 -- 172.21.15.110:0/1185783439 >> v2:172.21.15.134:3300/0 conn(0x7f68f4142710 msgr2=0x7f68f414c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.609+0000 7f68f9b85640 1 --2- 172.21.15.110:0/1185783439 >> v2:172.21.15.134:3300/0 conn(0x7f68f4142710 0x7f68f414c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.609+0000 7f68f9b85640 1 -- 172.21.15.110:0/1185783439 >> v1:172.21.15.134:6789/0 conn(0x7f68f40b3a30 legacy=0x7f68f4142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.609+0000 7f68f9b85640 10 monclient: _finish_hunting 0 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.609+0000 7f68f9b85640 1 monclient: found mon.noname-a 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.609+0000 7f68f9b85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:27.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.609+0000 7f68f9b85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:27.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.609+0000 7f68f9b85640 1 -- 172.21.15.110:0/1185783439 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f40ab0c0 con 0x7f68f40b4ca0 2024-01-20T11:15:27.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.611+0000 7f68f9b85640 1 --2- 172.21.15.110:0/1185783439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b4ca0 0x7f68f413f4f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68f40b3860 tx=0x7f68e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9f2cada3e129c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 1 -- 172.21.15.110:0/1185783439 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e4031870 con 0x7f68f40b4ca0 2024-01-20T11:15:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 10 monclient: dump: 2024-01-20T11:15:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 10 monclient: _finish_auth 0 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 10 monclient: _check_auth_tickets 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 1 -- 172.21.15.110:0/1185783439 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e40319d0 con 0x7f68f40b4ca0 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68fbde9640 5 monclient: authenticate success, global_id 4281 2024-01-20T11:15:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68fbde9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 1 -- 172.21.15.110:0/1185783439 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e4031d60 con 0x7f68f40b4ca0 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68f8b83640 10 monclient: dump: 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:27.618 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:27.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68fbde9640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:27.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.616+0000 7f68fbde9640 4 set_mon_vals no callback set 2024-01-20T11:15:27.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 10 monclient: shutdown 2024-01-20T11:15:27.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:27.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 1 -- 172.21.15.110:0/1185783439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b4ca0 msgr2=0x7f68f413f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 1 --2- 172.21.15.110:0/1185783439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b4ca0 0x7f68f413f4f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68f40b3860 tx=0x7f68e402b910 comp rx=0 tx=0).stop 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 1 -- 172.21.15.110:0/1185783439 shutdown_connections 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 1 --2- 172.21.15.110:0/1185783439 >> v2:172.21.15.134:3300/0 conn(0x7f68f4142710 0x7f68f414c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 1 --2- 172.21.15.110:0/1185783439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b4ca0 0x7f68f413f4f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 1 -- 172.21.15.110:0/1185783439 >> 172.21.15.110:0/1185783439 conn(0x7f68f4066590 msgr2=0x7f68f40669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 1 -- 172.21.15.110:0/1185783439 shutdown_connections 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.617+0000 7f68fbde9640 1 -- 172.21.15.110:0/1185783439 wait complete. 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.618+0000 7f68fbde9640 10 monclient: build_initial_monmap 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.618+0000 7f68fbde9640 10 monclient: monmap: 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.618+0000 7f68fbde9640 1 librados: starting msgr at 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.618+0000 7f68fbde9640 1 librados: starting objecter 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.618+0000 7f68fbde9640 1 Processor -- start 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.618+0000 7f68fbde9640 1 -- start start 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.618+0000 7f68fbde9640 1 librados: setting wanted keys 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.618+0000 7f68fbde9640 1 librados: calling monclient init 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.618+0000 7f68fbde9640 10 monclient: init 2024-01-20T11:15:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b3a30 0x7f68f4145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient(hunting): picked mon.noname-a con 0x7f68f40b3a30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68f40b4ca0 0x7f68f40abac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient(hunting): picked mon.noname-c con 0x7f68f40b4ca0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68f4142710 0x7f68f41da870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient(hunting): picked mon.noname-b con 0x7f68f4142710 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f40acc40 con 0x7f68f40b3a30 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f4143300 con 0x7f68f4142710 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f40aceb0 con 0x7f68f40b4ca0 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fbde9640 10 monclient(hunting): authenticate will time out at 1370.969429s 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68f9b85640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b3a30 0x7f68f4145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68f9384640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68f40b4ca0 0x7f68f40abac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.619+0000 7f68fa386640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68f4142710 0x7f68f41da870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9b85640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b3a30 0x7f68f4145cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39110/0 (socket says 172.21.15.110:39110) 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9b85640 1 -- 172.21.15.110:0/407790562 learned_addr learned my addr 172.21.15.110:0/407790562 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9b85640 10 monclient(hunting): get_auth_request con 0x7f68f40b3a30 auth_method 0 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9b85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9b85640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9b85640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9384640 10 monclient(hunting): get_auth_request con 0x7f68f40b4ca0 auth_method 0 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9384640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9384640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9384640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68fa386640 10 monclient(hunting): get_auth_request con 0x7f68f4142710 auth_method 0 2024-01-20T11:15:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68fa386640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68fa386640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68fa386640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9b85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9b85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9b85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9384640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9384640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.620+0000 7f68f9384640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.621+0000 7f68fa386640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.621+0000 7f68fa386640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:27.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.621+0000 7f68fa386640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:27.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2024-01-20T11:15:27.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 1 -- 172.21.15.110:0/407790562 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68f4142710 msgr2=0x7f68f41da870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:27.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68f4142710 0x7f68f41da870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:27.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 1 -- 172.21.15.110:0/407790562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b3a30 msgr2=0x7f68f4145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:27.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b3a30 0x7f68f4145cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:27.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 10 monclient: _finish_hunting 0 2024-01-20T11:15:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 1 monclient: found mon.noname-c 2024-01-20T11:15:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:15:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.623+0000 7f68f9384640 1 -- 172.21.15.110:0/407790562 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e40094d0 con 0x7f68f40b4ca0 2024-01-20T11:15:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.626+0000 7f68f9384640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68f40b4ca0 0x7f68f40abac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f68f000c340 tx=0x7f68f000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.626+0000 7f68ea7fc640 10 client.?.objecter ms_handle_connect 0x7f68f40b4ca0 2024-01-20T11:15:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.626+0000 7f68ea7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 1 -- 172.21.15.110:0/407790562 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f00052d0 con 0x7f68f40b4ca0 2024-01-20T11:15:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 10 monclient: dump: 2024-01-20T11:15:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 10 monclient: _finish_auth 0 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 1 -- 172.21.15.110:0/407790562 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f0005450 con 0x7f68f40b4ca0 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 1 -- 172.21.15.110:0/407790562 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f0007260 con 0x7f68f40b4ca0 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ea7fc640 10 monclient: dump: 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68fbde9640 5 monclient: authenticate success, global_id 4220 2024-01-20T11:15:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68fbde9640 10 monclient: _renew_subs 2024-01-20T11:15:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68ebfff640 4 set_mon_vals no callback set 2024-01-20T11:15:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68fbde9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f40ad190 con 0x7f68f40b4ca0 2024-01-20T11:15:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68fbde9640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68fbde9640 10 monclient: _renew_subs 2024-01-20T11:15:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68fbde9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f41daf90 con 0x7f68f40b4ca0 2024-01-20T11:15:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.630+0000 7f68fbde9640 1 librados: init done 2024-01-20T11:15:27.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.631+0000 7f68fbde9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:27.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.631+0000 7f68fbde9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:27.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.631+0000 7f68fbde9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:27.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.631+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f404e970 con 0x7f68f40b4ca0 2024-01-20T11:15:27.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.890+0000 7f68ea7fc640 1 -- 172.21.15.110:0/407790562 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f68f00073c0 con 0x7f68f40b4ca0 2024-01-20T11:15:27.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.890+0000 7f68ea7fc640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f68d806c8b0 0x7f68d806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:27.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.891+0000 7f68f9b85640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f68d806c8b0 0x7f68d806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:27.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.891+0000 7f68f9b85640 10 monclient: get_auth_request con 0x7f68d806c8b0 auth_method 0 2024-01-20T11:15:27.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.891+0000 7f68f9b85640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f68d806c8b0 0x7f68d806ed70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f68e40090b0 tx=0x7f68e403c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:27.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.891+0000 7f68ea7fc640 10 client.4220.objecter ms_handle_connect 0x7f68d806c8b0 2024-01-20T11:15:27.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.912+0000 7f68ea7fc640 1 -- 172.21.15.110:0/407790562 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f68f0057a30 con 0x7f68f40b4ca0 2024-01-20T11:15:27.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.912+0000 7f68ea7fc640 10 client.4220.objecter ms_dispatch 0x7f68f4000b90 osd_map(24..24 src has 1..24) v4 2024-01-20T11:15:27.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.912+0000 7f68ea7fc640 3 client.4220.objecter handle_osd_map got epochs [24,24] > 0 2024-01-20T11:15:27.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.912+0000 7f68ea7fc640 3 client.4220.objecter handle_osd_map decoding full epoch 24 2024-01-20T11:15:27.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.912+0000 7f68ea7fc640 20 client.4220.objecter dump_active .. 0 homeless 2024-01-20T11:15:27.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.912+0000 7f68ea7fc640 1 -- 172.21.15.110:0/407790562 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f68f0053d70 con 0x7f68f40b4ca0 2024-01-20T11:15:27.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.912+0000 7f68ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:27.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:27.912+0000 7f68ea7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.146+0000 7f68fbde9640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-20T11:15:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.146+0000 7f68fbde9640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-20T11:15:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.146+0000 7f68fbde9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.146+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f68f40a9a50 con 0x7f68f40b4ca0 2024-01-20T11:15:28.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.554+0000 7f68ea7fc640 1 -- 172.21.15.110:0/407790562 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v25) v1 ==== 173+0+0 (secure 0 0 0) 0x7f68f0020020 con 0x7f68f40b4ca0 2024-01-20T11:15:28.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.554+0000 7f68ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-20T11:15:28.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.554+0000 7f68ea7fc640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-01-20T11:15:28.555 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-20T11:15:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.556+0000 7f68fbde9640 10 librados: watch_flush enter 2024-01-20T11:15:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.556+0000 7f68fbde9640 10 librados: watch_flush exit 2024-01-20T11:15:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.556+0000 7f68fbde9640 20 client.4220.objecter shutdown clearing up homeless session... 2024-01-20T11:15:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.556+0000 7f68fbde9640 10 client.4220.objecter successfully canceled tick 2024-01-20T11:15:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.556+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f68d806c8b0 msgr2=0x7f68d806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.556+0000 7f68fbde9640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f68d806c8b0 0x7f68d806ed70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f68e40090b0 tx=0x7f68e403c040 comp rx=0 tx=0).stop 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 10 monclient: shutdown 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68f40b4ca0 msgr2=0x7f68f40abac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68f40b4ca0 0x7f68f40abac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f68f000c340 tx=0x7f68f000c800 comp rx=0 tx=0).stop 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 shutdown_connections 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f68d806c8b0 0x7f68d806ed70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68f4142710 0x7f68f41da870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68f40b4ca0 0x7f68f40abac0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 --2- 172.21.15.110:0/407790562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40b3a30 0x7f68f4145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 >> 172.21.15.110:0/407790562 conn(0x7f68f4066590 msgr2=0x7f68f40b8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 shutdown_connections 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 -- 172.21.15.110:0/407790562 wait complete. 2024-01-20T11:15:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.557+0000 7f68fbde9640 1 librados: shutdown 2024-01-20T11:15:28.570 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-20T11:15:28.570 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-20T11:15:28.570 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-20T11:15:28.571 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-20T11:15:28.571 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-20T11:15:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.674+0000 7f3472997640 10 monclient: get_monmap_and_config 2024-01-20T11:15:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.674+0000 7f3472997640 10 monclient: build_initial_monmap 2024-01-20T11:15:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.674+0000 7f3472997640 10 monclient: monmap: 2024-01-20T11:15:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:28.675269+0000 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:28.675269+0000 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:28.675 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.675+0000 7f3472997640 1 Processor -- start 2024-01-20T11:15:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.675+0000 7f3472997640 1 -- start start 2024-01-20T11:15:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.675+0000 7f3472997640 10 monclient: init 2024-01-20T11:15:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.675+0000 7f3472997640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.675+0000 7f3472997640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-01-20T11:15:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.675+0000 7f3472997640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f346c0a96f0 0x7f346c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.675+0000 7f3472997640 10 monclient(hunting): picked mon.noname-c con 0x7f346c0a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3472997640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f346c13fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3472997640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f346c0a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3472997640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c0b25c0 con 0x7f346c0a9ad0 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3472997640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c0ab0a0 con 0x7f346c13fa10 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3472997640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3472997640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346c0aaea0 con 0x7f346c0a96f0 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3472997640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3472997640 10 monclient(hunting): authenticate will time out at 1372.026261s 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3470f34640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f346c0a9ad0 0x7f346c14c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60244/0 (socket says 172.21.15.110:60244) 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f3470f34640 1 -- 172.21.15.110:0/277340208 learned_addr learned my addr 172.21.15.110:0/277340208 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.676+0000 7f346bfff640 1 --2- 172.21.15.110:0/277340208 >> v2:172.21.15.134:3300/0 conn(0x7f346c0a96f0 0x7f346c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.677+0000 7f346bfff640 10 monclient(hunting): get_auth_request con 0x7f346c0a96f0 auth_method 0 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.677+0000 7f346bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.677+0000 7f346bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.677+0000 7f346bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.678+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.678+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.678+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.679+0000 7f346affd640 1 -- 172.21.15.110:0/277340208 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153378868 0 0) 0x7f346c0b25c0 con 0x7f346c0a9ad0 2024-01-20T11:15:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.679+0000 7f346affd640 10 monclient(hunting): _negotiate 2024-01-20T11:15:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.679+0000 7f346affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.679+0000 7f346affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.679+0000 7f346affd640 10 monclient(hunting): my global_id is 4287 2024-01-20T11:15:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.679+0000 7f346affd640 1 -- 172.21.15.110:0/277340208 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3454003520 con 0x7f346c0a9ad0 2024-01-20T11:15:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.681+0000 7f346bfff640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-01-20T11:15:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.681+0000 7f346bfff640 1 -- 172.21.15.110:0/277340208 >> v1:172.21.15.134:6789/0 conn(0x7f346c13fa10 legacy=0x7f346c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.681+0000 7f346bfff640 1 -- 172.21.15.110:0/277340208 >> v1:172.21.15.110:6789/0 conn(0x7f346c0a9ad0 legacy=0x7f346c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.681+0000 7f346bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.681+0000 7f346bfff640 1 monclient: found mon.noname-c 2024-01-20T11:15:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.681+0000 7f346bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.681+0000 7f346bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:15:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.681+0000 7f346bfff640 1 -- 172.21.15.110:0/277340208 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c0b3fb0 con 0x7f346c0a96f0 2024-01-20T11:15:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.687+0000 7f346bfff640 1 --2- 172.21.15.110:0/277340208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 0x7f346c13f4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f34600097a0 tx=0x7f346002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d278e667b4dcf8cc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 1 -- 172.21.15.110:0/277340208 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3460031830 con 0x7f346c0a96f0 2024-01-20T11:15:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 10 monclient: dump: 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 10 monclient: _finish_auth 0 2024-01-20T11:15:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 1 -- 172.21.15.110:0/277340208 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3460031990 con 0x7f346c0a96f0 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 1 -- 172.21.15.110:0/277340208 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3460031d20 con 0x7f346c0a96f0 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f346affd640 10 monclient: dump: 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:28.696 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:28.696 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:28.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f3472997640 5 monclient: authenticate success, global_id 4312 2024-01-20T11:15:28.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f3472997640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:28.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.693+0000 7f3472997640 4 set_mon_vals no callback set 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.694+0000 7f3472997640 10 monclient: shutdown 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.694+0000 7f3472997640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.694+0000 7f3472997640 1 -- 172.21.15.110:0/277340208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 msgr2=0x7f346c13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.694+0000 7f3472997640 1 --2- 172.21.15.110:0/277340208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 0x7f346c13f4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f34600097a0 tx=0x7f346002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.694+0000 7f3472997640 1 -- 172.21.15.110:0/277340208 shutdown_connections 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.694+0000 7f3472997640 1 --2- 172.21.15.110:0/277340208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 0x7f346c13f4d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.694+0000 7f3472997640 1 -- 172.21.15.110:0/277340208 >> 172.21.15.110:0/277340208 conn(0x7f346c0b7bc0 msgr2=0x7f346c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.694+0000 7f3472997640 1 -- 172.21.15.110:0/277340208 shutdown_connections 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.694+0000 7f3472997640 1 -- 172.21.15.110:0/277340208 wait complete. 2024-01-20T11:15:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.695+0000 7f3472997640 10 monclient: build_initial_monmap 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.695+0000 7f3472997640 10 monclient: monmap: 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.695+0000 7f3472997640 1 librados: starting msgr at 2024-01-20T11:15:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.695+0000 7f3472997640 1 librados: starting objecter 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.695+0000 7f3472997640 1 Processor -- start 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.695+0000 7f3472997640 1 -- start start 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.695+0000 7f3472997640 1 librados: setting wanted keys 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.695+0000 7f3472997640 1 librados: calling monclient init 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.695+0000 7f3472997640 10 monclient: init 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 0x7f346c137e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient(hunting): picked mon.noname-b con 0x7f346c0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f346c0a9ad0 0x7f346c0b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient(hunting): picked mon.noname-c con 0x7f346c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f346c0b13f0 0x7f346c1d79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient(hunting): picked mon.noname-a con 0x7f346c0b13f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0afac0 con 0x7f346c0b13f0 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c1d7510 con 0x7f346c0a96f0 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c138980 con 0x7f346c0a9ad0 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3472997640 10 monclient(hunting): authenticate will time out at 1372.046561s 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f346bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 0x7f346c137e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.696+0000 7f3470f34640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f346c0b13f0 0x7f346c1d79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f346c0a9ad0 0x7f346c0b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 0x7f346c137e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:51548/0 (socket says 172.21.15.110:51548) 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346bfff640 1 -- 172.21.15.110:0/2669049384 learned_addr learned my addr 172.21.15.110:0/2669049384 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346bfff640 10 monclient(hunting): get_auth_request con 0x7f346c0a96f0 auth_method 0 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346b7fe640 10 monclient(hunting): get_auth_request con 0x7f346c0a9ad0 auth_method 0 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f3470f34640 10 monclient(hunting): get_auth_request con 0x7f346c0b13f0 auth_method 0 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f3470f34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f3470f34640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f3470f34640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.697+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.698+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.698+0000 7f3470f34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:28.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.698+0000 7f3470f34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:28.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.698+0000 7f3470f34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:28.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.698+0000 7f346b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:28.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.698+0000 7f346b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:28.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.698+0000 7f346b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:28.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.700+0000 7f346b7fe640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-01-20T11:15:28.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f346b7fe640 1 -- 172.21.15.110:0/2669049384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 msgr2=0x7f346c137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:28.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f346b7fe640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 0x7f346c137e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:28.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f346b7fe640 1 -- 172.21.15.110:0/2669049384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f346c0b13f0 msgr2=0x7f346c1d79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:28.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f346b7fe640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f346c0b13f0 0x7f346c1d79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:28.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f346b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:15:28.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f346b7fe640 1 monclient: found mon.noname-c 2024-01-20T11:15:28.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f346b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:28.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f346b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:15:28.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f346b7fe640 1 -- 172.21.15.110:0/2669049384 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3460009400 con 0x7f346c0a9ad0 2024-01-20T11:15:28.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.701+0000 7f3470f34640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f346c0b13f0 0x7f346c1d79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:15:28.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.703+0000 7f346b7fe640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f346c0a9ad0 0x7f346c0b0eb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f346c0aac70 tx=0x7f345c00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:28.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.703+0000 7f3468ff9640 10 client.?.objecter ms_handle_connect 0x7f346c0a9ad0 2024-01-20T11:15:28.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.703+0000 7f3468ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.705+0000 7f3468ff9640 1 -- 172.21.15.110:0/2669049384 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c0052d0 con 0x7f346c0a9ad0 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.705+0000 7f3468ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.705+0000 7f3468ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.705+0000 7f3468ff9640 10 monclient: dump: 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:28.708 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.705+0000 7f3468ff9640 10 monclient: _finish_auth 0 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.705+0000 7f3468ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.705+0000 7f3468ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 5 monclient: authenticate success, global_id 4223 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 10 monclient: _renew_subs 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346c0ae0b0 con 0x7f346c0a9ad0 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 10 monclient: _renew_subs 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:28.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346c1d8110 con 0x7f346c0a9ad0 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 1 librados: init done 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.706+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346c13f2f0 con 0x7f346c0a9ad0 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.707+0000 7f3468ff9640 1 -- 172.21.15.110:0/2669049384 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c005430 con 0x7f346c0a9ad0 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.707+0000 7f3468ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.707+0000 7f3468ff9640 1 -- 172.21.15.110:0/2669049384 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c007250 con 0x7f346c0a9ad0 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.707+0000 7f3468ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.707+0000 7f3468ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.707+0000 7f3468ff9640 10 monclient: dump: 2024-01-20T11:15:28.710 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:28.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.707+0000 7f346a7fc640 4 set_mon_vals no callback set 2024-01-20T11:15:28.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.998+0000 7f3468ff9640 1 -- 172.21.15.110:0/2669049384 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f345c0073b0 con 0x7f346c0a9ad0 2024-01-20T11:15:28.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.999+0000 7f3468ff9640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f345806c860 0x7f345806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:28.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.999+0000 7f346bfff640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f345806c860 0x7f345806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:28.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:28.999+0000 7f346bfff640 10 monclient: get_auth_request con 0x7f345806c860 auth_method 0 2024-01-20T11:15:28.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.000+0000 7f3468ff9640 10 client.4223.objecter ms_handle_connect 0x7f345806c860 2024-01-20T11:15:29.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.000+0000 7f346bfff640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f345806c860 0x7f345806ed20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f346002b910 tx=0x7f346003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:29.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.021+0000 7f3468ff9640 1 -- 172.21.15.110:0/2669049384 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f345c007660 con 0x7f346c0a9ad0 2024-01-20T11:15:29.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.021+0000 7f3468ff9640 10 client.4223.objecter ms_dispatch 0x7f346c000b90 osd_map(25..25 src has 1..25) v4 2024-01-20T11:15:29.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.021+0000 7f3468ff9640 3 client.4223.objecter handle_osd_map got epochs [25,25] > 0 2024-01-20T11:15:29.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.021+0000 7f3468ff9640 3 client.4223.objecter handle_osd_map decoding full epoch 25 2024-01-20T11:15:29.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.021+0000 7f3468ff9640 20 client.4223.objecter dump_active .. 0 homeless 2024-01-20T11:15:29.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.021+0000 7f3468ff9640 1 -- 172.21.15.110:0/2669049384 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f345c053e00 con 0x7f346c0a9ad0 2024-01-20T11:15:29.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.021+0000 7f3468ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:29.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.021+0000 7f3468ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.314+0000 7f3472997640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-01-20T11:15:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.314+0000 7f3472997640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-01-20T11:15:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.314+0000 7f3472997640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.314+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f346c075490 con 0x7f346c0a9ad0 2024-01-20T11:15:29.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.449+0000 7f3468ff9640 1 -- 172.21.15.110:0/2669049384 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v8) v1 ==== 74+0+170774 (secure 0 0 0) 0x7f345c057420 con 0x7f346c0a9ad0 2024-01-20T11:15:29.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.449+0000 7f3468ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-01-20T11:15:29.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.449+0000 7f3468ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:29.452 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:15:29.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 10 librados: watch_flush enter 2024-01-20T11:15:29.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 10 librados: watch_flush exit 2024-01-20T11:15:29.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 20 client.4223.objecter shutdown clearing up homeless session... 2024-01-20T11:15:29.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 10 client.4223.objecter successfully canceled tick 2024-01-20T11:15:29.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f345806c860 msgr2=0x7f345806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:29.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f345806c860 0x7f345806ed20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f346002b910 tx=0x7f346003c040 comp rx=0 tx=0).stop 2024-01-20T11:15:29.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 10 monclient: shutdown 2024-01-20T11:15:29.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:29.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f346c0a9ad0 msgr2=0x7f346c0b0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f346c0a9ad0 0x7f346c0b0eb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f346c0aac70 tx=0x7f345c00c820 comp rx=0 tx=0).stop 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 shutdown_connections 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f345806c860 0x7f345806ed20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f346c0b13f0 0x7f346c1d79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f346c0a9ad0 0x7f346c0b0eb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 --2- 172.21.15.110:0/2669049384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f346c0a96f0 0x7f346c137e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 >> 172.21.15.110:0/2669049384 conn(0x7f346c0b7bc0 msgr2=0x7f346c0b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.451+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 shutdown_connections 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.452+0000 7f3472997640 1 -- 172.21.15.110:0/2669049384 wait complete. 2024-01-20T11:15:29.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.452+0000 7f3472997640 1 librados: shutdown 2024-01-20T11:15:29.466 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"active_gid":4164,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":813062206},{"type":"v1","addr":"172.21.15.134:6817","nonce":813062206}]},"active_addr":"172.21.15.134:6817/813062206","active_change":"2024-01-20T11:14:40.465884+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":4105,"name":"y","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":12,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":4048766302}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":2339790961}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":4150982747}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":2336856841}]}]} 2024-01-20T11:15:29.471 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-20T11:15:29.471 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-20T11:15:29.471 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-20T11:15:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.575+0000 7fefde415640 10 monclient: get_monmap_and_config 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.575+0000 7fefde415640 10 monclient: build_initial_monmap 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.575+0000 7fefde415640 10 monclient: monmap: 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:29.575635+0000 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:29.575635+0000 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:29.576 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:29.576 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:29.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.575+0000 7fefde415640 1 Processor -- start 2024-01-20T11:15:29.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.575+0000 7fefde415640 1 -- start start 2024-01-20T11:15:29.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.575+0000 7fefde415640 10 monclient: init 2024-01-20T11:15:29.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:29.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-01-20T11:15:29.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefd80a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:29.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fefd80a9ad0 0x7fefd81423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:29.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 10 monclient(hunting): picked mon.noname-a con 0x7fefd80a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefd8142920 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd80b25c0 con 0x7fefd80a96f0 2024-01-20T11:15:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd813fbe0 con 0x7fefd8142920 2024-01-20T11:15:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd80aaea0 con 0x7fefd80a9ad0 2024-01-20T11:15:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.576+0000 7fefde415640 10 monclient(hunting): authenticate will time out at 1372.926671s 2024-01-20T11:15:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.577+0000 7fefdc9b2640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fefd8142920 0x7fefd814c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:53582/0 (socket says 172.21.15.110:53582) 2024-01-20T11:15:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.577+0000 7fefdc9b2640 1 -- 172.21.15.110:0/1234341055 learned_addr learned my addr 172.21.15.110:0/1234341055 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:29.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.588+0000 7fefd77fe640 1 --2- 172.21.15.110:0/1234341055 >> v2:172.21.15.110:3300/0 conn(0x7fefd80a9ad0 0x7fefd81423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:29.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.589+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263041251 0 0) 0x7fefd813fbe0 con 0x7fefd8142920 2024-01-20T11:15:29.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.589+0000 7fefd6ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:15:29.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.589+0000 7fefd6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:29.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.589+0000 7fefd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:29.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.589+0000 7fefd6ffd640 10 monclient(hunting): my global_id is 4321 2024-01-20T11:15:29.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.589+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc0003520 con 0x7fefd8142920 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 568618232 0 0) 0x7fefc0003520 con 0x7fefd8142920 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 >> v2:172.21.15.110:3300/0 conn(0x7fefd80a9ad0 msgr2=0x7fefd81423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 --2- 172.21.15.110:0/1234341055 >> v2:172.21.15.110:3300/0 conn(0x7fefd80a9ad0 0x7fefd81423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 >> v1:172.21.15.110:6789/0 conn(0x7fefd80a96f0 legacy=0x7fefd813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 monclient: found mon.noname-c-legacy 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.134:6789/0 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd80ab0a0 con 0x7fefd8142920 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2748620985 0 0) 0x7fefd00019f0 con 0x7fefd8142920 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient: dump: 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:29.594 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] but i'm connected to v1:172.21.15.134:6789/0, reconnecting 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 >> v1:172.21.15.134:6789/0 conn(0x7fefd8142920 legacy=0x7fefd814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefc0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient(hunting): picked mon.b con 0x7fefc0003970 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefc00047c0 0x7fefc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient(hunting): picked mon.c con 0x7fefc00047c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefc0005380 0x7fefc000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient(hunting): picked mon.a con 0x7fefc0005380 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80aaea0 con 0x7fefc0005380 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc000c220 con 0x7fefc0003970 2024-01-20T11:15:29.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:29.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc000c3e0 con 0x7fefc00047c0 2024-01-20T11:15:29.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:29.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.592+0000 7fefd6ffd640 10 monclient(hunting): _finish_auth 0 2024-01-20T11:15:29.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.594+0000 7fefd7fff640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefc0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:29.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.596+0000 7fefd7fff640 10 monclient(hunting): get_auth_request con 0x7fefc0003970 auth_method 0 2024-01-20T11:15:29.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.596+0000 7fefd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:29.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.596+0000 7fefd7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:29.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.596+0000 7fefd7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:15:29.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.596+0000 7fefd77fe640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefc00047c0 0x7fefc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:29.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.597+0000 7fefd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:29.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.597+0000 7fefd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:29.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.597+0000 7fefd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:15:29.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.597+0000 7fefd77fe640 10 monclient(hunting): get_auth_request con 0x7fefc00047c0 auth_method 0 2024-01-20T11:15:29.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.597+0000 7fefd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:29.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.597+0000 7fefd77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:29.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.597+0000 7fefd77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:15:29.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.598+0000 7fefd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:29.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.598+0000 7fefd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:29.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.598+0000 7fefd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:15:29.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 10 monclient(hunting): handle_auth_done global_id 4321 payload 293 2024-01-20T11:15:29.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 1 -- 172.21.15.110:0/1234341055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefc00047c0 msgr2=0x7fefc0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:29.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefc00047c0 0x7fefc0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 1 -- 172.21.15.110:0/1234341055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefc0005380 msgr2=0x7fefc000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:29.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefc0005380 0x7fefc000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 1 monclient: found mon.b 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 1 -- 172.21.15.110:0/1234341055 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd80ab0a0 con 0x7fefc0003970 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 10 monclient: _finish_auth 0 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 10 monclient: _check_auth_tickets 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefde415640 5 monclient: authenticate success, global_id 4321 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefdc9b2640 1 -- 172.21.15.110:0/1234341055 reap_dead start 2024-01-20T11:15:29.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.599+0000 7fefde415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:29.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.601+0000 7fefd7fff640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefc0003d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fefc8001650 tx=0x7fefc00057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b4b1963d1b4e726 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:29.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8005270 con 0x7fefc0003970 2024-01-20T11:15:29.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:29.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:29.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 10 monclient: dump: 2024-01-20T11:15:29.605 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:29.605 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:29.605 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.605 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc802e070 con 0x7fefc0003970 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 1 -- 172.21.15.110:0/1234341055 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8003610 con 0x7fefc0003970 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefde415640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefde415640 4 set_mon_vals no callback set 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:29.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.605+0000 7fefd6ffd640 10 monclient: dump: 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:29.607 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:29.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.606+0000 7fefde415640 10 monclient: shutdown 2024-01-20T11:15:29.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.606+0000 7fefde415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:29.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.606+0000 7fefde415640 1 -- 172.21.15.110:0/1234341055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 msgr2=0x7fefc0003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:29.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.606+0000 7fefde415640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefc0003d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fefc8001650 tx=0x7fefc00057f0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.606+0000 7fefde415640 1 -- 172.21.15.110:0/1234341055 shutdown_connections 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.606+0000 7fefde415640 1 --2- 172.21.15.110:0/1234341055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefc0003d70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.606+0000 7fefde415640 1 -- 172.21.15.110:0/1234341055 >> 172.21.15.110:0/1234341055 conn(0x7fefd80b7bc0 msgr2=0x7fefd80b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.606+0000 7fefde415640 1 -- 172.21.15.110:0/1234341055 shutdown_connections 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.606+0000 7fefde415640 1 -- 172.21.15.110:0/1234341055 wait complete. 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.607+0000 7fefde415640 10 monclient: build_initial_monmap 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.607+0000 7fefde415640 10 monclient: monmap: 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.607+0000 7fefde415640 1 librados: starting msgr at 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.607+0000 7fefde415640 1 librados: starting objecter 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.607+0000 7fefde415640 1 Processor -- start 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.607+0000 7fefde415640 1 -- start start 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.607+0000 7fefde415640 1 librados: setting wanted keys 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.607+0000 7fefde415640 1 librados: calling monclient init 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.607+0000 7fefde415640 10 monclient: init 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefd8144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient(hunting): picked mon.noname-b con 0x7fefc0003970 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:29.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefd81454a0 0x7fefd813d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient(hunting): picked mon.noname-a con 0x7fefd81454a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefd8144610 0x7fefd81db440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient(hunting): picked mon.noname-c con 0x7fefd8144610 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd81daf20 con 0x7fefd81454a0 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd81dabb0 con 0x7fefc0003970 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd81db980 con 0x7fefd8144610 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.608+0000 7fefde415640 10 monclient(hunting): authenticate will time out at 1372.958914s 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefdc9b2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefd8144610 0x7fefd81db440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefd7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefd8144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefd7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefd8144ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:57996/0 (socket says 172.21.15.110:57996) 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefd7fff640 1 -- 172.21.15.110:0/537243903 learned_addr learned my addr 172.21.15.110:0/537243903 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:29.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefd7fff640 10 monclient(hunting): get_auth_request con 0x7fefc0003970 auth_method 0 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefd7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefdc9b2640 10 monclient(hunting): get_auth_request con 0x7fefd8144610 auth_method 0 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefdc9b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefdc9b2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.609+0000 7fefdc9b2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.610+0000 7fefd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.610+0000 7fefd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.610+0000 7fefd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.610+0000 7fefdc9b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.610+0000 7fefdc9b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:29.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.610+0000 7fefdc9b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:29.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-01-20T11:15:29.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 1 -- 172.21.15.110:0/537243903 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefd8144610 msgr2=0x7fefd81db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:29.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefd8144610 0x7fefd81db440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 1 -- 172.21.15.110:0/537243903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefd81454a0 msgr2=0x7fefd813d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:29.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefd81454a0 0x7fefd813d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:29.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:29.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 1 monclient: found mon.noname-b 2024-01-20T11:15:29.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:29.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:29.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.612+0000 7fefd7fff640 1 -- 172.21.15.110:0/537243903 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd8135ef0 con 0x7fefc0003970 2024-01-20T11:15:29.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.615+0000 7fefd7fff640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefd8144ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fefc8001770 tx=0x7fefc8000fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:29.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.615+0000 7fefd4ff9640 10 client.?.objecter ms_handle_connect 0x7fefc0003970 2024-01-20T11:15:29.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.615+0000 7fefd4ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 1 -- 172.21.15.110:0/537243903 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8007240 con 0x7fefc0003970 2024-01-20T11:15:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 10 monclient: dump: 2024-01-20T11:15:29.671 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 10 monclient: _finish_auth 0 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 1 -- 172.21.15.110:0/537243903 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc80073a0 con 0x7fefc0003970 2024-01-20T11:15:29.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 1 -- 172.21.15.110:0/537243903 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc80067b0 con 0x7fefc0003970 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd4ff9640 10 monclient: dump: 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:29.673 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefde415640 5 monclient: authenticate success, global_id 4324 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefde415640 10 monclient: _renew_subs 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefde415640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefd67fc640 4 set_mon_vals no callback set 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd8136080 con 0x7fefc0003970 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefde415640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefde415640 10 monclient: _renew_subs 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefde415640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd81364e0 con 0x7fefc0003970 2024-01-20T11:15:29.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.671+0000 7fefde415640 1 librados: init done 2024-01-20T11:15:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.672+0000 7fefde415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.672+0000 7fefde415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.672+0000 7fefde415640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:29.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.672+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd80b7a30 con 0x7fefc0003970 2024-01-20T11:15:29.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.926+0000 7fefd4ff9640 1 -- 172.21.15.110:0/537243903 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fefc8030030 con 0x7fefc0003970 2024-01-20T11:15:29.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.926+0000 7fefd4ff9640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fefc406c8e0 0x7fefc406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:29.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.926+0000 7fefd77fe640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fefc406c8e0 0x7fefc406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:29.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.926+0000 7fefd77fe640 10 monclient: get_auth_request con 0x7fefc406c8e0 auth_method 0 2024-01-20T11:15:29.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.927+0000 7fefd77fe640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fefc406c8e0 0x7fefc406eda0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fefd813f000 tx=0x7fefcc00a570 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:29.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.927+0000 7fefd4ff9640 10 client.4324.objecter ms_handle_connect 0x7fefc406c8e0 2024-01-20T11:15:29.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.928+0000 7fefd4ff9640 1 -- 172.21.15.110:0/537243903 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fefc8079430 con 0x7fefc0003970 2024-01-20T11:15:29.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.928+0000 7fefd4ff9640 10 client.4324.objecter ms_dispatch 0x7fefd8000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:29.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.928+0000 7fefd4ff9640 3 client.4324.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:29.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.928+0000 7fefd4ff9640 3 client.4324.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:29.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.928+0000 7fefd4ff9640 20 client.4324.objecter dump_active .. 0 homeless 2024-01-20T11:15:29.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.940+0000 7fefd4ff9640 1 -- 172.21.15.110:0/537243903 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fefc8075070 con 0x7fefc0003970 2024-01-20T11:15:29.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.940+0000 7fefd4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:29.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:29.940+0000 7fefd4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:30.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.114+0000 7fefde415640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:30.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.114+0000 7fefde415640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:30.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.114+0000 7fefde415640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:30.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.114+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefd81db090 con 0x7fefc0003970 2024-01-20T11:15:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.148+0000 7fefd4ff9640 1 -- 172.21.15.110:0/537243903 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14712 (secure 0 0 0) 0x7fefc8078690 con 0x7fefc0003970 2024-01-20T11:15:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.148+0000 7fefd4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.148+0000 7fefd4ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:30.148 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:15:30.149 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":26,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:15:29.487735+0000","last_up_change":"2024-01-20T11:15:22.093857+0000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-20T11:15:13.023751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-20T11:15:27.038641+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6817","nonce":3175427153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6819","nonce":3175427153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6825","nonce":3175427153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6822","nonce":3175427153}]},"public_addr":"172.21.15.110:6817/3175427153","cluster_addr":"172.21.15.110:6819/3175427153","heartbeat_back_addr":"172.21.15.110:6825/3175427153","heartbeat_front_addr":"172.21.15.110:6822/3175427153","state":["exists","up"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6801","nonce":3373744578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6803","nonce":3373744578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6807","nonce":3373744578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6805","nonce":3373744578}]},"public_addr":"172.21.15.110:6801/3373744578","cluster_addr":"172.21.15.110:6803/3373744578","heartbeat_back_addr":"172.21.15.110:6807/3373744578","heartbeat_front_addr":"172.21.15.110:6805/3373744578","state":["exists","up"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6809","nonce":470946803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6811","nonce":470946803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6815","nonce":470946803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6813","nonce":470946803}]},"public_addr":"172.21.15.110:6809/470946803","cluster_addr":"172.21.15.110:6811/470946803","heartbeat_back_addr":"172.21.15.110:6815/470946803","heartbeat_front_addr":"172.21.15.110:6813/470946803","state":["exists","up"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6821","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6823","nonce":1038822226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6827","nonce":1038822226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6831","nonce":1038822226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6829","nonce":1038822226}]},"public_addr":"172.21.15.110:6823/1038822226","cluster_addr":"172.21.15.110:6827/1038822226","heartbeat_back_addr":"172.21.15.110:6831/1038822226","heartbeat_front_addr":"172.21.15.110:6829/1038822226","state":["exists","up"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6824","nonce":2594530523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6827","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6829","nonce":2594530523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6833","nonce":2594530523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6831","nonce":2594530523}]},"public_addr":"172.21.15.134:6824/2594530523","cluster_addr":"172.21.15.134:6829/2594530523","heartbeat_back_addr":"172.21.15.134:6833/2594530523","heartbeat_front_addr":"172.21.15.134:6831/2594530523","state":["exists","up"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6819","nonce":1586889676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6821","nonce":1586889676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6828","nonce":1586889676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6823","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6825","nonce":1586889676}]},"public_addr":"172.21.15.134:6819/1586889676","cluster_addr":"172.21.15.134:6821/1586889676","heartbeat_back_addr":"172.21.15.134:6828/1586889676","heartbeat_front_addr":"172.21.15.134:6825/1586889676","state":["exists","up"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6801","nonce":2798305725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6803","nonce":2798305725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6807","nonce":2798305725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6805","nonce":2798305725}]},"public_addr":"172.21.15.134:6801/2798305725","cluster_addr":"172.21.15.134:6803/2798305725","heartbeat_back_addr":"172.21.15.134:6807/2798305725","heartbeat_front_addr":"172.21.15.134:6805/2798305725","state":["exists","up"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6809","nonce":604027796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6811","nonce":604027796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6815","nonce":604027796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6813","nonce":604027796}]},"public_addr":"172.21.15.134:6809/604027796","cluster_addr":"172.21.15.134:6811/604027796","heartbeat_back_addr":"172.21.15.134:6815/604027796","heartbeat_front_addr":"172.21.15.134:6813/604027796","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.482016+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.271832+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:08.974739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.596210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:10.058818+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:10.303347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:08.463225+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.463577+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:15:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 10 librados: watch_flush enter 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 10 librados: watch_flush exit 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 20 client.4324.objecter shutdown clearing up homeless session... 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 10 client.4324.objecter successfully canceled tick 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fefc406c8e0 msgr2=0x7fefc406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fefc406c8e0 0x7fefc406eda0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fefd813f000 tx=0x7fefcc00a570 comp rx=0 tx=0).stop 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 10 monclient: shutdown 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 msgr2=0x7fefd8144ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefd8144ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fefc8001770 tx=0x7fefc8000fb0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 shutdown_connections 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fefc406c8e0 0x7fefc406eda0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefd8144610 0x7fefd81db440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefd81454a0 0x7fefd813d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 --2- 172.21.15.110:0/537243903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc0003970 0x7fefd8144ed0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 >> 172.21.15.110:0/537243903 conn(0x7fefd80b7bc0 msgr2=0x7fefd80b71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:30.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 shutdown_connections 2024-01-20T11:15:30.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 -- 172.21.15.110:0/537243903 wait complete. 2024-01-20T11:15:30.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.150+0000 7fefde415640 1 librados: shutdown 2024-01-20T11:15:30.161 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-20T11:15:30.161 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.260+0000 7feeac2da640 10 monclient: get_monmap_and_config 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.260+0000 7feeac2da640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.260+0000 7feeac2da640 10 monclient: monmap: 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:30.260703+0000 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:30.260703+0000 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:30.260 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:30.261 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:30.261 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:30.261 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:30.261 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.260+0000 7feeac2da640 1 Processor -- start 2024-01-20T11:15:30.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.260+0000 7feeac2da640 1 -- start start 2024-01-20T11:15:30.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.260+0000 7feeac2da640 10 monclient: init 2024-01-20T11:15:30.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feea40a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feea40a9ad0 0x7feea41423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 10 monclient(hunting): picked mon.noname-a con 0x7feea40a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feea4142920 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea40b25c0 con 0x7feea4142920 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea413fbe0 con 0x7feea40a96f0 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea40aaea0 con 0x7feea40a9ad0 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.261+0000 7feeac2da640 10 monclient(hunting): authenticate will time out at 1373.611690s 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.262+0000 7feea9875640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feea40a9ad0 0x7feea41423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.262+0000 7feeaa076640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7feea40a96f0 0x7feea413f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:53594/0 (socket says 172.21.15.110:53594) 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.262+0000 7feeaa076640 1 -- 172.21.15.110:0/4155773080 learned_addr learned my addr 172.21.15.110:0/4155773080 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.262+0000 7feea9875640 10 monclient(hunting): get_auth_request con 0x7feea40a9ad0 auth_method 0 2024-01-20T11:15:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.262+0000 7feea9875640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.262+0000 7feea9875640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.262+0000 7feea9875640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.263+0000 7feea9875640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.263+0000 7feea9875640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.263+0000 7feea9875640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.264+0000 7feea9074640 1 -- 172.21.15.110:0/4155773080 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865446345 0 0) 0x7feea413fbe0 con 0x7feea40a96f0 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.264+0000 7feea9074640 10 monclient(hunting): _negotiate 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.264+0000 7feea9074640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.264+0000 7feea9074640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.264+0000 7feea9074640 10 monclient(hunting): my global_id is 4327 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.264+0000 7feea9074640 1 -- 172.21.15.110:0/4155773080 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee90003520 con 0x7feea40a96f0 2024-01-20T11:15:30.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.265+0000 7feea9875640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-01-20T11:15:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.265+0000 7feea9875640 1 -- 172.21.15.110:0/4155773080 >> v1:172.21.15.134:6789/0 conn(0x7feea40a96f0 legacy=0x7feea413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.265+0000 7feea9875640 1 -- 172.21.15.110:0/4155773080 >> v1:172.21.15.110:6789/0 conn(0x7feea4142920 legacy=0x7feea414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.265+0000 7feea9875640 10 monclient: _finish_hunting 0 2024-01-20T11:15:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.265+0000 7feea9875640 1 monclient: found mon.noname-a 2024-01-20T11:15:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.265+0000 7feea9875640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.265+0000 7feea9875640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:30.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.265+0000 7feea9875640 1 -- 172.21.15.110:0/4155773080 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea40ab0a0 con 0x7feea40a9ad0 2024-01-20T11:15:30.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.269+0000 7feea9875640 1 --2- 172.21.15.110:0/4155773080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea40a9ad0 0x7feea41423e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee940097a0 tx=0x7fee9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5637096995bb1407 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 1 -- 172.21.15.110:0/4155773080 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94031830 con 0x7feea40a9ad0 2024-01-20T11:15:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 10 monclient: dump: 2024-01-20T11:15:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 10 monclient: _finish_auth 0 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 10 monclient: _check_auth_tickets 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 1 -- 172.21.15.110:0/4155773080 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee94031990 con 0x7feea40a9ad0 2024-01-20T11:15:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 1 -- 172.21.15.110:0/4155773080 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94031d20 con 0x7feea40a9ad0 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feea9074640 10 monclient: dump: 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feeac2da640 5 monclient: authenticate success, global_id 4293 2024-01-20T11:15:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feeac2da640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:30.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.272+0000 7feeac2da640 4 set_mon_vals no callback set 2024-01-20T11:15:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.273+0000 7feeac2da640 10 monclient: shutdown 2024-01-20T11:15:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.273+0000 7feeac2da640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.273+0000 7feeac2da640 1 -- 172.21.15.110:0/4155773080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea40a9ad0 msgr2=0x7feea41423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.273+0000 7feeac2da640 1 --2- 172.21.15.110:0/4155773080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea40a9ad0 0x7feea41423e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee940097a0 tx=0x7fee9402b8d0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.273+0000 7feeac2da640 1 -- 172.21.15.110:0/4155773080 shutdown_connections 2024-01-20T11:15:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.273+0000 7feeac2da640 1 --2- 172.21.15.110:0/4155773080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea40a9ad0 0x7feea41423e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.273+0000 7feeac2da640 1 -- 172.21.15.110:0/4155773080 >> 172.21.15.110:0/4155773080 conn(0x7feea40b7bc0 msgr2=0x7feea40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.273+0000 7feeac2da640 1 -- 172.21.15.110:0/4155773080 shutdown_connections 2024-01-20T11:15:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.273+0000 7feeac2da640 1 -- 172.21.15.110:0/4155773080 wait complete. 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.274+0000 7feeac2da640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.274+0000 7feeac2da640 10 monclient: monmap: 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.274+0000 7feeac2da640 1 librados: starting msgr at 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.274+0000 7feeac2da640 1 librados: starting objecter 2024-01-20T11:15:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.274+0000 7feeac2da640 1 Processor -- start 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.274+0000 7feeac2da640 1 -- start start 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.274+0000 7feeac2da640 1 librados: setting wanted keys 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.274+0000 7feeac2da640 1 librados: calling monclient init 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.274+0000 7feeac2da640 10 monclient: init 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feea40a96f0 0x7feea4138570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient(hunting): picked mon.noname-c con 0x7feea40a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea40a9ad0 0x7feea4138ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient(hunting): picked mon.noname-b con 0x7feea40a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea4142920 0x7feea41dae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient(hunting): picked mon.noname-a con 0x7feea4142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea41dbb50 con 0x7feea4142920 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea4138ff0 con 0x7feea40a9ad0 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea4139720 con 0x7feea40a96f0 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeac2da640 10 monclient(hunting): authenticate will time out at 1373.625681s 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.275+0000 7feeaa877640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea4142920 0x7feea41dae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feea9875640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea40a9ad0 0x7feea4138ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa076640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feea40a96f0 0x7feea4138570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa877640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea4142920 0x7feea41dae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48338/0 (socket says 172.21.15.110:48338) 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa877640 1 -- 172.21.15.110:0/26613267 learned_addr learned my addr 172.21.15.110:0/26613267 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa877640 10 monclient(hunting): get_auth_request con 0x7feea4142920 auth_method 0 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa877640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa877640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa877640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feea9875640 10 monclient(hunting): get_auth_request con 0x7feea40a9ad0 auth_method 0 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feea9875640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feea9875640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feea9875640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa076640 10 monclient(hunting): get_auth_request con 0x7feea40a96f0 auth_method 0 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa076640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa076640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa076640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feea9875640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feea9875640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feea9875640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa877640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa877640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.276+0000 7feeaa877640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.277+0000 7feeaa076640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.277+0000 7feeaa076640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:30.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.277+0000 7feeaa076640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:30.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-01-20T11:15:30.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 1 -- 172.21.15.110:0/26613267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feea40a96f0 msgr2=0x7feea4138570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:30.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feea40a96f0 0x7feea4138570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 1 -- 172.21.15.110:0/26613267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea4142920 msgr2=0x7feea41dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea4142920 0x7feea41dae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 10 monclient: _finish_hunting 0 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 1 monclient: found mon.noname-b 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.279+0000 7feea9875640 1 -- 172.21.15.110:0/26613267 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee94009400 con 0x7feea40a9ad0 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.282+0000 7feea9875640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea40a9ad0 0x7feea4138ab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fee94033000 tx=0x7fee940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.282+0000 7fee9affd640 10 client.?.objecter ms_handle_connect 0x7feea40a9ad0 2024-01-20T11:15:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.282+0000 7fee9affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 1 -- 172.21.15.110:0/26613267 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9403b9f0 con 0x7feea40a9ad0 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 10 monclient: dump: 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:30.287 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 10 monclient: _finish_auth 0 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 1 -- 172.21.15.110:0/26613267 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9403bb70 con 0x7feea40a9ad0 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feeac2da640 5 monclient: authenticate success, global_id 4330 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feeac2da640 10 monclient: _renew_subs 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feeac2da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea4139940 con 0x7feea40a9ad0 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 1 -- 172.21.15.110:0/26613267 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94045400 con 0x7feea40a9ad0 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:30.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7fee9affd640 10 monclient: dump: 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feeac2da640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feeac2da640 10 monclient: _renew_subs 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feea8873640 4 set_mon_vals no callback set 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feeac2da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea41db560 con 0x7feea40a9ad0 2024-01-20T11:15:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.286+0000 7feeac2da640 1 librados: init done 2024-01-20T11:15:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.287+0000 7feeac2da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.287+0000 7feeac2da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.287+0000 7feeac2da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.287+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea411fba0 con 0x7feea40a9ad0 2024-01-20T11:15:30.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.307+0000 7fee9affd640 1 -- 172.21.15.110:0/26613267 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fee9404e440 con 0x7feea40a9ad0 2024-01-20T11:15:30.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.307+0000 7fee9affd640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee8806c8e0 0x7fee8806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.307+0000 7feeaa076640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee8806c8e0 0x7fee8806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:30.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.307+0000 7feeaa076640 10 monclient: get_auth_request con 0x7fee8806c8e0 auth_method 0 2024-01-20T11:15:30.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.308+0000 7feeaa076640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee8806c8e0 0x7fee8806eda0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feea00011e0 tx=0x7feea0007d30 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:30.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.308+0000 7fee9affd640 10 client.4330.objecter ms_handle_connect 0x7fee8806c8e0 2024-01-20T11:15:30.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.542+0000 7fee9affd640 1 -- 172.21.15.110:0/26613267 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fee9407a710 con 0x7feea40a9ad0 2024-01-20T11:15:30.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.542+0000 7fee9affd640 10 client.4330.objecter ms_dispatch 0x7feea4000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:30.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.542+0000 7fee9affd640 3 client.4330.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:30.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.542+0000 7fee9affd640 3 client.4330.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:30.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.542+0000 7fee9affd640 20 client.4330.objecter dump_active .. 0 homeless 2024-01-20T11:15:30.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.552+0000 7fee9affd640 1 -- 172.21.15.110:0/26613267 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fee94076a50 con 0x7feea40a9ad0 2024-01-20T11:15:30.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.552+0000 7fee9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:30.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.552+0000 7fee9affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:30.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.755+0000 7feeac2da640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:30.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.755+0000 7feeac2da640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:30.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.755+0000 7feeac2da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:30.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.755+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feea4075490 con 0x7feea40a9ad0 2024-01-20T11:15:30.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.782+0000 7fee9affd640 1 -- 172.21.15.110:0/26613267 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14712 (secure 0 0 0) 0x7fee94035090 con 0x7feea40a9ad0 2024-01-20T11:15:30.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.782+0000 7fee9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-20T11:15:30.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.782+0000 7fee9affd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:30.783 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:15:30.783 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":26,"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","created":"2024-01-20T11:14:25.570462+0000","modified":"2024-01-20T11:15:29.487735+0000","last_up_change":"2024-01-20T11:15:22.093857+0000","last_in_change":"2024-01-20T11:14:30.958130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-20T11:15:13.023751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-20T11:15:27.038641+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34615fe4-d784-4baa-93fe-23f986a84457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6817","nonce":3175427153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6819","nonce":3175427153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6825","nonce":3175427153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3175427153},{"type":"v1","addr":"172.21.15.110:6822","nonce":3175427153}]},"public_addr":"172.21.15.110:6817/3175427153","cluster_addr":"172.21.15.110:6819/3175427153","heartbeat_back_addr":"172.21.15.110:6825/3175427153","heartbeat_front_addr":"172.21.15.110:6822/3175427153","state":["exists","up"]},{"osd":1,"uuid":"09fd7abf-c068-48f2-8166-35df5fa549b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6801","nonce":3373744578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6803","nonce":3373744578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6807","nonce":3373744578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":3373744578},{"type":"v1","addr":"172.21.15.110:6805","nonce":3373744578}]},"public_addr":"172.21.15.110:6801/3373744578","cluster_addr":"172.21.15.110:6803/3373744578","heartbeat_back_addr":"172.21.15.110:6807/3373744578","heartbeat_front_addr":"172.21.15.110:6805/3373744578","state":["exists","up"]},{"osd":2,"uuid":"90618c4b-85ed-4ae8-8ae4-c8627385de33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6809","nonce":470946803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6811","nonce":470946803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6815","nonce":470946803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":470946803},{"type":"v1","addr":"172.21.15.110:6813","nonce":470946803}]},"public_addr":"172.21.15.110:6809/470946803","cluster_addr":"172.21.15.110:6811/470946803","heartbeat_back_addr":"172.21.15.110:6815/470946803","heartbeat_front_addr":"172.21.15.110:6813/470946803","state":["exists","up"]},{"osd":3,"uuid":"f83babc8-dca0-4c13-9bdd-b643ece19536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6821","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6823","nonce":1038822226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6827","nonce":1038822226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6831","nonce":1038822226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1038822226},{"type":"v1","addr":"172.21.15.110:6829","nonce":1038822226}]},"public_addr":"172.21.15.110:6823/1038822226","cluster_addr":"172.21.15.110:6827/1038822226","heartbeat_back_addr":"172.21.15.110:6831/1038822226","heartbeat_front_addr":"172.21.15.110:6829/1038822226","state":["exists","up"]},{"osd":4,"uuid":"27eba313-a00b-4455-bd52-efe2b7792812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6824","nonce":2594530523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6827","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6829","nonce":2594530523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6833","nonce":2594530523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":2594530523},{"type":"v1","addr":"172.21.15.134:6831","nonce":2594530523}]},"public_addr":"172.21.15.134:6824/2594530523","cluster_addr":"172.21.15.134:6829/2594530523","heartbeat_back_addr":"172.21.15.134:6833/2594530523","heartbeat_front_addr":"172.21.15.134:6831/2594530523","state":["exists","up"]},{"osd":5,"uuid":"1747bfe4-6d9b-4d04-90ef-3d2d36bda8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6819","nonce":1586889676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6821","nonce":1586889676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6828","nonce":1586889676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6823","nonce":1586889676},{"type":"v1","addr":"172.21.15.134:6825","nonce":1586889676}]},"public_addr":"172.21.15.134:6819/1586889676","cluster_addr":"172.21.15.134:6821/1586889676","heartbeat_back_addr":"172.21.15.134:6828/1586889676","heartbeat_front_addr":"172.21.15.134:6825/1586889676","state":["exists","up"]},{"osd":6,"uuid":"7fdee016-2887-40d6-ad61-4e4eb6df9fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6801","nonce":2798305725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6803","nonce":2798305725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6807","nonce":2798305725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2798305725},{"type":"v1","addr":"172.21.15.134:6805","nonce":2798305725}]},"public_addr":"172.21.15.134:6801/2798305725","cluster_addr":"172.21.15.134:6803/2798305725","heartbeat_back_addr":"172.21.15.134:6807/2798305725","heartbeat_front_addr":"172.21.15.134:6805/2798305725","state":["exists","up"]},{"osd":7,"uuid":"a63e46f9-9dd8-41d1-afc6-c5c317345707","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6809","nonce":604027796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6811","nonce":604027796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6815","nonce":604027796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":604027796},{"type":"v1","addr":"172.21.15.134:6813","nonce":604027796}]},"public_addr":"172.21.15.134:6809/604027796","cluster_addr":"172.21.15.134:6811/604027796","heartbeat_back_addr":"172.21.15.134:6815/604027796","heartbeat_front_addr":"172.21.15.134:6813/604027796","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.482016+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.271832+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:08.974739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.596210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:10.058818+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:10.303347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:08.463225+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-20T11:15:09.463577+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-20T11:15:30.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.784+0000 7feeac2da640 10 librados: watch_flush enter 2024-01-20T11:15:30.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.784+0000 7feeac2da640 10 librados: watch_flush exit 2024-01-20T11:15:30.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.784+0000 7feeac2da640 20 client.4330.objecter shutdown clearing up homeless session... 2024-01-20T11:15:30.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.784+0000 7feeac2da640 10 client.4330.objecter successfully canceled tick 2024-01-20T11:15:30.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee8806c8e0 msgr2=0x7fee8806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:30.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee8806c8e0 0x7fee8806eda0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feea00011e0 tx=0x7feea0007d30 comp rx=0 tx=0).stop 2024-01-20T11:15:30.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 10 monclient: shutdown 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea40a9ad0 msgr2=0x7feea4138ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea40a9ad0 0x7feea4138ab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fee94033000 tx=0x7fee940046c0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 shutdown_connections 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee8806c8e0 0x7fee8806eda0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea4142920 0x7feea41dae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea40a9ad0 0x7feea4138ab0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 --2- 172.21.15.110:0/26613267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feea40a96f0 0x7feea4138570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 >> 172.21.15.110:0/26613267 conn(0x7feea40b7bc0 msgr2=0x7feea40b73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 shutdown_connections 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 -- 172.21.15.110:0/26613267 wait complete. 2024-01-20T11:15:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.785+0000 7feeac2da640 1 librados: shutdown 2024-01-20T11:15:30.796 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-20T11:15:30.796 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-20T11:15:30.797 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-20T11:15:30.797 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-20T11:15:30.797 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-20T11:15:30.798 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-20T11:15:30.798 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-20T11:15:30.799 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-20T11:15:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.969+0000 7f91ca856640 10 monclient: get_monmap_and_config 2024-01-20T11:15:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.969+0000 7f91ca856640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.969+0000 7f91ca856640 10 monclient: monmap: 2024-01-20T11:15:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:30.970262+0000 2024-01-20T11:15:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:30.970262+0000 2024-01-20T11:15:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:30.971 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.969+0000 7f3789c4f640 10 monclient: get_monmap_and_config 2024-01-20T11:15:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.969+0000 7f3789c4f640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.969+0000 7f3789c4f640 10 monclient: monmap: 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:30.969683+0000 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:30.969683+0000 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:30.972 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.970+0000 7f3789c4f640 1 Processor -- start 2024-01-20T11:15:30.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.969+0000 7f8854946640 10 monclient: get_monmap_and_config 2024-01-20T11:15:30.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.969+0000 7f8854946640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.969+0000 7f8854946640 10 monclient: monmap: 2024-01-20T11:15:30.973 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.973 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:30.970298+0000 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:30.970298+0000 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.970+0000 7f8854946640 1 Processor -- start 2024-01-20T11:15:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.970+0000 7f8854946640 1 -- start start 2024-01-20T11:15:30.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.970+0000 7f8854946640 10 monclient: init 2024-01-20T11:15:30.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.970+0000 7f3789c4f640 1 -- start start 2024-01-20T11:15:30.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.970+0000 7f3789c4f640 10 monclient: init 2024-01-20T11:15:30.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f91ca856640 1 Processor -- start 2024-01-20T11:15:30.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37840ae120 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37840aeb00 0x7f37840acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 10 monclient(hunting): picked mon.noname-b con 0x7f37840aeb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f37840ad4e0 0x7f37840ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 10 monclient(hunting): picked mon.noname-a con 0x7f37840ad4e0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37840ac650 con 0x7f37840ae120 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37840ab810 con 0x7f37840ad4e0 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840aef20 con 0x7f37840aeb00 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f3789c4f640 10 monclient(hunting): authenticate will time out at 1374.321536s 2024-01-20T11:15:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f91ca856640 1 -- start start 2024-01-20T11:15:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f91ca856640 10 monclient: init 2024-01-20T11:15:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-01-20T11:15:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8848034c00 0x7f88480c3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 10 monclient(hunting): picked mon.noname-b con 0x7f8848034c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f884802dfc0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f88480c4870 0x7f88480c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 10 monclient(hunting): picked mon.noname-a con 0x7f88480c4870 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884802f340 con 0x7f884802dfc0 2024-01-20T11:15:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8848034fe0 con 0x7f88480c4870 2024-01-20T11:15:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8848030780 con 0x7f8848034c00 2024-01-20T11:15:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f8854946640 10 monclient(hunting): authenticate will time out at 1374.321888s 2024-01-20T11:15:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f12adf89640 10 monclient: get_monmap_and_config 2024-01-20T11:15:30.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f12adf89640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:30.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-01-20T11:15:30.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.971+0000 7f12adf89640 10 monclient: monmap: 2024-01-20T11:15:30.979 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.979 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:30.971801+0000 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:30.971801+0000 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:30.980 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91c40b0c00 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91c4143b90 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f91c40aa740 0x7f91c40ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 10 monclient(hunting): picked mon.noname-a con 0x7f91c40aa740 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c40ab240 con 0x7f91c40b0c00 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c40ab6d0 con 0x7f91c4143b90 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c40ac770 con 0x7f91c40aa740 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f91ca856640 10 monclient(hunting): authenticate will time out at 1374.322463s 2024-01-20T11:15:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f3783fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f37840ad4e0 0x7f37840ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f3783fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f37840ad4e0 0x7f37840ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48348/0 (socket says 172.21.15.110:48348) 2024-01-20T11:15:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f3783fff640 1 -- 172.21.15.110:0/3774042537 learned_addr learned my addr 172.21.15.110:0/3774042537 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f3783fff640 10 monclient(hunting): get_auth_request con 0x7f37840ad4e0 auth_method 0 2024-01-20T11:15:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f3783fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f3783fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f3783fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f12adf89640 1 Processor -- start 2024-01-20T11:15:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f12adf89640 1 -- start start 2024-01-20T11:15:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f12adf89640 10 monclient: init 2024-01-20T11:15:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f884ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8848034c00 0x7f88480c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f884ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8848034c00 0x7f88480c3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33660/0 (socket says 172.21.15.110:33660) 2024-01-20T11:15:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f884ed9d640 1 -- 172.21.15.110:0/2367531897 learned_addr learned my addr 172.21.15.110:0/2367531897 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f884ed9d640 10 monclient(hunting): get_auth_request con 0x7f8848034c00 auth_method 0 2024-01-20T11:15:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f884ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f884ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f884ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f91c37fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f91c4143b90 0x7f91c40acb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:53614/0 (socket says 172.21.15.110:53614) 2024-01-20T11:15:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f5fdef1a640 10 monclient: get_monmap_and_config 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f5fdef1a640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.972+0000 7f5fdef1a640 10 monclient: monmap: 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:30.973283+0000 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:30.973283+0000 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:30.986 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f91c37fe640 1 -- 172.21.15.110:0/4109202669 learned_addr learned my addr 172.21.15.110:0/4109202669 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.973+0000 7f3782ffd640 1 --2- 172.21.15.110:0/3774042537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37840aeb00 0x7f37840acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f12adf89640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f5fdef1a640 1 Processor -- start 2024-01-20T11:15:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f12adf89640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-20T11:15:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f12adf89640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f12a80aeb90 0x7f12a813b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f3783fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f3783fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f3783fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f12adf89640 10 monclient(hunting): picked mon.noname-c con 0x7f12a80aeb90 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f12adf89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12a813b890 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f12adf89640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f12a80ad6b0 0x7f12a8148810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f12adf89640 10 monclient(hunting): picked mon.noname-a con 0x7f12a80ad6b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7fc33b11d640 10 monclient: get_monmap_and_config 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7fc33b11d640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7fc33b11d640 10 monclient: monmap: 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:30.975314+0000 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:30.975314+0000 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:30.990 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f12adf89640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a80ac650 con 0x7f12a813b890 2024-01-20T11:15:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f12adf89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f12adf89640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a80aef70 con 0x7f12a80ad6b0 2024-01-20T11:15:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f12adf89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f12adf89640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a80ab810 con 0x7f12a80aeb90 2024-01-20T11:15:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f12adf89640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f12adf89640 10 monclient(hunting): authenticate will time out at 1374.325632s 2024-01-20T11:15:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f3782ffd640 10 monclient(hunting): get_auth_request con 0x7f37840aeb00 auth_method 0 2024-01-20T11:15:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f3782ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f3782ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f3782ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f5fdef1a640 1 -- start start 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.974+0000 7f5fdef1a640 10 monclient: init 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5fd80af440 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80aab00 0x7f5fd80ad670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd80aab00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5fd813aca0 0x7f5fd813b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd813aca0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80ac860 con 0x7f5fd80af440 2024-01-20T11:15:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd80ac5f0 con 0x7f5fd813aca0 2024-01-20T11:15:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd80af820 con 0x7f5fd80aab00 2024-01-20T11:15:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f5fdef1a640 10 monclient(hunting): authenticate will time out at 1374.325548s 2024-01-20T11:15:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f884ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f884ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f884ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7fc33b11d640 1 Processor -- start 2024-01-20T11:15:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f6ccacb4640 10 monclient: get_monmap_and_config 2024-01-20T11:15:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f6ccacb4640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7f6ccacb4640 10 monclient: monmap: 2024-01-20T11:15:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:30.976500+0000 2024-01-20T11:15:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:30.976500+0000 2024-01-20T11:15:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:30.996 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7fc33b11d640 1 -- start start 2024-01-20T11:15:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7fc33b11d640 10 monclient: init 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7faf8a70f640 10 monclient: get_monmap_and_config 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.975+0000 7faf8a70f640 10 monclient: build_initial_monmap 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7faf8a70f640 10 monclient: monmap: 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:30.976586+0000 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:30.976586+0000 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:30.998 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:30.998 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:30.998 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:30.998 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:30.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:30.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc33406fa90 0x7fc33413cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 10 monclient(hunting): picked mon.noname-c con 0x7fc33406fa90 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 0x7fc334143470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 10 monclient(hunting): picked mon.noname-b con 0x7fc33413d500 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3341439b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc334071790 con 0x7fc3341439b0 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33413d940 con 0x7fc33406fa90 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc334071a70 con 0x7fc33413d500 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.976+0000 7fc33b11d640 10 monclient(hunting): authenticate will time out at 1374.326986s 2024-01-20T11:15:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f6ccacb4640 1 Processor -- start 2024-01-20T11:15:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f91c8df3640 1 --2- 172.21.15.110:0/4109202669 >> v2:172.21.15.110:3300/0 conn(0x7f91c40aa740 0x7f91c40ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f6ccacb4640 1 -- start start 2024-01-20T11:15:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f6ccacb4640 10 monclient: init 2024-01-20T11:15:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-01-20T11:15:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f3783fff640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-01-20T11:15:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f3783fff640 1 -- 172.21.15.110:0/3774042537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37840aeb00 msgr2=0x7f37840acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f3783fff640 1 --2- 172.21.15.110:0/3774042537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37840aeb00 0x7f37840acfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f3783fff640 1 -- 172.21.15.110:0/3774042537 >> v1:172.21.15.134:6789/0 conn(0x7f37840ae120 legacy=0x7f378413b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f3783fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f3783fff640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f3783fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f3783fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f3783fff640 1 -- 172.21.15.110:0/3774042537 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37840ac8a0 con 0x7f37840ad4e0 2024-01-20T11:15:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7faf8a70f640 1 Processor -- start 2024-01-20T11:15:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7faf8a70f640 1 -- start start 2024-01-20T11:15:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7faf8a70f640 10 monclient: init 2024-01-20T11:15:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faf8406f920 0x7faf8406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 10 monclient(hunting): picked mon.noname-a con 0x7faf8406f920 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf8413d230 0x7faf8413d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 10 monclient(hunting): picked mon.noname-b con 0x7faf8413d230 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf84143950 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840715b0 con 0x7faf84143950 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf84071890 con 0x7faf8406f920 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf84071b70 con 0x7faf8413d230 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7faf8a70f640 10 monclient(hunting): authenticate will time out at 1374.328437s 2024-01-20T11:15:31.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.977+0000 7f12a77fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f12a80aeb90 0x7f12a813b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cc40b04c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:31.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6cc40705b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:31.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40aecd0 0x7f6cc414a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f91c8df3640 10 monclient(hunting): get_auth_request con 0x7f91c40aa740 auth_method 0 2024-01-20T11:15:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f91c8df3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f91c8df3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f91c8df3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 10 monclient(hunting): picked mon.noname-b con 0x7f6cc40aecd0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc40b10f0 con 0x7f6cc40b04c0 2024-01-20T11:15:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc406fa60 con 0x7f6cc40705b0 2024-01-20T11:15:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc40b08a0 con 0x7f6cc40aecd0 2024-01-20T11:15:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f6ccacb4640 10 monclient(hunting): authenticate will time out at 1374.328782s 2024-01-20T11:15:31.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7fc3396ba640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc3341439b0 0x7fc33413fc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:53638/0 (socket says 172.21.15.110:53638) 2024-01-20T11:15:31.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7fc3396ba640 1 -- 172.21.15.110:0/1284778848 learned_addr learned my addr 172.21.15.110:0/1284778848 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f884ed9d640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-01-20T11:15:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f884ed9d640 1 -- 172.21.15.110:0/2367531897 >> v2:172.21.15.110:3300/0 conn(0x7f88480c4870 msgr2=0x7f88480c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f884ed9d640 1 --2- 172.21.15.110:0/2367531897 >> v2:172.21.15.110:3300/0 conn(0x7f88480c4870 0x7f88480c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f884ed9d640 1 -- 172.21.15.110:0/2367531897 >> v1:172.21.15.110:6789/0 conn(0x7f884802dfc0 legacy=0x7f88480c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f884ed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f884ed9d640 1 monclient: found mon.noname-b 2024-01-20T11:15:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f884ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f884ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f884ed9d640 1 -- 172.21.15.110:0/2367531897 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884802c630 con 0x7f8848034c00 2024-01-20T11:15:31.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f12a6ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f12a813b890 0x7f12a80acfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:53622/0 (socket says 172.21.15.110:53622) 2024-01-20T11:15:31.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.978+0000 7f12a6ffd640 1 -- 172.21.15.110:0/574011621 learned_addr learned my addr 172.21.15.110:0/574011621 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:31.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f12a77fe640 10 monclient(hunting): get_auth_request con 0x7f12a80aeb90 auth_method 0 2024-01-20T11:15:31.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f12a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f12a77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f12a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f91c8df3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f91c8df3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.979+0000 7f91c8df3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.980+0000 7f5fdd717640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80aab00 0x7f5fd80ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.981+0000 7f5fde719640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5fd813aca0 0x7f5fd813b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.981+0000 7f884ed9d640 1 --2- 172.21.15.110:0/2367531897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8848034c00 0x7f88480c3570 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8840009870 tx=0x7f884002b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2bb6ef21413a6c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.982+0000 7f91c8df3640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2024-01-20T11:15:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.982+0000 7f91c8df3640 1 -- 172.21.15.110:0/4109202669 >> v1:172.21.15.134:6789/0 conn(0x7f91c4143b90 legacy=0x7f91c40acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.982+0000 7f91c8df3640 1 -- 172.21.15.110:0/4109202669 >> v1:172.21.15.110:6789/0 conn(0x7f91c40b0c00 legacy=0x7f91c4143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.982+0000 7f91c8df3640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.982+0000 7f91c8df3640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.982+0000 7f91c8df3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.982+0000 7f91c8df3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.982+0000 7f91c8df3640 1 -- 172.21.15.110:0/4109202669 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c40ad770 con 0x7f91c40aa740 2024-01-20T11:15:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f3783fff640 1 --2- 172.21.15.110:0/3774042537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37840ad4e0 0x7f37840ad930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3778009870 tx=0x7f377802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecaae6125d25bdbf server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f5fdd717640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80aab00 0x7f5fd80ad670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33674/0 (socket says 172.21.15.110:33674) 2024-01-20T11:15:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f5fdd717640 1 -- 172.21.15.110:0/4221860459 learned_addr learned my addr 172.21.15.110:0/4221860459 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f5fdd717640 10 monclient(hunting): get_auth_request con 0x7f5fd80aab00 auth_method 0 2024-01-20T11:15:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f5fdd717640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f5fdd717640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f5fdd717640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f5fdd717640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f5fdd717640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.983+0000 7f5fdd717640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7f12a77fe640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-01-20T11:15:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7f12a77fe640 1 -- 172.21.15.110:0/574011621 >> v2:172.21.15.110:3300/0 conn(0x7f12a80ad6b0 msgr2=0x7f12a8148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7f12a77fe640 1 --2- 172.21.15.110:0/574011621 >> v2:172.21.15.110:3300/0 conn(0x7f12a80ad6b0 0x7f12a8148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7f12a77fe640 1 -- 172.21.15.110:0/574011621 >> v1:172.21.15.134:6789/0 conn(0x7f12a813b890 legacy=0x7f12a80acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7f12a77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7f12a77fe640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7f12a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7f12a77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7f12a77fe640 1 -- 172.21.15.110:0/574011621 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a80ac8a0 con 0x7f12a80aeb90 2024-01-20T11:15:31.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.984+0000 7fc333fff640 1 --2- 172.21.15.110:0/1284778848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 0x7fc334143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fde719640 10 monclient(hunting): get_auth_request con 0x7f5fd813aca0 auth_method 0 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fde719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fde719640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fde719640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fdd717640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fdd717640 1 -- 172.21.15.110:0/4221860459 >> v2:172.21.15.110:3300/0 conn(0x7f5fd813aca0 msgr2=0x7f5fd813b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fdd717640 1 --2- 172.21.15.110:0/4221860459 >> v2:172.21.15.110:3300/0 conn(0x7f5fd813aca0 0x7f5fd813b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fdd717640 1 -- 172.21.15.110:0/4221860459 >> v1:172.21.15.110:6789/0 conn(0x7f5fd80af440 legacy=0x7f5fd80acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fdd717640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fdd717640 1 monclient: found mon.noname-b 2024-01-20T11:15:31.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fdd717640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fdd717640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.985+0000 7f5fdd717640 1 -- 172.21.15.110:0/4221860459 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd80acb40 con 0x7f5fd80aab00 2024-01-20T11:15:31.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.987+0000 7f91c8df3640 1 --2- 172.21.15.110:0/4109202669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c40aa740 0x7f91c40ad230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f91b80097a0 tx=0x7f91b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1284d8d55589650 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.989+0000 7f6cc9251640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40aecd0 0x7f6cc414a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.989+0000 7fc333fff640 10 monclient(hunting): get_auth_request con 0x7fc33413d500 auth_method 0 2024-01-20T11:15:31.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.989+0000 7fc333fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.989+0000 7fc333fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.989+0000 7fc333fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.991+0000 7f6cc9251640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40aecd0 0x7f6cc414a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33688/0 (socket says 172.21.15.110:33688) 2024-01-20T11:15:31.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.991+0000 7f6cc9251640 1 -- 172.21.15.110:0/1798914858 learned_addr learned my addr 172.21.15.110:0/1798914858 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.991+0000 7f6cc9251640 10 monclient(hunting): get_auth_request con 0x7f6cc40aecd0 auth_method 0 2024-01-20T11:15:31.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.991+0000 7f6cc9251640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.991+0000 7f6cc9251640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.991+0000 7f6cc9251640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.991+0000 7fc333fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.991+0000 7fc333fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.991+0000 7fc333fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7fc338eb9640 1 --2- 172.21.15.110:0/1284778848 >> v2:172.21.15.134:3300/0 conn(0x7fc33406fa90 0x7fc33413cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7f6cc9251640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7f6cc9251640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7f6cc9251640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7fc3337fe640 1 -- 172.21.15.110:0/1284778848 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410957778 0 0) 0x7fc334071790 con 0x7fc3341439b0 2024-01-20T11:15:31.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7fc3337fe640 10 monclient(hunting): _negotiate 2024-01-20T11:15:31.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7fc3337fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7fc3337fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7fc3337fe640 10 monclient(hunting): my global_id is 4342 2024-01-20T11:15:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7f5fdd717640 1 --2- 172.21.15.110:0/4221860459 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80aab00 0x7f5fd80ad670 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8008870 tx=0x7f5fc802df00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7168d2976c3d37d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.992+0000 7fc3337fe640 1 -- 172.21.15.110:0/1284778848 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc320003520 con 0x7fc3341439b0 2024-01-20T11:15:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.993+0000 7fc338eb9640 10 monclient(hunting): get_auth_request con 0x7fc33406fa90 auth_method 0 2024-01-20T11:15:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.993+0000 7fc338eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.993+0000 7fc338eb9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.993+0000 7fc338eb9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.993+0000 7fc338eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.993+0000 7fc338eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.993+0000 7fc338eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.994+0000 7f12a77fe640 1 --2- 172.21.15.110:0/574011621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a80aeb90 0x7f12a813b350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f12a80b1260 tx=0x7f129c02c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d8812c2aaef9550 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.994+0000 7f6cc9251640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.994+0000 7f6cc9251640 1 -- 172.21.15.110:0/1798914858 >> v1:172.21.15.134:6789/0 conn(0x7f6cc40705b0 legacy=0x7f6cc4071d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.994+0000 7f6cc9251640 1 -- 172.21.15.110:0/1798914858 >> v1:172.21.15.110:6789/0 conn(0x7f6cc40b04c0 legacy=0x7f6cc413c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.994+0000 7f6cc9251640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.994+0000 7f6cc9251640 1 monclient: found mon.noname-b 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.994+0000 7f6cc9251640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.994+0000 7f6cc9251640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.994+0000 7f6cc9251640 1 -- 172.21.15.110:0/1798914858 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc4070090 con 0x7f6cc40aecd0 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.996+0000 7fc338eb9640 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2024-01-20T11:15:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.996+0000 7fc338eb9640 1 -- 172.21.15.110:0/1284778848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 msgr2=0x7fc334143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.996+0000 7fc338eb9640 1 --2- 172.21.15.110:0/1284778848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 0x7fc334143470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.996+0000 7fc338eb9640 1 -- 172.21.15.110:0/1284778848 >> v1:172.21.15.134:6789/0 conn(0x7fc3341439b0 legacy=0x7fc33413fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.996+0000 7fc338eb9640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.996+0000 7fc338eb9640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.996+0000 7fc338eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.996+0000 7fc338eb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.996+0000 7fc338eb9640 1 -- 172.21.15.110:0/1284778848 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334071d50 con 0x7fc33406fa90 2024-01-20T11:15:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 1 -- 172.21.15.110:0/4109202669 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b802e070 con 0x7f91c40aa740 2024-01-20T11:15:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 10 monclient: dump: 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 1 -- 172.21.15.110:0/4109202669 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b80315a0 con 0x7f91c40aa740 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 1 -- 172.21.15.110:0/4109202669 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b8031930 con 0x7f91c40aa740 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91c2ffd640 10 monclient: dump: 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91ca856640 5 monclient: authenticate success, global_id 4302 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91ca856640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f91ca856640 4 set_mon_vals no callback set 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f37827fc640 1 -- 172.21.15.110:0/3774042537 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778031640 con 0x7f37840ad4e0 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:30.999+0000 7f37827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 10 monclient: dump: 2024-01-20T11:15:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 1 -- 172.21.15.110:0/3774042537 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37780317a0 con 0x7f37840ad4e0 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 1 -- 172.21.15.110:0/3774042537 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778031b30 con 0x7f37840ad4e0 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 5 monclient: authenticate success, global_id 4299 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 4 set_mon_vals no callback set 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f37827fc640 10 monclient: dump: 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.030 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.030 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.030 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f91ca856640 10 monclient: shutdown 2024-01-20T11:15:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f91ca856640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f91ca856640 1 -- 172.21.15.110:0/4109202669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c40aa740 msgr2=0x7f91c40ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f91ca856640 1 --2- 172.21.15.110:0/4109202669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c40aa740 0x7f91c40ad230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f91b80097a0 tx=0x7f91b802b840 comp rx=0 tx=0).stop 2024-01-20T11:15:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f91ca856640 1 -- 172.21.15.110:0/4109202669 shutdown_connections 2024-01-20T11:15:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f91ca856640 1 --2- 172.21.15.110:0/4109202669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c40aa740 0x7f91c40ad230 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f91ca856640 1 -- 172.21.15.110:0/4109202669 >> 172.21.15.110:0/4109202669 conn(0x7f91c40b3e60 msgr2=0x7f91c40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f91ca856640 1 -- 172.21.15.110:0/4109202669 shutdown_connections 2024-01-20T11:15:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f91ca856640 1 -- 172.21.15.110:0/4109202669 wait complete. 2024-01-20T11:15:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 10 monclient: shutdown 2024-01-20T11:15:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 1 -- 172.21.15.110:0/3774042537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37840ad4e0 msgr2=0x7f37840ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 1 --2- 172.21.15.110:0/3774042537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37840ad4e0 0x7f37840ad930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3778009870 tx=0x7f377802b8f0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 1 -- 172.21.15.110:0/3774042537 shutdown_connections 2024-01-20T11:15:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 1 --2- 172.21.15.110:0/3774042537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37840ad4e0 0x7f37840ad930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 1 --2- 172.21.15.110:0/3774042537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37840aeb00 0x7f37840acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.000+0000 7f3789c4f640 1 -- 172.21.15.110:0/3774042537 >> 172.21.15.110:0/3774042537 conn(0x7f37840b3b10 msgr2=0x7f37840b3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f3789c4f640 1 -- 172.21.15.110:0/3774042537 shutdown_connections 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f91ca856640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f91ca856640 10 monclient: monmap: 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.034 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.034 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.034 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.034 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.034 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f3789c4f640 1 -- 172.21.15.110:0/3774042537 wait complete. 2024-01-20T11:15:31.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f6cc9251640 1 --2- 172.21.15.110:0/1798914858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40aecd0 0x7f6cc414a860 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00d180 tx=0x7f6cbc02e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4db45207bc62f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f91ca856640 1 librados: starting msgr at 2024-01-20T11:15:31.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f91ca856640 1 librados: starting objecter 2024-01-20T11:15:31.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f3789c4f640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f91ca856640 1 Processor -- start 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.001+0000 7f3789c4f640 10 monclient: monmap: 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.036 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 1 -- start start 2024-01-20T11:15:31.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 1 librados: setting wanted keys 2024-01-20T11:15:31.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 1 librados: calling monclient init 2024-01-20T11:15:31.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient: init 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f3789c4f640 1 librados: starting msgr at 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f3789c4f640 1 librados: starting objecter 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f3789c4f640 1 Processor -- start 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f3789c4f640 1 -- start start 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f3789c4f640 1 librados: setting wanted keys 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f91c40aa740 0x7f91c414a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient(hunting): picked mon.noname-c con 0x7f91c40aa740 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91c40b0c00 0x7f91c4146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient(hunting): picked mon.noname-b con 0x7f91c40b0c00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c4143b90 0x7f91c41440e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient(hunting): picked mon.noname-a con 0x7f91c4143b90 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c4138830 con 0x7f91c4143b90 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c4148230 con 0x7f91c40b0c00 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c4148480 con 0x7f91c40aa740 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f91ca856640 10 monclient(hunting): authenticate will time out at 1374.352849s 2024-01-20T11:15:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f3789c4f640 1 librados: calling monclient init 2024-01-20T11:15:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7f3789c4f640 10 monclient: init 2024-01-20T11:15:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7faf88cac640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7faf84143950 0x7faf8413fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46200/0 (socket says 172.21.15.110:46200) 2024-01-20T11:15:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7faf88cac640 1 -- 172.21.15.110:0/2419214985 learned_addr learned my addr 172.21.15.110:0/2419214985 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7faf837fe640 1 --2- 172.21.15.110:0/2419214985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf8413d230 0x7faf8413d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.002+0000 7fc338eb9640 1 --2- 172.21.15.110:0/1284778848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc33406fa90 0x7fc33413cfc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc324009d90 tx=0x7fc32402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3c25627c845d315 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:15:31.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37840ad4e0 0x7f378412e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient(hunting): picked mon.noname-b con 0x7f37840ad4e0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37840ae120 0x7f37841302a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7faf837fe640 10 monclient(hunting): get_auth_request con 0x7faf8413d230 auth_method 0 2024-01-20T11:15:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7faf837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7faf837fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7faf837fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient(hunting): picked mon.noname-a con 0x7f37840ae120 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37840aeb00 0x7f378406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient(hunting): picked mon.noname-c con 0x7f37840aeb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378412f3c0 con 0x7f37840ae120 2024-01-20T11:15:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3784071e60 con 0x7f37840ad4e0 2024-01-20T11:15:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3784071b80 con 0x7f37840aeb00 2024-01-20T11:15:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f3789c4f640 10 monclient(hunting): authenticate will time out at 1374.353382s 2024-01-20T11:15:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7faf83fff640 1 --2- 172.21.15.110:0/2419214985 >> v2:172.21.15.110:3300/0 conn(0x7faf8406f920 0x7faf8406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 1 -- 172.21.15.110:0/574011621 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c008df0 con 0x7f12a80aeb90 2024-01-20T11:15:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7faf837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7faf837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7faf837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 10 monclient: dump: 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.046 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 1 -- 172.21.15.110:0/574011621 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c00a0d0 con 0x7f12a80aeb90 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12adf89640 5 monclient: authenticate success, global_id 4336 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12adf89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12adf89640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 1 -- 172.21.15.110:0/574011621 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c00a460 con 0x7f12a80aeb90 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12adf89640 4 set_mon_vals no callback set 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.003+0000 7f12a67fc640 10 monclient: dump: 2024-01-20T11:15:31.047 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7faf83fff640 10 monclient(hunting): get_auth_request con 0x7faf8406f920 auth_method 0 2024-01-20T11:15:31.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7faf83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7faf83fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7faf83fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 10 monclient: shutdown 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 1 -- 172.21.15.110:0/574011621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a80aeb90 msgr2=0x7f12a813b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 1 --2- 172.21.15.110:0/574011621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a80aeb90 0x7f12a813b350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f12a80b1260 tx=0x7f129c02c9f0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 1 -- 172.21.15.110:0/574011621 shutdown_connections 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 1 --2- 172.21.15.110:0/574011621 >> v2:172.21.15.110:3300/0 conn(0x7f12a80ad6b0 0x7f12a8148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 1 --2- 172.21.15.110:0/574011621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a80aeb90 0x7f12a813b350 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 1 -- 172.21.15.110:0/574011621 >> 172.21.15.110:0/574011621 conn(0x7f12a80b3b10 msgr2=0x7f12a80b3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 1 -- 172.21.15.110:0/574011621 shutdown_connections 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.004+0000 7f12adf89640 1 -- 172.21.15.110:0/574011621 wait complete. 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.005+0000 7f12adf89640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.005+0000 7f12adf89640 10 monclient: monmap: 2024-01-20T11:15:31.051 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.006+0000 7f12adf89640 1 librados: starting msgr at 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.006+0000 7f12adf89640 1 librados: starting objecter 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.006+0000 7f12adf89640 1 Processor -- start 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.006+0000 7f12adf89640 1 -- start start 2024-01-20T11:15:31.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.006+0000 7f12adf89640 1 librados: setting wanted keys 2024-01-20T11:15:31.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.006+0000 7f12adf89640 1 librados: calling monclient init 2024-01-20T11:15:31.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.006+0000 7f12adf89640 10 monclient: init 2024-01-20T11:15:31.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.006+0000 7f91c8df3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c4143b90 0x7f91c41440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:31.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a80ad6b0 0x7f12a80717f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient(hunting): picked mon.noname-b con 0x7f12a80ad6b0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12a80aeb90 0x7f12a81d6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient(hunting): picked mon.noname-c con 0x7f12a80aeb90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a813b890 0x7f12a8138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient(hunting): picked mon.noname-a con 0x7f12a813b890 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a813c7e0 con 0x7f12a813b890 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8137bc0 con 0x7f12a80ad6b0 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8142230 con 0x7f12a80aeb90 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f91c37fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91c40b0c00 0x7f91c4146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7faf837fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-01-20T11:15:31.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7faf837fe640 1 -- 172.21.15.110:0/2419214985 >> v2:172.21.15.110:3300/0 conn(0x7faf8406f920 msgr2=0x7faf8406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f12adf89640 10 monclient(hunting): authenticate will time out at 1374.357691s 2024-01-20T11:15:31.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7faf837fe640 1 --2- 172.21.15.110:0/2419214985 >> v2:172.21.15.110:3300/0 conn(0x7faf8406f920 0x7faf8406fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7faf837fe640 1 -- 172.21.15.110:0/2419214985 >> v1:172.21.15.110:6789/0 conn(0x7faf84143950 legacy=0x7faf8413fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7faf837fe640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7faf837fe640 1 monclient: found mon.noname-b 2024-01-20T11:15:31.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7faf837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7faf837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7faf837fe640 1 -- 172.21.15.110:0/2419214985 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf84143430 con 0x7faf8413d230 2024-01-20T11:15:31.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f91c8df3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c4143b90 0x7f91c41440e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48414/0 (socket says 172.21.15.110:48414) 2024-01-20T11:15:31.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f91c8df3640 1 -- 172.21.15.110:0/1352040462 learned_addr learned my addr 172.21.15.110:0/1352040462 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f91c8df3640 10 monclient(hunting): get_auth_request con 0x7f91c4143b90 auth_method 0 2024-01-20T11:15:31.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f91c8df3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f91c8df3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.007+0000 7f91c8df3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 1 -- 172.21.15.110:0/1284778848 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32402cbd0 con 0x7fc33406fa90 2024-01-20T11:15:31.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 10 monclient: dump: 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 1 -- 172.21.15.110:0/1284778848 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32402cd30 con 0x7fc33406fa90 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 1 -- 172.21.15.110:0/1284778848 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32402cea0 con 0x7fc33406fa90 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc3337fe640 10 monclient: dump: 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.061 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc33b11d640 5 monclient: authenticate success, global_id 4348 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc33b11d640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7fc33b11d640 4 set_mon_vals no callback set 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7f91c37fe640 10 monclient(hunting): get_auth_request con 0x7f91c40b0c00 auth_method 0 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7f91c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7f91c37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7f91c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.008+0000 7f37837fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37840ad4e0 0x7f378412e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 10 monclient: shutdown 2024-01-20T11:15:31.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 1 -- 172.21.15.110:0/1284778848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc33406fa90 msgr2=0x7fc33413cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 1 --2- 172.21.15.110:0/1284778848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc33406fa90 0x7fc33413cfc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc324009d90 tx=0x7fc32402be20 comp rx=0 tx=0).stop 2024-01-20T11:15:31.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 1 -- 172.21.15.110:0/1284778848 shutdown_connections 2024-01-20T11:15:31.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 1 --2- 172.21.15.110:0/1284778848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 0x7fc334143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 1 --2- 172.21.15.110:0/1284778848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc33406fa90 0x7fc33413cfc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 1 -- 172.21.15.110:0/1284778848 >> 172.21.15.110:0/1284778848 conn(0x7fc3340b57e0 msgr2=0x7fc3340b5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 1 -- 172.21.15.110:0/1284778848 shutdown_connections 2024-01-20T11:15:31.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7f91c8df3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7f91c8df3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7f91c8df3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.009+0000 7fc33b11d640 1 -- 172.21.15.110:0/1284778848 wait complete. 2024-01-20T11:15:31.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7f91c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7f91c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7f91c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7fc33b11d640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7fc33b11d640 10 monclient: monmap: 2024-01-20T11:15:31.066 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.066 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.066 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7fc33b11d640 1 librados: starting msgr at 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7fc33b11d640 1 librados: starting objecter 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7fc33b11d640 1 Processor -- start 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7fc33b11d640 1 -- start start 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7fc33b11d640 1 librados: setting wanted keys 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7fc33b11d640 1 librados: calling monclient init 2024-01-20T11:15:31.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7fc33b11d640 10 monclient: init 2024-01-20T11:15:31.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7f37837fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37840ad4e0 0x7f378412e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58046/0 (socket says 172.21.15.110:58046) 2024-01-20T11:15:31.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7f37837fe640 1 -- 172.21.15.110:0/268917190 learned_addr learned my addr 172.21.15.110:0/268917190 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7f37837fe640 10 monclient(hunting): get_auth_request con 0x7f37840ad4e0 auth_method 0 2024-01-20T11:15:31.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7f37837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7f37837fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.010+0000 7f37837fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:15:31.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc33406fa90 0x7fc33413e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient(hunting): picked mon.noname-a con 0x7fc33406fa90 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 0x7fc33413eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient(hunting): picked mon.noname-c con 0x7fc33413d500 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3341439b0 0x7fc33406b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient(hunting): picked mon.noname-b con 0x7fc3341439b0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33413e3c0 con 0x7fc33406fa90 2024-01-20T11:15:31.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3340b4df0 con 0x7fc3341439b0 2024-01-20T11:15:31.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc334140b60 con 0x7fc33413d500 2024-01-20T11:15:31.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.011+0000 7fc33b11d640 10 monclient(hunting): authenticate will time out at 1374.361678s 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 1 -- 172.21.15.110:0/4221860459 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc802ecb0 con 0x7f5fd80aab00 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 10 monclient: dump: 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.071 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 1 -- 172.21.15.110:0/4221860459 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc8032070 con 0x7f5fd80aab00 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdef1a640 5 monclient: authenticate success, global_id 4238 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdef1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdef1a640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdef1a640 4 set_mon_vals no callback set 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 1 -- 172.21.15.110:0/4221860459 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8038620 con 0x7f5fd80aab00 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f5fdcf16640 10 monclient: dump: 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.073 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-01-20T11:15:31.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 1 -- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f91c40aa740 msgr2=0x7f91c414a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f91c40aa740 0x7f91c414a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 1 -- 172.21.15.110:0/1352040462 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91c40b0c00 msgr2=0x7f91c4146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91c40b0c00 0x7f91c4146c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.013+0000 7f91c8df3640 1 -- 172.21.15.110:0/1352040462 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b8009400 con 0x7f91c4143b90 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 1 -- 172.21.15.110:0/2367531897 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8840031870 con 0x7f8848034c00 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 10 monclient: dump: 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.075 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f8854946640 5 monclient: authenticate success, global_id 4232 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f8854946640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 1 -- 172.21.15.110:0/2367531897 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88400319d0 con 0x7f8848034c00 2024-01-20T11:15:31.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f8854946640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f8854946640 4 set_mon_vals no callback set 2024-01-20T11:15:31.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 10 monclient: shutdown 2024-01-20T11:15:31.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 1 -- 172.21.15.110:0/4221860459 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80aab00 msgr2=0x7f5fd80ad670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 1 --2- 172.21.15.110:0/4221860459 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80aab00 0x7f5fd80ad670 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8008870 tx=0x7f5fc802df00 comp rx=0 tx=0).stop 2024-01-20T11:15:31.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 1 -- 172.21.15.110:0/4221860459 shutdown_connections 2024-01-20T11:15:31.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 1 --2- 172.21.15.110:0/4221860459 >> v2:172.21.15.110:3300/0 conn(0x7f5fd813aca0 0x7f5fd813b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 1 --2- 172.21.15.110:0/4221860459 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80aab00 0x7f5fd80ad670 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 1 -- 172.21.15.110:0/4221860459 >> 172.21.15.110:0/4221860459 conn(0x7f5fd80b3df0 msgr2=0x7f5fd80b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 1 -- 172.21.15.110:0/2367531897 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8840031d60 con 0x7f8848034c00 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f884dd9b640 10 monclient: dump: 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.078 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.079 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.079 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 1 -- 172.21.15.110:0/4221860459 shutdown_connections 2024-01-20T11:15:31.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.014+0000 7f5fdef1a640 1 -- 172.21.15.110:0/4221860459 wait complete. 2024-01-20T11:15:31.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f37837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f37837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f37837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 10 monclient: shutdown 2024-01-20T11:15:31.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 1 -- 172.21.15.110:0/2367531897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8848034c00 msgr2=0x7f88480c3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 1 --2- 172.21.15.110:0/2367531897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8848034c00 0x7f88480c3570 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8840009870 tx=0x7f884002b910 comp rx=0 tx=0).stop 2024-01-20T11:15:31.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 1 -- 172.21.15.110:0/2367531897 shutdown_connections 2024-01-20T11:15:31.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 1 --2- 172.21.15.110:0/2367531897 >> v2:172.21.15.110:3300/0 conn(0x7f88480c4870 0x7f88480c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 1 --2- 172.21.15.110:0/2367531897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8848034c00 0x7f88480c3570 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 1 -- 172.21.15.110:0/2367531897 >> 172.21.15.110:0/2367531897 conn(0x7f8848005e90 msgr2=0x7f88480062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f5fdef1a640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f5fdef1a640 10 monclient: monmap: 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.081 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.082 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.082 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 1 -- 172.21.15.110:0/2367531897 shutdown_connections 2024-01-20T11:15:31.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f8854946640 1 -- 172.21.15.110:0/2367531897 wait complete. 2024-01-20T11:15:31.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f5fdef1a640 1 librados: starting msgr at 2024-01-20T11:15:31.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f5fdef1a640 1 librados: starting objecter 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f5fdef1a640 1 Processor -- start 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f5fdef1a640 1 -- start start 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f5fdef1a640 1 librados: setting wanted keys 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f5fdef1a640 1 librados: calling monclient init 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f5fdef1a640 10 monclient: init 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.015+0000 7f12a77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a80ad6b0 0x7f12a80717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f8854946640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f8854946640 10 monclient: monmap: 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.083 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd80aab00 0x7f5fd806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd80aab00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80af440 0x7f5fd81d6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f8854946640 1 librados: starting msgr at 2024-01-20T11:15:31.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f8854946640 1 librados: starting objecter 2024-01-20T11:15:31.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd80af440 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd813aca0 0x7f5fd81d2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd813aca0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd806fa20 con 0x7f5fd813aca0 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd81d6600 con 0x7f5fd80aab00 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd806d3c0 con 0x7f5fd80af440 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f5fdef1a640 10 monclient(hunting): authenticate will time out at 1374.366603s 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f8854946640 1 Processor -- start 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f8854946640 1 -- start start 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f8854946640 1 librados: setting wanted keys 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f8854946640 1 librados: calling monclient init 2024-01-20T11:15:31.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.016+0000 7f8854946640 10 monclient: init 2024-01-20T11:15:31.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f12a77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a80ad6b0 0x7f12a80717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58058/0 (socket says 172.21.15.110:58058) 2024-01-20T11:15:31.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f12a77fe640 1 -- 172.21.15.110:0/587004102 learned_addr learned my addr 172.21.15.110:0/587004102 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f12a77fe640 10 monclient(hunting): get_auth_request con 0x7f12a80ad6b0 auth_method 0 2024-01-20T11:15:31.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f12a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f12a77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f12a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:15:31.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f884802dfc0 0x7f884802a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient(hunting): picked mon.noname-c con 0x7f884802dfc0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8848034c00 0x7f8848037410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient(hunting): picked mon.noname-a con 0x7f8848034c00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88480c4870 0x7f8848037dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient(hunting): picked mon.noname-b con 0x7f88480c4870 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88480b7ae0 con 0x7f8848034c00 2024-01-20T11:15:31.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8848029a40 con 0x7f88480c4870 2024-01-20T11:15:31.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8848037950 con 0x7f884802dfc0 2024-01-20T11:15:31.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.017+0000 7f8854946640 10 monclient(hunting): authenticate will time out at 1374.367548s 2024-01-20T11:15:31.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.018+0000 7faf837fe640 1 --2- 172.21.15.110:0/2419214985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf8413d230 0x7faf8413d650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faf78009790 tx=0x7faf7802fc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5bdeba0b87d4dbf server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.019+0000 7f12a7fff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a813b890 0x7f12a8138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2024-01-20T11:15:31.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 1 -- 172.21.15.110:0/268917190 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37840aeb00 msgr2=0x7f378406eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37840aeb00 0x7f378406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 1 -- 172.21.15.110:0/268917190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37840ae120 msgr2=0x7f37841302a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37840ae120 0x7f37841302a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 1 monclient: found mon.noname-b 2024-01-20T11:15:31.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f37837fe640 1 -- 172.21.15.110:0/268917190 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37780094d0 con 0x7f37840ad4e0 2024-01-20T11:15:31.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f91c8df3640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c4143b90 0x7f91c41440e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91b80097a0 tx=0x7f91b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f91c0ff9640 10 client.?.objecter ms_handle_connect 0x7f91c4143b90 2024-01-20T11:15:31.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.020+0000 7f91c0ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.021+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.021+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.021+0000 7f12a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.022+0000 7f5fde719640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd813aca0 0x7f5fd81d2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.022+0000 7f12a7fff640 10 monclient(hunting): get_auth_request con 0x7f12a813b890 auth_method 0 2024-01-20T11:15:31.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.022+0000 7f12a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.022+0000 7f12a7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.022+0000 7f12a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.023+0000 7f5fde719640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd813aca0 0x7f5fd81d2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48460/0 (socket says 172.21.15.110:48460) 2024-01-20T11:15:31.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.023+0000 7f5fde719640 1 -- 172.21.15.110:0/157855777 learned_addr learned my addr 172.21.15.110:0/157855777 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.023+0000 7f5fde719640 10 monclient(hunting): get_auth_request con 0x7f5fd813aca0 auth_method 0 2024-01-20T11:15:31.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.023+0000 7f5fde719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.023+0000 7f5fde719640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.023+0000 7f5fde719640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2024-01-20T11:15:31.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 1 -- 172.21.15.110:0/587004102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12a80aeb90 msgr2=0x7f12a81d6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12a80aeb90 0x7f12a81d6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 1 -- 172.21.15.110:0/587004102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a813b890 msgr2=0x7f12a8138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a813b890 0x7f12a8138220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 1 monclient: found mon.noname-b 2024-01-20T11:15:31.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.024+0000 7f12a77fe640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c00ae30 con 0x7f12a80ad6b0 2024-01-20T11:15:31.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.025+0000 7f5fde719640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.025+0000 7f5fde719640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.025+0000 7f5fde719640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 1 -- 172.21.15.110:0/2419214985 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf780309d0 con 0x7faf8413d230 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 10 monclient: dump: 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.097 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 1 -- 172.21.15.110:0/2419214985 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf78030b30 con 0x7faf8413d230 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 1 -- 172.21.15.110:0/2419214985 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78038440 con 0x7faf8413d230 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf82ffd640 10 monclient: dump: 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf8a70f640 5 monclient: authenticate success, global_id 4247 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf8a70f640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7faf8a70f640 4 set_mon_vals no callback set 2024-01-20T11:15:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7f37837fe640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37840ad4e0 0x7f378412e940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f377c00c8a0 tx=0x7f377c00cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7f376bfff640 10 client.?.objecter ms_handle_connect 0x7f37840ad4e0 2024-01-20T11:15:31.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.026+0000 7f376bfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7fc333fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 0x7fc33413eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 10 monclient: shutdown 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 1 -- 172.21.15.110:0/2419214985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf8413d230 msgr2=0x7faf8413d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 1 --2- 172.21.15.110:0/2419214985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf8413d230 0x7faf8413d650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faf78009790 tx=0x7faf7802fc20 comp rx=0 tx=0).stop 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 1 -- 172.21.15.110:0/2419214985 shutdown_connections 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 1 --2- 172.21.15.110:0/2419214985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf8413d230 0x7faf8413d650 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 1 --2- 172.21.15.110:0/2419214985 >> v2:172.21.15.110:3300/0 conn(0x7faf8406f920 0x7faf8406fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 1 -- 172.21.15.110:0/2419214985 >> 172.21.15.110:0/2419214985 conn(0x7faf84066d60 msgr2=0x7faf840b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 1 -- 172.21.15.110:0/2419214985 shutdown_connections 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.027+0000 7faf8a70f640 1 -- 172.21.15.110:0/2419214985 wait complete. 2024-01-20T11:15:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7faf8a70f640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7fc333fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 0x7fc33413eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33730/0 (socket says 172.21.15.110:33730) 2024-01-20T11:15:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7fc333fff640 1 -- 172.21.15.110:0/1796048816 learned_addr learned my addr 172.21.15.110:0/1796048816 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7fc333fff640 10 monclient(hunting): get_auth_request con 0x7fc33413d500 auth_method 0 2024-01-20T11:15:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7fc333fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7fc333fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7fc333fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7faf8a70f640 10 monclient: monmap: 2024-01-20T11:15:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7f128ffff640 10 client.?.objecter ms_handle_connect 0x7f12a80ad6b0 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7f128ffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7f12a77fe640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a80ad6b0 0x7f12a80717f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f129c000c00 tx=0x7f129c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7f884ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f884802dfc0 0x7f884802a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7faf8a70f640 1 librados: starting msgr at 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7faf8a70f640 1 librados: starting objecter 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7faf8a70f640 1 Processor -- start 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7faf8a70f640 1 -- start start 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7faf8a70f640 1 librados: setting wanted keys 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7faf8a70f640 1 librados: calling monclient init 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.028+0000 7faf8a70f640 10 monclient: init 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf8406f920 0x7faf841dbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 10 monclient(hunting): picked mon.noname-a con 0x7faf8406f920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf8413d230 0x7faf841dc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 10 monclient(hunting): picked mon.noname-c con 0x7faf8413d230 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf84143950 0x7faf8413e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 10 monclient(hunting): picked mon.noname-b con 0x7faf84143950 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf841db9f0 con 0x7faf8406f920 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8406b700 con 0x7faf84143950 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.029+0000 7faf8a70f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8406b870 con 0x7faf8413d230 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7faf8a70f640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7faf8a70f640 10 monclient(hunting): authenticate will time out at 1374.380205s 2024-01-20T11:15:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-01-20T11:15:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 1 -- 172.21.15.110:0/157855777 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80af440 msgr2=0x7f5fd81d6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd80af440 0x7f5fd81d6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 1 -- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd80aab00 msgr2=0x7f5fd806e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd80aab00 0x7f5fd806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7f5fde719640 1 -- 172.21.15.110:0/157855777 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc8008550 con 0x7f5fd813aca0 2024-01-20T11:15:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7fc333fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7fc333fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.030+0000 7fc333fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f884ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f884802dfc0 0x7f884802a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33742/0 (socket says 172.21.15.110:33742) 2024-01-20T11:15:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f884ed9d640 1 -- 172.21.15.110:0/1085817970 learned_addr learned my addr 172.21.15.110:0/1085817970 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f884ed9d640 10 monclient(hunting): get_auth_request con 0x7f884802dfc0 auth_method 0 2024-01-20T11:15:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f884ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f884ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f884ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 1 -- 172.21.15.110:0/1798914858 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc02ea60 con 0x7f6cc40aecd0 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 10 monclient: dump: 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 1 -- 172.21.15.110:0/1798914858 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc02ebc0 con 0x7f6cc40aecd0 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 1 -- 172.21.15.110:0/1798914858 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc009600 con 0x7f6cc40aecd0 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6cc37fe640 10 monclient: dump: 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6ccacb4640 5 monclient: authenticate success, global_id 4244 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6ccacb4640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.031+0000 7f6ccacb4640 4 set_mon_vals no callback set 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f376bfff640 1 -- 172.21.15.110:0/268917190 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c003b70 con 0x7f37840ad4e0 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f376bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f376bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f376bfff640 10 monclient: dump: 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.114 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f376bfff640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f376bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f376bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f3789c4f640 5 monclient: authenticate success, global_id 4354 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f3789c4f640 10 monclient: _renew_subs 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f3789c4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378412f5e0 con 0x7f37840ad4e0 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f884ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f884ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f884ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f3789c4f640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f3789c4f640 10 monclient: _renew_subs 2024-01-20T11:15:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f3789c4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378412fba0 con 0x7f37840ad4e0 2024-01-20T11:15:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f3789c4f640 1 librados: init done 2024-01-20T11:15:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f6ccacb4640 10 monclient: shutdown 2024-01-20T11:15:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f6ccacb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1798914858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40aecd0 msgr2=0x7f6cc414a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f6ccacb4640 1 --2- 172.21.15.110:0/1798914858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40aecd0 0x7f6cc414a860 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00d180 tx=0x7f6cbc02e0c0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1798914858 shutdown_connections 2024-01-20T11:15:31.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f6ccacb4640 1 --2- 172.21.15.110:0/1798914858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40aecd0 0x7f6cc414a860 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1798914858 >> 172.21.15.110:0/1798914858 conn(0x7f6cc40b1340 msgr2=0x7f6cc40b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.032+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1798914858 shutdown_connections 2024-01-20T11:15:31.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.033+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1798914858 wait complete. 2024-01-20T11:15:31.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.033+0000 7f3789c4f640 10 client.4354.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.033+0000 7f3789c4f640 15 client.4354.objecter _session_command_op_assign -1 1 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.033+0000 7f3789c4f640 20 client.4354.objecter _get_session osd=-1 returning homeless 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.033+0000 7f3789c4f640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.033+0000 7f3789c4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.033+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f37840ab3b0 con 0x7f37840ad4e0 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.033+0000 7f6ccacb4640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.033+0000 7f6ccacb4640 10 monclient: monmap: 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.119 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.120 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.120 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.120 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.120 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.120 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.120 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.120 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.034+0000 7fc338eb9640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc33406fa90 0x7fc33413e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.034+0000 7f6ccacb4640 1 librados: starting msgr at 2024-01-20T11:15:31.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.034+0000 7f6ccacb4640 1 librados: starting objecter 2024-01-20T11:15:31.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.034+0000 7f6ccacb4640 1 Processor -- start 2024-01-20T11:15:31.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.034+0000 7f6ccacb4640 1 -- start start 2024-01-20T11:15:31.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.034+0000 7f6ccacb4640 1 librados: setting wanted keys 2024-01-20T11:15:31.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.034+0000 7f6ccacb4640 1 librados: calling monclient init 2024-01-20T11:15:31.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.034+0000 7f6ccacb4640 10 monclient: init 2024-01-20T11:15:31.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f5fde719640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd813aca0 0x7f5fd81d2f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400d5d0 tx=0x7f5fd400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f5fce7fc640 10 client.?.objecter ms_handle_connect 0x7f5fd813aca0 2024-01-20T11:15:31.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f5fce7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:15:31.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cc40705b0 0x7f6cc40adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f376bfff640 1 -- 172.21.15.110:0/268917190 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c0018b0 con 0x7f37840ad4e0 2024-01-20T11:15:31.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f376bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f376bfff640 1 -- 172.21.15.110:0/268917190 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c00a850 con 0x7f37840ad4e0 2024-01-20T11:15:31.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f376bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f376bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:31.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient(hunting): picked mon.noname-a con 0x7f6cc40705b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cc40aecd0 0x7f6cc40a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f376bfff640 10 monclient: dump: 2024-01-20T11:15:31.124 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.124 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.124 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.124 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.124 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.125 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient(hunting): picked mon.noname-b con 0x7f6cc40aecd0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40b04c0 0x7f6cc41d3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient(hunting): picked mon.noname-c con 0x7f6cc40b04c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc413fb20 con 0x7f6cc40705b0 2024-01-20T11:15:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc413fd30 con 0x7f6cc40aecd0 2024-01-20T11:15:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc40a96a0 con 0x7f6cc40b04c0 2024-01-20T11:15:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f6ccacb4640 10 monclient(hunting): authenticate will time out at 1374.385642s 2024-01-20T11:15:31.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7f3781ffb640 4 set_mon_vals no callback set 2024-01-20T11:15:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-01-20T11:15:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 1 -- 172.21.15.110:0/1796048816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3341439b0 msgr2=0x7fc33406b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3341439b0 0x7fc33406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 1 -- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc33406fa90 msgr2=0x7fc33413e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc33406fa90 0x7fc33413e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7fc333fff640 1 -- 172.21.15.110:0/1796048816 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc324009a70 con 0x7fc33413d500 2024-01-20T11:15:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7faf83fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf8406f920 0x7faf841dbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7faf83fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf8406f920 0x7faf841dbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48482/0 (socket says 172.21.15.110:48482) 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7faf83fff640 1 -- 172.21.15.110:0/1859501030 learned_addr learned my addr 172.21.15.110:0/1859501030 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7faf83fff640 10 monclient(hunting): get_auth_request con 0x7faf8406f920 auth_method 0 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7faf83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7faf83fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.035+0000 7faf83fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.037+0000 7faf83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.037+0000 7faf83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.037+0000 7faf83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 1 -- 172.21.15.110:0/1859501030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf8413d230 msgr2=0x7faf841dc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf8413d230 0x7faf841dc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 1 -- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf84143950 msgr2=0x7faf8413e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf84143950 0x7faf8413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7faf83fff640 1 -- 172.21.15.110:0/1859501030 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf780093a0 con 0x7faf8406f920 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 1 -- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88480c4870 msgr2=0x7f8848037dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88480c4870 0x7f8848037dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 1 -- 172.21.15.110:0/1085817970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8848034c00 msgr2=0x7f8848037410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8848034c00 0x7f8848037410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.040+0000 7f884ed9d640 1 -- 172.21.15.110:0/1085817970 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88400094d0 con 0x7f884802dfc0 2024-01-20T11:15:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.045+0000 7fc333fff640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 0x7fc33413eda0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e9b0 tx=0x7fc31c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.045+0000 7fc3317fa640 10 client.?.objecter ms_handle_connect 0x7fc33413d500 2024-01-20T11:15:31.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.045+0000 7fc3317fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7faf83fff640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf8406f920 0x7faf841dbe10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faf7400d5f0 tx=0x7faf7400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7faf80ff9640 10 client.?.objecter ms_handle_connect 0x7faf8406f920 2024-01-20T11:15:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7faf80ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f884ed9d640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f884802dfc0 0x7f884802a0d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8840009870 tx=0x7f88400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 1 -- 172.21.15.110:0/1352040462 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b8032840 con 0x7f91c4143b90 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 10 monclient: dump: 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.134 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f88377fe640 10 client.?.objecter ms_handle_connect 0x7f884802dfc0 2024-01-20T11:15:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f88377fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 1 -- 172.21.15.110:0/1352040462 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b80329a0 con 0x7f91c4143b90 2024-01-20T11:15:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91ca856640 5 monclient: authenticate success, global_id 4311 2024-01-20T11:15:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91ca856640 10 monclient: _renew_subs 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91ca856640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c4148760 con 0x7f91c4143b90 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 1 -- 172.21.15.110:0/1352040462 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b8032d50 con 0x7f91c4143b90 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c0ff9640 10 monclient: dump: 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91ca856640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91ca856640 10 monclient: _renew_subs 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91ca856640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c4138ea0 con 0x7f91c4143b90 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91c27fc640 4 set_mon_vals no callback set 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.046+0000 7f91ca856640 1 librados: init done 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f91ca856640 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f91ca856640 15 client.4311.objecter _session_command_op_assign -1 1 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f91ca856640 20 client.4311.objecter _get_session osd=-1 returning homeless 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f91ca856640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f91ca856640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f91c40aaed0 con 0x7f91c4143b90 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fce7fc640 1 -- 172.21.15.110:0/157855777 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd40125f0 con 0x7f5fd813aca0 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fce7fc640 10 monclient: dump: 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fce7fc640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fce7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fdef1a640 5 monclient: authenticate success, global_id 4317 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fdef1a640 10 monclient: _renew_subs 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fdef1a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.047+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd80ab7c0 con 0x7f5fd813aca0 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 10 monclient: _renew_subs 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd813cff0 con 0x7f5fd813aca0 2024-01-20T11:15:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 1 librados: init done 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fce7fc640 1 -- 172.21.15.110:0/157855777 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd4012770 con 0x7f5fd813aca0 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fce7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fcffff640 4 set_mon_vals no callback set 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fce7fc640 1 -- 172.21.15.110:0/157855777 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4012b20 con 0x7f5fd813aca0 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fce7fc640 10 monclient: dump: 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.142 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.143 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 10 client.4317.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 15 client.4317.objecter _session_command_op_assign -1 1 2024-01-20T11:15:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 20 client.4317.objecter _get_session osd=-1 returning homeless 2024-01-20T11:15:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.048+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5fc0002910 con 0x7f5fd813aca0 2024-01-20T11:15:31.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.056+0000 7f6cc9251640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40b04c0 0x7f6cc41d3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f6cc9251640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40b04c0 0x7f6cc41d3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33764/0 (socket says 172.21.15.110:33764) 2024-01-20T11:15:31.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f6cc9251640 1 -- 172.21.15.110:0/1173476578 learned_addr learned my addr 172.21.15.110:0/1173476578 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f6cc9251640 10 monclient(hunting): get_auth_request con 0x7f6cc40b04c0 auth_method 0 2024-01-20T11:15:31.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f6cc9251640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f6cc9251640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f6cc9251640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 1 -- 172.21.15.110:0/1085817970 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884003ad30 con 0x7f884802dfc0 2024-01-20T11:15:31.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 10 monclient: dump: 2024-01-20T11:15:31.145 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.145 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.145 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.145 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.145 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 1 -- 172.21.15.110:0/1085817970 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884003ae90 con 0x7f884802dfc0 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 1 -- 172.21.15.110:0/1085817970 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8840044610 con 0x7f884802dfc0 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f88377fe640 10 monclient: dump: 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f8854946640 5 monclient: authenticate success, global_id 4253 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f8854946640 10 monclient: _renew_subs 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f884d59a640 4 set_mon_vals no callback set 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f8854946640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8848029cb0 con 0x7f884802dfc0 2024-01-20T11:15:31.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f8854946640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f8854946640 10 monclient: _renew_subs 2024-01-20T11:15:31.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f8854946640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8848038310 con 0x7f884802dfc0 2024-01-20T11:15:31.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.058+0000 7f8854946640 1 librados: init done 2024-01-20T11:15:31.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f376bfff640 1 -- 172.21.15.110:0/268917190 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f377c00a9b0 con 0x7f37840ad4e0 2024-01-20T11:15:31.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f376bfff640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377006c860 0x7f377006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f8854946640 10 client.4253.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f8854946640 15 client.4253.objecter _session_command_op_assign -1 1 2024-01-20T11:15:31.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f8854946640 20 client.4253.objecter _get_session osd=-1 returning homeless 2024-01-20T11:15:31.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f8854946640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f8854946640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f884802eaf0 con 0x7f884802dfc0 2024-01-20T11:15:31.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f6cc9251640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f6cc9251640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.059+0000 7f6cc9251640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.060+0000 7f3782ffd640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377006c860 0x7f377006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.060+0000 7f3782ffd640 10 monclient: get_auth_request con 0x7f377006c860 auth_method 0 2024-01-20T11:15:31.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.060+0000 7f3782ffd640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377006c860 0x7f377006ed20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3784134800 tx=0x7f3774008040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.060+0000 7f376bfff640 10 client.4354.objecter ms_handle_connect 0x7f377006c860 2024-01-20T11:15:31.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 1 -- 172.21.15.110:0/268917190 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f377c0591e0 con 0x7f37840ad4e0 2024-01-20T11:15:31.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 10 client.4354.objecter ms_dispatch 0x7f3784000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 3 client.4354.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 3 client.4354.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 10 client.4354.objecter checking command 1 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.110:6808/470946803,v1:172.21.15.110:6809/470946803] conn(0x7f3770072c40 0x7f3770075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 20 client.4354.objecter _get_session s=0x7f3770072570 osd=2 3 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 20 client.4354.objecter put_session s=0x7f3770072570 osd=2 3 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 15 client.4354.objecter _session_command_op_remove -1 1 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 20 client.4354.objecter _get_session s=0x7f3770072570 osd=2 3 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 20 client.4354.objecter get_session s=0x7f3770072570 osd=2 3 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 15 client.4354.objecter _session_command_op_assign 2 1 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 20 client.4354.objecter put_session s=0x7f3770072570 osd=2 4 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 10 client.4354.objecter _send_command 1 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 1 -- 172.21.15.110:0/268917190 --> [v2:172.21.15.110:6808/470946803,v1:172.21.15.110:6809/470946803] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3770075710 con 0x7f3770072c40 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.061+0000 7f376bfff640 20 client.4354.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f3783fff640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.110:6808/470946803,v1:172.21.15.110:6809/470946803] conn(0x7f3770072c40 0x7f3770075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f3783fff640 10 monclient: get_auth_request con 0x7f3770072c40 auth_method 0 2024-01-20T11:15:31.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c03e070 con 0x7f12a80ad6b0 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 10 monclient: dump: 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.155 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c007d50 con 0x7f12a80ad6b0 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12adf89640 5 monclient: authenticate success, global_id 4357 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12a5ffb640 4 set_mon_vals no callback set 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12adf89640 10 monclient: _renew_subs 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12adf89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12adf89640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a8138760 con 0x7f12a80ad6b0 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12adf89640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12adf89640 10 monclient: _renew_subs 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12adf89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12adf89640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a813bf80 con 0x7f12a80ad6b0 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f12adf89640 1 librados: init done 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c03e070 con 0x7f12a80ad6b0 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.063+0000 7f128ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f128ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f128ffff640 10 monclient: dump: 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.158 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-01-20T11:15:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 1 -- 172.21.15.110:0/1173476578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cc40aecd0 msgr2=0x7f6cc40a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cc40aecd0 0x7f6cc40a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 1 -- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cc40705b0 msgr2=0x7f6cc40adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cc40705b0 0x7f6cc40adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f6cc9251640 1 -- 172.21.15.110:0/1173476578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc00bd40 con 0x7f6cc40b04c0 2024-01-20T11:15:31.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f376bfff640 1 -- 172.21.15.110:0/268917190 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f377c054e20 con 0x7f37840ad4e0 2024-01-20T11:15:31.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f376bfff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-20T11:15:31.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f12adf89640 10 client.4357.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f12adf89640 15 client.4357.objecter _session_command_op_assign -1 1 2024-01-20T11:15:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f12adf89640 20 client.4357.objecter _get_session osd=-1 returning homeless 2024-01-20T11:15:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f12adf89640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f12adf89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.064+0000 7f12adf89640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f12a80ab3b0 con 0x7f12a80ad6b0 2024-01-20T11:15:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.066+0000 7f3783fff640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.110:6808/470946803,v1:172.21.15.110:6809/470946803] conn(0x7f3770072c40 0x7f3770075060 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.066+0000 7f376bfff640 10 client.4354.objecter ms_handle_connect 0x7f3770072c40 2024-01-20T11:15:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.081+0000 7f91c0ff9640 1 -- 172.21.15.110:0/1352040462 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f91b804d3a0 con 0x7f91c4143b90 2024-01-20T11:15:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.082+0000 7f91c0ff9640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f91b006c8e0 0x7f91b006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.082+0000 7f91c3fff640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f91b006c8e0 0x7f91b006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.082+0000 7f91c3fff640 10 monclient: get_auth_request con 0x7f91b006c8e0 auth_method 0 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.082+0000 7f91c0ff9640 10 client.4311.objecter ms_handle_connect 0x7f91b006c8e0 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.082+0000 7f91c3fff640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f91b006c8e0 0x7f91b006eda0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f91bc001c60 tx=0x7f91bc007d10 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 1 -- 172.21.15.110:0/1352040462 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f91b807a860 con 0x7f91c4143b90 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 10 client.4311.objecter ms_dispatch 0x7f91c4000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 3 client.4311.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 3 client.4311.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 10 client.4311.objecter checking command 1 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:6816/3175427153,v1:172.21.15.110:6817/3175427153] conn(0x7f91b0072cc0 0x7f91b00750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 20 client.4311.objecter _get_session s=0x7f91b00725f0 osd=0 3 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 20 client.4311.objecter put_session s=0x7f91b00725f0 osd=0 3 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 15 client.4311.objecter _session_command_op_remove -1 1 2024-01-20T11:15:31.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 20 client.4311.objecter _get_session s=0x7f91b00725f0 osd=0 3 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 20 client.4311.objecter get_session s=0x7f91b00725f0 osd=0 3 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 15 client.4311.objecter _session_command_op_assign 0 1 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 20 client.4311.objecter put_session s=0x7f91b00725f0 osd=0 4 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 10 client.4311.objecter _send_command 1 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 1 -- 172.21.15.110:0/1352040462 --> [v2:172.21.15.110:6816/3175427153,v1:172.21.15.110:6817/3175427153] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91b0075790 con 0x7f91b0072cc0 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 20 client.4311.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 1 -- 172.21.15.110:0/1352040462 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91b80764a0 con 0x7f91c4143b90 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.088+0000 7f91c0ff9640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.089+0000 7f91c37fe640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:6816/3175427153,v1:172.21.15.110:6817/3175427153] conn(0x7f91b0072cc0 0x7f91b00750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.089+0000 7f91c37fe640 10 monclient: get_auth_request con 0x7f91b0072cc0 auth_method 0 2024-01-20T11:15:31.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.091+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f129c042070 con 0x7f12a80ad6b0 2024-01-20T11:15:31.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.091+0000 7f128ffff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f129406c8c0 0x7f129406ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.092+0000 7f12a6ffd640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f129406c8c0 0x7f129406ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.092+0000 7f12a6ffd640 10 monclient: get_auth_request con 0x7f129406c8c0 auth_method 0 2024-01-20T11:15:31.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.092+0000 7f128ffff640 10 client.4357.objecter ms_handle_connect 0x7f129406c8c0 2024-01-20T11:15:31.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.092+0000 7f12a6ffd640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f129406c8c0 0x7f129406ed80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f12a813cd80 tx=0x7f1298007a40 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f91c0ff9640 10 client.4311.objecter ms_handle_connect 0x7f91b0072cc0 2024-01-20T11:15:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f91c37fe640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:6816/3175427153,v1:172.21.15.110:6817/3175427153] conn(0x7f91b0072cc0 0x7f91b00750e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f88377fe640 1 -- 172.21.15.110:0/1085817970 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f884004d450 con 0x7f884802dfc0 2024-01-20T11:15:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f129c07a0b0 con 0x7f12a80ad6b0 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 10 client.4357.objecter ms_dispatch 0x7f12a8000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 3 client.4357.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 3 client.4357.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 10 client.4357.objecter checking command 1 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 0x7f12940750c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 20 client.4357.objecter _get_session s=0x7f12940725d0 osd=5 3 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 20 client.4357.objecter put_session s=0x7f12940725d0 osd=5 3 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 15 client.4357.objecter _session_command_op_remove -1 1 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 20 client.4357.objecter _get_session s=0x7f12940725d0 osd=5 3 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 20 client.4357.objecter get_session s=0x7f12940725d0 osd=5 3 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 15 client.4357.objecter _session_command_op_assign 5 1 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 20 client.4357.objecter put_session s=0x7f12940725d0 osd=5 4 2024-01-20T11:15:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 10 client.4357.objecter _send_command 1 2024-01-20T11:15:31.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1294075770 con 0x7f1294072ca0 2024-01-20T11:15:31.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 20 client.4357.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f129c0763f0 con 0x7f12a80ad6b0 2024-01-20T11:15:31.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.094+0000 7f128ffff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-20T11:15:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.095+0000 7f88377fe640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f883c06c860 0x7f883c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.095+0000 7f884e59c640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f883c06c860 0x7f883c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.095+0000 7f884e59c640 10 monclient: get_auth_request con 0x7f883c06c860 auth_method 0 2024-01-20T11:15:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.095+0000 7f884e59c640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f883c06c860 0x7f883c06ed20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8848036eb0 tx=0x7f8844007990 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.095+0000 7f88377fe640 10 client.4253.objecter ms_handle_connect 0x7f883c06c860 2024-01-20T11:15:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.102+0000 7f88377fe640 1 -- 172.21.15.110:0/1085817970 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f884002d040 con 0x7f884802dfc0 2024-01-20T11:15:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.102+0000 7f88377fe640 10 client.4253.objecter ms_dispatch 0x7f8850000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.102+0000 7f88377fe640 3 client.4253.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.102+0000 7f88377fe640 3 client.4253.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.102+0000 7f88377fe640 10 client.4253.objecter checking command 1 2024-01-20T11:15:31.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.102+0000 7f88377fe640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6808/604027796,v1:172.21.15.134:6809/604027796] conn(0x7f883c072c40 0x7f883c075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.102+0000 7f6cc9251640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40b04c0 0x7f6cc41d3550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc038040 tx=0x7f6cbc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 20 client.4253.objecter _get_session s=0x7f883c072570 osd=7 3 2024-01-20T11:15:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 20 client.4253.objecter put_session s=0x7f883c072570 osd=7 3 2024-01-20T11:15:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 15 client.4253.objecter _session_command_op_remove -1 1 2024-01-20T11:15:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 20 client.4253.objecter _get_session s=0x7f883c072570 osd=7 3 2024-01-20T11:15:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 20 client.4253.objecter get_session s=0x7f883c072570 osd=7 3 2024-01-20T11:15:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 15 client.4253.objecter _session_command_op_assign 7 1 2024-01-20T11:15:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 20 client.4253.objecter put_session s=0x7f883c072570 osd=7 4 2024-01-20T11:15:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 10 client.4253.objecter _send_command 1 2024-01-20T11:15:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 1 -- 172.21.15.110:0/1085817970 --> [v2:172.21.15.134:6808/604027796,v1:172.21.15.134:6809/604027796] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f883c075710 con 0x7f883c072c40 2024-01-20T11:15:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f6cc17fa640 10 client.?.objecter ms_handle_connect 0x7f6cc40b04c0 2024-01-20T11:15:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f6cc17fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 20 client.4253.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 1 -- 172.21.15.110:0/1085817970 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8840077210 con 0x7f884802dfc0 2024-01-20T11:15:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f88377fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-20T11:15:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f884f59e640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6808/604027796,v1:172.21.15.134:6809/604027796] conn(0x7f883c072c40 0x7f883c075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.103+0000 7f884f59e640 10 monclient: get_auth_request con 0x7f883c072c40 auth_method 0 2024-01-20T11:15:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.107+0000 7f88377fe640 10 client.4253.objecter ms_handle_connect 0x7f883c072c40 2024-01-20T11:15:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.107+0000 7f884f59e640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6808/604027796,v1:172.21.15.134:6809/604027796] conn(0x7f883c072c40 0x7f883c075060 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 1 -- 172.21.15.110:0/1796048816 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31c009670 con 0x7fc33413d500 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 10 monclient: dump: 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.174 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 1 -- 172.21.15.110:0/1796048816 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31c00c9a0 con 0x7fc33413d500 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3317fa640 1 -- 172.21.15.110:0/1796048816 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31c00cd50 con 0x7fc33413d500 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc33b11d640 5 monclient: authenticate success, global_id 4250 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc33b11d640 10 monclient: _renew_subs 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc33b11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3340b4a10 con 0x7fc33413d500 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.108+0000 7fc3327fc640 4 set_mon_vals no callback set 2024-01-20T11:15:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc33b11d640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc33b11d640 10 monclient: _renew_subs 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc33b11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33413f4c0 con 0x7fc33413d500 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc33b11d640 1 librados: init done 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc3317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc3317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.109+0000 7fc3317fa640 10 monclient: dump: 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.177 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.177 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.177 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7fc33b11d640 10 client.4250.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7fc33b11d640 15 client.4250.objecter _session_command_op_assign -1 1 2024-01-20T11:15:31.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7fc33b11d640 20 client.4250.objecter _get_session osd=-1 returning homeless 2024-01-20T11:15:31.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7fc33b11d640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7fc33b11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc334070190 con 0x7fc33413d500 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 1 -- 172.21.15.110:0/1173476578 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc036800 con 0x7f6cc40b04c0 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 10 monclient: dump: 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 1 -- 172.21.15.110:0/1173476578 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc036980 con 0x7f6cc40b04c0 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 1 -- 172.21.15.110:0/1173476578 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc036d30 con 0x7f6cc40b04c0 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc17fa640 10 monclient: dump: 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6ccacb4640 5 monclient: authenticate success, global_id 4256 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6cc2ffd640 4 set_mon_vals no callback set 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6ccacb4640 10 monclient: _renew_subs 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6ccacb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc40a9840 con 0x7f6cc40b04c0 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6ccacb4640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6ccacb4640 10 monclient: _renew_subs 2024-01-20T11:15:31.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6ccacb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc40aa400 con 0x7f6cc40b04c0 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.110+0000 7f6ccacb4640 1 librados: init done 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.112+0000 7f6ccacb4640 10 client.4256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.112+0000 7f6ccacb4640 15 client.4256.objecter _session_command_op_assign -1 1 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.112+0000 7f6ccacb4640 20 client.4256.objecter _get_session osd=-1 returning homeless 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.112+0000 7f6ccacb4640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.112+0000 7f6ccacb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.112+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6cc4072460 con 0x7f6cc40b04c0 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.116+0000 7f376bfff640 1 -- 172.21.15.110:0/268917190 <== osd.2 v2:172.21.15.110:6808/470946803 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3770075710 con 0x7f3770072c40 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.116+0000 7f376bfff640 10 client.4354.objecter ms_dispatch 0x7f3784000b90 command_reply(tid 1: 0 ) v1 2024-01-20T11:15:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.116+0000 7f376bfff640 10 client.4354.objecter _finish_command 1 = system:0 2024-01-20T11:15:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.116+0000 7f376bfff640 20 client.4354.objecter put_session s=0x7f3770072570 osd=2 4 2024-01-20T11:15:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.116+0000 7f376bfff640 15 client.4354.objecter _session_command_op_remove 2 1 2024-01-20T11:15:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7f5fce7fc640 1 -- 172.21.15.110:0/157855777 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5fd4012c80 con 0x7f5fd813aca0 2024-01-20T11:15:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 1 -- 172.21.15.110:0/1859501030 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf740125f0 con 0x7faf8406f920 2024-01-20T11:15:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 10 monclient: dump: 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 1 -- 172.21.15.110:0/1859501030 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf74012750 con 0x7faf8406f920 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 1 -- 172.21.15.110:0/1859501030 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf74012b00 con 0x7faf8406f920 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf80ff9640 10 monclient: dump: 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.185 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf8a70f640 5 monclient: authenticate success, global_id 4323 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf8a70f640 10 monclient: _renew_subs 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf827fc640 4 set_mon_vals no callback set 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf8a70f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8413dc70 con 0x7faf8406f920 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf8a70f640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf8a70f640 10 monclient: _renew_subs 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf8a70f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8413ee50 con 0x7faf8406f920 2024-01-20T11:15:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7faf8a70f640 1 librados: init done 2024-01-20T11:15:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.118+0000 7f5fce7fc640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5fbc06c8b0 0x7f5fbc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7f5fddf18640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5fbc06c8b0 0x7f5fbc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7f5fddf18640 10 monclient: get_auth_request con 0x7f5fbc06c8b0 auth_method 0 2024-01-20T11:15:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7faf8a70f640 10 client.4323.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7faf8a70f640 15 client.4323.objecter _session_command_op_assign -1 1 2024-01-20T11:15:31.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7faf8a70f640 20 client.4323.objecter _get_session osd=-1 returning homeless 2024-01-20T11:15:31.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7faf8a70f640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7faf8a70f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faf84075490 con 0x7faf8406f920 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7f5fce7fc640 10 client.4317.objecter ms_handle_connect 0x7f5fbc06c8b0 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.119+0000 7f5fddf18640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5fbc06c8b0 0x7f5fbc06ed70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0001c60 tx=0x7f5fd0007d10 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 1 -- 172.21.15.110:0/157855777 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5fd4059730 con 0x7f5fd813aca0 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 10 client.4317.objecter ms_dispatch 0x7f5fd8000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 3 client.4317.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 3 client.4317.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 10 client.4317.objecter checking command 1 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6800/2798305725,v1:172.21.15.134:6801/2798305725] conn(0x7f5fbc072c90 0x7f5fbc0750b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 20 client.4317.objecter _get_session s=0x7f5fbc0725c0 osd=6 3 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 20 client.4317.objecter put_session s=0x7f5fbc0725c0 osd=6 3 2024-01-20T11:15:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 15 client.4317.objecter _session_command_op_remove -1 1 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 20 client.4317.objecter _get_session s=0x7f5fbc0725c0 osd=6 3 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 20 client.4317.objecter get_session s=0x7f5fbc0725c0 osd=6 3 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 15 client.4317.objecter _session_command_op_assign 6 1 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 20 client.4317.objecter put_session s=0x7f5fbc0725c0 osd=6 4 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 10 client.4317.objecter _send_command 1 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 1 -- 172.21.15.110:0/157855777 --> [v2:172.21.15.134:6800/2798305725,v1:172.21.15.134:6801/2798305725] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fbc075760 con 0x7f5fbc072c90 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.124+0000 7f5fce7fc640 20 client.4317.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.125+0000 7f5fdd717640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6800/2798305725,v1:172.21.15.134:6801/2798305725] conn(0x7f5fbc072c90 0x7f5fbc0750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.125+0000 7f5fdd717640 10 monclient: get_auth_request con 0x7f5fbc072c90 auth_method 0 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.127+0000 7f5fce7fc640 1 -- 172.21.15.110:0/157855777 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fd4055370 con 0x7f5fd813aca0 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.127+0000 7f5fce7fc640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.129+0000 7f5fce7fc640 10 client.4317.objecter ms_handle_connect 0x7f5fbc072c90 2024-01-20T11:15:31.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.129+0000 7f5fdd717640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6800/2798305725,v1:172.21.15.134:6801/2798305725] conn(0x7f5fbc072c90 0x7f5fbc0750b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 10 client.4354.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 15 client.4354.objecter _session_command_op_assign -1 2 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 20 client.4354.objecter _get_session s=0x7f3770072570 osd=2 3 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 20 client.4354.objecter put_session s=0x7f3770072570 osd=2 3 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 20 client.4354.objecter _get_session s=0x7f3770072570 osd=2 3 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 15 client.4354.objecter _session_command_op_remove -1 2 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 20 client.4354.objecter get_session s=0x7f3770072570 osd=2 3 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 15 client.4354.objecter _session_command_op_assign 2 2 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 20 client.4354.objecter put_session s=0x7f3770072570 osd=2 4 2024-01-20T11:15:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 10 client.4354.objecter _send_command 2 2024-01-20T11:15:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.134+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 --> [v2:172.21.15.110:6808/470946803,v1:172.21.15.110:6809/470946803] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f37840ab3b0 con 0x7f3770072c40 2024-01-20T11:15:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.136+0000 7fc3317fa640 1 -- 172.21.15.110:0/1796048816 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc31c013470 con 0x7fc33413d500 2024-01-20T11:15:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.136+0000 7fc3317fa640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc31406c8b0 0x7fc31406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.137+0000 7fc338eb9640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc31406c8b0 0x7fc31406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.137+0000 7fc338eb9640 10 monclient: get_auth_request con 0x7fc31406c8b0 auth_method 0 2024-01-20T11:15:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.137+0000 7fc3317fa640 10 client.4250.objecter ms_handle_connect 0x7fc31406c8b0 2024-01-20T11:15:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.137+0000 7fc338eb9640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc31406c8b0 0x7fc31406ed70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc324036000 tx=0x7fc324030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.145+0000 7f376bfff640 1 -- 172.21.15.110:0/268917190 <== osd.2 v2:172.21.15.110:6808/470946803 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3778004040 con 0x7f3770072c40 2024-01-20T11:15:31.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.145+0000 7f376bfff640 10 client.4354.objecter ms_dispatch 0x7f3784000b90 command_reply(tid 2: 0 ) v1 2024-01-20T11:15:31.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.145+0000 7f376bfff640 10 client.4354.objecter _finish_command 2 = system:0 2024-01-20T11:15:31.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.145+0000 7f376bfff640 20 client.4354.objecter put_session s=0x7f3770072570 osd=2 4 2024-01-20T11:15:31.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.145+0000 7f376bfff640 15 client.4354.objecter _session_command_op_remove 2 2 2024-01-20T11:15:31.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 10 librados: watch_flush enter 2024-01-20T11:15:31.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 10 librados: watch_flush exit 2024-01-20T11:15:31.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 10 client.4354.objecter close_session for osd.2 2024-01-20T11:15:31.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 >> [v2:172.21.15.110:6808/470946803,v1:172.21.15.110:6809/470946803] conn(0x7f3770072c40 msgr2=0x7f3770075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.110:6808/470946803,v1:172.21.15.110:6809/470946803] conn(0x7f3770072c40 0x7f3770075060 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 20 client.4354.objecter put_session s=0x7f3770072570 osd=2 1 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 20 client.4354.objecter shutdown clearing up homeless session... 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 10 client.4354.objecter successfully canceled tick 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377006c860 msgr2=0x7f377006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377006c860 0x7f377006ed20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3784134800 tx=0x7f3774008040 comp rx=0 tx=0).stop 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 10 monclient: shutdown 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37840ad4e0 msgr2=0x7f378412e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 1 --2- 172.21.15.110:0/268917190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37840ad4e0 0x7f378412e940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f377c00c8a0 tx=0x7f377c00cd60 comp rx=0 tx=0).stop 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3783fff640 1 -- 172.21.15.110:0/268917190 reap_dead start 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 shutdown_connections 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 >> 172.21.15.110:0/268917190 conn(0x7f37840b3b10 msgr2=0x7f37840b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.146+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 shutdown_connections 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.147+0000 7f3789c4f640 1 -- 172.21.15.110:0/268917190 wait complete. 2024-01-20T11:15:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.147+0000 7f3789c4f640 1 librados: shutdown 2024-01-20T11:15:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.150+0000 7f91c0ff9640 1 -- 172.21.15.110:0/1352040462 <== osd.0 v2:172.21.15.110:6816/3175427153 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f91b0075790 con 0x7f91b0072cc0 2024-01-20T11:15:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.150+0000 7f91c0ff9640 10 client.4311.objecter ms_dispatch 0x7f91c4000b90 command_reply(tid 1: 0 ) v1 2024-01-20T11:15:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.150+0000 7f91c0ff9640 10 client.4311.objecter _finish_command 1 = system:0 2024-01-20T11:15:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.150+0000 7f91c0ff9640 20 client.4311.objecter put_session s=0x7f91b00725f0 osd=0 4 2024-01-20T11:15:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.150+0000 7f91c0ff9640 15 client.4311.objecter _session_command_op_remove 0 1 2024-01-20T11:15:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.152+0000 7fc3317fa640 1 -- 172.21.15.110:0/1796048816 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc31c058e80 con 0x7fc33413d500 2024-01-20T11:15:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.152+0000 7fc3317fa640 10 client.4250.objecter ms_dispatch 0x7fc334000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.152+0000 7fc3317fa640 3 client.4250.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.152+0000 7fc3317fa640 3 client.4250.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 10 client.4250.objecter checking command 1 2024-01-20T11:15:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:6821/1038822226,v1:172.21.15.110:6823/1038822226] conn(0x7fc314072ce0 0x7fc314075100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 20 client.4250.objecter _get_session s=0x7fc314072610 osd=3 3 2024-01-20T11:15:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 20 client.4250.objecter put_session s=0x7fc314072610 osd=3 3 2024-01-20T11:15:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 15 client.4250.objecter _session_command_op_remove -1 1 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 20 client.4250.objecter _get_session s=0x7fc314072610 osd=3 3 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 20 client.4250.objecter get_session s=0x7fc314072610 osd=3 3 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 15 client.4250.objecter _session_command_op_assign 3 1 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 20 client.4250.objecter put_session s=0x7fc314072610 osd=3 4 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 10 client.4250.objecter _send_command 1 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 1 -- 172.21.15.110:0/1796048816 --> [v2:172.21.15.110:6821/1038822226,v1:172.21.15.110:6823/1038822226] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3140757b0 con 0x7fc314072ce0 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 20 client.4250.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 1 -- 172.21.15.110:0/1796048816 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc31c054ac0 con 0x7fc33413d500 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.153+0000 7fc3317fa640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.154+0000 7fc3396ba640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:6821/1038822226,v1:172.21.15.110:6823/1038822226] conn(0x7fc314072ce0 0x7fc314075100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.154+0000 7fc3396ba640 10 monclient: get_auth_request con 0x7fc314072ce0 auth_method 0 2024-01-20T11:15:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.155+0000 7f88377fe640 1 -- 172.21.15.110:0/1085817970 <== osd.7 v2:172.21.15.134:6808/604027796 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f883c075710 con 0x7f883c072c40 2024-01-20T11:15:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.155+0000 7f88377fe640 10 client.4253.objecter ms_dispatch 0x7f8850000b90 command_reply(tid 1: 0 ) v1 2024-01-20T11:15:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.155+0000 7f88377fe640 10 client.4253.objecter _finish_command 1 = system:0 2024-01-20T11:15:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.155+0000 7f88377fe640 20 client.4253.objecter put_session s=0x7f883c072570 osd=7 4 2024-01-20T11:15:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.155+0000 7f88377fe640 15 client.4253.objecter _session_command_op_remove 7 1 2024-01-20T11:15:31.200 INFO:teuthology.orchestra.run.smithi110.stdout:73014444038 2024-01-20T11:15:31.200 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-20T11:15:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.160+0000 7fc3317fa640 10 client.4250.objecter ms_handle_connect 0x7fc314072ce0 2024-01-20T11:15:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.160+0000 7fc3396ba640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:6821/1038822226,v1:172.21.15.110:6823/1038822226] conn(0x7fc314072ce0 0x7fc314075100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.166+0000 7faf80ff9640 1 -- 172.21.15.110:0/1859501030 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7faf74012c60 con 0x7faf8406f920 2024-01-20T11:15:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.166+0000 7faf80ff9640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf7006c890 0x7faf7006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.166+0000 7faf837fe640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf7006c890 0x7faf7006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.166+0000 7faf837fe640 10 monclient: get_auth_request con 0x7faf7006c890 auth_method 0 2024-01-20T11:15:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.167+0000 7faf837fe640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf7006c890 0x7faf7006ed50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faf7803a000 tx=0x7faf78032000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.167+0000 7faf80ff9640 10 client.4323.objecter ms_handle_connect 0x7faf7006c890 2024-01-20T11:15:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-20T11:15:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 15 client.4311.objecter _session_command_op_assign -1 2 2024-01-20T11:15:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 20 client.4311.objecter _get_session s=0x7f91b00725f0 osd=0 3 2024-01-20T11:15:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 20 client.4311.objecter put_session s=0x7f91b00725f0 osd=0 3 2024-01-20T11:15:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 20 client.4311.objecter _get_session s=0x7f91b00725f0 osd=0 3 2024-01-20T11:15:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 15 client.4311.objecter _session_command_op_remove -1 2 2024-01-20T11:15:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 20 client.4311.objecter get_session s=0x7f91b00725f0 osd=0 3 2024-01-20T11:15:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 15 client.4311.objecter _session_command_op_assign 0 2 2024-01-20T11:15:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 20 client.4311.objecter put_session s=0x7f91b00725f0 osd=0 4 2024-01-20T11:15:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 10 client.4311.objecter _send_command 2 2024-01-20T11:15:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.169+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 --> [v2:172.21.15.110:6816/3175427153,v1:172.21.15.110:6817/3175427153] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f91c4075490 con 0x7f91b0072cc0 2024-01-20T11:15:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.172+0000 7f6cc17fa640 1 -- 172.21.15.110:0/1173476578 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6cbc04a460 con 0x7f6cc40b04c0 2024-01-20T11:15:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.172+0000 7f6cc17fa640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6ca806c8b0 0x7f6ca806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.172+0000 7f6cc8a50640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6ca806c8b0 0x7f6ca806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.172+0000 7f6cc8a50640 10 monclient: get_auth_request con 0x7f6ca806c8b0 auth_method 0 2024-01-20T11:15:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7f6cc17fa640 10 client.4256.objecter ms_handle_connect 0x7f6ca806c8b0 2024-01-20T11:15:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7f6cc8a50640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6ca806c8b0 0x7f6ca806ed70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009650 tx=0x7f6cb8006ba0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 1 -- 172.21.15.110:0/1859501030 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faf74059160 con 0x7faf8406f920 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 10 client.4323.objecter ms_dispatch 0x7faf84000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 3 client.4323.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 3 client.4323.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 10 client.4323.objecter checking command 1 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6822/2594530523,v1:172.21.15.134:6824/2594530523] conn(0x7faf70073590 0x7faf70075a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 20 client.4323.objecter _get_session s=0x7faf70072ec0 osd=4 3 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 20 client.4323.objecter put_session s=0x7faf70072ec0 osd=4 3 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 15 client.4323.objecter _session_command_op_remove -1 1 2024-01-20T11:15:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 20 client.4323.objecter _get_session s=0x7faf70072ec0 osd=4 3 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 20 client.4323.objecter get_session s=0x7faf70072ec0 osd=4 3 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 15 client.4323.objecter _session_command_op_assign 4 1 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 20 client.4323.objecter put_session s=0x7faf70072ec0 osd=4 4 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 10 client.4323.objecter _send_command 1 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 1 -- 172.21.15.110:0/1859501030 --> [v2:172.21.15.134:6822/2594530523,v1:172.21.15.134:6824/2594530523] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf70076100 con 0x7faf70073590 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 20 client.4323.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 1 -- 172.21.15.110:0/1859501030 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf74012f10 con 0x7faf8406f920 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.173+0000 7faf80ff9640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.174+0000 7faf88cac640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6822/2594530523,v1:172.21.15.134:6824/2594530523] conn(0x7faf70073590 0x7faf70075a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.174+0000 7faf88cac640 10 monclient: get_auth_request con 0x7faf70073590 auth_method 0 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.175+0000 7f5fce7fc640 1 -- 172.21.15.110:0/157855777 <== osd.6 v2:172.21.15.134:6800/2798305725 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5fbc075760 con 0x7f5fbc072c90 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.175+0000 7f5fce7fc640 10 client.4317.objecter ms_dispatch 0x7f5fd8000b90 command_reply(tid 1: 0 ) v1 2024-01-20T11:15:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.175+0000 7f5fce7fc640 10 client.4317.objecter _finish_command 1 = system:0 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.175+0000 7f5fce7fc640 20 client.4317.objecter put_session s=0x7f5fbc0725c0 osd=6 4 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.175+0000 7f5fce7fc640 15 client.4317.objecter _session_command_op_remove 6 1 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.178+0000 7f8854946640 10 client.4253.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 15 client.4253.objecter _session_command_op_assign -1 2 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 20 client.4253.objecter _get_session s=0x7f883c072570 osd=7 3 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 20 client.4253.objecter put_session s=0x7f883c072570 osd=7 3 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 20 client.4253.objecter _get_session s=0x7f883c072570 osd=7 3 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 15 client.4253.objecter _session_command_op_remove -1 2 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 20 client.4253.objecter get_session s=0x7f883c072570 osd=7 3 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 15 client.4253.objecter _session_command_op_assign 7 2 2024-01-20T11:15:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 20 client.4253.objecter put_session s=0x7f883c072570 osd=7 4 2024-01-20T11:15:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 10 client.4253.objecter _send_command 2 2024-01-20T11:15:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.179+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 --> [v2:172.21.15.134:6808/604027796,v1:172.21.15.134:6809/604027796] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f88480b7800 con 0x7f883c072c40 2024-01-20T11:15:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 1 -- 172.21.15.110:0/1173476578 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6cbc07c6f0 con 0x7f6cc40b04c0 2024-01-20T11:15:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 10 client.4256.objecter ms_dispatch 0x7f6cc4000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 3 client.4256.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 3 client.4256.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 10 client.4256.objecter checking command 1 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:6800/3373744578,v1:172.21.15.110:6801/3373744578] conn(0x7f6ca8072c90 0x7f6ca80750b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 20 client.4256.objecter _get_session s=0x7f6ca80725c0 osd=1 3 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 20 client.4256.objecter put_session s=0x7f6ca80725c0 osd=1 3 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 15 client.4256.objecter _session_command_op_remove -1 1 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 20 client.4256.objecter _get_session s=0x7f6ca80725c0 osd=1 3 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 20 client.4256.objecter get_session s=0x7f6ca80725c0 osd=1 3 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 15 client.4256.objecter _session_command_op_assign 1 1 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 20 client.4256.objecter put_session s=0x7f6ca80725c0 osd=1 4 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 10 client.4256.objecter _send_command 1 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 1 -- 172.21.15.110:0/1173476578 --> [v2:172.21.15.110:6800/3373744578,v1:172.21.15.110:6801/3373744578] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ca8075760 con 0x7f6ca8072c90 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 20 client.4256.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 1 -- 172.21.15.110:0/1173476578 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cbc036e90 con 0x7f6cc40b04c0 2024-01-20T11:15:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7f6cc17fa640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-20T11:15:31.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7faf80ff9640 10 client.4323.objecter ms_handle_connect 0x7faf70073590 2024-01-20T11:15:31.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.180+0000 7faf88cac640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6822/2594530523,v1:172.21.15.134:6824/2594530523] conn(0x7faf70073590 0x7faf70075a30 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.182+0000 7f6cc3fff640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:6800/3373744578,v1:172.21.15.110:6801/3373744578] conn(0x7f6ca8072c90 0x7f6ca80750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.182+0000 7f6cc3fff640 10 monclient: get_auth_request con 0x7f6ca8072c90 auth_method 0 2024-01-20T11:15:31.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91c0ff9640 1 -- 172.21.15.110:0/1352040462 <== osd.0 v2:172.21.15.110:6816/3175427153 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f91c4075490 con 0x7f91b0072cc0 2024-01-20T11:15:31.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91c0ff9640 10 client.4311.objecter ms_dispatch 0x7f91c4000b90 command_reply(tid 2: 0 ) v1 2024-01-20T11:15:31.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91c0ff9640 10 client.4311.objecter _finish_command 2 = system:0 2024-01-20T11:15:31.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91c0ff9640 20 client.4311.objecter put_session s=0x7f91b00725f0 osd=0 4 2024-01-20T11:15:31.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91c0ff9640 15 client.4311.objecter _session_command_op_remove 0 2 2024-01-20T11:15:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 10 librados: watch_flush enter 2024-01-20T11:15:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 10 librados: watch_flush exit 2024-01-20T11:15:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 10 client.4311.objecter close_session for osd.0 2024-01-20T11:15:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:6816/3175427153,v1:172.21.15.110:6817/3175427153] conn(0x7f91b0072cc0 msgr2=0x7f91b00750e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:6816/3175427153,v1:172.21.15.110:6817/3175427153] conn(0x7f91b0072cc0 0x7f91b00750e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 20 client.4311.objecter put_session s=0x7f91b00725f0 osd=0 1 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 20 client.4311.objecter shutdown clearing up homeless session... 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 10 client.4311.objecter successfully canceled tick 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f91b006c8e0 msgr2=0x7f91b006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f91b006c8e0 0x7f91b006eda0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f91bc001c60 tx=0x7f91bc007d10 comp rx=0 tx=0).stop 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 10 monclient: shutdown 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c4143b90 msgr2=0x7f91c41440e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91ca856640 1 --2- 172.21.15.110:0/1352040462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91c4143b90 0x7f91c41440e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91b80097a0 tx=0x7f91b80046c0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.183+0000 7f91c8df3640 1 -- 172.21.15.110:0/1352040462 reap_dead start 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.184+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 shutdown_connections 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.184+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 >> 172.21.15.110:0/1352040462 conn(0x7f91c40b3e60 msgr2=0x7f91c40aeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.184+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 shutdown_connections 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.184+0000 7f91ca856640 1 -- 172.21.15.110:0/1352040462 wait complete. 2024-01-20T11:15:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.184+0000 7f91ca856640 1 librados: shutdown 2024-01-20T11:15:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.186+0000 7f6cc3fff640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:6800/3373744578,v1:172.21.15.110:6801/3373744578] conn(0x7f6ca8072c90 0x7f6ca80750b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.186+0000 7f6cc17fa640 10 client.4256.objecter ms_handle_connect 0x7f6ca8072c90 2024-01-20T11:15:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f88377fe640 1 -- 172.21.15.110:0/1085817970 <== osd.7 v2:172.21.15.134:6808/604027796 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f88480b7800 con 0x7f883c072c40 2024-01-20T11:15:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f88377fe640 10 client.4253.objecter ms_dispatch 0x7f8850000b90 command_reply(tid 2: 0 ) v1 2024-01-20T11:15:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f88377fe640 10 client.4253.objecter _finish_command 2 = system:0 2024-01-20T11:15:31.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f88377fe640 20 client.4253.objecter put_session s=0x7f883c072570 osd=7 4 2024-01-20T11:15:31.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f88377fe640 15 client.4253.objecter _session_command_op_remove 7 2 2024-01-20T11:15:31.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f8854946640 10 librados: watch_flush enter 2024-01-20T11:15:31.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f8854946640 10 librados: watch_flush exit 2024-01-20T11:15:31.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f8854946640 10 client.4253.objecter close_session for osd.7 2024-01-20T11:15:31.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6808/604027796,v1:172.21.15.134:6809/604027796] conn(0x7f883c072c40 msgr2=0x7f883c075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f8854946640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6808/604027796,v1:172.21.15.134:6809/604027796] conn(0x7f883c072c40 0x7f883c075060 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f8854946640 20 client.4253.objecter put_session s=0x7f883c072570 osd=7 1 2024-01-20T11:15:31.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f8854946640 20 client.4253.objecter shutdown clearing up homeless session... 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.196+0000 7f8854946640 10 client.4253.objecter successfully canceled tick 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f883c06c860 msgr2=0x7f883c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f883c06c860 0x7f883c06ed20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8848036eb0 tx=0x7f8844007990 comp rx=0 tx=0).stop 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 10 monclient: shutdown 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f884802dfc0 msgr2=0x7f884802a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 1 --2- 172.21.15.110:0/1085817970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f884802dfc0 0x7f884802a0d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8840009870 tx=0x7f88400046c0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f884f59e640 1 -- 172.21.15.110:0/1085817970 reap_dead start 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 shutdown_connections 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 >> 172.21.15.110:0/1085817970 conn(0x7f8848005e90 msgr2=0x7f88480c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 shutdown_connections 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.197+0000 7f8854946640 1 -- 172.21.15.110:0/1085817970 wait complete. 2024-01-20T11:15:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.198+0000 7f8854946640 1 librados: shutdown 2024-01-20T11:15:31.218 INFO:teuthology.orchestra.run.smithi110.stdout:73014444039 2024-01-20T11:15:31.219 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-20T11:15:31.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 10 client.4317.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-20T11:15:31.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 15 client.4317.objecter _session_command_op_assign -1 2 2024-01-20T11:15:31.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 20 client.4317.objecter _get_session s=0x7f5fbc0725c0 osd=6 3 2024-01-20T11:15:31.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 20 client.4317.objecter put_session s=0x7f5fbc0725c0 osd=6 3 2024-01-20T11:15:31.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 20 client.4317.objecter _get_session s=0x7f5fbc0725c0 osd=6 3 2024-01-20T11:15:31.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 15 client.4317.objecter _session_command_op_remove -1 2 2024-01-20T11:15:31.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 20 client.4317.objecter get_session s=0x7f5fbc0725c0 osd=6 3 2024-01-20T11:15:31.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 15 client.4317.objecter _session_command_op_assign 6 2 2024-01-20T11:15:31.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 20 client.4317.objecter put_session s=0x7f5fbc0725c0 osd=6 4 2024-01-20T11:15:31.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 10 client.4317.objecter _send_command 2 2024-01-20T11:15:31.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.205+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 --> [v2:172.21.15.134:6800/2798305725,v1:172.21.15.134:6801/2798305725] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5fc0001240 con 0x7f5fbc072c90 2024-01-20T11:15:31.220 INFO:teuthology.orchestra.run.smithi110.stdout:90194313223 2024-01-20T11:15:31.221 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-20T11:15:31.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fce7fc640 1 -- 172.21.15.110:0/157855777 <== osd.6 v2:172.21.15.134:6800/2798305725 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5fc0001240 con 0x7f5fbc072c90 2024-01-20T11:15:31.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fce7fc640 10 client.4317.objecter ms_dispatch 0x7f5fd8000b90 command_reply(tid 2: 0 ) v1 2024-01-20T11:15:31.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fce7fc640 10 client.4317.objecter _finish_command 2 = system:0 2024-01-20T11:15:31.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fce7fc640 20 client.4317.objecter put_session s=0x7f5fbc0725c0 osd=6 4 2024-01-20T11:15:31.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fce7fc640 15 client.4317.objecter _session_command_op_remove 6 2 2024-01-20T11:15:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 10 librados: watch_flush enter 2024-01-20T11:15:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 10 librados: watch_flush exit 2024-01-20T11:15:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 10 client.4317.objecter close_session for osd.6 2024-01-20T11:15:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6800/2798305725,v1:172.21.15.134:6801/2798305725] conn(0x7f5fbc072c90 msgr2=0x7f5fbc0750b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6800/2798305725,v1:172.21.15.134:6801/2798305725] conn(0x7f5fbc072c90 0x7f5fbc0750b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 20 client.4317.objecter put_session s=0x7f5fbc0725c0 osd=6 1 2024-01-20T11:15:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 20 client.4317.objecter shutdown clearing up homeless session... 2024-01-20T11:15:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 10 client.4317.objecter successfully canceled tick 2024-01-20T11:15:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5fbc06c8b0 msgr2=0x7f5fbc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5fbc06c8b0 0x7f5fbc06ed70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0001c60 tx=0x7f5fd0007d10 comp rx=0 tx=0).stop 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 10 monclient: shutdown 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd813aca0 msgr2=0x7f5fd81d2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 --2- 172.21.15.110:0/157855777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd813aca0 0x7f5fd81d2f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400d5d0 tx=0x7f5fd400da90 comp rx=0 tx=0).stop 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fde719640 1 -- 172.21.15.110:0/157855777 reap_dead start 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 shutdown_connections 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 >> 172.21.15.110:0/157855777 conn(0x7f5fd80b3df0 msgr2=0x7f5fd80a9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 shutdown_connections 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.211+0000 7f5fdef1a640 1 -- 172.21.15.110:0/157855777 wait complete. 2024-01-20T11:15:31.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.212+0000 7f5fdef1a640 1 librados: shutdown 2024-01-20T11:15:31.226 INFO:teuthology.orchestra.run.smithi110.stdout:64424509448 2024-01-20T11:15:31.226 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-20T11:15:31.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.239+0000 7f6cc17fa640 1 -- 172.21.15.110:0/1173476578 <== osd.1 v2:172.21.15.110:6800/3373744578 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6ca8075760 con 0x7f6ca8072c90 2024-01-20T11:15:31.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.239+0000 7f6cc17fa640 10 client.4256.objecter ms_dispatch 0x7f6cc4000b90 command_reply(tid 1: 0 ) v1 2024-01-20T11:15:31.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.239+0000 7f6cc17fa640 10 client.4256.objecter _finish_command 1 = system:0 2024-01-20T11:15:31.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.239+0000 7f6cc17fa640 20 client.4256.objecter put_session s=0x7f6ca80725c0 osd=1 4 2024-01-20T11:15:31.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.239+0000 7f6cc17fa640 15 client.4256.objecter _session_command_op_remove 1 1 2024-01-20T11:15:31.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.255+0000 7fc3317fa640 1 -- 172.21.15.110:0/1796048816 <== osd.3 v2:172.21.15.110:6821/1038822226 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc3140757b0 con 0x7fc314072ce0 2024-01-20T11:15:31.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.255+0000 7fc3317fa640 10 client.4250.objecter ms_dispatch 0x7fc334000b90 command_reply(tid 1: 0 ) v1 2024-01-20T11:15:31.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.255+0000 7fc3317fa640 10 client.4250.objecter _finish_command 1 = system:0 2024-01-20T11:15:31.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.255+0000 7fc3317fa640 20 client.4250.objecter put_session s=0x7fc314072610 osd=3 4 2024-01-20T11:15:31.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.255+0000 7fc3317fa640 15 client.4250.objecter _session_command_op_remove 3 1 2024-01-20T11:15:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.260+0000 7faf80ff9640 1 -- 172.21.15.110:0/1859501030 <== osd.4 v2:172.21.15.134:6822/2594530523 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7faf70076100 con 0x7faf70073590 2024-01-20T11:15:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.260+0000 7faf80ff9640 10 client.4323.objecter ms_dispatch 0x7faf84000b90 command_reply(tid 1: 0 ) v1 2024-01-20T11:15:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.260+0000 7faf80ff9640 10 client.4323.objecter _finish_command 1 = system:0 2024-01-20T11:15:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.260+0000 7faf80ff9640 20 client.4323.objecter put_session s=0x7faf70072ec0 osd=4 4 2024-01-20T11:15:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.260+0000 7faf80ff9640 15 client.4323.objecter _session_command_op_remove 4 1 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 10 client.4256.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 15 client.4256.objecter _session_command_op_assign -1 2 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 20 client.4256.objecter _get_session s=0x7f6ca80725c0 osd=1 3 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 20 client.4256.objecter put_session s=0x7f6ca80725c0 osd=1 3 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 20 client.4256.objecter _get_session s=0x7f6ca80725c0 osd=1 3 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 15 client.4256.objecter _session_command_op_remove -1 2 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 20 client.4256.objecter get_session s=0x7f6ca80725c0 osd=1 3 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 15 client.4256.objecter _session_command_op_assign 1 2 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 20 client.4256.objecter put_session s=0x7f6ca80725c0 osd=1 4 2024-01-20T11:15:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 10 client.4256.objecter _send_command 2 2024-01-20T11:15:31.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.266+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 --> [v2:172.21.15.110:6800/3373744578,v1:172.21.15.110:6801/3373744578] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6cc4072460 con 0x7f6ca8072c90 2024-01-20T11:15:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6cc17fa640 1 -- 172.21.15.110:0/1173476578 <== osd.1 v2:172.21.15.110:6800/3373744578 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6cb0001190 con 0x7f6ca8072c90 2024-01-20T11:15:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6cc17fa640 10 client.4256.objecter ms_dispatch 0x7f6cc4000b90 command_reply(tid 2: 0 ) v1 2024-01-20T11:15:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6cc17fa640 10 client.4256.objecter _finish_command 2 = system:0 2024-01-20T11:15:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6cc17fa640 20 client.4256.objecter put_session s=0x7f6ca80725c0 osd=1 4 2024-01-20T11:15:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6cc17fa640 15 client.4256.objecter _session_command_op_remove 1 2 2024-01-20T11:15:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6ccacb4640 10 librados: watch_flush enter 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6ccacb4640 10 librados: watch_flush exit 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6ccacb4640 10 client.4256.objecter close_session for osd.1 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:6800/3373744578,v1:172.21.15.110:6801/3373744578] conn(0x7f6ca8072c90 msgr2=0x7f6ca80750b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6ccacb4640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:6800/3373744578,v1:172.21.15.110:6801/3373744578] conn(0x7f6ca8072c90 0x7f6ca80750b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6ccacb4640 20 client.4256.objecter put_session s=0x7f6ca80725c0 osd=1 1 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6ccacb4640 20 client.4256.objecter shutdown clearing up homeless session... 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.275+0000 7f6ccacb4640 10 client.4256.objecter successfully canceled tick 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.276+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6ca806c8b0 msgr2=0x7f6ca806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.276+0000 7f6ccacb4640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6ca806c8b0 0x7f6ca806ed70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009650 tx=0x7f6cb8006ba0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.276+0000 7f6ccacb4640 10 monclient: shutdown 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.276+0000 7f6ccacb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.276+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40b04c0 msgr2=0x7f6cc41d3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.276+0000 7f6ccacb4640 1 --2- 172.21.15.110:0/1173476578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cc40b04c0 0x7f6cc41d3550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc038040 tx=0x7f6cbc004a40 comp rx=0 tx=0).stop 2024-01-20T11:15:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.276+0000 7f6cc9251640 1 -- 172.21.15.110:0/1173476578 reap_dead start 2024-01-20T11:15:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.278+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 shutdown_connections 2024-01-20T11:15:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.278+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 >> 172.21.15.110:0/1173476578 conn(0x7f6cc40b1340 msgr2=0x7f6cc4071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.278+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 shutdown_connections 2024-01-20T11:15:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.278+0000 7f6ccacb4640 1 -- 172.21.15.110:0/1173476578 wait complete. 2024-01-20T11:15:31.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.279+0000 7f6ccacb4640 1 librados: shutdown 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 10 client.4323.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 15 client.4323.objecter _session_command_op_assign -1 2 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 20 client.4323.objecter _get_session s=0x7faf70072ec0 osd=4 3 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 20 client.4323.objecter put_session s=0x7faf70072ec0 osd=4 3 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 20 client.4323.objecter _get_session s=0x7faf70072ec0 osd=4 3 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 15 client.4323.objecter _session_command_op_remove -1 2 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 20 client.4323.objecter get_session s=0x7faf70072ec0 osd=4 3 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 15 client.4323.objecter _session_command_op_assign 4 2 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 20 client.4323.objecter put_session s=0x7faf70072ec0 osd=4 4 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 10 client.4323.objecter _send_command 2 2024-01-20T11:15:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.285+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 --> [v2:172.21.15.134:6822/2594530523,v1:172.21.15.134:6824/2594530523] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faf8413f350 con 0x7faf70073590 2024-01-20T11:15:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 10 client.4250.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 15 client.4250.objecter _session_command_op_assign -1 2 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 20 client.4250.objecter _get_session s=0x7fc314072610 osd=3 3 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 20 client.4250.objecter put_session s=0x7fc314072610 osd=3 3 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 20 client.4250.objecter _get_session s=0x7fc314072610 osd=3 3 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 15 client.4250.objecter _session_command_op_remove -1 2 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 20 client.4250.objecter get_session s=0x7fc314072610 osd=3 3 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 15 client.4250.objecter _session_command_op_assign 3 2 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 20 client.4250.objecter put_session s=0x7fc314072610 osd=3 4 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 10 client.4250.objecter _send_command 2 2024-01-20T11:15:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.287+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 --> [v2:172.21.15.110:6821/1038822226,v1:172.21.15.110:6823/1038822226] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc33406fe70 con 0x7fc314072ce0 2024-01-20T11:15:31.293 INFO:teuthology.orchestra.run.smithi110.stdout:73014444039 2024-01-20T11:15:31.294 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-20T11:15:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.296+0000 7faf80ff9640 1 -- 172.21.15.110:0/1859501030 <== osd.4 v2:172.21.15.134:6822/2594530523 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faf8413f350 con 0x7faf70073590 2024-01-20T11:15:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.296+0000 7faf80ff9640 10 client.4323.objecter ms_dispatch 0x7faf84000b90 command_reply(tid 2: 0 ) v1 2024-01-20T11:15:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.296+0000 7faf80ff9640 10 client.4323.objecter _finish_command 2 = system:0 2024-01-20T11:15:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.296+0000 7faf80ff9640 20 client.4323.objecter put_session s=0x7faf70072ec0 osd=4 4 2024-01-20T11:15:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.296+0000 7faf80ff9640 15 client.4323.objecter _session_command_op_remove 4 2 2024-01-20T11:15:31.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.297+0000 7faf8a70f640 10 librados: watch_flush enter 2024-01-20T11:15:31.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 10 librados: watch_flush exit 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 10 client.4323.objecter close_session for osd.4 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6822/2594530523,v1:172.21.15.134:6824/2594530523] conn(0x7faf70073590 msgr2=0x7faf70075a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6822/2594530523,v1:172.21.15.134:6824/2594530523] conn(0x7faf70073590 0x7faf70075a30 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 20 client.4323.objecter put_session s=0x7faf70072ec0 osd=4 1 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 20 client.4323.objecter shutdown clearing up homeless session... 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 10 client.4323.objecter successfully canceled tick 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf7006c890 msgr2=0x7faf7006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf7006c890 0x7faf7006ed50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faf7803a000 tx=0x7faf78032000 comp rx=0 tx=0).stop 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 10 monclient: shutdown 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf8406f920 msgr2=0x7faf841dbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 1 --2- 172.21.15.110:0/1859501030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf8406f920 0x7faf841dbe10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faf7400d5f0 tx=0x7faf7400dab0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf88cac640 1 -- 172.21.15.110:0/1859501030 reap_dead start 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 shutdown_connections 2024-01-20T11:15:31.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.298+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 >> 172.21.15.110:0/1859501030 conn(0x7faf84066d60 msgr2=0x7faf8413a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.299+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 shutdown_connections 2024-01-20T11:15:31.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.299+0000 7faf8a70f640 1 -- 172.21.15.110:0/1859501030 wait complete. 2024-01-20T11:15:31.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.299+0000 7faf8a70f640 1 librados: shutdown 2024-01-20T11:15:31.318 INFO:teuthology.orchestra.run.smithi110.stdout:94489280521 2024-01-20T11:15:31.318 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-20T11:15:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc3317fa640 1 -- 172.21.15.110:0/1796048816 <== osd.3 v2:172.21.15.110:6821/1038822226 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc3140757b0 con 0x7fc314072ce0 2024-01-20T11:15:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc3317fa640 10 client.4250.objecter ms_dispatch 0x7fc334000b90 command_reply(tid 2: 0 ) v1 2024-01-20T11:15:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc3317fa640 10 client.4250.objecter _finish_command 2 = system:0 2024-01-20T11:15:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc3317fa640 20 client.4250.objecter put_session s=0x7fc314072610 osd=3 4 2024-01-20T11:15:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc3317fa640 15 client.4250.objecter _session_command_op_remove 3 2 2024-01-20T11:15:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc33b11d640 10 librados: watch_flush enter 2024-01-20T11:15:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc33b11d640 10 librados: watch_flush exit 2024-01-20T11:15:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc33b11d640 10 client.4250.objecter close_session for osd.3 2024-01-20T11:15:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:6821/1038822226,v1:172.21.15.110:6823/1038822226] conn(0x7fc314072ce0 msgr2=0x7fc314075100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc33b11d640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:6821/1038822226,v1:172.21.15.110:6823/1038822226] conn(0x7fc314072ce0 0x7fc314075100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc33b11d640 20 client.4250.objecter put_session s=0x7fc314072610 osd=3 1 2024-01-20T11:15:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc33b11d640 20 client.4250.objecter shutdown clearing up homeless session... 2024-01-20T11:15:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.334+0000 7fc33b11d640 10 client.4250.objecter successfully canceled tick 2024-01-20T11:15:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.335+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc31406c8b0 msgr2=0x7fc31406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.335+0000 7fc33b11d640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc31406c8b0 0x7fc31406ed70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc324036000 tx=0x7fc324030000 comp rx=0 tx=0).stop 2024-01-20T11:15:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.335+0000 7fc33b11d640 10 monclient: shutdown 2024-01-20T11:15:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.335+0000 7fc33b11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.335+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 msgr2=0x7fc33413eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.335+0000 7fc33b11d640 1 --2- 172.21.15.110:0/1796048816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc33413d500 0x7fc33413eda0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e9b0 tx=0x7fc31c00ee70 comp rx=0 tx=0).stop 2024-01-20T11:15:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.335+0000 7fc3396ba640 1 -- 172.21.15.110:0/1796048816 reap_dead start 2024-01-20T11:15:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.336+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 shutdown_connections 2024-01-20T11:15:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.336+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 >> 172.21.15.110:0/1796048816 conn(0x7fc3340b57e0 msgr2=0x7fc334138c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.337+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 shutdown_connections 2024-01-20T11:15:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.338+0000 7fc33b11d640 1 -- 172.21.15.110:0/1796048816 wait complete. 2024-01-20T11:15:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.338+0000 7fc33b11d640 1 librados: shutdown 2024-01-20T11:15:31.356 INFO:teuthology.orchestra.run.smithi110.stdout:90194313224 2024-01-20T11:15:31.357 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-20T11:15:31.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.375+0000 7feba412c640 10 monclient: get_monmap_and_config 2024-01-20T11:15:31.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.375+0000 7feba412c640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.375+0000 7feba412c640 10 monclient: monmap: 2024-01-20T11:15:31.375 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.375 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.375 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:31.375823+0000 2024-01-20T11:15:31.376 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:31.375823+0000 2024-01-20T11:15:31.376 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.376 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.376 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:31.376 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:31.376 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:31.376 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:31.376 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:31.376 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.376+0000 7feba412c640 1 Processor -- start 2024-01-20T11:15:31.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.376+0000 7feba412c640 1 -- start start 2024-01-20T11:15:31.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.376+0000 7feba412c640 10 monclient: init 2024-01-20T11:15:31.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.377+0000 7feba412c640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.377+0000 7feba412c640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-01-20T11:15:31.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.377+0000 7feba412c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7feb9c0a96f0 0x7feb9c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.377+0000 7feba412c640 10 monclient(hunting): picked mon.noname-c con 0x7feb9c0a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:31.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.377+0000 7feba412c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb9c13fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:31.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.377+0000 7feba412c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb9c0a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:31.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.378+0000 7feba412c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9c0b25c0 con 0x7feb9c13fa10 2024-01-20T11:15:31.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.378+0000 7feba412c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9c0ab0a0 con 0x7feb9c0a9ad0 2024-01-20T11:15:31.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.378+0000 7feba412c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.378+0000 7feba412c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9c0aaea0 con 0x7feb9c0a96f0 2024-01-20T11:15:31.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.378+0000 7feba412c640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.378+0000 7feba412c640 10 monclient(hunting): authenticate will time out at 1374.728395s 2024-01-20T11:15:31.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.378+0000 7feba1ec8640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7feb9c0a96f0 0x7feb9c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.379+0000 7feba16c7640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7feb9c13fa10 0x7feb9c1423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46222/0 (socket says 172.21.15.110:46222) 2024-01-20T11:15:31.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.379+0000 7feba16c7640 1 -- 172.21.15.110:0/3750840410 learned_addr learned my addr 172.21.15.110:0/3750840410 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:31.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.379+0000 7feba1ec8640 10 monclient(hunting): get_auth_request con 0x7feb9c0a96f0 auth_method 0 2024-01-20T11:15:31.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.379+0000 7feba1ec8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.379+0000 7feba1ec8640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.379+0000 7feba1ec8640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.380+0000 7feba1ec8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.380+0000 7feba1ec8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.380+0000 7feba1ec8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.382+0000 7feba0ec6640 1 -- 172.21.15.110:0/3750840410 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543492387 0 0) 0x7feb9c0b25c0 con 0x7feb9c13fa10 2024-01-20T11:15:31.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.382+0000 7feba0ec6640 10 monclient(hunting): _negotiate 2024-01-20T11:15:31.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.382+0000 7feba0ec6640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.382+0000 7feba0ec6640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.382+0000 7feba0ec6640 10 monclient(hunting): my global_id is 4326 2024-01-20T11:15:31.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.382+0000 7feba0ec6640 1 -- 172.21.15.110:0/3750840410 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb88003520 con 0x7feb9c13fa10 2024-01-20T11:15:31.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.384+0000 7feba1ec8640 10 monclient(hunting): handle_auth_done global_id 4360 payload 274 2024-01-20T11:15:31.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.384+0000 7feba1ec8640 1 -- 172.21.15.110:0/3750840410 >> v1:172.21.15.134:6789/0 conn(0x7feb9c0a9ad0 legacy=0x7feb9c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.384+0000 7feba1ec8640 1 -- 172.21.15.110:0/3750840410 >> v1:172.21.15.110:6789/0 conn(0x7feb9c13fa10 legacy=0x7feb9c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.384+0000 7feba1ec8640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.384+0000 7feba1ec8640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.384+0000 7feba1ec8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.384+0000 7feba1ec8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.384+0000 7feba1ec8640 1 -- 172.21.15.110:0/3750840410 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c0b3fb0 con 0x7feb9c0a96f0 2024-01-20T11:15:31.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.385+0000 7f71c8d08640 10 monclient: get_monmap_and_config 2024-01-20T11:15:31.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.385+0000 7f71c8d08640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.385+0000 7f71c8d08640 10 monclient: monmap: 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:31.385946+0000 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:31.385946+0000 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:31.389 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:31.389 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:31.389 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.386+0000 7f71c8d08640 1 Processor -- start 2024-01-20T11:15:31.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.386+0000 7f71c8d08640 1 -- start start 2024-01-20T11:15:31.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.386+0000 7f71c8d08640 10 monclient: init 2024-01-20T11:15:31.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7feba1ec8640 1 --2- 172.21.15.110:0/3750840410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb9c0a96f0 0x7feb9c13f4d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feb980097a0 tx=0x7feb9802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81eb2c39659e4448 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f71c40a96f0 0x7f71c413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 10 monclient(hunting): picked mon.noname-c con 0x7f71c40a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71c413fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71c40a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c40b25c0 con 0x7f71c40a9ad0 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c40ab0a0 con 0x7f71c413fa10 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c40aaea0 con 0x7f71c40a96f0 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f71c8d08640 10 monclient(hunting): authenticate will time out at 1374.737790s 2024-01-20T11:15:31.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f793f31d640 10 monclient: get_monmap_and_config 2024-01-20T11:15:31.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f793f31d640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.387+0000 7f793f31d640 10 monclient: monmap: 2024-01-20T11:15:31.392 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:31.388422+0000 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:31.388422+0000 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.388+0000 7f793f31d640 1 Processor -- start 2024-01-20T11:15:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f71c2d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f71c40a9ad0 0x7f71c414c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46238/0 (socket says 172.21.15.110:46238) 2024-01-20T11:15:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f71c2d9d640 1 -- 172.21.15.110:0/583092178 learned_addr learned my addr 172.21.15.110:0/583092178 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.388+0000 7f793f31d640 1 -- start start 2024-01-20T11:15:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.388+0000 7f793f31d640 10 monclient: init 2024-01-20T11:15:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f71c259c640 1 --2- 172.21.15.110:0/583092178 >> v2:172.21.15.134:3300/0 conn(0x7f71c40a96f0 0x7f71c413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79380a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f79380a9ad0 0x7f79381423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 10 monclient(hunting): picked mon.noname-a con 0x7f79380a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7938142920 0x7f793814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 10 monclient(hunting): picked mon.noname-c con 0x7f7938142920 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79380b25c0 con 0x7f79380a96f0 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79380aaea0 con 0x7f79380a9ad0 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f793813fbe0 con 0x7f7938142920 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.389+0000 7f793f31d640 10 monclient(hunting): authenticate will time out at 1374.740178s 2024-01-20T11:15:31.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.391+0000 7f71c259c640 10 monclient(hunting): get_auth_request con 0x7f71c40a96f0 auth_method 0 2024-01-20T11:15:31.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.391+0000 7f71c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.391+0000 7f71c259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.391+0000 7f71c259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f71c159a640 1 -- 172.21.15.110:0/583092178 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267066375 0 0) 0x7f71c40b25c0 con 0x7f71c40a9ad0 2024-01-20T11:15:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f71c159a640 10 monclient(hunting): _negotiate 2024-01-20T11:15:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f71c159a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f71c159a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f71c159a640 10 monclient(hunting): my global_id is 4329 2024-01-20T11:15:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f71c159a640 1 -- 172.21.15.110:0/583092178 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ac003520 con 0x7f71c40a9ad0 2024-01-20T11:15:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f71c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f71c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f71c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.392+0000 7f793c8b8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f79380a9ad0 0x7f79381423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7f793d0b9640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f79380a96f0 0x7f793813f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46250/0 (socket says 172.21.15.110:46250) 2024-01-20T11:15:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7f793d0b9640 1 -- 172.21.15.110:0/4162101223 learned_addr learned my addr 172.21.15.110:0/4162101223 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 1 -- 172.21.15.110:0/3750840410 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98031830 con 0x7feb9c0a96f0 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 10 monclient: dump: 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 1 -- 172.21.15.110:0/3750840410 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb98031990 con 0x7feb9c0a96f0 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 1 -- 172.21.15.110:0/3750840410 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98031d20 con 0x7feb9c0a96f0 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba0ec6640 10 monclient: dump: 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba412c640 5 monclient: authenticate success, global_id 4360 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba412c640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.393+0000 7feba412c640 4 set_mon_vals no callback set 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7feba412c640 10 monclient: shutdown 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7feba412c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7feba412c640 1 -- 172.21.15.110:0/3750840410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb9c0a96f0 msgr2=0x7feb9c13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7feba412c640 1 --2- 172.21.15.110:0/3750840410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb9c0a96f0 0x7feb9c13f4d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feb980097a0 tx=0x7feb9802b8d0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7f71c259c640 10 monclient(hunting): handle_auth_done global_id 4366 payload 274 2024-01-20T11:15:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7f71c259c640 1 -- 172.21.15.110:0/583092178 >> v1:172.21.15.134:6789/0 conn(0x7f71c413fa10 legacy=0x7f71c41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7f71c259c640 1 -- 172.21.15.110:0/583092178 >> v1:172.21.15.110:6789/0 conn(0x7f71c40a9ad0 legacy=0x7f71c414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7feba412c640 1 -- 172.21.15.110:0/3750840410 shutdown_connections 2024-01-20T11:15:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7feba412c640 1 --2- 172.21.15.110:0/3750840410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb9c0a96f0 0x7feb9c13f4d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7feba412c640 1 -- 172.21.15.110:0/3750840410 >> 172.21.15.110:0/3750840410 conn(0x7feb9c0b7bc0 msgr2=0x7feb9c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7f71c259c640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7f71c259c640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.395+0000 7feba412c640 1 -- 172.21.15.110:0/3750840410 shutdown_connections 2024-01-20T11:15:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.395+0000 7feba412c640 1 -- 172.21.15.110:0/3750840410 wait complete. 2024-01-20T11:15:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7f71c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7f71c259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.394+0000 7f71c259c640 1 -- 172.21.15.110:0/583092178 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c40b3fb0 con 0x7f71c40a96f0 2024-01-20T11:15:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7feba412c640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7feba412c640 10 monclient: monmap: 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.407 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7fd804bb6640 10 monclient: get_monmap_and_config 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7fd804bb6640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7fd804bb6640 10 monclient: monmap: 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:31.396789+0000 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:31.396789+0000 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:31.409 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:31.409 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:31.409 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:31.409 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7f793d8ba640 1 --2- 172.21.15.110:0/4162101223 >> v2:172.21.15.134:3300/0 conn(0x7f7938142920 0x7f793814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7feba412c640 1 librados: starting msgr at 2024-01-20T11:15:31.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7feba412c640 1 librados: starting objecter 2024-01-20T11:15:31.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7feba412c640 1 Processor -- start 2024-01-20T11:15:31.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7feba412c640 1 -- start start 2024-01-20T11:15:31.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7feba412c640 1 librados: setting wanted keys 2024-01-20T11:15:31.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7feba412c640 1 librados: calling monclient init 2024-01-20T11:15:31.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.396+0000 7feba412c640 10 monclient: init 2024-01-20T11:15:31.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7fd804bb6640 1 Processor -- start 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb9c0a96f0 0x7feb9c1d7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient(hunting): picked mon.noname-a con 0x7feb9c0a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb9c0a9ad0 0x7feb9c139d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient(hunting): picked mon.noname-b con 0x7feb9c0a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb9c1d7eb0 0x7feb9c13a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient(hunting): picked mon.noname-c con 0x7feb9c1d7eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c1d74e0 con 0x7feb9c0a96f0 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c13a780 con 0x7feb9c0a9ad0 2024-01-20T11:15:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c1cb520 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7feba412c640 10 monclient(hunting): authenticate will time out at 1374.748020s 2024-01-20T11:15:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7f793d8ba640 10 monclient(hunting): get_auth_request con 0x7f7938142920 auth_method 0 2024-01-20T11:15:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7f793d8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7f793d8ba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7f793d8ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7fd804bb6640 1 -- start start 2024-01-20T11:15:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.397+0000 7fd804bb6640 10 monclient: init 2024-01-20T11:15:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-01-20T11:15:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8000663b0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd800066790 0x7fd80013b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 10 monclient(hunting): picked mon.noname-a con 0x7fd800066790 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd80013bb70 0x7fd80013e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7feba26c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb9c1d7eb0 0x7feb9c13a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 10 monclient(hunting): picked mon.noname-c con 0x7fd80013bb70 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80013bfa0 con 0x7fd8000663b0 2024-01-20T11:15:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd800070d10 con 0x7fd800066790 2024-01-20T11:15:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd800070860 con 0x7fd80013bb70 2024-01-20T11:15:31.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7fd804bb6640 10 monclient(hunting): authenticate will time out at 1374.748944s 2024-01-20T11:15:31.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7f793d8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7f793d8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.398+0000 7f793d8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.399+0000 7feba26c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb9c1d7eb0 0x7feb9c13a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33770/0 (socket says 172.21.15.110:33770) 2024-01-20T11:15:31.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.399+0000 7feba26c9640 1 -- 172.21.15.110:0/501189539 learned_addr learned my addr 172.21.15.110:0/501189539 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.399+0000 7feba26c9640 10 monclient(hunting): get_auth_request con 0x7feb9c1d7eb0 auth_method 0 2024-01-20T11:15:31.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.399+0000 7feba26c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.399+0000 7feba26c9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.399+0000 7feba26c9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.399+0000 7feba16c7640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb9c0a9ad0 0x7feb9c139d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.400+0000 7feba26c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.400+0000 7feba26c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.400+0000 7feba26c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.400+0000 7f793d8ba640 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2024-01-20T11:15:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.401+0000 7f793d8ba640 1 -- 172.21.15.110:0/4162101223 >> v2:172.21.15.110:3300/0 conn(0x7f79380a9ad0 msgr2=0x7f79381423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.401+0000 7f793d8ba640 1 --2- 172.21.15.110:0/4162101223 >> v2:172.21.15.110:3300/0 conn(0x7f79380a9ad0 0x7f79381423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.401+0000 7f793d8ba640 1 -- 172.21.15.110:0/4162101223 >> v1:172.21.15.110:6789/0 conn(0x7f79380a96f0 legacy=0x7f793813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.401+0000 7f793d8ba640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.401+0000 7f793d8ba640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.401+0000 7f793d8ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.401+0000 7f793d8ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.401+0000 7f793d8ba640 1 -- 172.21.15.110:0/4162101223 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79380ab0a0 con 0x7f7938142920 2024-01-20T11:15:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.402+0000 7feba16c7640 10 monclient(hunting): get_auth_request con 0x7feb9c0a9ad0 auth_method 0 2024-01-20T11:15:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.402+0000 7feba16c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.402+0000 7feba16c7640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.402+0000 7feba16c7640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.403+0000 7feba16c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.403+0000 7feba16c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.403+0000 7feba16c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.403+0000 7feba26c9640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.404+0000 7feba26c9640 1 -- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb9c0a9ad0 msgr2=0x7feb9c139d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.404+0000 7feba26c9640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb9c0a9ad0 0x7feb9c139d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.404+0000 7feba26c9640 1 -- 172.21.15.110:0/501189539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb9c0a96f0 msgr2=0x7feb9c1d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.404+0000 7feba26c9640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb9c0a96f0 0x7feb9c1d7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.404+0000 7feba26c9640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.404+0000 7feba26c9640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.404+0000 7feba26c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.404+0000 7feba26c9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.404+0000 7feba26c9640 1 -- 172.21.15.110:0/501189539 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98009400 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.406+0000 7feba26c9640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb9c1d7eb0 0x7feb9c13a240 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb9400c730 tx=0x7feb9400cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.406+0000 7feb927fc640 10 client.?.objecter ms_handle_connect 0x7feb9c1d7eb0 2024-01-20T11:15:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.406+0000 7feb927fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.408+0000 7f793d8ba640 1 --2- 172.21.15.110:0/4162101223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7938142920 0x7f793814c850 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7938075f80 tx=0x7f793002be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62f1740ea830a56d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7fd7f7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd800066790 0x7fd80013b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feb927fc640 1 -- 172.21.15.110:0/501189539 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94006d50 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feb927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feb927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feb927fc640 10 monclient: dump: 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feb927fc640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feb927fc640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7fd7fed9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd80013bb70 0x7fd80013e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7fd7f7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd800066790 0x7fd80013b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48532/0 (socket says 172.21.15.110:48532) 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7fd7f7fff640 1 -- 172.21.15.110:0/816300125 learned_addr learned my addr 172.21.15.110:0/816300125 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7fd7f7fff640 10 monclient(hunting): get_auth_request con 0x7fd800066790 auth_method 0 2024-01-20T11:15:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7fd7f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7fd7f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7fd7f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feb927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feba412c640 5 monclient: authenticate success, global_id 4259 2024-01-20T11:15:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feba412c640 10 monclient: _renew_subs 2024-01-20T11:15:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feba412c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9c1407e0 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feba412c640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feba412c640 10 monclient: _renew_subs 2024-01-20T11:15:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feba412c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7fd7fed9d640 10 monclient(hunting): get_auth_request con 0x7fd80013bb70 auth_method 0 2024-01-20T11:15:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7fd7fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7fd7fed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7fd7fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9c1cbf90 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.409+0000 7feba412c640 1 librados: init done 2024-01-20T11:15:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7fd7f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7fd7f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7fd7f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7feba412c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7feba412c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7feba412c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.410+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9c1d72b0 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.413+0000 7fd7f7fff640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-01-20T11:15:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.413+0000 7fd7f7fff640 1 -- 172.21.15.110:0/816300125 >> v2:172.21.15.134:3300/0 conn(0x7fd80013bb70 msgr2=0x7fd80013e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.413+0000 7fd7f7fff640 1 --2- 172.21.15.110:0/816300125 >> v2:172.21.15.134:3300/0 conn(0x7fd80013bb70 0x7fd80013e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.413+0000 7fd7f7fff640 1 -- 172.21.15.110:0/816300125 >> v1:172.21.15.110:6789/0 conn(0x7fd8000663b0 legacy=0x7fd80006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.413+0000 7fd7f7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.413+0000 7fd7f7fff640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.413+0000 7fd7f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.413+0000 7fd7f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.413+0000 7fd7f7fff640 1 -- 172.21.15.110:0/816300125 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80006f490 con 0x7fd800066790 2024-01-20T11:15:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 1 -- 172.21.15.110:0/4162101223 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793002cbe0 con 0x7f7938142920 2024-01-20T11:15:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 10 monclient: dump: 2024-01-20T11:15:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 1 -- 172.21.15.110:0/4162101223 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793002cd40 con 0x7f7938142920 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 1 -- 172.21.15.110:0/4162101223 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930034660 con 0x7f7938142920 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f792ffff640 10 monclient: dump: 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f793f31d640 5 monclient: authenticate success, global_id 4369 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f793f31d640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.415+0000 7f793f31d640 4 set_mon_vals no callback set 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 10 monclient: shutdown 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 1 -- 172.21.15.110:0/4162101223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7938142920 msgr2=0x7f793814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 1 --2- 172.21.15.110:0/4162101223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7938142920 0x7f793814c850 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7938075f80 tx=0x7f793002be30 comp rx=0 tx=0).stop 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 1 -- 172.21.15.110:0/4162101223 shutdown_connections 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 1 --2- 172.21.15.110:0/4162101223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7938142920 0x7f793814c850 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 1 --2- 172.21.15.110:0/4162101223 >> v2:172.21.15.110:3300/0 conn(0x7f79380a9ad0 0x7f79381423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 1 -- 172.21.15.110:0/4162101223 >> 172.21.15.110:0/4162101223 conn(0x7f79380b7bc0 msgr2=0x7f79380b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 1 -- 172.21.15.110:0/4162101223 shutdown_connections 2024-01-20T11:15:31.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f793f31d640 1 -- 172.21.15.110:0/4162101223 wait complete. 2024-01-20T11:15:31.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.416+0000 7f71c259c640 1 --2- 172.21.15.110:0/583092178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 0x7f71c413f4d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f71b80097a0 tx=0x7f71b802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12abc56f4baf7589 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.417+0000 7f793f31d640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.417+0000 7f793f31d640 10 monclient: monmap: 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.417+0000 7f793f31d640 1 librados: starting msgr at 2024-01-20T11:15:31.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.417+0000 7f793f31d640 1 librados: starting objecter 2024-01-20T11:15:31.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.417+0000 7f793f31d640 1 Processor -- start 2024-01-20T11:15:31.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.417+0000 7f793f31d640 1 -- start start 2024-01-20T11:15:31.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.417+0000 7f793f31d640 1 librados: setting wanted keys 2024-01-20T11:15:31.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.417+0000 7f793f31d640 1 librados: calling monclient init 2024-01-20T11:15:31.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.417+0000 7f793f31d640 10 monclient: init 2024-01-20T11:15:31.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7fd7f7fff640 1 --2- 172.21.15.110:0/816300125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 0x7fd80013b630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd8000b0490 tx=0x7fd7e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a300bb687e1f5a19 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:15:31.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79380a96f0 0x7f79381390c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient(hunting): picked mon.noname-c con 0x7f79380a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79380a9ad0 0x7f79381df2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient(hunting): picked mon.noname-b con 0x7f79380a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7938142920 0x7f79381df7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient(hunting): picked mon.noname-a con 0x7f7938142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79380b18f0 con 0x7f7938142920 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79381dee20 con 0x7f79380a9ad0 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79380ad560 con 0x7f79380a96f0 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.418+0000 7f793f31d640 10 monclient(hunting): authenticate will time out at 1374.768868s 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.420+0000 7f793d8ba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7938142920 0x7f79381df7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f793c8b8640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79380a9ad0 0x7f79381df2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 1 -- 172.21.15.110:0/583092178 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b802e070 con 0x7f71c40a96f0 2024-01-20T11:15:31.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 10 monclient: dump: 2024-01-20T11:15:31.443 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.443 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.443 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.443 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 1 -- 172.21.15.110:0/583092178 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b80315a0 con 0x7f71c40a96f0 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 1 -- 172.21.15.110:0/583092178 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b802e070 con 0x7f71c40a96f0 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c159a640 10 monclient: dump: 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c8d08640 5 monclient: authenticate success, global_id 4366 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c8d08640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.421+0000 7f71c8d08640 4 set_mon_vals no callback set 2024-01-20T11:15:31.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f793c8b8640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79380a9ad0 0x7f79381df2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58142/0 (socket says 172.21.15.110:58142) 2024-01-20T11:15:31.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f793c8b8640 1 -- 172.21.15.110:0/1172510983 learned_addr learned my addr 172.21.15.110:0/1172510983 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f793c8b8640 10 monclient(hunting): get_auth_request con 0x7f79380a9ad0 auth_method 0 2024-01-20T11:15:31.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f793c8b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f793c8b8640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f793c8b8640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f71c8d08640 10 monclient: shutdown 2024-01-20T11:15:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f71c8d08640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f71c8d08640 1 -- 172.21.15.110:0/583092178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 msgr2=0x7f71c413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.422+0000 7f71c8d08640 1 --2- 172.21.15.110:0/583092178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 0x7f71c413f4d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f71b80097a0 tx=0x7f71b802b840 comp rx=0 tx=0).stop 2024-01-20T11:15:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.423+0000 7f793c8b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.423+0000 7f793c8b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.423+0000 7f793c8b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.423+0000 7f71c8d08640 1 -- 172.21.15.110:0/583092178 shutdown_connections 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.423+0000 7f71c8d08640 1 --2- 172.21.15.110:0/583092178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 0x7f71c413f4d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.423+0000 7f71c8d08640 1 -- 172.21.15.110:0/583092178 >> 172.21.15.110:0/583092178 conn(0x7f71c40b7bc0 msgr2=0x7f71c40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.423+0000 7f71c8d08640 1 -- 172.21.15.110:0/583092178 shutdown_connections 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.423+0000 7f71c8d08640 1 -- 172.21.15.110:0/583092178 wait complete. 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.424+0000 7f71c8d08640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.424+0000 7f71c8d08640 10 monclient: monmap: 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.450 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.424+0000 7f793d0b9640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79380a96f0 0x7f79381390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.424+0000 7f71c8d08640 1 librados: starting msgr at 2024-01-20T11:15:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.424+0000 7f71c8d08640 1 librados: starting objecter 2024-01-20T11:15:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 1 Processor -- start 2024-01-20T11:15:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 1 -- start start 2024-01-20T11:15:31.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 1 librados: setting wanted keys 2024-01-20T11:15:31.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f793d8ba640 10 monclient(hunting): get_auth_request con 0x7f7938142920 auth_method 0 2024-01-20T11:15:31.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f793d8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f793d8ba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f793d8ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 1 librados: calling monclient init 2024-01-20T11:15:31.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f793d8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f793d8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 10 monclient: init 2024-01-20T11:15:31.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f793d8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 0x7f71c4070d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 10 monclient(hunting): picked mon.noname-b con 0x7f71c40a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.425+0000 7f71c8d08640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f71c40a9ad0 0x7f71c406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 10 monclient(hunting): picked mon.noname-c con 0x7f71c40a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71c406e1f0 0x7f71c41de740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 10 monclient(hunting): picked mon.noname-a con 0x7f71c406e1f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40ab8a0 con 0x7f71c406e1f0 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c4071540 con 0x7f71c40a96f0 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40abb10 con 0x7f71c40a9ad0 2024-01-20T11:15:31.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f71c8d08640 10 monclient(hunting): authenticate will time out at 1374.776425s 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 1 -- 172.21.15.110:0/1172510983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79380a96f0 msgr2=0x7f79381390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79380a96f0 0x7f79381390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 1 -- 172.21.15.110:0/1172510983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7938142920 msgr2=0x7f79381df7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7938142920 0x7f79381df7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 1 monclient: found mon.noname-b 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.426+0000 7f793c8b8640 1 -- 172.21.15.110:0/1172510983 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7930009a70 con 0x7f79380a9ad0 2024-01-20T11:15:31.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 1 -- 172.21.15.110:0/816300125 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e80317e0 con 0x7fd800066790 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 10 monclient: dump: 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 1 -- 172.21.15.110:0/816300125 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e8031940 con 0x7fd800066790 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 1 -- 172.21.15.110:0/816300125 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8031cd0 con 0x7fd800066790 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd7fdd9b640 10 monclient: dump: 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd804bb6640 5 monclient: authenticate success, global_id 4335 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd804bb6640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.427+0000 7fd804bb6640 4 set_mon_vals no callback set 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 10 monclient: shutdown 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 1 -- 172.21.15.110:0/816300125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 msgr2=0x7fd80013b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 1 --2- 172.21.15.110:0/816300125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 0x7fd80013b630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd8000b0490 tx=0x7fd7e802b880 comp rx=0 tx=0).stop 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 1 -- 172.21.15.110:0/816300125 shutdown_connections 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 1 --2- 172.21.15.110:0/816300125 >> v2:172.21.15.134:3300/0 conn(0x7fd80013bb70 0x7fd80013e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 1 --2- 172.21.15.110:0/816300125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 0x7fd80013b630 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 1 -- 172.21.15.110:0/816300125 >> 172.21.15.110:0/816300125 conn(0x7fd8000b3f50 msgr2=0x7fd8000b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 1 -- 172.21.15.110:0/816300125 shutdown_connections 2024-01-20T11:15:31.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7fd804bb6640 1 -- 172.21.15.110:0/816300125 wait complete. 2024-01-20T11:15:31.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.428+0000 7f71c259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 0x7f71c4070d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7fd804bb6640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7fd804bb6640 10 monclient: monmap: 2024-01-20T11:15:31.463 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7fd804bb6640 1 librados: starting msgr at 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7fd804bb6640 1 librados: starting objecter 2024-01-20T11:15:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7fd804bb6640 1 Processor -- start 2024-01-20T11:15:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7f71c259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 0x7f71c4070d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58158/0 (socket says 172.21.15.110:58158) 2024-01-20T11:15:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7f71c259c640 1 -- 172.21.15.110:0/1641043361 learned_addr learned my addr 172.21.15.110:0/1641043361 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7f71c259c640 10 monclient(hunting): get_auth_request con 0x7f71c40a96f0 auth_method 0 2024-01-20T11:15:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7f71c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7f71c259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7f71c259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7fd804bb6640 1 -- start start 2024-01-20T11:15:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7fd804bb6640 1 librados: setting wanted keys 2024-01-20T11:15:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7fd804bb6640 1 librados: calling monclient init 2024-01-20T11:15:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.429+0000 7fd804bb6640 10 monclient: init 2024-01-20T11:15:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8000663b0 0x7fd8001384d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient(hunting): picked mon.noname-b con 0x7fd8000663b0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 0x7fd8000b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient(hunting): picked mon.noname-a con 0x7fd800066790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd80013bb70 0x7fd8000b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient(hunting): picked mon.noname-c con 0x7fd80013bb70 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8000b2f70 con 0x7fd800066790 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8000b20d0 con 0x7fd8000663b0 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd800134740 con 0x7fd80013bb70 2024-01-20T11:15:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7fd804bb6640 10 monclient(hunting): authenticate will time out at 1374.780705s 2024-01-20T11:15:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7f793c8b8640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79380a9ad0 0x7f79381df2b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f793400c820 tx=0x7f793400cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7f792dffb640 10 client.?.objecter ms_handle_connect 0x7f79380a9ad0 2024-01-20T11:15:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.430+0000 7f792dffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.431+0000 7f71c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.431+0000 7f71c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.431+0000 7f71c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 1 -- 172.21.15.110:0/1641043361 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f71c40a9ad0 msgr2=0x7f71c406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f71c40a9ad0 0x7f71c406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 1 -- 172.21.15.110:0/1641043361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71c406e1f0 msgr2=0x7f71c41de740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71c406e1f0 0x7f71c41de740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 1 monclient: found mon.noname-b 2024-01-20T11:15:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.434+0000 7f71c259c640 1 -- 172.21.15.110:0/1641043361 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b8009400 con 0x7f71c40a96f0 2024-01-20T11:15:31.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.435+0000 7fd7fe59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8000663b0 0x7fd8001384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.436+0000 7fd7fe59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8000663b0 0x7fd8001384d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58170/0 (socket says 172.21.15.110:58170) 2024-01-20T11:15:31.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.436+0000 7fd7fe59c640 1 -- 172.21.15.110:0/3488353251 learned_addr learned my addr 172.21.15.110:0/3488353251 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.436+0000 7fd7fe59c640 10 monclient(hunting): get_auth_request con 0x7fd8000663b0 auth_method 0 2024-01-20T11:15:31.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.436+0000 7fd7fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.436+0000 7fd7fe59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.436+0000 7fd7fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.437+0000 7f71c259c640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 0x7f71c4070d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f71b8000c00 tx=0x7f71b80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.437+0000 7f71b2ffd640 10 client.?.objecter ms_handle_connect 0x7f71c40a96f0 2024-01-20T11:15:31.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.437+0000 7f71b2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.437+0000 7fd7f7fff640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 0x7fd8000b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.438+0000 7fd7f7fff640 10 monclient(hunting): get_auth_request con 0x7fd800066790 auth_method 0 2024-01-20T11:15:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.438+0000 7fd7f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.438+0000 7fd7f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.438+0000 7fd7f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.439+0000 7fd7f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.439+0000 7fd7f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.439+0000 7fd7f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71b2ffd640 1 -- 172.21.15.110:0/1641043361 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b802e070 con 0x7f71c40a96f0 2024-01-20T11:15:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71b2ffd640 10 monclient: dump: 2024-01-20T11:15:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71b2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71b2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71c8d08640 5 monclient: authenticate success, global_id 4381 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71c8d08640 10 monclient: _renew_subs 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.440+0000 7f71c8d08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.441+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c40abdf0 con 0x7f71c40a96f0 2024-01-20T11:15:31.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.441+0000 7f71c8d08640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.441+0000 7f71c8d08640 10 monclient: _renew_subs 2024-01-20T11:15:31.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.441+0000 7f71c8d08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.441+0000 7fd7fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.441+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c4138890 con 0x7f71c40a96f0 2024-01-20T11:15:31.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.441+0000 7f71c8d08640 1 librados: init done 2024-01-20T11:15:31.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.441+0000 7fd7fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.441+0000 7fd7fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7f71c8d08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7f71c8d08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7f71c8d08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c4075490 con 0x7f71c40a96f0 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 1 -- 172.21.15.110:0/3488353251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd80013bb70 msgr2=0x7fd8000b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd80013bb70 0x7fd8000b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 1 -- 172.21.15.110:0/3488353251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8000663b0 msgr2=0x7fd8001384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8000663b0 0x7fd8001384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.442+0000 7fd7f7fff640 1 -- 172.21.15.110:0/3488353251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e80094d0 con 0x7fd800066790 2024-01-20T11:15:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.444+0000 7feb927fc640 1 -- 172.21.15.110:0/501189539 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb94001790 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.445+0000 7feb927fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.445+0000 7feb93fff640 4 set_mon_vals no callback set 2024-01-20T11:15:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.445+0000 7feb927fc640 1 -- 172.21.15.110:0/501189539 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94003b70 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.445+0000 7feb927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.445+0000 7feb927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.445+0000 7feb927fc640 10 monclient: dump: 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.446+0000 7feb927fc640 1 -- 172.21.15.110:0/501189539 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7feb9400a7f0 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.446+0000 7feb927fc640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feb8006c860 0x7feb8006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.446+0000 7fd7f7fff640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 0x7fd8000b24f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8033000 tx=0x7fd7e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.446+0000 7fd7f6ffd640 10 client.?.objecter ms_handle_connect 0x7fd800066790 2024-01-20T11:15:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.446+0000 7fd7f6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.446+0000 7feba1ec8640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feb8006c860 0x7feb8006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.446+0000 7feba1ec8640 10 monclient: get_auth_request con 0x7feb8006c860 auth_method 0 2024-01-20T11:15:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.447+0000 7feb927fc640 10 client.4259.objecter ms_handle_connect 0x7feb8006c860 2024-01-20T11:15:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.447+0000 7feba1ec8640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feb8006c860 0x7feb8006ed20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb98036000 tx=0x7feb98030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.448+0000 7f71b2ffd640 1 -- 172.21.15.110:0/1641043361 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b8032640 con 0x7f71c40a96f0 2024-01-20T11:15:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.448+0000 7f71b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.448+0000 7f71b2ffd640 1 -- 172.21.15.110:0/1641043361 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8031f10 con 0x7f71c40a96f0 2024-01-20T11:15:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.448+0000 7f71b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.448+0000 7f71b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.448+0000 7f71b2ffd640 10 monclient: dump: 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.448+0000 7f71c0d99640 4 set_mon_vals no callback set 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.450+0000 7fdbcc226640 10 monclient: get_monmap_and_config 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.450+0000 7fdbcc226640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.450+0000 7fdbcc226640 10 monclient: monmap: 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:31.451063+0000 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:31.451063+0000 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:31.488 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fdbcc226640 1 Processor -- start 2024-01-20T11:15:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7feb927fc640 1 -- 172.21.15.110:0/501189539 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feb9400ce40 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7feb927fc640 10 client.4259.objecter ms_dispatch 0x7feb9c000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7feb927fc640 3 client.4259.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7feb927fc640 3 client.4259.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7feb927fc640 20 client.4259.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fdbcc226640 1 -- start start 2024-01-20T11:15:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fdbcc226640 10 monclient: init 2024-01-20T11:15:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 1 -- 172.21.15.110:0/3488353251 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e80329c0 con 0x7fd800066790 2024-01-20T11:15:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 10 monclient: dump: 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 1 -- 172.21.15.110:0/3488353251 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e8032b40 con 0x7fd800066790 2024-01-20T11:15:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd804bb6640 5 monclient: authenticate success, global_id 4341 2024-01-20T11:15:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd804bb6640 10 monclient: _renew_subs 2024-01-20T11:15:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd804bb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8000b3ca0 con 0x7fd800066790 2024-01-20T11:15:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-01-20T11:15:31.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdbc40a96f0 0x7fdbc413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc40a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:31.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdbc413fa10 0x7fdbc41423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc413fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:31.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbc40a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:31.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40b25c0 con 0x7fdbc40a9ad0 2024-01-20T11:15:31.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc40aaea0 con 0x7fdbc40a96f0 2024-01-20T11:15:31.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc40ab0a0 con 0x7fdbc413fa10 2024-01-20T11:15:31.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.452+0000 7fdbcc226640 10 monclient(hunting): authenticate will time out at 1374.802765s 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd804bb6640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7fd59a640 4 set_mon_vals no callback set 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 1 -- 172.21.15.110:0/3488353251 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e80453f0 con 0x7fd800066790 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd7f6ffd640 10 monclient: dump: 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd804bb6640 10 monclient: _renew_subs 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd804bb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd800134c60 con 0x7fd800066790 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.451+0000 7fd804bb6640 1 librados: init done 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.453+0000 7fd804bb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.453+0000 7fd804bb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.453+0000 7fd804bb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.453+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd800067cd0 con 0x7fd800066790 2024-01-20T11:15:31.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.469+0000 7fa6c91cf640 10 monclient: get_monmap_and_config 2024-01-20T11:15:31.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.469+0000 7fa6c91cf640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.469+0000 7fa6c91cf640 10 monclient: monmap: 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:31.470059+0000 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:31.470059+0000 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 1 Processor -- start 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 1 -- start start 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient: init 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa6c40b4d10 0x7fa6c40b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c40b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6c40b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa6c4145980 0x7fa6c4145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c4145980 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c4146370 con 0x7fa6c40b17c0 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c40b50f0 con 0x7fa6c40b4d10 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c40aade0 con 0x7fa6c4145980 2024-01-20T11:15:31.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.470+0000 7fa6c91cf640 10 monclient(hunting): authenticate will time out at 1374.821099s 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.516+0000 7fa22a7d0640 10 monclient: get_monmap_and_config 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.516+0000 7fa22a7d0640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.516+0000 7fa22a7d0640 10 monclient: monmap: 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:31.516883+0000 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:31.516883+0000 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:31.517 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:31.517 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:31.517 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.516+0000 7fa22a7d0640 1 Processor -- start 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 1 -- start start 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient: init 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa2240a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa22413fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa2240a9ad0 0x7fa224142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient(hunting): picked mon.noname-a con 0x7fa2240a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:31.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240b25c0 con 0x7fa22413fa10 2024-01-20T11:15:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2240aaea0 con 0x7fa2240a9ad0 2024-01-20T11:15:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2240ab0a0 con 0x7fa2240a96f0 2024-01-20T11:15:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.517+0000 7fa22a7d0640 10 monclient(hunting): authenticate will time out at 1374.867968s 2024-01-20T11:15:31.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.694+0000 7f71b2ffd640 1 -- 172.21.15.110:0/1641043361 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f71b80329b0 con 0x7f71c40a96f0 2024-01-20T11:15:31.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.694+0000 7f71b2ffd640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f71a406c860 0x7f71a406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.694+0000 7f71c1d9b640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f71a406c860 0x7f71a406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.695+0000 7f71c1d9b640 10 monclient: get_auth_request con 0x7f71a406c860 auth_method 0 2024-01-20T11:15:31.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.695+0000 7feb927fc640 1 -- 172.21.15.110:0/501189539 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7feb94054ec0 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.695+0000 7feb927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.695+0000 7feb927fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:31.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.695+0000 7f71b2ffd640 10 client.4381.objecter ms_handle_connect 0x7f71a406c860 2024-01-20T11:15:31.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.695+0000 7f71c1d9b640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f71a406c860 0x7f71a406ed20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f71c40b7790 tx=0x7f71b4006ba0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7fa223fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7f71b2ffd640 1 -- 172.21.15.110:0/1641043361 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f71b807aab0 con 0x7f71c40a96f0 2024-01-20T11:15:31.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7f71b2ffd640 10 client.4381.objecter ms_dispatch 0x7f71c4000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7f71b2ffd640 3 client.4381.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7f71b2ffd640 3 client.4381.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7f71b2ffd640 20 client.4381.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7fa223fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33786/0 (socket says 172.21.15.110:33786) 2024-01-20T11:15:31.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7fa223fff640 1 -- 172.21.15.110:0/2979393457 learned_addr learned my addr 172.21.15.110:0/2979393457 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7fa223fff640 10 monclient(hunting): get_auth_request con 0x7fa2240a96f0 auth_method 0 2024-01-20T11:15:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7fa223fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7fa223fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.700+0000 7fa223fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.701+0000 7fa223fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.701+0000 7fa223fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.701+0000 7fa223fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.703+0000 7fa223fff640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-01-20T11:15:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.703+0000 7fa223fff640 1 -- 172.21.15.110:0/2979393457 >> v2:172.21.15.110:3300/0 conn(0x7fa2240a9ad0 msgr2=0x7fa224142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.703+0000 7fa223fff640 1 --2- 172.21.15.110:0/2979393457 >> v2:172.21.15.110:3300/0 conn(0x7fa2240a9ad0 0x7fa224142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.703+0000 7fa223fff640 1 -- 172.21.15.110:0/2979393457 >> v1:172.21.15.110:6789/0 conn(0x7fa22413fa10 legacy=0x7fa2241423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.703+0000 7fa223fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.703+0000 7fa223fff640 1 monclient: found mon.noname-b 2024-01-20T11:15:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.703+0000 7fa223fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.703+0000 7fa223fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.703+0000 7fa223fff640 1 -- 172.21.15.110:0/2979393457 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240b3fb0 con 0x7fa2240a96f0 2024-01-20T11:15:31.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.705+0000 7fa223fff640 1 --2- 172.21.15.110:0/2979393457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413f4d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa214009870 tx=0x7fa21402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f12ef9e4c17834ba server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 1 -- 172.21.15.110:0/2979393457 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2140317c0 con 0x7fa2240a96f0 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 10 monclient: dump: 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 1 -- 172.21.15.110:0/2979393457 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa214031920 con 0x7fa2240a96f0 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa22a7d0640 5 monclient: authenticate success, global_id 4262 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa22a7d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa22a7d0640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 1 -- 172.21.15.110:0/2979393457 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214031cb0 con 0x7fa2240a96f0 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa22a7d0640 4 set_mon_vals no callback set 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa222ffd640 10 monclient: dump: 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.712 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa22a7d0640 10 monclient: shutdown 2024-01-20T11:15:31.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa22a7d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa22a7d0640 1 -- 172.21.15.110:0/2979393457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 msgr2=0x7fa22413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.710+0000 7fa22a7d0640 1 --2- 172.21.15.110:0/2979393457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413f4d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa214009870 tx=0x7fa21402b860 comp rx=0 tx=0).stop 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.711+0000 7fa22a7d0640 1 -- 172.21.15.110:0/2979393457 shutdown_connections 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.711+0000 7fa22a7d0640 1 --2- 172.21.15.110:0/2979393457 >> v2:172.21.15.110:3300/0 conn(0x7fa2240a9ad0 0x7fa224142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.711+0000 7fa22a7d0640 1 --2- 172.21.15.110:0/2979393457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413f4d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.711+0000 7fa22a7d0640 1 -- 172.21.15.110:0/2979393457 >> 172.21.15.110:0/2979393457 conn(0x7fa2240b7bc0 msgr2=0x7fa2240b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.711+0000 7fa22a7d0640 1 -- 172.21.15.110:0/2979393457 shutdown_connections 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.711+0000 7fa22a7d0640 1 -- 172.21.15.110:0/2979393457 wait complete. 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.711+0000 7f71b2ffd640 1 -- 172.21.15.110:0/1641043361 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f71b80766f0 con 0x7f71c40a96f0 2024-01-20T11:15:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.711+0000 7f71b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.711+0000 7f71b2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:31.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.712+0000 7fa22a7d0640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.712+0000 7fa22a7d0640 10 monclient: monmap: 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.712+0000 7fa22a7d0640 1 librados: starting msgr at 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.712+0000 7fa22a7d0640 1 librados: starting objecter 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.712+0000 7fa22a7d0640 1 Processor -- start 2024-01-20T11:15:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.712+0000 7fa22a7d0640 1 -- start start 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.712+0000 7fa22a7d0640 1 librados: setting wanted keys 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.712+0000 7fa22a7d0640 1 librados: calling monclient init 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.712+0000 7fa22a7d0640 10 monclient: init 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient(hunting): picked mon.noname-c con 0x7fa2240a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2240a9ad0 0x7fa2241dae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa2240a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa22413ada0 0x7fa2241db350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient(hunting): picked mon.noname-a con 0x7fa22413ada0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2240ac3b0 con 0x7fa22413ada0 2024-01-20T11:15:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241da8f0 con 0x7fa2240a9ad0 2024-01-20T11:15:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa224137ff0 con 0x7fa2240a96f0 2024-01-20T11:15:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.713+0000 7fa22a7d0640 10 monclient(hunting): authenticate will time out at 1375.063934s 2024-01-20T11:15:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7fa223fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7fdbc97c1640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdbc413fa10 0x7fdbc41423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7fa223fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33792/0 (socket says 172.21.15.110:33792) 2024-01-20T11:15:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7fa223fff640 1 -- 172.21.15.110:0/4198319818 learned_addr learned my addr 172.21.15.110:0/4198319818 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7fa223fff640 10 monclient(hunting): get_auth_request con 0x7fa2240a96f0 auth_method 0 2024-01-20T11:15:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7fa223fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7fa223fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7fa223fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7f792dffb640 1 -- 172.21.15.110:0/1172510983 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934004220 con 0x7f79380a9ad0 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7f792dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7f792dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7f792dffb640 10 monclient: dump: 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.722 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.722 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7f792dffb640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7f792dffb640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7f792dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.714+0000 7f793f31d640 5 monclient: authenticate success, global_id 4378 2024-01-20T11:15:31.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f793f31d640 10 monclient: _renew_subs 2024-01-20T11:15:31.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f793f31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7938070320 con 0x7f79380a9ad0 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7fa223fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7fa223fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7fa223fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f793f31d640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f793f31d640 10 monclient: _renew_subs 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f793f31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7938070a90 con 0x7f79380a9ad0 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f793f31d640 1 librados: init done 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f792dffb640 1 -- 172.21.15.110:0/1172510983 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7934004380 con 0x7f79380a9ad0 2024-01-20T11:15:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f792dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f792dffb640 1 -- 172.21.15.110:0/1172510983 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934007a70 con 0x7f79380a9ad0 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f792dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f792dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f792dffb640 10 monclient: dump: 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.725 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7fdbc97c1640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdbc413fa10 0x7fdbc41423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58180/0 (socket says 172.21.15.110:58180) 2024-01-20T11:15:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7fdbc97c1640 1 -- 172.21.15.110:0/17443598 learned_addr learned my addr 172.21.15.110:0/17443598 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7fdbc97c1640 10 monclient(hunting): get_auth_request con 0x7fdbc413fa10 auth_method 0 2024-01-20T11:15:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7fdbc97c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7fdbc97c1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7fdbc97c1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.715+0000 7f792f7fe640 4 set_mon_vals no callback set 2024-01-20T11:15:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.716+0000 7fdbc97c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.716+0000 7fdbc97c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.716+0000 7f793f31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.716+0000 7fdbc97c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.716+0000 7f793f31d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.716+0000 7f793f31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.716+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7918005080 con 0x7f79380a9ad0 2024-01-20T11:15:31.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.718+0000 7fa223fff640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-01-20T11:15:31.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.718+0000 7fa223fff640 1 -- 172.21.15.110:0/4198319818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2240a9ad0 msgr2=0x7fa2241dae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.718+0000 7fa223fff640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2240a9ad0 0x7fa2241dae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.719+0000 7fa223fff640 1 -- 172.21.15.110:0/4198319818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa22413ada0 msgr2=0x7fa2241db350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.719+0000 7fa223fff640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa22413ada0 0x7fa2241db350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.719+0000 7fa223fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.719+0000 7fa223fff640 1 monclient: found mon.noname-c 2024-01-20T11:15:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.719+0000 7fa223fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.719+0000 7fa223fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.719+0000 7fa223fff640 1 -- 172.21.15.110:0/4198319818 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140094d0 con 0x7fa2240a96f0 2024-01-20T11:15:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.722+0000 7fdbc9fc2640 1 --2- 172.21.15.110:0/17443598 >> v2:172.21.15.110:3300/0 conn(0x7fdbc40a96f0 0x7fdbc413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.722+0000 7fdbc9fc2640 10 monclient(hunting): get_auth_request con 0x7fdbc40a96f0 auth_method 0 2024-01-20T11:15:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.722+0000 7fdbc9fc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.722+0000 7fdbc9fc2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.722+0000 7fdbc9fc2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.722+0000 7fa223fff640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413b560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa214000c00 tx=0x7fa2140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.722+0000 7fa220ff9640 10 client.?.objecter ms_handle_connect 0x7fa2240a96f0 2024-01-20T11:15:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.722+0000 7fa220ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.723+0000 7fdbc9fc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.723+0000 7fdbc9fc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.723+0000 7fdbc9fc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.723+0000 7fa6c2d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa6c40b4d10 0x7fa6c40b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.724+0000 7fa6c2d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa6c40b4d10 0x7fa6c40b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48584/0 (socket says 172.21.15.110:48584) 2024-01-20T11:15:31.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.724+0000 7fa6c2d9d640 1 -- 172.21.15.110:0/528688684 learned_addr learned my addr 172.21.15.110:0/528688684 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.724+0000 7fa6c2d9d640 10 monclient(hunting): get_auth_request con 0x7fa6c40b4d10 auth_method 0 2024-01-20T11:15:31.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.724+0000 7fa6c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.724+0000 7fa6c2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.724+0000 7fa6c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fdbc9fc2640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-01-20T11:15:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fdbc9fc2640 1 -- 172.21.15.110:0/17443598 >> v2:172.21.15.134:3300/0 conn(0x7fdbc413fa10 msgr2=0x7fdbc41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fdbc9fc2640 1 --2- 172.21.15.110:0/17443598 >> v2:172.21.15.134:3300/0 conn(0x7fdbc413fa10 0x7fdbc41423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fdbc9fc2640 1 -- 172.21.15.110:0/17443598 >> v1:172.21.15.134:6789/0 conn(0x7fdbc40a9ad0 legacy=0x7fdbc4142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fdbc9fc2640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fdbc9fc2640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fdbc9fc2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fdbc9fc2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fdbc9fc2640 1 -- 172.21.15.110:0/17443598 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40b3fb0 con 0x7fdbc40a96f0 2024-01-20T11:15:31.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa220ff9640 1 -- 172.21.15.110:0/4198319818 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214032860 con 0x7fa2240a96f0 2024-01-20T11:15:31.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa220ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa220ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:31.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa220ff9640 10 monclient: dump: 2024-01-20T11:15:31.735 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.735 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.735 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.735 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.735 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa220ff9640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa220ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa220ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa22a7d0640 5 monclient: authenticate success, global_id 4265 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa22a7d0640 10 monclient: _renew_subs 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa22a7d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2241381b0 con 0x7fa2240a96f0 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa22a7d0640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa22a7d0640 10 monclient: _renew_subs 2024-01-20T11:15:31.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa22a7d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22413aa50 con 0x7fa2240a96f0 2024-01-20T11:15:31.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.726+0000 7fa22a7d0640 1 librados: init done 2024-01-20T11:15:31.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.727+0000 7fa6c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.727+0000 7fa6c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.727+0000 7fa6c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa22a7d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa22a7d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa22a7d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa224075490 con 0x7fa2240a96f0 2024-01-20T11:15:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa220ff9640 1 -- 172.21.15.110:0/4198319818 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2140329c0 con 0x7fa2240a96f0 2024-01-20T11:15:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa220ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa220ff9640 1 -- 172.21.15.110:0/4198319818 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214032d30 con 0x7fa2240a96f0 2024-01-20T11:15:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa220ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.732+0000 7fa6c2d9d640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-01-20T11:15:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.732+0000 7fa6c2d9d640 1 -- 172.21.15.110:0/528688684 >> v2:172.21.15.134:3300/0 conn(0x7fa6c4145980 msgr2=0x7fa6c4145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.732+0000 7fa6c2d9d640 1 --2- 172.21.15.110:0/528688684 >> v2:172.21.15.134:3300/0 conn(0x7fa6c4145980 0x7fa6c4145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.732+0000 7fa6c2d9d640 1 -- 172.21.15.110:0/528688684 >> v1:172.21.15.134:6789/0 conn(0x7fa6c40b17c0 legacy=0x7fa6c40b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.732+0000 7fa6c2d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.732+0000 7fa6c2d9d640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.732+0000 7fa6c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.732+0000 7fa6c2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.732+0000 7fa6c2d9d640 1 -- 172.21.15.110:0/528688684 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4145460 con 0x7fa6c40b4d10 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa2227fc640 4 set_mon_vals no callback set 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa220ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.731+0000 7fa220ff9640 10 monclient: dump: 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.741 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.741 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.741 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.737+0000 7fdbc9fc2640 1 --2- 172.21.15.110:0/17443598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a96f0 0x7fdbc413f4d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdbc406d0f0 tx=0x7fdbc002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fad9de98022aca server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.756+0000 7fd7f6ffd640 1 -- 172.21.15.110:0/3488353251 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd7e804e450 con 0x7fd800066790 2024-01-20T11:15:31.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.756+0000 7fd7f6ffd640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7e006c8e0 0x7fd7e006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.757+0000 7fd7fe59c640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7e006c8e0 0x7fd7e006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.757+0000 7fd7fe59c640 10 monclient: get_auth_request con 0x7fd7e006c8e0 auth_method 0 2024-01-20T11:15:31.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.757+0000 7fd7f6ffd640 10 client.4341.objecter ms_handle_connect 0x7fd7e006c8e0 2024-01-20T11:15:31.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.757+0000 7fd7fe59c640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7e006c8e0 0x7fd7e006eda0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00018b0 tx=0x7fd7f0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.772+0000 7fd7f6ffd640 1 -- 172.21.15.110:0/3488353251 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd7e807af10 con 0x7fd800066790 2024-01-20T11:15:31.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.772+0000 7fd7f6ffd640 10 client.4341.objecter ms_dispatch 0x7fd800000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:31.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.772+0000 7fd7f6ffd640 3 client.4341.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:31.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.772+0000 7fd7f6ffd640 3 client.4341.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:31.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.772+0000 7fd7f6ffd640 20 client.4341.objecter dump_active .. 0 homeless 2024-01-20T11:15:31.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.772+0000 7fd7f6ffd640 1 -- 172.21.15.110:0/3488353251 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd7e8076b50 con 0x7fd800066790 2024-01-20T11:15:31.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.772+0000 7fd7f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.772+0000 7fd7f6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:31.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.775+0000 7fa6c2d9d640 1 --2- 172.21.15.110:0/528688684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 0x7fa6c40b3da0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40aa1c0 tx=0x7fa6b402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b50dd5806dadf7 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 1 -- 172.21.15.110:0/17443598 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0031710 con 0x7fdbc40a96f0 2024-01-20T11:15:31.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 10 monclient: dump: 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 1 -- 172.21.15.110:0/17443598 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc002f050 con 0x7fdbc40a96f0 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 1 -- 172.21.15.110:0/17443598 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0031a90 con 0x7fdbc40a96f0 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbc8fc0640 10 monclient: dump: 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.782 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.783 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbcc226640 5 monclient: authenticate success, global_id 4344 2024-01-20T11:15:31.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbcc226640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.780+0000 7fdbcc226640 4 set_mon_vals no callback set 2024-01-20T11:15:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 10 monclient: shutdown 2024-01-20T11:15:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 1 -- 172.21.15.110:0/17443598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a96f0 msgr2=0x7fdbc413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 1 --2- 172.21.15.110:0/17443598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a96f0 0x7fdbc413f4d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdbc406d0f0 tx=0x7fdbc002b7b0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 1 -- 172.21.15.110:0/17443598 shutdown_connections 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 1 --2- 172.21.15.110:0/17443598 >> v2:172.21.15.134:3300/0 conn(0x7fdbc413fa10 0x7fdbc41423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 1 --2- 172.21.15.110:0/17443598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a96f0 0x7fdbc413f4d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 1 -- 172.21.15.110:0/17443598 >> 172.21.15.110:0/17443598 conn(0x7fdbc40b7bc0 msgr2=0x7fdbc40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 1 -- 172.21.15.110:0/17443598 shutdown_connections 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.781+0000 7fdbcc226640 1 -- 172.21.15.110:0/17443598 wait complete. 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.782+0000 7fdbcc226640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.782+0000 7fdbcc226640 10 monclient: monmap: 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.783+0000 7fdbcc226640 1 librados: starting msgr at 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.783+0000 7fdbcc226640 1 librados: starting objecter 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.783+0000 7fdbcc226640 1 Processor -- start 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.783+0000 7fdbcc226640 1 -- start start 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.783+0000 7fdbcc226640 1 librados: setting wanted keys 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.783+0000 7fdbcc226640 1 librados: calling monclient init 2024-01-20T11:15:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.783+0000 7fdbcc226640 10 monclient: init 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.784+0000 7fdbcc226640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.784+0000 7fdbcc226640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.784+0000 7fdbcc226640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbc40a96f0 0x7fdbc413b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.784+0000 7fdbcc226640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc40a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.784+0000 7fdbcc226640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a9ad0 0x7fdbc41384c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.784+0000 7fdbcc226640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc40a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.784+0000 7fdbcc226640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc40ab550 0x7fdbc4138a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.785+0000 7fdbcc226640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc40ab550 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.785+0000 7fdbcc226640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.785+0000 7fdbcc226640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40abef0 con 0x7fdbc40a9ad0 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.785+0000 7fdbcc226640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.785+0000 7fdbcc226640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40ac1d0 con 0x7fdbc40a96f0 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.785+0000 7fdbcc226640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.785+0000 7fdbcc226640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40ac4b0 con 0x7fdbc40ab550 2024-01-20T11:15:31.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.785+0000 7fdbcc226640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.785+0000 7fdbcc226640 10 monclient(hunting): authenticate will time out at 1375.135760s 2024-01-20T11:15:31.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.817+0000 7fa220ff9640 1 -- 172.21.15.110:0/4198319818 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa21404e450 con 0x7fa2240a96f0 2024-01-20T11:15:31.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.818+0000 7fa220ff9640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa21006c860 0x7fa21006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.818+0000 7fa2237fe640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa21006c860 0x7fa21006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.818+0000 7fa2237fe640 10 monclient: get_auth_request con 0x7fa21006c860 auth_method 0 2024-01-20T11:15:31.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.819+0000 7fa220ff9640 10 client.4265.objecter ms_handle_connect 0x7fa21006c860 2024-01-20T11:15:31.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.819+0000 7fa2237fe640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa21006c860 0x7fa21006ed20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa218006fd0 tx=0x7fa218006bc0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.825+0000 7fdbc97c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a9ad0 0x7fdbc41384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.826+0000 7fdbc97c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a9ad0 0x7fdbc41384c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48604/0 (socket says 172.21.15.110:48604) 2024-01-20T11:15:31.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.826+0000 7fdbc97c1640 1 -- 172.21.15.110:0/1426275856 learned_addr learned my addr 172.21.15.110:0/1426275856 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.826+0000 7fdbc97c1640 10 monclient(hunting): get_auth_request con 0x7fdbc40a9ad0 auth_method 0 2024-01-20T11:15:31.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.826+0000 7fdbc97c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.826+0000 7fdbc97c1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.826+0000 7fdbc97c1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.828+0000 7fdbc97c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:31.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.828+0000 7fdbc97c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:31.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.828+0000 7fdbc97c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:31.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-01-20T11:15:31.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 1 -- 172.21.15.110:0/1426275856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc40ab550 msgr2=0x7fdbc4138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc40ab550 0x7fdbc4138a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 1 -- 172.21.15.110:0/1426275856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbc40a96f0 msgr2=0x7fdbc413b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbc40a96f0 0x7fdbc413b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 10 monclient: _finish_hunting 0 2024-01-20T11:15:31.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 1 monclient: found mon.noname-a 2024-01-20T11:15:31.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:31.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.831+0000 7fdbc97c1640 1 -- 172.21.15.110:0/1426275856 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc0009400 con 0x7fdbc40a9ad0 2024-01-20T11:15:31.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 1 -- 172.21.15.110:0/528688684 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4031750 con 0x7fa6c40b4d10 2024-01-20T11:15:31.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 10 monclient: dump: 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 1 -- 172.21.15.110:0/528688684 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b40318b0 con 0x7fa6c40b4d10 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 1 -- 172.21.15.110:0/528688684 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4031c40 con 0x7fa6c40b4d10 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c91cf640 5 monclient: authenticate success, global_id 4347 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c91cf640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c91cf640 4 set_mon_vals no callback set 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.833+0000 7fa6c1d9b640 10 monclient: dump: 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.835 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.836 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.836 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.836 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.834+0000 7fa6c91cf640 10 monclient: shutdown 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.834+0000 7fa6c91cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.834+0000 7fa6c91cf640 1 -- 172.21.15.110:0/528688684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 msgr2=0x7fa6c40b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.834+0000 7fa6c91cf640 1 --2- 172.21.15.110:0/528688684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 0x7fa6c40b3da0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40aa1c0 tx=0x7fa6b402b7f0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.834+0000 7fa6c91cf640 1 -- 172.21.15.110:0/528688684 shutdown_connections 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.834+0000 7fa6c91cf640 1 --2- 172.21.15.110:0/528688684 >> v2:172.21.15.134:3300/0 conn(0x7fa6c4145980 0x7fa6c4145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.834+0000 7fa6c91cf640 1 --2- 172.21.15.110:0/528688684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 0x7fa6c40b3da0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.834+0000 7fa6c91cf640 1 -- 172.21.15.110:0/528688684 >> 172.21.15.110:0/528688684 conn(0x7fa6c4066590 msgr2=0x7fa6c40669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.835+0000 7fa6c91cf640 1 -- 172.21.15.110:0/528688684 shutdown_connections 2024-01-20T11:15:31.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.835+0000 7fa6c91cf640 1 -- 172.21.15.110:0/528688684 wait complete. 2024-01-20T11:15:31.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.836+0000 7fdbc97c1640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a9ad0 0x7fdbc41384c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400e870 tx=0x7fdbb400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.836+0000 7fa6c91cf640 10 monclient: build_initial_monmap 2024-01-20T11:15:31.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.836+0000 7fdbba7fc640 10 client.?.objecter ms_handle_connect 0x7fdbc40a9ad0 2024-01-20T11:15:31.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.836+0000 7fdbba7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:31.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.836+0000 7fa6c91cf640 10 monclient: monmap: 2024-01-20T11:15:31.840 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:31.840 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:31.840 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:31.840 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:31.840 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:31.840 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.836+0000 7fa6c91cf640 1 librados: starting msgr at 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.836+0000 7fa6c91cf640 1 librados: starting objecter 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.837+0000 7fa6c91cf640 1 Processor -- start 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.837+0000 7fa6c91cf640 1 -- start start 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.837+0000 7fa6c91cf640 1 librados: setting wanted keys 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.837+0000 7fa6c91cf640 1 librados: calling monclient init 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.837+0000 7fa6c91cf640 10 monclient: init 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:15:31.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6c40b17c0 0x7fa6c4142650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c40b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 0x7fa6c413fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c40b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c4145980 0x7fa6c4140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa6c4145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c4140780 con 0x7fa6c40b4d10 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c4141be0 con 0x7fa6c4145980 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c4141ec0 con 0x7fa6c40b17c0 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:31.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.838+0000 7fa6c91cf640 10 monclient(hunting): authenticate will time out at 1375.188514s 2024-01-20T11:15:31.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 1 -- 172.21.15.110:0/1426275856 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb400ee60 con 0x7fdbc40a9ad0 2024-01-20T11:15:31.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:31.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 10 monclient: dump: 2024-01-20T11:15:31.843 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.843 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 10 monclient: _finish_auth 0 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 1 -- 172.21.15.110:0/1426275856 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb400c940 con 0x7fdbc40a9ad0 2024-01-20T11:15:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 1 -- 172.21.15.110:0/1426275856 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb400ccb0 con 0x7fdbc40a9ad0 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.842+0000 7fdbba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbba7fc640 10 monclient: dump: 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:31.845 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbcc226640 5 monclient: authenticate success, global_id 4350 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbcc226640 10 monclient: _renew_subs 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbbbfff640 4 set_mon_vals no callback set 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbcc226640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc413b910 con 0x7fdbc40a9ad0 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbcc226640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbcc226640 10 monclient: _renew_subs 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbcc226640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc41dacf0 con 0x7fdbc40a9ad0 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fdbcc226640 1 librados: init done 2024-01-20T11:15:31.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.843+0000 7fa6c259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 0x7fa6c413fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fdbcc226640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fdbcc226640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:31.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fdbcc226640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc413c010 con 0x7fdbc40a9ad0 2024-01-20T11:15:31.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fa6c259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 0x7fa6c413fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48618/0 (socket says 172.21.15.110:48618) 2024-01-20T11:15:31.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fa6c259c640 1 -- 172.21.15.110:0/2506143461 learned_addr learned my addr 172.21.15.110:0/2506143461 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:31.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fa6c259c640 10 monclient(hunting): get_auth_request con 0x7fa6c40b4d10 auth_method 0 2024-01-20T11:15:31.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fa6c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:31.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fa6c259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:31.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.844+0000 7fa6c259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:31.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.912+0000 7f71c8d08640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-20T11:15:31.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.912+0000 7f71c8d08640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-20T11:15:31.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.912+0000 7f71c8d08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:31.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.912+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f71c406d9f0 con 0x7f71c40a96f0 2024-01-20T11:15:31.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.963+0000 7fd804bb6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-20T11:15:31.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.963+0000 7fd804bb6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-20T11:15:31.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.963+0000 7fd804bb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:31.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.963+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd8001345a0 con 0x7fd800066790 2024-01-20T11:15:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.982+0000 7feba412c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-20T11:15:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.982+0000 7feba412c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-20T11:15:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.983+0000 7feba412c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.983+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7feb9c075490 con 0x7feb9c1d7eb0 2024-01-20T11:15:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.998+0000 7f792dffb640 1 -- 172.21.15.110:0/1172510983 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7934007bd0 con 0x7f79380a9ad0 2024-01-20T11:15:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.998+0000 7f792dffb640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f791c06c890 0x7f791c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.998+0000 7f793d0b9640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f791c06c890 0x7f791c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.999+0000 7f793d0b9640 10 monclient: get_auth_request con 0x7f791c06c890 auth_method 0 2024-01-20T11:15:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.999+0000 7f793d0b9640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f791c06c890 0x7f791c06ed50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7928007fd0 tx=0x7f7928009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:31.999+0000 7f792dffb640 10 client.4378.objecter ms_handle_connect 0x7f791c06c890 2024-01-20T11:15:32.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.016+0000 7f792dffb640 1 -- 172.21.15.110:0/1172510983 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f79340584e0 con 0x7f79380a9ad0 2024-01-20T11:15:32.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.016+0000 7f792dffb640 10 client.4378.objecter ms_dispatch 0x7f7938000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:32.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.016+0000 7f792dffb640 3 client.4378.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:32.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.016+0000 7f792dffb640 3 client.4378.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:32.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.016+0000 7f792dffb640 20 client.4378.objecter dump_active .. 0 homeless 2024-01-20T11:15:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.027+0000 7f792dffb640 1 -- 172.21.15.110:0/1172510983 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7934054820 con 0x7f79380a9ad0 2024-01-20T11:15:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.027+0000 7f792dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.027+0000 7f792dffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.028+0000 7fa6c359e640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c4145980 0x7fa6c4140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.028+0000 7fa6c359e640 10 monclient(hunting): get_auth_request con 0x7fa6c4145980 auth_method 0 2024-01-20T11:15:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.028+0000 7fa6c359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.028+0000 7fa6c359e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.028+0000 7fa6c359e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.031+0000 7fa6c359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.031+0000 7fa6c359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.031+0000 7fa6c359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.034+0000 7fa6c359e640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.035+0000 7fa6c359e640 1 -- 172.21.15.110:0/2506143461 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6c40b17c0 msgr2=0x7fa6c4142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.035+0000 7fa6c359e640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6c40b17c0 0x7fa6c4142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.035+0000 7fa6c359e640 1 -- 172.21.15.110:0/2506143461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 msgr2=0x7fa6c413fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.035+0000 7fa6c359e640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 0x7fa6c413fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.035+0000 7fa6c359e640 10 monclient: _finish_hunting 0 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.035+0000 7fa6c359e640 1 monclient: found mon.noname-b 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.035+0000 7fa6c359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.035+0000 7fa6c359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.035+0000 7fa6c359e640 1 -- 172.21.15.110:0/2506143461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b40094d0 con 0x7fa6c4145980 2024-01-20T11:15:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.037+0000 7fa6c359e640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c4145980 0x7fa6c4140240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00be20 tx=0x7fa6bc00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.037+0000 7fa6ab7fe640 10 client.?.objecter ms_handle_connect 0x7fa6c4145980 2024-01-20T11:15:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.037+0000 7fa6ab7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.039+0000 7f71b2ffd640 1 -- 172.21.15.110:0/1641043361 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f71b8079d10 con 0x7f71c40a96f0 2024-01-20T11:15:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.039+0000 7f71b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-20T11:15:32.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.039+0000 7f71b2ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:32.040 INFO:teuthology.orchestra.run.smithi110.stdout:90194313222 2024-01-20T11:15:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 1 -- 172.21.15.110:0/2506143461 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00cdf0 con 0x7fa6c4145980 2024-01-20T11:15:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 10 monclient: dump: 2024-01-20T11:15:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 1 -- 172.21.15.110:0/2506143461 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc014440 con 0x7fa6c4145980 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c91cf640 5 monclient: authenticate success, global_id 4390 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c91cf640 10 monclient: _renew_subs 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c91cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c4142f50 con 0x7fa6c4145980 2024-01-20T11:15:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 10 librados: watch_flush enter 2024-01-20T11:15:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 10 librados: watch_flush exit 2024-01-20T11:15:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 20 client.4381.objecter shutdown clearing up homeless session... 2024-01-20T11:15:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 10 client.4381.objecter successfully canceled tick 2024-01-20T11:15:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 1 -- 172.21.15.110:0/2506143461 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0147f0 con 0x7fa6c4145980 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6ab7fe640 10 monclient: dump: 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:32.046 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c91cf640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c91cf640 10 monclient: _renew_subs 2024-01-20T11:15:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c91cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c413f290 con 0x7fa6c4145980 2024-01-20T11:15:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c91cf640 1 librados: init done 2024-01-20T11:15:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7fa6c159a640 4 set_mon_vals no callback set 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f71a406c860 msgr2=0x7f71a406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f71a406c860 0x7f71a406ed20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f71c40b7790 tx=0x7f71b4006ba0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 10 monclient: shutdown 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 msgr2=0x7f71c4070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 0x7f71c4070d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f71b8000c00 tx=0x7f71b80046c0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 shutdown_connections 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f71a406c860 0x7f71a406ed20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.042+0000 7f71c8d08640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71c406e1f0 0x7f71c41de740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7f71c8d08640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f71c40a9ad0 0x7f71c406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7f71c8d08640 1 --2- 172.21.15.110:0/1641043361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71c40a96f0 0x7f71c4070d00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 >> 172.21.15.110:0/1641043361 conn(0x7f71c40b7bc0 msgr2=0x7f71c40b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:32.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 shutdown_connections 2024-01-20T11:15:32.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7f71c8d08640 1 -- 172.21.15.110:0/1641043361 wait complete. 2024-01-20T11:15:32.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7f71c8d08640 1 librados: shutdown 2024-01-20T11:15:32.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7fa6c91cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:32.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7fa6c91cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:32.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7fa6c91cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:32.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.043+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c4075490 con 0x7fa6c4145980 2024-01-20T11:15:32.057 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313223 got 90194313222 for osd.7 2024-01-20T11:15:32.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.062+0000 7fa6ab7fe640 1 -- 172.21.15.110:0/2506143461 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa6bc01d3b0 con 0x7fa6c4145980 2024-01-20T11:15:32.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.062+0000 7fa6ab7fe640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa6b006c860 0x7fa6b006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:32.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.062+0000 7fa6c2d9d640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa6b006c860 0x7fa6b006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:32.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.062+0000 7fa6c2d9d640 10 monclient: get_auth_request con 0x7fa6b006c860 auth_method 0 2024-01-20T11:15:32.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.063+0000 7fa6c2d9d640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa6b006c860 0x7fa6b006ed20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4035000 tx=0x7fa6b4030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:32.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.063+0000 7fa6ab7fe640 10 client.4390.objecter ms_handle_connect 0x7fa6b006c860 2024-01-20T11:15:32.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.065+0000 7fa6ab7fe640 1 -- 172.21.15.110:0/2506143461 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa6bc058cd0 con 0x7fa6c4145980 2024-01-20T11:15:32.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.065+0000 7fa6ab7fe640 10 client.4390.objecter ms_dispatch 0x7fa6c4000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:32.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.065+0000 7fa6ab7fe640 3 client.4390.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:32.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.065+0000 7fa6ab7fe640 3 client.4390.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:32.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.065+0000 7fa6ab7fe640 20 client.4390.objecter dump_active .. 0 homeless 2024-01-20T11:15:32.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.093+0000 7fa220ff9640 1 -- 172.21.15.110:0/4198319818 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa214039310 con 0x7fa2240a96f0 2024-01-20T11:15:32.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.093+0000 7fa220ff9640 10 client.4265.objecter ms_dispatch 0x7fa224000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:32.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.093+0000 7fa220ff9640 3 client.4265.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:32.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.093+0000 7fa220ff9640 3 client.4265.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:32.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.093+0000 7fa220ff9640 20 client.4265.objecter dump_active .. 0 homeless 2024-01-20T11:15:32.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.093+0000 7fa220ff9640 1 -- 172.21.15.110:0/4198319818 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa214076f50 con 0x7fa2240a96f0 2024-01-20T11:15:32.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.093+0000 7fa220ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:32.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.093+0000 7fa220ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:32.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.111+0000 7feb927fc640 1 -- 172.21.15.110:0/501189539 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb940584e0 con 0x7feb9c1d7eb0 2024-01-20T11:15:32.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.111+0000 7feb927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-20T11:15:32.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.111+0000 7feb927fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:32.112 INFO:teuthology.orchestra.run.smithi110.stdout:73014444037 2024-01-20T11:15:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 10 librados: watch_flush enter 2024-01-20T11:15:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 10 librados: watch_flush exit 2024-01-20T11:15:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 20 client.4259.objecter shutdown clearing up homeless session... 2024-01-20T11:15:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 10 client.4259.objecter successfully canceled tick 2024-01-20T11:15:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feb8006c860 msgr2=0x7feb8006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feb8006c860 0x7feb8006ed20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb98036000 tx=0x7feb98030000 comp rx=0 tx=0).stop 2024-01-20T11:15:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 10 monclient: shutdown 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb9c1d7eb0 msgr2=0x7feb9c13a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb9c1d7eb0 0x7feb9c13a240 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb9400c730 tx=0x7feb9400cbf0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 shutdown_connections 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feb8006c860 0x7feb8006ed20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb9c1d7eb0 0x7feb9c13a240 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb9c0a9ad0 0x7feb9c139d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 --2- 172.21.15.110:0/501189539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb9c0a96f0 0x7feb9c1d7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.113+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 >> 172.21.15.110:0/501189539 conn(0x7feb9c0b7bc0 msgr2=0x7feb9c13aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.114+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 shutdown_connections 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.114+0000 7feba412c640 1 -- 172.21.15.110:0/501189539 wait complete. 2024-01-20T11:15:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.114+0000 7feba412c640 1 librados: shutdown 2024-01-20T11:15:32.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.115+0000 7fdbba7fc640 1 -- 172.21.15.110:0/1426275856 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdbb4013470 con 0x7fdbc40a9ad0 2024-01-20T11:15:32.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.116+0000 7fdbba7fc640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdba806c860 0x7fdba806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:32.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.116+0000 7fdbc9fc2640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdba806c860 0x7fdba806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:32.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.116+0000 7fdbc9fc2640 10 monclient: get_auth_request con 0x7fdba806c860 auth_method 0 2024-01-20T11:15:32.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.116+0000 7fdbba7fc640 10 client.4350.objecter ms_handle_connect 0x7fdba806c860 2024-01-20T11:15:32.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.116+0000 7fdbc9fc2640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdba806c860 0x7fdba806ed20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0035000 tx=0x7fdbc0030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:32.125 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444038 got 73014444037 for osd.2 2024-01-20T11:15:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.131+0000 7fdbba7fc640 1 -- 172.21.15.110:0/1426275856 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdbb4059150 con 0x7fdbc40a9ad0 2024-01-20T11:15:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.131+0000 7fdbba7fc640 10 client.4350.objecter ms_dispatch 0x7fdbc4000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.131+0000 7fdbba7fc640 3 client.4350.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.131+0000 7fdbba7fc640 3 client.4350.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.131+0000 7fdbba7fc640 20 client.4350.objecter dump_active .. 0 homeless 2024-01-20T11:15:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.131+0000 7fdbba7fc640 1 -- 172.21.15.110:0/1426275856 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fdbb4054d90 con 0x7fdbc40a9ad0 2024-01-20T11:15:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.131+0000 7fdbba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.131+0000 7fdbba7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:32.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.142+0000 7fd7f6ffd640 1 -- 172.21.15.110:0/3488353251 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd8001345a0 con 0x7fd800066790 2024-01-20T11:15:32.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.143+0000 7fd7f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-20T11:15:32.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.143+0000 7fd7f6ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:32.143 INFO:teuthology.orchestra.run.smithi110.stdout:64424509445 2024-01-20T11:15:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 10 librados: watch_flush enter 2024-01-20T11:15:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 10 librados: watch_flush exit 2024-01-20T11:15:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 20 client.4341.objecter shutdown clearing up homeless session... 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 10 client.4341.objecter successfully canceled tick 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7e006c8e0 msgr2=0x7fd7e006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7e006c8e0 0x7fd7e006eda0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00018b0 tx=0x7fd7f0009040 comp rx=0 tx=0).stop 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 10 monclient: shutdown 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 msgr2=0x7fd8000b24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 0x7fd8000b24f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8033000 tx=0x7fd7e80046c0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 shutdown_connections 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7e006c8e0 0x7fd7e006eda0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd80013bb70 0x7fd8000b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd800066790 0x7fd8000b24f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 --2- 172.21.15.110:0/3488353251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8000663b0 0x7fd8001384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 >> 172.21.15.110:0/3488353251 conn(0x7fd8000b3f50 msgr2=0x7fd80006e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 shutdown_connections 2024-01-20T11:15:32.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 -- 172.21.15.110:0/3488353251 wait complete. 2024-01-20T11:15:32.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.145+0000 7fd804bb6640 1 librados: shutdown 2024-01-20T11:15:32.158 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509445 for osd.6 2024-01-20T11:15:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.302+0000 7f793f31d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-20T11:15:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.302+0000 7f793f31d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-20T11:15:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.302+0000 7f793f31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.302+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7918005600 con 0x7f79380a9ad0 2024-01-20T11:15:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.310+0000 7fa6ab7fe640 1 -- 172.21.15.110:0/2506143461 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa6bc054910 con 0x7fa6c4145980 2024-01-20T11:15:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.310+0000 7fa6ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.310+0000 7fa6ab7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.317+0000 7fdbcc226640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-20T11:15:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.317+0000 7fdbcc226640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-20T11:15:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.317+0000 7fdbcc226640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.317+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdbc4075490 con 0x7fdbc40a9ad0 2024-01-20T11:15:32.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.321+0000 7fdbba7fc640 1 -- 172.21.15.110:0/1426275856 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdbb4021090 con 0x7fdbc40a9ad0 2024-01-20T11:15:32.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.322+0000 7fdbba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-20T11:15:32.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.322+0000 7fdbba7fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:32.322 INFO:teuthology.orchestra.run.smithi110.stdout:73014444038 2024-01-20T11:15:32.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 10 librados: watch_flush enter 2024-01-20T11:15:32.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 10 librados: watch_flush exit 2024-01-20T11:15:32.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 20 client.4350.objecter shutdown clearing up homeless session... 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 10 client.4350.objecter successfully canceled tick 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdba806c860 msgr2=0x7fdba806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdba806c860 0x7fdba806ed20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0035000 tx=0x7fdbc0030000 comp rx=0 tx=0).stop 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 10 monclient: shutdown 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a9ad0 msgr2=0x7fdbc41384c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a9ad0 0x7fdbc41384c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400e870 tx=0x7fdbb400ed30 comp rx=0 tx=0).stop 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 shutdown_connections 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdba806c860 0x7fdba806ed20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc40ab550 0x7fdbc4138a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc40a9ad0 0x7fdbc41384c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 --2- 172.21.15.110:0/1426275856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbc40a96f0 0x7fdbc413b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.325+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 >> 172.21.15.110:0/1426275856 conn(0x7fdbc40b7bc0 msgr2=0x7fdbc40b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.326+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 shutdown_connections 2024-01-20T11:15:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.326+0000 7fdbcc226640 1 -- 172.21.15.110:0/1426275856 wait complete. 2024-01-20T11:15:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.326+0000 7fdbcc226640 1 librados: shutdown 2024-01-20T11:15:32.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444039 got 73014444038 for osd.1 2024-01-20T11:15:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.371+0000 7fa22a7d0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-20T11:15:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.371+0000 7fa22a7d0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-20T11:15:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.371+0000 7fa22a7d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.371+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa224069850 con 0x7fa2240a96f0 2024-01-20T11:15:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.371+0000 7f792dffb640 1 -- 172.21.15.110:0/1172510983 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f793401e020 con 0x7f79380a9ad0 2024-01-20T11:15:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.371+0000 7f792dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-20T11:15:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.371+0000 7f792dffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:32.372 INFO:teuthology.orchestra.run.smithi110.stdout:73014444038 2024-01-20T11:15:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 10 librados: watch_flush enter 2024-01-20T11:15:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 10 librados: watch_flush exit 2024-01-20T11:15:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 20 client.4378.objecter shutdown clearing up homeless session... 2024-01-20T11:15:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 10 client.4378.objecter successfully canceled tick 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f791c06c890 msgr2=0x7f791c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f791c06c890 0x7f791c06ed50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7928007fd0 tx=0x7f7928009040 comp rx=0 tx=0).stop 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 10 monclient: shutdown 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79380a9ad0 msgr2=0x7f79381df2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79380a9ad0 0x7f79381df2b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f793400c820 tx=0x7f793400cce0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 shutdown_connections 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f791c06c890 0x7f791c06ed50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7938142920 0x7f79381df7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79380a9ad0 0x7f79381df2b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 --2- 172.21.15.110:0/1172510983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79380a96f0 0x7f79381390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 >> 172.21.15.110:0/1172510983 conn(0x7f79380b7bc0 msgr2=0x7f79380b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 shutdown_connections 2024-01-20T11:15:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 -- 172.21.15.110:0/1172510983 wait complete. 2024-01-20T11:15:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.374+0000 7f793f31d640 1 librados: shutdown 2024-01-20T11:15:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.376+0000 7fa220ff9640 1 -- 172.21.15.110:0/4198319818 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa21407a570 con 0x7fa2240a96f0 2024-01-20T11:15:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.376+0000 7fa220ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-20T11:15:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.376+0000 7fa220ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:32.376 INFO:teuthology.orchestra.run.smithi110.stdout:90194313223 2024-01-20T11:15:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 10 librados: watch_flush enter 2024-01-20T11:15:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 10 librados: watch_flush exit 2024-01-20T11:15:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 20 client.4265.objecter shutdown clearing up homeless session... 2024-01-20T11:15:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 10 client.4265.objecter successfully canceled tick 2024-01-20T11:15:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa21006c860 msgr2=0x7fa21006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa21006c860 0x7fa21006ed20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa218006fd0 tx=0x7fa218006bc0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 10 monclient: shutdown 2024-01-20T11:15:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 msgr2=0x7fa22413b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413b560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa214000c00 tx=0x7fa2140046c0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 shutdown_connections 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa21006c860 0x7fa21006ed20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa22413ada0 0x7fa2241db350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2240a9ad0 0x7fa2241dae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.378+0000 7fa22a7d0640 1 --2- 172.21.15.110:0/4198319818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2240a96f0 0x7fa22413b560 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.379+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 >> 172.21.15.110:0/4198319818 conn(0x7fa2240b7bc0 msgr2=0x7fa22413c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.379+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 shutdown_connections 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.379+0000 7fa22a7d0640 1 -- 172.21.15.110:0/4198319818 wait complete. 2024-01-20T11:15:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.379+0000 7fa22a7d0640 1 librados: shutdown 2024-01-20T11:15:32.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444039 got 73014444038 for osd.0 2024-01-20T11:15:32.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313224 got 90194313223 for osd.3 2024-01-20T11:15:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.502+0000 7fa6c91cf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-20T11:15:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.502+0000 7fa6c91cf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-20T11:15:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.502+0000 7fa6c91cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.502+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa6c413f290 con 0x7fa6c4145980 2024-01-20T11:15:32.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.508+0000 7fa6ab7fe640 1 -- 172.21.15.110:0/2506143461 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa6bc019070 con 0x7fa6c4145980 2024-01-20T11:15:32.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.508+0000 7fa6ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-20T11:15:32.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.508+0000 7fa6ab7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:32.508 INFO:teuthology.orchestra.run.smithi110.stdout:94489280520 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 10 librados: watch_flush enter 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 10 librados: watch_flush exit 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 20 client.4390.objecter shutdown clearing up homeless session... 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 10 client.4390.objecter successfully canceled tick 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa6b006c860 msgr2=0x7fa6b006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa6b006c860 0x7fa6b006ed20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4035000 tx=0x7fa6b4030000 comp rx=0 tx=0).stop 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 10 monclient: shutdown 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c4145980 msgr2=0x7fa6c4140240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c4145980 0x7fa6c4140240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00be20 tx=0x7fa6bc00bf10 comp rx=0 tx=0).stop 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 shutdown_connections 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa6b006c860 0x7fa6b006ed20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c4145980 0x7fa6c4140240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6c40b4d10 0x7fa6c413fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 --2- 172.21.15.110:0/2506143461 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6c40b17c0 0x7fa6c4142650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:32.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 >> 172.21.15.110:0/2506143461 conn(0x7fa6c4066590 msgr2=0x7fa6c4144c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:32.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 shutdown_connections 2024-01-20T11:15:32.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 -- 172.21.15.110:0/2506143461 wait complete. 2024-01-20T11:15:32.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:32.510+0000 7fa6c91cf640 1 librados: shutdown 2024-01-20T11:15:32.521 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280521 got 94489280520 for osd.4 2024-01-20T11:15:33.007 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:24.031 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:33.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-20T11:15:33.062 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:24.086 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:33.075 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:01:24.081 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:33.094 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:24.118 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:33.126 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-20T11:15:33.134 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:01:24.141 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:33.154 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:24.178 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:33.158 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-20T11:15:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.167+0000 7fd8814cf640 10 monclient: get_monmap_and_config 2024-01-20T11:15:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.167+0000 7fd8814cf640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.167+0000 7fd8814cf640 10 monclient: monmap: 2024-01-20T11:15:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:33.167982+0000 2024-01-20T11:15:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:33.167982+0000 2024-01-20T11:15:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:33.168 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.168+0000 7fd8814cf640 1 Processor -- start 2024-01-20T11:15:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.168+0000 7fd8814cf640 1 -- start start 2024-01-20T11:15:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.168+0000 7fd8814cf640 10 monclient: init 2024-01-20T11:15:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-01-20T11:15:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd87c143540 0x7fd87c13d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd87c143540 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87c070c80 0x7fd87c0b0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd87c070c80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd87c143fc0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c071ce0 con 0x7fd87c143fc0 2024-01-20T11:15:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87c06f7b0 con 0x7fd87c143540 2024-01-20T11:15:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c06f8f0 con 0x7fd87c070c80 2024-01-20T11:15:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.169+0000 7fd8814cf640 10 monclient(hunting): authenticate will time out at 1376.519573s 2024-01-20T11:15:33.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.197+0000 7fd87affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd87c143540 0x7fd87c13d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.227+0000 7fd87affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd87c143540 0x7fd87c13d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48632/0 (socket says 172.21.15.110:48632) 2024-01-20T11:15:33.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.227+0000 7fd87affd640 1 -- 172.21.15.110:0/1632321689 learned_addr learned my addr 172.21.15.110:0/1632321689 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.227+0000 7fd87affd640 10 monclient(hunting): get_auth_request con 0x7fd87c143540 auth_method 0 2024-01-20T11:15:33.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.227+0000 7fd87affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.227+0000 7fd87affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.227+0000 7fd87affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.231+0000 7fd87affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.231+0000 7fd87affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.231+0000 7fd87affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.233+0000 7f607bee7640 10 monclient: get_monmap_and_config 2024-01-20T11:15:33.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.233+0000 7f607bee7640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.233+0000 7f607bee7640 10 monclient: monmap: 2024-01-20T11:15:33.233 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.233 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.233 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:33.234085+0000 2024-01-20T11:15:33.233 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:33.234085+0000 2024-01-20T11:15:33.234 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.234 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.234 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:33.234 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:33.234 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:33.234 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:33.234 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:33.234 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.234+0000 7f607bee7640 1 Processor -- start 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.234+0000 7f607bee7640 1 -- start start 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.234+0000 7f607bee7640 10 monclient: init 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.234+0000 7f607bee7640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.234+0000 7f607bee7640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.234+0000 7f607bee7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60740a96f0 0x7f607413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.234+0000 7f607bee7640 10 monclient(hunting): picked mon.noname-b con 0x7f60740a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.234+0000 7f607bee7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f607413fa10 0x7f60741423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.234+0000 7f607bee7640 10 monclient(hunting): picked mon.noname-a con 0x7f607413fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:33.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f60740a9ad0 0x7f6074142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 10 monclient(hunting): picked mon.noname-c con 0x7f60740a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:33.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60740aaea0 con 0x7f607413fa10 2024-01-20T11:15:33.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60740b25c0 con 0x7f60740a9ad0 2024-01-20T11:15:33.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60740ab0a0 con 0x7f60740a96f0 2024-01-20T11:15:33.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.235+0000 7f607bee7640 10 monclient(hunting): authenticate will time out at 1376.585306s 2024-01-20T11:15:33.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.252+0000 7fd87a7fc640 1 --2- 172.21.15.110:0/1632321689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87c070c80 0x7fd87c0b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.257+0000 7fd87a7fc640 10 monclient(hunting): get_auth_request con 0x7fd87c070c80 auth_method 0 2024-01-20T11:15:33.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.257+0000 7fd87a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.257+0000 7fd87a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.257+0000 7fd87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.265+0000 7f16497c0640 10 monclient: get_monmap_and_config 2024-01-20T11:15:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.265+0000 7f16497c0640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.265+0000 7f16497c0640 10 monclient: monmap: 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:33.266264+0000 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:33.266264+0000 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:33.266 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 1 Processor -- start 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 1 -- start start 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient: init 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16440a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16440a9ad0 0x7f16441423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient(hunting): picked mon.noname-b con 0x7f16440a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1644142920 0x7f164414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient(hunting): picked mon.noname-c con 0x7f1644142920 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16440b25c0 con 0x7f16440a96f0 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16440aaea0 con 0x7f1644142920 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164413fbe0 con 0x7f16440a9ad0 2024-01-20T11:15:33.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.266+0000 7f16497c0640 10 monclient(hunting): authenticate will time out at 1376.617069s 2024-01-20T11:15:33.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.277+0000 7fd87affd640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2024-01-20T11:15:33.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.277+0000 7fd87affd640 1 -- 172.21.15.110:0/1632321689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87c070c80 msgr2=0x7fd87c0b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.277+0000 7fd87affd640 1 --2- 172.21.15.110:0/1632321689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87c070c80 0x7fd87c0b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.277+0000 7fd87affd640 1 -- 172.21.15.110:0/1632321689 >> v1:172.21.15.110:6789/0 conn(0x7fd87c143fc0 legacy=0x7fd87c13fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.277+0000 7fd87affd640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.277+0000 7fd87affd640 1 monclient: found mon.noname-a 2024-01-20T11:15:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.277+0000 7fd87affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.277+0000 7fd87affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.277+0000 7fd87affd640 1 -- 172.21.15.110:0/1632321689 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c06ffb0 con 0x7fd87c143540 2024-01-20T11:15:33.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.285+0000 7f607a484640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f60740a9ad0 0x7f6074142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.285+0000 7f607a484640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f60740a9ad0 0x7f6074142920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58230/0 (socket says 172.21.15.110:58230) 2024-01-20T11:15:33.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.285+0000 7f607a484640 1 -- 172.21.15.110:0/2977993175 learned_addr learned my addr 172.21.15.110:0/2977993175 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.285+0000 7f607a484640 10 monclient(hunting): get_auth_request con 0x7f60740a9ad0 auth_method 0 2024-01-20T11:15:33.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.285+0000 7f607a484640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.285+0000 7f607a484640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.285+0000 7f607a484640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.287+0000 7f607a484640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.287+0000 7f607a484640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.287+0000 7f607a484640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.288+0000 7f16437fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1644142920 0x7f164414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.288+0000 7f16437fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1644142920 0x7f164414c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58242/0 (socket says 172.21.15.110:58242) 2024-01-20T11:15:33.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.288+0000 7f16437fe640 1 -- 172.21.15.110:0/736517150 learned_addr learned my addr 172.21.15.110:0/736517150 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.288+0000 7f16437fe640 10 monclient(hunting): get_auth_request con 0x7f1644142920 auth_method 0 2024-01-20T11:15:33.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.288+0000 7f16437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.288+0000 7f16437fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.288+0000 7f16437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 10 monclient(hunting): handle_auth_done global_id 4393 payload 274 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 1 -- 172.21.15.110:0/2977993175 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60740a96f0 msgr2=0x7f607413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 1 --2- 172.21.15.110:0/2977993175 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60740a96f0 0x7f607413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 1 -- 172.21.15.110:0/2977993175 >> v2:172.21.15.110:3300/0 conn(0x7f607413fa10 msgr2=0x7f60741423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 1 --2- 172.21.15.110:0/2977993175 >> v2:172.21.15.110:3300/0 conn(0x7f607413fa10 0x7f60741423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 1 monclient: found mon.noname-c 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.290+0000 7f607a484640 1 -- 172.21.15.110:0/2977993175 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60740b3fb0 con 0x7f60740a9ad0 2024-01-20T11:15:33.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.295+0000 7f16437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.295+0000 7f16437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.295+0000 7f16437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.325+0000 7f16427fc640 1 --2- 172.21.15.110:0/736517150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16440a9ad0 0x7f16441423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.332+0000 7f16427fc640 10 monclient(hunting): get_auth_request con 0x7f16440a9ad0 auth_method 0 2024-01-20T11:15:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.332+0000 7f16427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.332+0000 7f16427fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.332+0000 7f16427fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.335+0000 7f16437fe640 10 monclient(hunting): handle_auth_done global_id 4396 payload 274 2024-01-20T11:15:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.335+0000 7f16437fe640 1 -- 172.21.15.110:0/736517150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16440a9ad0 msgr2=0x7f16441423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.336+0000 7f16437fe640 1 --2- 172.21.15.110:0/736517150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16440a9ad0 0x7f16441423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.336+0000 7f16437fe640 1 -- 172.21.15.110:0/736517150 >> v1:172.21.15.134:6789/0 conn(0x7f16440a96f0 legacy=0x7f164413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.336+0000 7f16437fe640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.336+0000 7f16437fe640 1 monclient: found mon.noname-c 2024-01-20T11:15:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.336+0000 7f16437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.336+0000 7f16437fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.336+0000 7f16437fe640 1 -- 172.21.15.110:0/736517150 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16440ab0a0 con 0x7f1644142920 2024-01-20T11:15:33.338 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-20T11:15:33.342 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:24.367 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:33.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.366+0000 7fd87affd640 1 --2- 172.21.15.110:0/1632321689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 0x7fd87c13d180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009870 tx=0x7fd86c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219d9cf2dee01b60 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.379+0000 7fd879ffb640 1 -- 172.21.15.110:0/1632321689 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c031640 con 0x7fd87c143540 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 10 monclient: dump: 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.380 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 1 -- 172.21.15.110:0/1632321689 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c0317a0 con 0x7fd87c143540 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd8814cf640 5 monclient: authenticate success, global_id 4356 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd8814cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd8814cf640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 1 -- 172.21.15.110:0/1632321689 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c031b30 con 0x7fd87c143540 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd8814cf640 4 set_mon_vals no callback set 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.380+0000 7fd879ffb640 10 monclient: dump: 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.381 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.382 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.382 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.382 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.382 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.382 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.382 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.382 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.382 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 10 monclient: shutdown 2024-01-20T11:15:33.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:33.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 1 -- 172.21.15.110:0/1632321689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 msgr2=0x7fd87c13d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 1 --2- 172.21.15.110:0/1632321689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 0x7fd87c13d180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009870 tx=0x7fd86c02b8f0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 1 -- 172.21.15.110:0/1632321689 shutdown_connections 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 1 --2- 172.21.15.110:0/1632321689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87c070c80 0x7fd87c0b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 1 --2- 172.21.15.110:0/1632321689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 0x7fd87c13d180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 1 -- 172.21.15.110:0/1632321689 >> 172.21.15.110:0/1632321689 conn(0x7fd87c0b5f00 msgr2=0x7fd87c0b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 1 -- 172.21.15.110:0/1632321689 shutdown_connections 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 1 -- 172.21.15.110:0/1632321689 wait complete. 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.381+0000 7fd8814cf640 10 monclient: monmap: 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.385 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.382+0000 7fd8814cf640 1 librados: starting msgr at 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.382+0000 7fd8814cf640 1 librados: starting objecter 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.382+0000 7fd8814cf640 1 Processor -- start 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.382+0000 7fd8814cf640 1 -- start start 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.382+0000 7fd8814cf640 1 librados: setting wanted keys 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.382+0000 7fd8814cf640 1 librados: calling monclient init 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.382+0000 7fd8814cf640 10 monclient: init 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87c070c80 0x7fd87c140620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient(hunting): picked mon.noname-c con 0x7fd87c070c80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 0x7fd87c1d70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd87c143540 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd87c143fc0 0x7fd87c06ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd87c143fc0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c137920 con 0x7fd87c143540 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c13f910 con 0x7fd87c143fc0 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c06c750 con 0x7fd87c070c80 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.383+0000 7fd8814cf640 10 monclient(hunting): authenticate will time out at 1376.733352s 2024-01-20T11:15:33.387 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-20T11:15:33.391 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-20T11:15:33.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.394+0000 7f607a484640 1 --2- 172.21.15.110:0/2977993175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60740a9ad0 0x7f6074142920 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f60741465c0 tx=0x7f606c02be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=931ff64b624836c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.398 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:01:24.404 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:33.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.402+0000 7fd87a7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 0x7fd87c1d70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.404+0000 7fd87a7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 0x7fd87c1d70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48650/0 (socket says 172.21.15.110:48650) 2024-01-20T11:15:33.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.404+0000 7fd87a7fc640 1 -- 172.21.15.110:0/401005217 learned_addr learned my addr 172.21.15.110:0/401005217 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.404+0000 7fd87a7fc640 10 monclient(hunting): get_auth_request con 0x7fd87c143540 auth_method 0 2024-01-20T11:15:33.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.404+0000 7fd87a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.404+0000 7fd87a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.404+0000 7fd87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.414+0000 7f16437fe640 1 --2- 172.21.15.110:0/736517150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1644142920 0x7f164414c850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f163c009e50 tx=0x7f163c02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74b7ec0b5af5ddcb server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.430+0000 7fd87b7fe640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd87c143fc0 0x7fd87c06ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.432+0000 7fd87b7fe640 10 monclient(hunting): get_auth_request con 0x7fd87c143fc0 auth_method 0 2024-01-20T11:15:33.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.432+0000 7fd87b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.432+0000 7fd87b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.432+0000 7fd87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.438+0000 7fd87b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.438+0000 7fd87b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.438+0000 7fd87b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.440+0000 7fd87a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.440+0000 7fd87a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.440+0000 7fd87a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 10 monclient(hunting): handle_auth_done global_id 4399 payload 824 2024-01-20T11:15:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 1 -- 172.21.15.110:0/401005217 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87c070c80 msgr2=0x7fd87c140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87c070c80 0x7fd87c140620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 1 -- 172.21.15.110:0/401005217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 msgr2=0x7fd87c1d70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 0x7fd87c1d70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 1 monclient: found mon.noname-b 2024-01-20T11:15:33.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.444+0000 7fd87b7fe640 1 -- 172.21.15.110:0/401005217 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0094d0 con 0x7fd87c143fc0 2024-01-20T11:15:33.452 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:24.476 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.458+0000 7f40d1890640 10 monclient: get_monmap_and_config 2024-01-20T11:15:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.458+0000 7f40d1890640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.458+0000 7f40d1890640 10 monclient: monmap: 2024-01-20T11:15:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:33.459134+0000 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:33.459134+0000 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 1 Processor -- start 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 1 -- start start 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 10 monclient: init 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-01-20T11:15:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc0a96f0 0x7f40cc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 10 monclient(hunting): picked mon.noname-b con 0x7f40cc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f40cc13fa10 0x7f40cc1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 10 monclient(hunting): picked mon.noname-a con 0x7f40cc13fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.459+0000 7f40d1890640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f40cc0a9ad0 0x7f40cc142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.460+0000 7f40d1890640 10 monclient(hunting): picked mon.noname-c con 0x7f40cc0a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.460+0000 7f40d1890640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.460+0000 7f40d1890640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40cc0aaea0 con 0x7f40cc13fa10 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.460+0000 7f40d1890640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.460+0000 7f40d1890640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40cc0b25c0 con 0x7f40cc0a9ad0 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.460+0000 7f40d1890640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.460+0000 7f40d1890640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc0ab0a0 con 0x7f40cc0a96f0 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.460+0000 7f40d1890640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.460+0000 7f40d1890640 10 monclient(hunting): authenticate will time out at 1376.810223s 2024-01-20T11:15:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.482+0000 7f40caffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc0a96f0 0x7f40cc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.489+0000 7f40caffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc0a96f0 0x7f40cc13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33864/0 (socket says 172.21.15.110:33864) 2024-01-20T11:15:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.489+0000 7f40caffd640 1 -- 172.21.15.110:0/3970470897 learned_addr learned my addr 172.21.15.110:0/3970470897 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.489+0000 7f40caffd640 10 monclient(hunting): get_auth_request con 0x7f40cc0a96f0 auth_method 0 2024-01-20T11:15:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.489+0000 7f40caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.489+0000 7f40caffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.489+0000 7f40caffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.499+0000 7f40caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.499+0000 7f40caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.499+0000 7f40caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-01-20T11:15:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 1 -- 172.21.15.110:0/3970470897 >> v2:172.21.15.134:3300/0 conn(0x7f40cc0a9ad0 msgr2=0x7f40cc142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 1 --2- 172.21.15.110:0/3970470897 >> v2:172.21.15.134:3300/0 conn(0x7f40cc0a9ad0 0x7f40cc142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 1 -- 172.21.15.110:0/3970470897 >> v2:172.21.15.110:3300/0 conn(0x7f40cc13fa10 msgr2=0x7f40cc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 1 --2- 172.21.15.110:0/3970470897 >> v2:172.21.15.110:3300/0 conn(0x7f40cc13fa10 0x7f40cc1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 1 monclient: found mon.noname-b 2024-01-20T11:15:33.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.502+0000 7f40caffd640 1 -- 172.21.15.110:0/3970470897 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc0b3fb0 con 0x7f40cc0a96f0 2024-01-20T11:15:33.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.511+0000 7fd87b7fe640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd87c143fc0 0x7fd87c06ce00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd87400c700 tx=0x7fd87400cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.511+0000 7fd8637fe640 10 client.?.objecter ms_handle_connect 0x7fd87c143fc0 2024-01-20T11:15:33.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.511+0000 7fd8637fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:33.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.512+0000 7f0cc0fd0640 10 monclient: get_monmap_and_config 2024-01-20T11:15:33.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.512+0000 7f0cc0fd0640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.512+0000 7f0cc0fd0640 10 monclient: monmap: 2024-01-20T11:15:33.512 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.512 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.512 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:33.513206+0000 2024-01-20T11:15:33.512 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:33.513206+0000 2024-01-20T11:15:33.513 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.513 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.513 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:33.513 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:33.513 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:33.513 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:33.513 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:33.513 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.513+0000 7f0cc0fd0640 1 Processor -- start 2024-01-20T11:15:33.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.513+0000 7f0cc0fd0640 1 -- start start 2024-01-20T11:15:33.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.513+0000 7f0cc0fd0640 10 monclient: init 2024-01-20T11:15:33.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-01-20T11:15:33.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0cbc0a96f0 0x7f0cbc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f0cbc0a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc13fa10 0x7f0cbc1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f0cbc13fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0cbc0a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc0b25c0 con 0x7f0cbc0a9ad0 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cbc0aaea0 con 0x7f0cbc0a96f0 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc0ab0a0 con 0x7f0cbc13fa10 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.514+0000 7f0cc0fd0640 10 monclient(hunting): authenticate will time out at 1376.864366s 2024-01-20T11:15:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.516+0000 7f072c839640 10 monclient: get_monmap_and_config 2024-01-20T11:15:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.516+0000 7f072c839640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.516+0000 7f072c839640 10 monclient: monmap: 2024-01-20T11:15:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:33.516665+0000 2024-01-20T11:15:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:33.516665+0000 2024-01-20T11:15:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.517 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:33.517 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:33.517 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:33.517 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:33.517 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:33.517 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.516+0000 7f072c839640 1 Processor -- start 2024-01-20T11:15:33.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.516+0000 7f072c839640 1 -- start start 2024-01-20T11:15:33.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.516+0000 7f072c839640 10 monclient: init 2024-01-20T11:15:33.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-20T11:15:33.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07280a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:33.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f07280a9ad0 0x7f07281423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 10 monclient(hunting): picked mon.noname-a con 0x7f07280a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:33.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0728142920 addr v1:172.21.15.134:6789/0 2024-01-20T11:15:33.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07280b25c0 con 0x7f07280a96f0 2024-01-20T11:15:33.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072813fbe0 con 0x7f0728142920 2024-01-20T11:15:33.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07280aaea0 con 0x7f07280a9ad0 2024-01-20T11:15:33.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.517+0000 7f072c839640 10 monclient(hunting): authenticate will time out at 1376.867481s 2024-01-20T11:15:33.522 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-20T11:15:33.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.529+0000 7f071dd9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f07280a9ad0 0x7f07281423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.531+0000 7f071dd9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f07280a9ad0 0x7f07281423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48676/0 (socket says 172.21.15.110:48676) 2024-01-20T11:15:33.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.531+0000 7f071dd9b640 1 -- 172.21.15.110:0/3902727922 learned_addr learned my addr 172.21.15.110:0/3902727922 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.531+0000 7f071dd9b640 10 monclient(hunting): get_auth_request con 0x7f07280a9ad0 auth_method 0 2024-01-20T11:15:33.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.531+0000 7f071dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.531+0000 7f071dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.531+0000 7f071dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.534+0000 7f071dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.534+0000 7f071dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.534+0000 7f071dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.537+0000 7f071dd9b640 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2024-01-20T11:15:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.537+0000 7f071dd9b640 1 -- 172.21.15.110:0/3902727922 >> v1:172.21.15.134:6789/0 conn(0x7f0728142920 legacy=0x7f072814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.537+0000 7f071dd9b640 1 -- 172.21.15.110:0/3902727922 >> v1:172.21.15.110:6789/0 conn(0x7f07280a96f0 legacy=0x7f072813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.537+0000 7f071dd9b640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.537+0000 7f071dd9b640 1 monclient: found mon.noname-a 2024-01-20T11:15:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.537+0000 7f071dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.537+0000 7f071dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.537+0000 7f071dd9b640 1 -- 172.21.15.110:0/3902727922 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07280ab0a0 con 0x7f07280a9ad0 2024-01-20T11:15:33.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 1 -- 172.21.15.110:0/2977993175 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c02ca10 con 0x7f60740a9ad0 2024-01-20T11:15:33.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:33.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 10 monclient: dump: 2024-01-20T11:15:33.540 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.540 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.540 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.540 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.540 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 1 -- 172.21.15.110:0/2977993175 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606c02cb70 con 0x7f60740a9ad0 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f607bee7640 5 monclient: authenticate success, global_id 4393 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f607bee7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f607bee7640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f607bee7640 4 set_mon_vals no callback set 2024-01-20T11:15:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 1 -- 172.21.15.110:0/2977993175 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c0343f0 con 0x7f60740a9ad0 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.540+0000 7f6078c81640 10 monclient: dump: 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.542 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 1 -- 172.21.15.110:0/736517150 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c02c9e0 con 0x7f1644142920 2024-01-20T11:15:33.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:33.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 10 monclient: dump: 2024-01-20T11:15:33.543 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.543 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 1 -- 172.21.15.110:0/736517150 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c02cb40 con 0x7f1644142920 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f16497c0640 5 monclient: authenticate success, global_id 4396 2024-01-20T11:15:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 1 -- 172.21.15.110:0/736517150 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c0343f0 con 0x7f1644142920 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f1641ffb640 10 monclient: dump: 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f16497c0640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f16497c0640 4 set_mon_vals no callback set 2024-01-20T11:15:33.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 10 monclient: shutdown 2024-01-20T11:15:33.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 1 -- 172.21.15.110:0/2977993175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60740a9ad0 msgr2=0x7f6074142920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 1 --2- 172.21.15.110:0/2977993175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60740a9ad0 0x7f6074142920 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f60741465c0 tx=0x7f606c02be70 comp rx=0 tx=0).stop 2024-01-20T11:15:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 1 -- 172.21.15.110:0/2977993175 shutdown_connections 2024-01-20T11:15:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 1 --2- 172.21.15.110:0/2977993175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60740a9ad0 0x7f6074142920 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 1 --2- 172.21.15.110:0/2977993175 >> v2:172.21.15.110:3300/0 conn(0x7f607413fa10 0x7f60741423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 1 --2- 172.21.15.110:0/2977993175 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60740a96f0 0x7f607413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 1 -- 172.21.15.110:0/2977993175 >> 172.21.15.110:0/2977993175 conn(0x7f60740b7bc0 msgr2=0x7f60740b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 1 -- 172.21.15.110:0/2977993175 shutdown_connections 2024-01-20T11:15:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.541+0000 7f607bee7640 1 -- 172.21.15.110:0/2977993175 wait complete. 2024-01-20T11:15:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 10 monclient: shutdown 2024-01-20T11:15:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 1 -- 172.21.15.110:0/736517150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1644142920 msgr2=0x7f164414c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 1 --2- 172.21.15.110:0/736517150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1644142920 0x7f164414c850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f163c009e50 tx=0x7f163c02be40 comp rx=0 tx=0).stop 2024-01-20T11:15:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 1 -- 172.21.15.110:0/736517150 shutdown_connections 2024-01-20T11:15:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 1 --2- 172.21.15.110:0/736517150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1644142920 0x7f164414c850 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 1 --2- 172.21.15.110:0/736517150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16440a9ad0 0x7f16441423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 1 -- 172.21.15.110:0/736517150 >> 172.21.15.110:0/736517150 conn(0x7f16440b7bc0 msgr2=0x7f16440b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:33.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 1 -- 172.21.15.110:0/736517150 shutdown_connections 2024-01-20T11:15:33.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 1 -- 172.21.15.110:0/736517150 wait complete. 2024-01-20T11:15:33.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f607bee7640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f607bee7640 10 monclient: monmap: 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f607bee7640 1 librados: starting msgr at 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f607bee7640 1 librados: starting objecter 2024-01-20T11:15:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f607bee7640 1 Processor -- start 2024-01-20T11:15:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f607bee7640 1 -- start start 2024-01-20T11:15:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f607bee7640 1 librados: setting wanted keys 2024-01-20T11:15:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f607bee7640 1 librados: calling monclient init 2024-01-20T11:15:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f607bee7640 10 monclient: init 2024-01-20T11:15:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.542+0000 7f16497c0640 10 monclient: monmap: 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f16497c0640 1 librados: starting msgr at 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f16497c0640 1 librados: starting objecter 2024-01-20T11:15:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f16497c0640 1 Processor -- start 2024-01-20T11:15:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:15:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60740a96f0 0x7f60740abf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient(hunting): picked mon.noname-b con 0x7f60740a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60740a9ad0 0x7f6074143550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient(hunting): picked mon.noname-a con 0x7f60740a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60740acaf0 0x7f60741dab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f16497c0640 1 -- start start 2024-01-20T11:15:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient(hunting): picked mon.noname-c con 0x7f60740acaf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60740ae450 con 0x7f60740a9ad0 2024-01-20T11:15:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60741da3c0 con 0x7f60740a96f0 2024-01-20T11:15:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60741da610 con 0x7f60740acaf0 2024-01-20T11:15:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f16497c0640 1 librados: setting wanted keys 2024-01-20T11:15:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f16497c0640 1 librados: calling monclient init 2024-01-20T11:15:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f607bee7640 10 monclient(hunting): authenticate will time out at 1376.893810s 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.543+0000 7f16497c0640 10 monclient: init 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16440a96f0 0x7f16441d78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient(hunting): picked mon.noname-a con 0x7f16440a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16440a9ad0 0x7f16441d7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient(hunting): picked mon.noname-c con 0x7f16440a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1644142920 0x7f1644138680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1644142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164411fd00 con 0x7f16440a96f0 2024-01-20T11:15:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16441d8500 con 0x7f1644142920 2024-01-20T11:15:33.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1644139830 con 0x7f16440a9ad0 2024-01-20T11:15:33.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f16497c0640 10 monclient(hunting): authenticate will time out at 1376.894532s 2024-01-20T11:15:33.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.544+0000 7f071dd9b640 1 --2- 172.21.15.110:0/3902727922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280a9ad0 0x7f07281423e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0710009870 tx=0x7f071002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e378eb7aa1777c60 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.546+0000 7f0cba59c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0cbc0a96f0 0x7f0cbc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.548+0000 7f1642ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16440a96f0 0x7f16441d78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.548+0000 7f6079482640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60740a9ad0 0x7f6074143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f1642ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16440a96f0 0x7f16441d78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48692/0 (socket says 172.21.15.110:48692) 2024-01-20T11:15:33.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f1642ffd640 1 -- 172.21.15.110:0/3680534729 learned_addr learned my addr 172.21.15.110:0/3680534729 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f1642ffd640 10 monclient(hunting): get_auth_request con 0x7f16440a96f0 auth_method 0 2024-01-20T11:15:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f1642ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f1642ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f1642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f0cba59c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0cbc0a96f0 0x7f0cbc13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58278/0 (socket says 172.21.15.110:58278) 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f0cba59c640 1 -- 172.21.15.110:0/2805202518 learned_addr learned my addr 172.21.15.110:0/2805202518 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f0cba59c640 10 monclient(hunting): get_auth_request con 0x7f0cbc0a96f0 auth_method 0 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f0cba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f0cba59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.549+0000 7f0cba59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7f6079482640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60740a9ad0 0x7f6074143550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48686/0 (socket says 172.21.15.110:48686) 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7f6079482640 1 -- 172.21.15.110:0/2006676911 learned_addr learned my addr 172.21.15.110:0/2006676911 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7f6079482640 10 monclient(hunting): get_auth_request con 0x7f60740a9ad0 auth_method 0 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7f6079482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7f6079482640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7f6079482640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 1 -- 172.21.15.110:0/401005217 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874006d50 con 0x7fd87c143fc0 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 10 monclient: dump: 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 1 -- 172.21.15.110:0/401005217 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd874003970 con 0x7fd87c143fc0 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8814cf640 5 monclient: authenticate success, global_id 4399 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8814cf640 10 monclient: _renew_subs 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8814cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87c13d9c0 con 0x7fd87c143fc0 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 1 -- 172.21.15.110:0/401005217 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874003d20 con 0x7fd87c143fc0 2024-01-20T11:15:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8797fa640 4 set_mon_vals no callback set 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.551+0000 7fd8637fe640 10 monclient: dump: 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.552+0000 7fd8814cf640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.552+0000 7fd8814cf640 10 monclient: _renew_subs 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.552+0000 7fd8814cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.552+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c1d7620 con 0x7fd87c143fc0 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.552+0000 7fd8814cf640 1 librados: init done 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.553+0000 7fd8814cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.553+0000 7fd8814cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.553+0000 7fd8814cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.553+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87c06c8e0 con 0x7fd87c143fc0 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.554+0000 7f1642ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.554+0000 7f1642ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.554+0000 7f1642ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.555+0000 7f6079482640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.555+0000 7f6079482640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.555+0000 7f6079482640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 1 -- 172.21.15.110:0/3902727922 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07100317e0 con 0x7f07280a9ad0 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 10 monclient: dump: 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 1 -- 172.21.15.110:0/3902727922 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0710031940 con 0x7f07280a9ad0 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f072c839640 5 monclient: authenticate success, global_id 4362 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f072c839640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 1 -- 172.21.15.110:0/3902727922 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710031cd0 con 0x7f07280a9ad0 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f0725d9b640 10 monclient: dump: 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.569 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f072c839640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.556+0000 7f072c839640 4 set_mon_vals no callback set 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.557+0000 7f072c839640 10 monclient: shutdown 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.557+0000 7f072c839640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.557+0000 7f072c839640 1 -- 172.21.15.110:0/3902727922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280a9ad0 msgr2=0x7f07281423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.557+0000 7f072c839640 1 --2- 172.21.15.110:0/3902727922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280a9ad0 0x7f07281423e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0710009870 tx=0x7f071002b880 comp rx=0 tx=0).stop 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.557+0000 7f072c839640 1 -- 172.21.15.110:0/3902727922 shutdown_connections 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.557+0000 7f072c839640 1 --2- 172.21.15.110:0/3902727922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280a9ad0 0x7f07281423e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.557+0000 7f072c839640 1 -- 172.21.15.110:0/3902727922 >> 172.21.15.110:0/3902727922 conn(0x7f07280b7bc0 msgr2=0x7f07280b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.558+0000 7f072c839640 1 -- 172.21.15.110:0/3902727922 shutdown_connections 2024-01-20T11:15:33.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.558+0000 7f072c839640 1 -- 172.21.15.110:0/3902727922 wait complete. 2024-01-20T11:15:33.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.558+0000 7f0cba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.558+0000 7f0cba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.558+0000 7f0cba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.559+0000 7f072c839640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.559+0000 7f072c839640 10 monclient: monmap: 2024-01-20T11:15:33.573 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.573 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.559+0000 7f072c839640 1 librados: starting msgr at 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.559+0000 7f072c839640 1 librados: starting objecter 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.560+0000 7f072c839640 1 Processor -- start 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.560+0000 7f072c839640 1 -- start start 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.560+0000 7f072c839640 1 librados: setting wanted keys 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.560+0000 7f072c839640 1 librados: calling monclient init 2024-01-20T11:15:33.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.560+0000 7f072c839640 10 monclient: init 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07280a96f0 0x7f07280b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient(hunting): picked mon.noname-b con 0x7f07280a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280a9ad0 0x7f07281d77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient(hunting): picked mon.noname-a con 0x7f07280a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0728142920 0x7f07281d7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient(hunting): picked mon.noname-c con 0x7f0728142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07280acb20 con 0x7f07280a9ad0 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07281c8320 con 0x7f07280a96f0 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07281c8600 con 0x7f0728142920 2024-01-20T11:15:33.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.561+0000 7f072c839640 10 monclient(hunting): authenticate will time out at 1376.912040s 2024-01-20T11:15:33.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-01-20T11:15:33.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 1 -- 172.21.15.110:0/3680534729 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16440a9ad0 msgr2=0x7f16441d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16440a9ad0 0x7f16441d7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 1 -- 172.21.15.110:0/3680534729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1644142920 msgr2=0x7f1644138680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1644142920 0x7f1644138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 1 monclient: found mon.noname-a 2024-01-20T11:15:33.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.562+0000 7f1642ffd640 1 -- 172.21.15.110:0/3680534729 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c009b30 con 0x7f16440a96f0 2024-01-20T11:15:33.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.578+0000 7f40caffd640 1 --2- 172.21.15.110:0/3970470897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc0a96f0 0x7f40cc13f4d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f40cc1465c0 tx=0x7f40bc02b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38a8a4822cef12a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.588+0000 7f607a484640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60740acaf0 0x7f60741dab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.590+0000 7f0cb9d9b640 1 --2- 172.21.15.110:0/2805202518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc13fa10 0x7f0cbc1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f1642ffd640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16440a96f0 0x7f16441d78d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f163400c410 tx=0x7f163400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f162b7fe640 10 client.?.objecter ms_handle_connect 0x7f16440a96f0 2024-01-20T11:15:33.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f162b7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:33.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-01-20T11:15:33.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 1 -- 172.21.15.110:0/2006676911 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60740acaf0 msgr2=0x7f60741dab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60740acaf0 0x7f60741dab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 1 -- 172.21.15.110:0/2006676911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60740a96f0 msgr2=0x7f60740abf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60740a96f0 0x7f60740abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 1 monclient: found mon.noname-a 2024-01-20T11:15:33.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.592+0000 7f6079482640 1 -- 172.21.15.110:0/2006676911 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c009b30 con 0x7f60740a9ad0 2024-01-20T11:15:33.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.599+0000 7f6079482640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60740a9ad0 0x7f6074143550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f606400b390 tx=0x7f606400b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.599+0000 7f606a7fc640 10 client.?.objecter ms_handle_connect 0x7f60740a9ad0 2024-01-20T11:15:33.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.599+0000 7f606a7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.602+0000 7f0cb9d9b640 10 monclient(hunting): get_auth_request con 0x7f0cbc13fa10 auth_method 0 2024-01-20T11:15:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.602+0000 7f0cb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.602+0000 7f0cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.602+0000 7f0cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.609+0000 7f071dd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280a9ad0 0x7f07281d77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.609+0000 7f0cb9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.609+0000 7f0cb9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.609+0000 7f0cb9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 1 -- 172.21.15.110:0/3970470897 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc0317e0 con 0x7f40cc0a96f0 2024-01-20T11:15:33.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:33.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 10 monclient: dump: 2024-01-20T11:15:33.611 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.611 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.611 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 1 -- 172.21.15.110:0/3970470897 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc031940 con 0x7f40cc0a96f0 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40d1890640 5 monclient: authenticate success, global_id 4274 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40d1890640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40d1890640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 1 -- 172.21.15.110:0/3970470897 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc031cd0 con 0x7f40cc0a96f0 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40d1890640 4 set_mon_vals no callback set 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f40c9ffb640 10 monclient: dump: 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.613 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 1 -- 172.21.15.110:0/3680534729 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16340035f0 con 0x7f16440a96f0 2024-01-20T11:15:33.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:33.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 10 monclient: dump: 2024-01-20T11:15:33.614 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.614 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 1 -- 172.21.15.110:0/3680534729 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1634003750 con 0x7f16440a96f0 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 1 -- 172.21.15.110:0/3680534729 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1634003b00 con 0x7f16440a96f0 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f162b7fe640 10 monclient: dump: 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.611+0000 7f16417fa640 4 set_mon_vals no callback set 2024-01-20T11:15:33.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 10 monclient: shutdown 2024-01-20T11:15:33.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:33.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 1 -- 172.21.15.110:0/3970470897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc0a96f0 msgr2=0x7f40cc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 1 --2- 172.21.15.110:0/3970470897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc0a96f0 0x7f40cc13f4d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f40cc1465c0 tx=0x7f40bc02b880 comp rx=0 tx=0).stop 2024-01-20T11:15:33.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f16497c0640 5 monclient: authenticate success, global_id 4365 2024-01-20T11:15:33.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 1 -- 172.21.15.110:0/3970470897 shutdown_connections 2024-01-20T11:15:33.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f16497c0640 10 monclient: _renew_subs 2024-01-20T11:15:33.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 1 --2- 172.21.15.110:0/3970470897 >> v2:172.21.15.134:3300/0 conn(0x7f40cc0a9ad0 0x7f40cc142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f16497c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 1 --2- 172.21.15.110:0/3970470897 >> v2:172.21.15.110:3300/0 conn(0x7f40cc13fa10 0x7f40cc1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1644139a50 con 0x7f16440a96f0 2024-01-20T11:15:33.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 1 --2- 172.21.15.110:0/3970470897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc0a96f0 0x7f40cc13f4d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 1 -- 172.21.15.110:0/3970470897 >> 172.21.15.110:0/3970470897 conn(0x7f40cc0b7bc0 msgr2=0x7f40cc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:33.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f16497c0640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:33.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f16497c0640 10 monclient: _renew_subs 2024-01-20T11:15:33.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f16497c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1644138da0 con 0x7f16440a96f0 2024-01-20T11:15:33.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f16497c0640 1 librados: init done 2024-01-20T11:15:33.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 1 -- 172.21.15.110:0/3970470897 shutdown_connections 2024-01-20T11:15:33.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f40d1890640 1 -- 172.21.15.110:0/3970470897 wait complete. 2024-01-20T11:15:33.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.612+0000 7f0726d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0728142920 0x7f07281d7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.613+0000 7f40d1890640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.613+0000 7f40d1890640 10 monclient: monmap: 2024-01-20T11:15:33.621 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.621 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.613+0000 7f16497c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.613+0000 7f16497c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.613+0000 7f16497c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.613+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16440af820 con 0x7f16440a96f0 2024-01-20T11:15:33.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f40d1890640 1 librados: starting msgr at 2024-01-20T11:15:33.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f40d1890640 1 librados: starting objecter 2024-01-20T11:15:33.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f40d1890640 1 Processor -- start 2024-01-20T11:15:33.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f0cb9d9b640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-01-20T11:15:33.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f40d1890640 1 -- start start 2024-01-20T11:15:33.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f0cb9d9b640 1 -- 172.21.15.110:0/2805202518 >> v2:172.21.15.134:3300/0 conn(0x7f0cbc0a96f0 msgr2=0x7f0cbc13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f0cb9d9b640 1 --2- 172.21.15.110:0/2805202518 >> v2:172.21.15.134:3300/0 conn(0x7f0cbc0a96f0 0x7f0cbc13f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f0cb9d9b640 1 -- 172.21.15.110:0/2805202518 >> v1:172.21.15.134:6789/0 conn(0x7f0cbc0a9ad0 legacy=0x7f0cbc142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f0cb9d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f0cb9d9b640 1 monclient: found mon.noname-b 2024-01-20T11:15:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f0cb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f0cb9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f0cb9d9b640 1 -- 172.21.15.110:0/2805202518 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc0b3fb0 con 0x7f0cbc13fa10 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f40d1890640 1 librados: setting wanted keys 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f40d1890640 1 librados: calling monclient init 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.614+0000 7f40d1890640 10 monclient: init 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40cc0a96f0 0x7f40cc0b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient(hunting): picked mon.noname-a con 0x7f40cc0a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40cc0a9ad0 0x7f40cc13acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient(hunting): picked mon.noname-b con 0x7f40cc0a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc13ca40 0x7f40cc140bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient(hunting): picked mon.noname-c con 0x7f40cc13ca40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc141100 con 0x7f40cc0a96f0 2024-01-20T11:15:33.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc13c540 con 0x7f40cc0a9ad0 2024-01-20T11:15:33.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc144510 con 0x7f40cc13ca40 2024-01-20T11:15:33.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f40d1890640 10 monclient(hunting): authenticate will time out at 1376.965635s 2024-01-20T11:15:33.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f0726d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0728142920 0x7f07281d7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33904/0 (socket says 172.21.15.110:33904) 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f0726d9d640 1 -- 172.21.15.110:0/2531179753 learned_addr learned my addr 172.21.15.110:0/2531179753 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f0726d9d640 10 monclient(hunting): get_auth_request con 0x7f0728142920 auth_method 0 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f0726d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f0726d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.615+0000 7f0726d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.619+0000 7f071dd9b640 10 monclient(hunting): get_auth_request con 0x7f07280a9ad0 auth_method 0 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.619+0000 7f071dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.619+0000 7f071dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.619+0000 7f071dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.620+0000 7f0cb9d9b640 1 --2- 172.21.15.110:0/2805202518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc13fa10 0x7f0cbc1423e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00097a0 tx=0x7f0cb002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52248f99740e69ba server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.620+0000 7f0726d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.620+0000 7f0726d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.620+0000 7f0726d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 1 -- 172.21.15.110:0/2006676911 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064011830 con 0x7f60740a9ad0 2024-01-20T11:15:33.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:33.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 10 monclient: dump: 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 1 -- 172.21.15.110:0/2006676911 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6064011990 con 0x7f60740a9ad0 2024-01-20T11:15:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f607bee7640 5 monclient: authenticate success, global_id 4368 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f607bee7640 10 monclient: _renew_subs 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f607bee7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60741db070 con 0x7f60740a9ad0 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 1 -- 172.21.15.110:0/2006676911 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064011d20 con 0x7f60740a9ad0 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606a7fc640 10 monclient: dump: 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f606bfff640 4 set_mon_vals no callback set 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f607bee7640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f607bee7640 10 monclient: _renew_subs 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f607bee7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60740ad180 con 0x7f60740a9ad0 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.624+0000 7f607bee7640 1 librados: init done 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.625+0000 7f607bee7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.625+0000 7f607bee7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.625+0000 7f607bee7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:33.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.625+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6074075490 con 0x7f60740a9ad0 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 1 -- 172.21.15.110:0/2531179753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07280a96f0 msgr2=0x7f07280b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07280a96f0 0x7f07280b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 1 -- 172.21.15.110:0/2531179753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280a9ad0 msgr2=0x7f07281d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280a9ad0 0x7f07281d77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 1 monclient: found mon.noname-c 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.629+0000 7f0726d9d640 1 -- 172.21.15.110:0/2531179753 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07100094d0 con 0x7f0728142920 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.638+0000 7f6e6ec15640 10 monclient: get_monmap_and_config 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.638+0000 7f6e6ec15640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.638+0000 7f6e6ec15640 10 monclient: monmap: 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:33.638824+0000 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:33.638824+0000 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:33.638 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:33.639 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:33.639 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.638+0000 7f6e6ec15640 1 Processor -- start 2024-01-20T11:15:33.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 1 -- start start 2024-01-20T11:15:33.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient: init 2024-01-20T11:15:33.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-01-20T11:15:33.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6e680a96f0 0x7f6e6813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient(hunting): picked mon.noname-a con 0x7f6e680a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e6813fa10 0x7f6e681423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient(hunting): picked mon.noname-b con 0x7f6e6813fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e680a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e680b25c0 con 0x7f6e680a9ad0 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e680aaea0 con 0x7f6e680a96f0 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e680ab0a0 con 0x7f6e6813fa10 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.639+0000 7f6e6ec15640 10 monclient(hunting): authenticate will time out at 1376.989972s 2024-01-20T11:15:33.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.642+0000 7f0726d9d640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0728142920 0x7f07281d7d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f072000c780 tx=0x7f072000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.642+0000 7f071f7fe640 10 client.?.objecter ms_handle_connect 0x7f0728142920 2024-01-20T11:15:33.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.642+0000 7f071f7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 1 -- 172.21.15.110:0/2805202518 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb002e070 con 0x7f0cbc13fa10 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 10 monclient: dump: 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cc0fd0640 5 monclient: authenticate success, global_id 4277 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cc0fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 1 -- 172.21.15.110:0/2805202518 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb00315a0 con 0x7f0cbc13fa10 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 1 -- 172.21.15.110:0/2805202518 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb002e070 con 0x7f0cbc13fa10 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cb959a640 10 monclient: dump: 2024-01-20T11:15:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cc0fd0640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f0cc0fd0640 4 set_mon_vals no callback set 2024-01-20T11:15:33.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.665+0000 7f40cb7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc13ca40 0x7f40cc140bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 10 monclient: shutdown 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/2805202518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc13fa10 msgr2=0x7f0cbc1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 1 --2- 172.21.15.110:0/2805202518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc13fa10 0x7f0cbc1423e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00097a0 tx=0x7f0cb002b840 comp rx=0 tx=0).stop 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/2805202518 shutdown_connections 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 1 --2- 172.21.15.110:0/2805202518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc13fa10 0x7f0cbc1423e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 1 --2- 172.21.15.110:0/2805202518 >> v2:172.21.15.134:3300/0 conn(0x7f0cbc0a96f0 0x7f0cbc13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/2805202518 >> 172.21.15.110:0/2805202518 conn(0x7f0cbc0b7bc0 msgr2=0x7f0cbc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/2805202518 shutdown_connections 2024-01-20T11:15:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.666+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/2805202518 wait complete. 2024-01-20T11:15:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f40cb7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc13ca40 0x7f40cc140bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33914/0 (socket says 172.21.15.110:33914) 2024-01-20T11:15:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f40cb7fe640 1 -- 172.21.15.110:0/128006686 learned_addr learned my addr 172.21.15.110:0/128006686 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f40cb7fe640 10 monclient(hunting): get_auth_request con 0x7f40cc13ca40 auth_method 0 2024-01-20T11:15:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f40cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f40cb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f40cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f6e67fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e6813fa10 0x7f6e681423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f0cc0fd0640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f0cc0fd0640 10 monclient: monmap: 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f0cc0fd0640 1 librados: starting msgr at 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.667+0000 7f0cc0fd0640 1 librados: starting objecter 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 1 Processor -- start 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 1 -- start start 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 1 librados: setting wanted keys 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 1 librados: calling monclient init 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 10 monclient: init 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f40cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f40cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f40cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cbc0a96f0 0x7f0cbc13b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f0cbc0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:33.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc0a9ad0 0x7f0cbc13b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f0cbc0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.668+0000 7f0cc0fd0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cbc13a560 0x7f0cbc1d7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.669+0000 7f0cc0fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f0cbc13a560 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:33.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.669+0000 7f0cc0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.669+0000 7f0cc0fd0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc0abf90 con 0x7f0cbc13a560 2024-01-20T11:15:33.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.669+0000 7f0cc0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.669+0000 7f0cc0fd0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc1d70e0 con 0x7f0cbc0a96f0 2024-01-20T11:15:33.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.669+0000 7f0cc0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.669+0000 7f0cc0fd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc137f60 con 0x7f0cbc0a9ad0 2024-01-20T11:15:33.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.669+0000 7f0cc0fd0640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.669+0000 7f0cc0fd0640 10 monclient(hunting): authenticate will time out at 1377.019217s 2024-01-20T11:15:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-01-20T11:15:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 1 -- 172.21.15.110:0/128006686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40cc0a9ad0 msgr2=0x7f40cc13acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40cc0a9ad0 0x7f40cc13acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 1 -- 172.21.15.110:0/128006686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40cc0a96f0 msgr2=0x7f40cc0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40cc0a96f0 0x7f40cc0b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 1 monclient: found mon.noname-c 2024-01-20T11:15:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.671+0000 7f40cb7fe640 1 -- 172.21.15.110:0/128006686 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc0094d0 con 0x7f40cc13ca40 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 1 -- 172.21.15.110:0/2531179753 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720003d80 con 0x7f0728142920 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 10 monclient: dump: 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 1 -- 172.21.15.110:0/2531179753 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072000ce90 con 0x7f0728142920 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072c839640 5 monclient: authenticate success, global_id 4280 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072c839640 10 monclient: _renew_subs 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072c839640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07281d8270 con 0x7f0728142920 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 1 -- 172.21.15.110:0/2531179753 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07200052b0 con 0x7f0728142920 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f071f7fe640 10 monclient: dump: 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072559a640 4 set_mon_vals no callback set 2024-01-20T11:15:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072c839640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:33.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072c839640 10 monclient: _renew_subs 2024-01-20T11:15:33.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072c839640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07280b07e0 con 0x7f0728142920 2024-01-20T11:15:33.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.674+0000 7f072c839640 1 librados: init done 2024-01-20T11:15:33.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f072c839640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f072c839640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f072c839640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0728075490 con 0x7f0728142920 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f6e67fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e6813fa10 0x7f6e681423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33916/0 (socket says 172.21.15.110:33916) 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f6e67fff640 1 -- 172.21.15.110:0/3492849748 learned_addr learned my addr 172.21.15.110:0/3492849748 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f6e67fff640 10 monclient(hunting): get_auth_request con 0x7f6e6813fa10 auth_method 0 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f6e67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f6e67fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.675+0000 7f6e67fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.679+0000 7f6e67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.679+0000 7f6e67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.679+0000 7f6e67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.680+0000 7f6e67fff640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2024-01-20T11:15:33.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.680+0000 7f6e67fff640 1 -- 172.21.15.110:0/3492849748 >> v2:172.21.15.110:3300/0 conn(0x7f6e680a96f0 msgr2=0x7f6e6813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.680+0000 7f6e67fff640 1 --2- 172.21.15.110:0/3492849748 >> v2:172.21.15.110:3300/0 conn(0x7f6e680a96f0 0x7f6e6813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.680+0000 7f6e67fff640 1 -- 172.21.15.110:0/3492849748 >> v1:172.21.15.110:6789/0 conn(0x7f6e680a9ad0 legacy=0x7f6e68142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.680+0000 7f6e67fff640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.680+0000 7f6e67fff640 1 monclient: found mon.noname-b 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.680+0000 7f6e67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.680+0000 7f6e67fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.680+0000 7f6e67fff640 1 -- 172.21.15.110:0/3492849748 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e680b3fb0 con 0x7f6e6813fa10 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.683+0000 7f40cb7fe640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc13ca40 0x7f40cc140bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40c400b9a0 tx=0x7f40c400be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.683+0000 7f40b3fff640 10 client.?.objecter ms_handle_connect 0x7f40cc13ca40 2024-01-20T11:15:33.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.683+0000 7f40b3fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:33.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.687+0000 7f6e67fff640 1 --2- 172.21.15.110:0/3492849748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e6813fa10 0x7f6e681423e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6e680b5720 tx=0x7f6e5402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d192f62048712e5b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.703+0000 7f0cb9d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc0a9ad0 0x7f0cbc13b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.716+0000 7f0cb9d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc0a9ad0 0x7f0cbc13b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33928/0 (socket says 172.21.15.110:33928) 2024-01-20T11:15:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.717+0000 7f0cb9d9b640 1 -- 172.21.15.110:0/266395249 learned_addr learned my addr 172.21.15.110:0/266395249 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:33.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.717+0000 7f0cb9d9b640 10 monclient(hunting): get_auth_request con 0x7f0cbc0a9ad0 auth_method 0 2024-01-20T11:15:33.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.717+0000 7f0cb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.717+0000 7f0cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.717+0000 7f0cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.834+0000 7fd8637fe640 1 -- 172.21.15.110:0/401005217 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd8740050d0 con 0x7fd87c143fc0 2024-01-20T11:15:33.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.834+0000 7fd8637fe640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd86806c8b0 0x7fd86806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.834+0000 7fd87affd640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd86806c8b0 0x7fd86806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.834+0000 7fd87affd640 10 monclient: get_auth_request con 0x7fd86806c8b0 auth_method 0 2024-01-20T11:15:33.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.835+0000 7fd87affd640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd86806c8b0 0x7fd86806ed70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd86c036000 tx=0x7fd86c030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.835+0000 7fd8637fe640 10 client.4399.objecter ms_handle_connect 0x7fd86806c8b0 2024-01-20T11:15:33.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.838+0000 7fd8637fe640 1 -- 172.21.15.110:0/401005217 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd874059070 con 0x7fd87c143fc0 2024-01-20T11:15:33.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.838+0000 7fd8637fe640 10 client.4399.objecter ms_dispatch 0x7fd87c000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:33.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.838+0000 7fd8637fe640 3 client.4399.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:33.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.838+0000 7fd8637fe640 3 client.4399.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:33.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.838+0000 7fd8637fe640 20 client.4399.objecter dump_active .. 0 homeless 2024-01-20T11:15:33.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.851+0000 7fd8637fe640 1 -- 172.21.15.110:0/401005217 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd874054cb0 con 0x7fd87c143fc0 2024-01-20T11:15:33.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.851+0000 7fd8637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.851+0000 7fd8637fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:33.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.855+0000 7f0cba59c640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cbc0a96f0 0x7f0cbc13b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.859+0000 7f0cba59c640 10 monclient(hunting): get_auth_request con 0x7f0cbc0a96f0 auth_method 0 2024-01-20T11:15:33.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.859+0000 7f0cba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:33.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.859+0000 7f0cba59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:33.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.859+0000 7f0cba59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:33.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.879+0000 7f0cba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:33.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.879+0000 7f0cba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:33.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.879+0000 7f0cba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.883+0000 7f0cba59c640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.884+0000 7f0cba59c640 1 -- 172.21.15.110:0/266395249 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc0a9ad0 msgr2=0x7f0cbc13b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.884+0000 7f0cba59c640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc0a9ad0 0x7f0cbc13b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.884+0000 7f0cba59c640 1 -- 172.21.15.110:0/266395249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cbc13a560 msgr2=0x7f0cbc1d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.884+0000 7f0cba59c640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cbc13a560 0x7f0cbc1d7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.884+0000 7f0cba59c640 10 monclient: _finish_hunting 0 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.884+0000 7f0cba59c640 1 monclient: found mon.noname-b 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.884+0000 7f0cba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.884+0000 7f0cba59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.884+0000 7f0cba59c640 1 -- 172.21.15.110:0/266395249 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb0009400 con 0x7f0cbc0a96f0 2024-01-20T11:15:33.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.892+0000 7f162b7fe640 1 -- 172.21.15.110:0/3680534729 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1634003c60 con 0x7f16440a96f0 2024-01-20T11:15:33.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.892+0000 7f162b7fe640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f163006c8b0 0x7f163006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.893+0000 7f16427fc640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f163006c8b0 0x7f163006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.893+0000 7f16427fc640 10 monclient: get_auth_request con 0x7f163006c8b0 auth_method 0 2024-01-20T11:15:33.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.893+0000 7f16427fc640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f163006c8b0 0x7f163006ed70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f16440af0f0 tx=0x7f1638008040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.893+0000 7f162b7fe640 10 client.4365.objecter ms_handle_connect 0x7f163006c8b0 2024-01-20T11:15:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.903+0000 7f0cba59c640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cbc0a96f0 0x7f0cbc13b170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00e850 tx=0x7f0cac00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.903+0000 7f0caaffd640 10 client.?.objecter ms_handle_connect 0x7f0cbc0a96f0 2024-01-20T11:15:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.903+0000 7f0caaffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e677fe640 1 -- 172.21.15.110:0/3492849748 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e540317c0 con 0x7f6e6813fa10 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e677fe640 10 monclient: dump: 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e677fe640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e677fe640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e677fe640 1 -- 172.21.15.110:0/3492849748 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e54031920 con 0x7f6e6813fa10 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e677fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e6ec15640 5 monclient: authenticate success, global_id 4286 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e6ec15640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.909+0000 7f6e6ec15640 4 set_mon_vals no callback set 2024-01-20T11:15:33.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 10 monclient: shutdown 2024-01-20T11:15:33.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:33.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 1 -- 172.21.15.110:0/3492849748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e6813fa10 msgr2=0x7f6e681423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 1 --2- 172.21.15.110:0/3492849748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e6813fa10 0x7f6e681423e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6e680b5720 tx=0x7f6e5402b860 comp rx=0 tx=0).stop 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 1 -- 172.21.15.110:0/3492849748 shutdown_connections 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 1 --2- 172.21.15.110:0/3492849748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e6813fa10 0x7f6e681423e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 1 --2- 172.21.15.110:0/3492849748 >> v2:172.21.15.110:3300/0 conn(0x7f6e680a96f0 0x7f6e6813f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 1 -- 172.21.15.110:0/3492849748 >> 172.21.15.110:0/3492849748 conn(0x7f6e680b7bc0 msgr2=0x7f6e680b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 1 -- 172.21.15.110:0/3492849748 shutdown_connections 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.910+0000 7f6e6ec15640 1 -- 172.21.15.110:0/3492849748 wait complete. 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.911+0000 7f6e6ec15640 10 monclient: build_initial_monmap 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.911+0000 7f6e6ec15640 10 monclient: monmap: 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:33.913 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.911+0000 7f6e6ec15640 1 librados: starting msgr at 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.911+0000 7f6e6ec15640 1 librados: starting objecter 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.911+0000 7f6e6ec15640 1 Processor -- start 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.911+0000 7f6e6ec15640 1 -- start start 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.911+0000 7f6e6ec15640 1 librados: setting wanted keys 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.911+0000 7f6e6ec15640 1 librados: calling monclient init 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.911+0000 7f6e6ec15640 10 monclient: init 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e680a96f0 0x7f6e6813d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient(hunting): picked mon.noname-b con 0x7f6e680a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e680a9ad0 0x7f6e6812f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient(hunting): picked mon.noname-c con 0x7f6e680a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e681410a0 0x7f6e68133510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient(hunting): picked mon.noname-a con 0x7f6e681410a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6813efd0 con 0x7f6e681410a0 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68141e20 con 0x7f6e680a96f0 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6813e480 con 0x7f6e680a9ad0 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.912+0000 7f6e6ec15640 10 monclient(hunting): authenticate will time out at 1377.262679s 2024-01-20T11:15:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.925+0000 7f162b7fe640 1 -- 172.21.15.110:0/3680534729 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1634058270 con 0x7f16440a96f0 2024-01-20T11:15:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.925+0000 7f162b7fe640 10 client.4365.objecter ms_dispatch 0x7f1644000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.925+0000 7f162b7fe640 3 client.4365.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.925+0000 7f162b7fe640 3 client.4365.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:33.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.925+0000 7f162b7fe640 20 client.4365.objecter dump_active .. 0 homeless 2024-01-20T11:15:33.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.925+0000 7f162b7fe640 1 -- 172.21.15.110:0/3680534729 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1634053eb0 con 0x7f16440a96f0 2024-01-20T11:15:33.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.925+0000 7f162b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.925+0000 7f162b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 1 -- 172.21.15.110:0/266395249 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00c940 con 0x7f0cbc0a96f0 2024-01-20T11:15:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 10 monclient: dump: 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 1 -- 172.21.15.110:0/266395249 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac00caa0 con 0x7f0cbc0a96f0 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cc0fd0640 5 monclient: authenticate success, global_id 4405 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cc0fd0640 10 monclient: _renew_subs 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cc0fd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc1d7b40 con 0x7f0cbc0a96f0 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 1 -- 172.21.15.110:0/266395249 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00ce10 con 0x7f0cbc0a96f0 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0caaffd640 10 monclient: dump: 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cc0fd0640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cc0fd0640 10 monclient: _renew_subs 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cc0fd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc1da5e0 con 0x7f0cbc0a96f0 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cc0fd0640 1 librados: init done 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.926+0000 7f0cb8d99640 4 set_mon_vals no callback set 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.927+0000 7f0cc0fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.927+0000 7f0cc0fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.927+0000 7f0cc0fd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.927+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc1d72a0 con 0x7f0cbc0a96f0 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 1 -- 172.21.15.110:0/128006686 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c400cba0 con 0x7f40cc13ca40 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 10 monclient: dump: 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.981 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 10 monclient: _finish_auth 0 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 10 monclient: _check_auth_tickets 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 1 -- 172.21.15.110:0/128006686 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c400cd20 con 0x7f40cc13ca40 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 1 -- 172.21.15.110:0/128006686 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c4014680 con 0x7f40cc13ca40 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40b3fff640 10 monclient: dump: 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:33.982 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40d1890640 5 monclient: authenticate success, global_id 4283 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40d1890640 10 monclient: _renew_subs 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40d1890640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40cc144790 con 0x7f40cc13ca40 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40c97fa640 4 set_mon_vals no callback set 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40d1890640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40d1890640 10 monclient: _renew_subs 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40d1890640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40cc144c50 con 0x7f40cc13ca40 2024-01-20T11:15:33.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.980+0000 7f40d1890640 1 librados: init done 2024-01-20T11:15:33.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.981+0000 7f40d1890640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.981+0000 7f40d1890640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:33.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.981+0000 7f40d1890640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:33.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.981+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40cc075490 con 0x7f40cc13ca40 2024-01-20T11:15:33.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.996+0000 7f071f7fe640 1 -- 172.21.15.110:0/2531179753 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f0720005410 con 0x7f0728142920 2024-01-20T11:15:33.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.997+0000 7f071f7fe640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f070806c8b0 0x7f070806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:33.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.997+0000 7f072659c640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f070806c8b0 0x7f070806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:33.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.997+0000 7f072659c640 10 monclient: get_auth_request con 0x7f070806c8b0 auth_method 0 2024-01-20T11:15:33.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.998+0000 7f071f7fe640 10 client.4280.objecter ms_handle_connect 0x7f070806c8b0 2024-01-20T11:15:33.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:33.998+0000 7f072659c640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f070806c8b0 0x7f070806ed70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0718006fd0 tx=0x7f0718006bc0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:34.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.007+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:34.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.007+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:34.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.007+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:34.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.007+0000 7f128effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:34.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.036+0000 7f071f7fe640 1 -- 172.21.15.110:0/2531179753 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f07200585c0 con 0x7f0728142920 2024-01-20T11:15:34.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.036+0000 7f071f7fe640 10 client.4280.objecter ms_dispatch 0x7f0728000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:34.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.036+0000 7f071f7fe640 3 client.4280.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:34.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.036+0000 7f071f7fe640 3 client.4280.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:34.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.036+0000 7f071f7fe640 20 client.4280.objecter dump_active .. 0 homeless 2024-01-20T11:15:34.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.036+0000 7f071f7fe640 1 -- 172.21.15.110:0/2531179753 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0720054200 con 0x7f0728142920 2024-01-20T11:15:34.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.036+0000 7f071f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:34.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.036+0000 7f071f7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:34.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.187+0000 7fd8814cf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-20T11:15:34.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.187+0000 7fd8814cf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-20T11:15:34.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.187+0000 7fd8814cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:34.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.187+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd87c067c50 con 0x7fd87c143fc0 2024-01-20T11:15:34.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.200+0000 7f6e6c9b1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e680a96f0 0x7f6e6813d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:34.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.201+0000 7f0caaffd640 1 -- 172.21.15.110:0/266395249 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f0cac013470 con 0x7f0cbc0a96f0 2024-01-20T11:15:34.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.202+0000 7f0caaffd640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0c9c06c8e0 0x7f0c9c06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:34.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.202+0000 7f0cb9d9b640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0c9c06c8e0 0x7f0c9c06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:34.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.202+0000 7f0cb9d9b640 10 monclient: get_auth_request con 0x7f0c9c06c8e0 auth_method 0 2024-01-20T11:15:34.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.203+0000 7f0caaffd640 10 client.4405.objecter ms_handle_connect 0x7f0c9c06c8e0 2024-01-20T11:15:34.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.203+0000 7f0cb9d9b640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0c9c06c8e0 0x7f0c9c06eda0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0cb002b880 tx=0x7f0cb0008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:34.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.204+0000 7f0caaffd640 1 -- 172.21.15.110:0/266395249 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0cac059240 con 0x7f0cbc0a96f0 2024-01-20T11:15:34.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.204+0000 7f0caaffd640 10 client.4405.objecter ms_dispatch 0x7f0cbc000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:34.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.204+0000 7f0caaffd640 3 client.4405.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:34.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.204+0000 7f0caaffd640 3 client.4405.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:34.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.204+0000 7f0caaffd640 20 client.4405.objecter dump_active .. 0 homeless 2024-01-20T11:15:34.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.214+0000 7f6e6c9b1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e680a96f0 0x7f6e6813d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:58348/0 (socket says 172.21.15.110:58348) 2024-01-20T11:15:34.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.214+0000 7f6e6c9b1640 1 -- 172.21.15.110:0/1369892941 learned_addr learned my addr 172.21.15.110:0/1369892941 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:34.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.214+0000 7f6e6c9b1640 10 monclient(hunting): get_auth_request con 0x7f6e680a96f0 auth_method 0 2024-01-20T11:15:34.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.214+0000 7f6e6c9b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:34.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.214+0000 7f6e6c9b1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:34.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.214+0000 7f6e6c9b1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:34.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.214+0000 7f0caaffd640 1 -- 172.21.15.110:0/266395249 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0cac054e80 con 0x7f0cbc0a96f0 2024-01-20T11:15:34.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.214+0000 7f0caaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:34.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.214+0000 7f0caaffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:34.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.234+0000 7f072c839640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-20T11:15:34.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.234+0000 7f072c839640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-20T11:15:34.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.234+0000 7f072c839640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:34.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.234+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f07281c8740 con 0x7f0728142920 2024-01-20T11:15:34.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.245+0000 7f16497c0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-20T11:15:34.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.245+0000 7f16497c0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-20T11:15:34.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.245+0000 7f16497c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:34.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.245+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f16440af820 con 0x7f16440a96f0 2024-01-20T11:15:34.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.256+0000 7f6e6c9b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:34.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.256+0000 7f6e6c9b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:34.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.256+0000 7f6e6c9b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:34.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-01-20T11:15:34.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 1 -- 172.21.15.110:0/1369892941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e680a9ad0 msgr2=0x7f6e6812f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e680a9ad0 0x7f6e6812f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 1 -- 172.21.15.110:0/1369892941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e681410a0 msgr2=0x7f6e68133510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e681410a0 0x7f6e68133510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 10 monclient: _finish_hunting 0 2024-01-20T11:15:34.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 1 monclient: found mon.noname-b 2024-01-20T11:15:34.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:34.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:15:34.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.266+0000 7f6e6c9b1640 1 -- 172.21.15.110:0/1369892941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e540094d0 con 0x7f6e680a96f0 2024-01-20T11:15:34.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.270+0000 7fd8637fe640 1 -- 172.21.15.110:0/401005217 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd8740582d0 con 0x7fd87c143fc0 2024-01-20T11:15:34.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.270+0000 7fd8637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-20T11:15:34.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.270+0000 7fd8637fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:34.271 INFO:teuthology.orchestra.run.smithi110.stdout:90194313222 2024-01-20T11:15:34.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 10 librados: watch_flush enter 2024-01-20T11:15:34.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 10 librados: watch_flush exit 2024-01-20T11:15:34.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 20 client.4399.objecter shutdown clearing up homeless session... 2024-01-20T11:15:34.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 10 client.4399.objecter successfully canceled tick 2024-01-20T11:15:34.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd86806c8b0 msgr2=0x7fd86806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd86806c8b0 0x7fd86806ed70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd86c036000 tx=0x7fd86c030000 comp rx=0 tx=0).stop 2024-01-20T11:15:34.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 10 monclient: shutdown 2024-01-20T11:15:34.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd87c143fc0 msgr2=0x7fd87c06ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd87c143fc0 0x7fd87c06ce00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd87400c700 tx=0x7fd87400cbc0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 shutdown_connections 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd86806c8b0 0x7fd86806ed70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.272+0000 7fd8814cf640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd87c143fc0 0x7fd87c06ce00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.273+0000 7fd8814cf640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87c143540 0x7fd87c1d70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.273+0000 7fd8814cf640 1 --2- 172.21.15.110:0/401005217 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87c070c80 0x7fd87c140620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.273+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 >> 172.21.15.110:0/401005217 conn(0x7fd87c0b5f00 msgr2=0x7fd87c13bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.273+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 shutdown_connections 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.273+0000 7fd8814cf640 1 -- 172.21.15.110:0/401005217 wait complete. 2024-01-20T11:15:34.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.273+0000 7fd8814cf640 1 librados: shutdown 2024-01-20T11:15:34.283 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313223 got 90194313222 for osd.7 2024-01-20T11:15:34.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.295+0000 7f6e6c9b1640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e680a96f0 0x7f6e6813d1d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00b4a0 tx=0x7f6e5c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:34.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.295+0000 7f6e657fa640 10 client.?.objecter ms_handle_connect 0x7f6e680a96f0 2024-01-20T11:15:34.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.295+0000 7f6e657fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:34.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.298+0000 7f6e657fa640 1 -- 172.21.15.110:0/1369892941 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c010610 con 0x7f6e680a96f0 2024-01-20T11:15:34.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:34.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:15:34.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 10 monclient: dump: 2024-01-20T11:15:34.299 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:34.299 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:34.299 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:34.299 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:34.299 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 10 monclient: _finish_auth 0 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 10 monclient: _check_auth_tickets 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 1 -- 172.21.15.110:0/1369892941 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c010770 con 0x7f6e680a96f0 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 1 -- 172.21.15.110:0/1369892941 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c010b20 con 0x7f6e680a96f0 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:15:34.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e657fa640 10 monclient: dump: 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 5 monclient: authenticate success, global_id 4408 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 10 monclient: _renew_subs 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:34.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6813e760 con 0x7f6e680a96f0 2024-01-20T11:15:34.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e66ffd640 4 set_mon_vals no callback set 2024-01-20T11:15:34.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:34.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 10 monclient: _renew_subs 2024-01-20T11:15:34.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:34.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6813eb80 con 0x7f6e680a96f0 2024-01-20T11:15:34.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 1 librados: init done 2024-01-20T11:15:34.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.299+0000 7f6e6ec15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:34.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.300+0000 7f6e6ec15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:34.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.300+0000 7f6e6ec15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:34.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.300+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6813ee30 con 0x7f6e680a96f0 2024-01-20T11:15:34.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.318+0000 7f6e657fa640 1 -- 172.21.15.110:0/1369892941 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6e5c010c80 con 0x7f6e680a96f0 2024-01-20T11:15:34.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.318+0000 7f6e657fa640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6e4c06c8b0 0x7f6e4c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:34.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.318+0000 7f6e67fff640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6e4c06c8b0 0x7f6e4c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:34.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.318+0000 7f6e67fff640 10 monclient: get_auth_request con 0x7f6e4c06c8b0 auth_method 0 2024-01-20T11:15:34.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.319+0000 7f6e657fa640 10 client.4408.objecter ms_handle_connect 0x7f6e4c06c8b0 2024-01-20T11:15:34.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.319+0000 7f6e67fff640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6e4c06c8b0 0x7f6e4c06ed70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6e54036000 tx=0x7f6e54030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:34.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.321+0000 7f6e657fa640 1 -- 172.21.15.110:0/1369892941 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6e5c0587e0 con 0x7f6e680a96f0 2024-01-20T11:15:34.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.321+0000 7f6e657fa640 10 client.4408.objecter ms_dispatch 0x7f6e68000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:34.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.321+0000 7f6e657fa640 3 client.4408.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:34.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.321+0000 7f6e657fa640 3 client.4408.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:34.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.321+0000 7f6e657fa640 20 client.4408.objecter dump_active .. 0 homeless 2024-01-20T11:15:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.326+0000 7f606a7fc640 1 -- 172.21.15.110:0/2006676911 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f606401a400 con 0x7f60740a9ad0 2024-01-20T11:15:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.326+0000 7f606a7fc640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f605806c860 0x7f605806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.327+0000 7f6079c83640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f605806c860 0x7f605806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.327+0000 7f6079c83640 10 monclient: get_auth_request con 0x7f605806c860 auth_method 0 2024-01-20T11:15:34.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.327+0000 7f606a7fc640 10 client.4368.objecter ms_handle_connect 0x7f605806c860 2024-01-20T11:15:34.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.327+0000 7f6079c83640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f605806c860 0x7f605806ed20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6070009710 tx=0x7f6070006c90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:34.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.341+0000 7f606a7fc640 1 -- 172.21.15.110:0/2006676911 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6064058940 con 0x7f60740a9ad0 2024-01-20T11:15:34.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.341+0000 7f606a7fc640 10 client.4368.objecter ms_dispatch 0x7f6074000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:34.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.341+0000 7f606a7fc640 3 client.4368.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:34.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.341+0000 7f606a7fc640 3 client.4368.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:34.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.341+0000 7f606a7fc640 20 client.4368.objecter dump_active .. 0 homeless 2024-01-20T11:15:34.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.341+0000 7f606a7fc640 1 -- 172.21.15.110:0/2006676911 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6064054c80 con 0x7f60740a9ad0 2024-01-20T11:15:34.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.341+0000 7f606a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:34.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.341+0000 7f606a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:34.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.419+0000 7f40b3fff640 1 -- 172.21.15.110:0/128006686 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f40c401d3b0 con 0x7f40cc13ca40 2024-01-20T11:15:34.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.420+0000 7f40b3fff640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b806c860 0x7f40b806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:34.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.420+0000 7f40caffd640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b806c860 0x7f40b806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:34.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.420+0000 7f40caffd640 10 monclient: get_auth_request con 0x7f40b806c860 auth_method 0 2024-01-20T11:15:34.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.421+0000 7f40b3fff640 10 client.4283.objecter ms_handle_connect 0x7f40b806c860 2024-01-20T11:15:34.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.421+0000 7f40caffd640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b806c860 0x7f40b806ed20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40bc036000 tx=0x7f40bc030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.434+0000 7f40b3fff640 1 -- 172.21.15.110:0/128006686 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f40c4058d00 con 0x7f40cc13ca40 2024-01-20T11:15:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.434+0000 7f40b3fff640 10 client.4283.objecter ms_dispatch 0x7f40cc000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.434+0000 7f40b3fff640 3 client.4283.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.434+0000 7f40b3fff640 3 client.4283.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.434+0000 7f40b3fff640 20 client.4283.objecter dump_active .. 0 homeless 2024-01-20T11:15:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.434+0000 7f40b3fff640 1 -- 172.21.15.110:0/128006686 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f40c4054940 con 0x7f40cc13ca40 2024-01-20T11:15:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.434+0000 7f40b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.434+0000 7f40b3fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:34.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.462+0000 7f162b7fe640 1 -- 172.21.15.110:0/3680534729 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f16440af820 con 0x7f16440a96f0 2024-01-20T11:15:34.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.462+0000 7f162b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-20T11:15:34.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.462+0000 7f162b7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:34.462 INFO:teuthology.orchestra.run.smithi110.stdout:64424509448 2024-01-20T11:15:34.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 10 librados: watch_flush enter 2024-01-20T11:15:34.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 10 librados: watch_flush exit 2024-01-20T11:15:34.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 20 client.4365.objecter shutdown clearing up homeless session... 2024-01-20T11:15:34.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 10 client.4365.objecter successfully canceled tick 2024-01-20T11:15:34.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f163006c8b0 msgr2=0x7f163006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f163006c8b0 0x7f163006ed70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f16440af0f0 tx=0x7f1638008040 comp rx=0 tx=0).stop 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 10 monclient: shutdown 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16440a96f0 msgr2=0x7f16441d78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16440a96f0 0x7f16441d78d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f163400c410 tx=0x7f163400c8d0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 shutdown_connections 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f163006c8b0 0x7f163006ed70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1644142920 0x7f1644138680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16440a9ad0 0x7f16441d7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 --2- 172.21.15.110:0/3680534729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16440a96f0 0x7f16441d78d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 >> 172.21.15.110:0/3680534729 conn(0x7f16440b7bc0 msgr2=0x7f16440b73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 shutdown_connections 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 -- 172.21.15.110:0/3680534729 wait complete. 2024-01-20T11:15:34.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.465+0000 7f16497c0640 1 librados: shutdown 2024-01-20T11:15:34.476 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.6 2024-01-20T11:15:34.476 DEBUG:teuthology.parallel:result is None 2024-01-20T11:15:34.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.530+0000 7f0cc0fd0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-20T11:15:34.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.530+0000 7f0cc0fd0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-20T11:15:34.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.530+0000 7f0cc0fd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:34.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.530+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0cbc075490 con 0x7f0cbc0a96f0 2024-01-20T11:15:34.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.530+0000 7f607bee7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-20T11:15:34.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.530+0000 7f607bee7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-20T11:15:34.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.530+0000 7f607bee7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:34.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.530+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f60741da780 con 0x7f60740a9ad0 2024-01-20T11:15:34.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.535+0000 7f606a7fc640 1 -- 172.21.15.110:0/2006676911 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f60640582a0 con 0x7f60740a9ad0 2024-01-20T11:15:34.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.535+0000 7f606a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-20T11:15:34.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.535+0000 7f606a7fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:34.535 INFO:teuthology.orchestra.run.smithi110.stdout:73014444038 2024-01-20T11:15:34.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 10 librados: watch_flush enter 2024-01-20T11:15:34.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 10 librados: watch_flush exit 2024-01-20T11:15:34.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 20 client.4368.objecter shutdown clearing up homeless session... 2024-01-20T11:15:34.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 10 client.4368.objecter successfully canceled tick 2024-01-20T11:15:34.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f605806c860 msgr2=0x7f605806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f605806c860 0x7f605806ed20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6070009710 tx=0x7f6070006c90 comp rx=0 tx=0).stop 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 10 monclient: shutdown 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60740a9ad0 msgr2=0x7f6074143550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60740a9ad0 0x7f6074143550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f606400b390 tx=0x7f606400b850 comp rx=0 tx=0).stop 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 shutdown_connections 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f605806c860 0x7f605806ed20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60740acaf0 0x7f60741dab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60740a9ad0 0x7f6074143550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 --2- 172.21.15.110:0/2006676911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60740a96f0 0x7f60740abf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.537+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 >> 172.21.15.110:0/2006676911 conn(0x7f60740b7bc0 msgr2=0x7f60740b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:34.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.538+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 shutdown_connections 2024-01-20T11:15:34.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.538+0000 7f607bee7640 1 -- 172.21.15.110:0/2006676911 wait complete. 2024-01-20T11:15:34.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.538+0000 7f607bee7640 1 librados: shutdown 2024-01-20T11:15:34.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.547+0000 7f071f7fe640 1 -- 172.21.15.110:0/2531179753 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07281c8740 con 0x7f0728142920 2024-01-20T11:15:34.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.547+0000 7f071f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-20T11:15:34.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.547+0000 7f071f7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:34.547 INFO:teuthology.orchestra.run.smithi110.stdout:90194313225 2024-01-20T11:15:34.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444038 got 73014444038 for osd.2 2024-01-20T11:15:34.550 DEBUG:teuthology.parallel:result is None 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 10 librados: watch_flush enter 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 10 librados: watch_flush exit 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 20 client.4280.objecter shutdown clearing up homeless session... 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 10 client.4280.objecter successfully canceled tick 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f070806c8b0 msgr2=0x7f070806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f070806c8b0 0x7f070806ed70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0718006fd0 tx=0x7f0718006bc0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 10 monclient: shutdown 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0728142920 msgr2=0x7f07281d7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0728142920 0x7f07281d7d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f072000c780 tx=0x7f072000cc40 comp rx=0 tx=0).stop 2024-01-20T11:15:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 shutdown_connections 2024-01-20T11:15:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f070806c8b0 0x7f070806ed70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0728142920 0x7f07281d7d30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280a9ad0 0x7f07281d77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 --2- 172.21.15.110:0/2531179753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07280a96f0 0x7f07280b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 >> 172.21.15.110:0/2531179753 conn(0x7f07280b7bc0 msgr2=0x7f072813b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 shutdown_connections 2024-01-20T11:15:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.549+0000 7f072c839640 1 -- 172.21.15.110:0/2531179753 wait complete. 2024-01-20T11:15:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.550+0000 7f072c839640 1 librados: shutdown 2024-01-20T11:15:34.560 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313224 got 90194313225 for osd.3 2024-01-20T11:15:34.560 DEBUG:teuthology.parallel:result is None 2024-01-20T11:15:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.630+0000 7f40d1890640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-20T11:15:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.630+0000 7f40d1890640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-20T11:15:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.630+0000 7f40d1890640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:15:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.630+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f40cc13c700 con 0x7f40cc13ca40 2024-01-20T11:15:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.632+0000 7f6e657fa640 1 -- 172.21.15.110:0/1369892941 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6e5c054420 con 0x7f6e680a96f0 2024-01-20T11:15:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.632+0000 7f6e657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.632+0000 7f6e657fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:34.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.636+0000 7f40b3fff640 1 -- 172.21.15.110:0/128006686 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f40c4019090 con 0x7f40cc13ca40 2024-01-20T11:15:34.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.636+0000 7f40b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-20T11:15:34.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.636+0000 7f40b3fff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:34.636 INFO:teuthology.orchestra.run.smithi110.stdout:73014444039 2024-01-20T11:15:34.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 10 librados: watch_flush enter 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 10 librados: watch_flush exit 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 20 client.4283.objecter shutdown clearing up homeless session... 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 10 client.4283.objecter successfully canceled tick 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b806c860 msgr2=0x7f40b806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b806c860 0x7f40b806ed20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40bc036000 tx=0x7f40bc030000 comp rx=0 tx=0).stop 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 10 monclient: shutdown 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc13ca40 msgr2=0x7f40cc140bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc13ca40 0x7f40cc140bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40c400b9a0 tx=0x7f40c400be60 comp rx=0 tx=0).stop 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 shutdown_connections 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b806c860 0x7f40b806ed20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40cc13ca40 0x7f40cc140bc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40cc0a9ad0 0x7f40cc13acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 --2- 172.21.15.110:0/128006686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40cc0a96f0 0x7f40cc0b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 >> 172.21.15.110:0/128006686 conn(0x7f40cc0b7bc0 msgr2=0x7f40cc1423e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:34.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f0caaffd640 1 -- 172.21.15.110:0/266395249 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0cac021090 con 0x7f0cbc0a96f0 2024-01-20T11:15:34.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f0caaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-20T11:15:34.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f0caaffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:34.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 shutdown_connections 2024-01-20T11:15:34.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 -- 172.21.15.110:0/128006686 wait complete. 2024-01-20T11:15:34.641 INFO:teuthology.orchestra.run.smithi110.stdout:73014444039 2024-01-20T11:15:34.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.638+0000 7f40d1890640 1 librados: shutdown 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 10 librados: watch_flush enter 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 10 librados: watch_flush exit 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 20 client.4405.objecter shutdown clearing up homeless session... 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 10 client.4405.objecter successfully canceled tick 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0c9c06c8e0 msgr2=0x7f0c9c06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0c9c06c8e0 0x7f0c9c06eda0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0cb002b880 tx=0x7f0cb0008030 comp rx=0 tx=0).stop 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 10 monclient: shutdown 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cbc0a96f0 msgr2=0x7f0cbc13b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cbc0a96f0 0x7f0cbc13b170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00e850 tx=0x7f0cac00ed10 comp rx=0 tx=0).stop 2024-01-20T11:15:34.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 shutdown_connections 2024-01-20T11:15:34.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0c9c06c8e0 0x7f0c9c06eda0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cbc13a560 0x7f0cbc1d7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0cbc0a9ad0 0x7f0cbc13b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 --2- 172.21.15.110:0/266395249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cbc0a96f0 0x7f0cbc13b170 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 >> 172.21.15.110:0/266395249 conn(0x7f0cbc0b7bc0 msgr2=0x7f0cbc0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:34.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 shutdown_connections 2024-01-20T11:15:34.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 -- 172.21.15.110:0/266395249 wait complete. 2024-01-20T11:15:34.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.641+0000 7f0cc0fd0640 1 librados: shutdown 2024-01-20T11:15:34.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444039 got 73014444039 for osd.1 2024-01-20T11:15:34.652 DEBUG:teuthology.parallel:result is None 2024-01-20T11:15:34.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444039 got 73014444039 for osd.0 2024-01-20T11:15:34.653 DEBUG:teuthology.parallel:result is None 2024-01-20T11:15:34.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.813+0000 7f6e6ec15640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-20T11:15:34.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.813+0000 7f6e6ec15640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-20T11:15:34.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.813+0000 7f6e6ec15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:34.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.813+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6e6813ee30 con 0x7f6e680a96f0 2024-01-20T11:15:34.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.819+0000 7f6e657fa640 1 -- 172.21.15.110:0/1369892941 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6e6813ee30 con 0x7f6e680a96f0 2024-01-20T11:15:34.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.819+0000 7f6e657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-20T11:15:34.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.819+0000 7f6e657fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:34.819 INFO:teuthology.orchestra.run.smithi110.stdout:94489280521 2024-01-20T11:15:34.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 10 librados: watch_flush enter 2024-01-20T11:15:34.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 10 librados: watch_flush exit 2024-01-20T11:15:34.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 20 client.4408.objecter shutdown clearing up homeless session... 2024-01-20T11:15:34.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 10 client.4408.objecter successfully canceled tick 2024-01-20T11:15:34.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6e4c06c8b0 msgr2=0x7f6e4c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6e4c06c8b0 0x7f6e4c06ed70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6e54036000 tx=0x7f6e54030000 comp rx=0 tx=0).stop 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 10 monclient: shutdown 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e680a96f0 msgr2=0x7f6e6813d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.821+0000 7f6e6ec15640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e680a96f0 0x7f6e6813d1d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00b4a0 tx=0x7f6e5c00b960 comp rx=0 tx=0).stop 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.822+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 shutdown_connections 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.822+0000 7f6e6ec15640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6e4c06c8b0 0x7f6e4c06ed70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.822+0000 7f6e6ec15640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e681410a0 0x7f6e68133510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.822+0000 7f6e6ec15640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e680a9ad0 0x7f6e6812f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.822+0000 7f6e6ec15640 1 --2- 172.21.15.110:0/1369892941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e680a96f0 0x7f6e6813d1d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.822+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 >> 172.21.15.110:0/1369892941 conn(0x7f6e680b7bc0 msgr2=0x7f6e681373a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.822+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 shutdown_connections 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.822+0000 7f6e6ec15640 1 -- 172.21.15.110:0/1369892941 wait complete. 2024-01-20T11:15:34.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:34.822+0000 7f6e6ec15640 1 librados: shutdown 2024-01-20T11:15:34.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280521 got 94489280521 for osd.4 2024-01-20T11:15:34.832 DEBUG:teuthology.parallel:result is None 2024-01-20T11:15:35.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.007+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:35.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.007+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:35.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.007+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:35.283 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.400+0000 7f5764415640 10 monclient: get_monmap_and_config 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.400+0000 7f5764415640 10 monclient: build_initial_monmap 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.400+0000 7f5764415640 10 monclient: monmap: 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:15:35.400822+0000 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:15:35.400822+0000 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:15:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:15:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:15:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:15:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:15:35.401 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.400+0000 7f5764415640 1 Processor -- start 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.400+0000 7f5764415640 1 -- start start 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.400+0000 7f5764415640 10 monclient: init 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f575c0a96f0 0x7f575c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient(hunting): picked mon.noname-c con 0x7f575c0a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f575c13fa10 0x7f575c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient(hunting): picked mon.noname-a con 0x7f575c13fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f575c0a9ad0 0x7f575c142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient(hunting): picked mon.noname-b con 0x7f575c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c0aaea0 con 0x7f575c13fa10 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c0b25c0 con 0x7f575c0a96f0 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0ab0a0 con 0x7f575c0a9ad0 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.401+0000 7f5764415640 10 monclient(hunting): authenticate will time out at 1378.751764s 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.402+0000 7f57619b0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f575c13fa10 0x7f575c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.402+0000 7f57619b0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f575c13fa10 0x7f575c1423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48740/0 (socket says 172.21.15.110:48740) 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.402+0000 7f57619b0640 1 -- 172.21.15.110:0/3484643791 learned_addr learned my addr 172.21.15.110:0/3484643791 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.402+0000 7f57619b0640 10 monclient(hunting): get_auth_request con 0x7f575c13fa10 auth_method 0 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.402+0000 7f57619b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.402+0000 7f57619b0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.402+0000 7f57619b0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.403+0000 7f57619b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.403+0000 7f57619b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.403+0000 7f57619b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2024-01-20T11:15:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 1 -- 172.21.15.110:0/3484643791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f575c0a9ad0 msgr2=0x7f575c142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:35.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 1 --2- 172.21.15.110:0/3484643791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f575c0a9ad0 0x7f575c142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 1 -- 172.21.15.110:0/3484643791 >> v2:172.21.15.134:3300/0 conn(0x7f575c0a96f0 msgr2=0x7f575c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:15:35.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 1 --2- 172.21.15.110:0/3484643791 >> v2:172.21.15.134:3300/0 conn(0x7f575c0a96f0 0x7f575c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 10 monclient: _finish_hunting 0 2024-01-20T11:15:35.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 1 monclient: found mon.noname-a 2024-01-20T11:15:35.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:35.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:35.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.404+0000 7f57619b0640 1 -- 172.21.15.110:0/3484643791 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0b3fb0 con 0x7f575c13fa10 2024-01-20T11:15:35.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.406+0000 7f57619b0640 1 --2- 172.21.15.110:0/3484643791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c13fa10 0x7f575c1423e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f575c0b5720 tx=0x7f575802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee937a75ee4f8ef7 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:35.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 1 -- 172.21.15.110:0/3484643791 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580317c0 con 0x7f575c13fa10 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 10 monclient: dump: 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 10 monclient: _finish_auth 0 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 10 monclient: _check_auth_tickets 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 1 -- 172.21.15.110:0/3484643791 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5758031920 con 0x7f575c13fa10 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f5764415640 5 monclient: authenticate success, global_id 4374 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f5764415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f5764415640 10 monclient: get_monmap_and_config success 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 1 -- 172.21.15.110:0/3484643791 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758031cb0 con 0x7f575c13fa10 2024-01-20T11:15:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f5764415640 4 set_mon_vals no callback set 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.410+0000 7f57611af640 10 monclient: dump: 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 10 monclient: shutdown 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 1 -- 172.21.15.110:0/3484643791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c13fa10 msgr2=0x7f575c1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 1 --2- 172.21.15.110:0/3484643791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c13fa10 0x7f575c1423e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f575c0b5720 tx=0x7f575802b860 comp rx=0 tx=0).stop 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 1 -- 172.21.15.110:0/3484643791 shutdown_connections 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 1 --2- 172.21.15.110:0/3484643791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f575c0a9ad0 0x7f575c142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 1 --2- 172.21.15.110:0/3484643791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c13fa10 0x7f575c1423e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 1 --2- 172.21.15.110:0/3484643791 >> v2:172.21.15.134:3300/0 conn(0x7f575c0a96f0 0x7f575c13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 1 -- 172.21.15.110:0/3484643791 >> 172.21.15.110:0/3484643791 conn(0x7f575c0b7bc0 msgr2=0x7f575c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 1 -- 172.21.15.110:0/3484643791 shutdown_connections 2024-01-20T11:15:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.411+0000 7f5764415640 1 -- 172.21.15.110:0/3484643791 wait complete. 2024-01-20T11:15:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.412+0000 7f5764415640 10 monclient: build_initial_monmap 2024-01-20T11:15:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.412+0000 7f5764415640 10 monclient: monmap: 2024-01-20T11:15:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:15:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:15:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:15:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:15:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.412+0000 7f5764415640 1 librados: starting msgr at 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.412+0000 7f5764415640 1 librados: starting objecter 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.412+0000 7f5764415640 1 Processor -- start 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 1 -- start start 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 1 librados: setting wanted keys 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 1 librados: calling monclient init 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient: init 2024-01-20T11:15:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient: _reopen_session rank -1 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f575c0a96f0 0x7f575c1d7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient(hunting): picked mon.noname-c con 0x7f575c0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f575c0a9ad0 0x7f575c1d7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient(hunting): picked mon.noname-b con 0x7f575c0a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1cbd60 0x7f575c1cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient(hunting): picked mon.noname-a con 0x7f575c1cbd60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0ab930 con 0x7f575c1cbd60 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0abb30 con 0x7f575c0a9ad0 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1d7360 con 0x7f575c0a96f0 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient(hunting): _renew_subs 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.413+0000 7f5764415640 10 monclient(hunting): authenticate will time out at 1378.763780s 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1cbd60 0x7f575c1cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1cbd60 0x7f575c1cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48744/0 (socket says 172.21.15.110:48744) 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 1 -- 172.21.15.110:0/309752555 learned_addr learned my addr 172.21.15.110:0/309752555 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 10 monclient(hunting): get_auth_request con 0x7f575c1cbd60 auth_method 0 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:15:35.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.414+0000 7f57629b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 1 -- 172.21.15.110:0/309752555 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f575c0a96f0 msgr2=0x7f575c1d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f575c0a96f0 0x7f575c1d7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 1 -- 172.21.15.110:0/309752555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f575c0a9ad0 msgr2=0x7f575c1d7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f575c0a9ad0 0x7f575c1d7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 10 monclient: _finish_hunting 0 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 1 monclient: found mon.noname-a 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:15:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.417+0000 7f57629b2640 1 -- 172.21.15.110:0/309752555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57580094d0 con 0x7f575c1cbd60 2024-01-20T11:15:35.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.420+0000 7f57629b2640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1cbd60 0x7f575c1cc1f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f575400b970 tx=0x7f575400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:35.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.420+0000 7f5752ffd640 10 client.?.objecter ms_handle_connect 0x7f575c1cbd60 2024-01-20T11:15:35.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.420+0000 7f5752ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 1 -- 172.21.15.110:0/309752555 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400cb40 con 0x7f575c1cbd60 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 10 monclient: dump: 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:35.425 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 10 monclient: _finish_auth 0 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 1 -- 172.21.15.110:0/309752555 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575400ccc0 con 0x7f575c1cbd60 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5764415640 5 monclient: authenticate success, global_id 4377 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 1 -- 172.21.15.110:0/309752555 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5754014680 con 0x7f575c1cbd60 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5764415640 10 monclient: _renew_subs 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5764415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c143740 con 0x7f575c1cbd60 2024-01-20T11:15:35.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f57609ae640 4 set_mon_vals no callback set 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5752ffd640 10 monclient: dump: 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:15:35.427 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:15:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5764415640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5764415640 10 monclient: _renew_subs 2024-01-20T11:15:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5764415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c1cc910 con 0x7f575c1cbd60 2024-01-20T11:15:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.424+0000 7f5764415640 1 librados: init done 2024-01-20T11:15:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.425+0000 7f5764415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.425+0000 7f5764415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.425+0000 7f5764415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.425+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c0b63b0 con 0x7f575c1cbd60 2024-01-20T11:15:35.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.445+0000 7f5752ffd640 1 -- 172.21.15.110:0/309752555 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f575401d3b0 con 0x7f575c1cbd60 2024-01-20T11:15:35.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.446+0000 7f5752ffd640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f574006c8e0 0x7f574006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:15:35.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.446+0000 7f57621b1640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f574006c8e0 0x7f574006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:15:35.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.446+0000 7f57621b1640 10 monclient: get_auth_request con 0x7f574006c8e0 auth_method 0 2024-01-20T11:15:35.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.447+0000 7f57621b1640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f574006c8e0 0x7f574006eda0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f574c009710 tx=0x7f574c006c90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:15:35.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.447+0000 7f5752ffd640 10 client.4377.objecter ms_handle_connect 0x7f574006c8e0 2024-01-20T11:15:35.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.686+0000 7f5752ffd640 1 -- 172.21.15.110:0/309752555 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f575400ce30 con 0x7f575c1cbd60 2024-01-20T11:15:35.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.686+0000 7f5752ffd640 10 client.4377.objecter ms_dispatch 0x7f575c000b90 osd_map(26..26 src has 1..26) v4 2024-01-20T11:15:35.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.686+0000 7f5752ffd640 3 client.4377.objecter handle_osd_map got epochs [26,26] > 0 2024-01-20T11:15:35.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.686+0000 7f5752ffd640 3 client.4377.objecter handle_osd_map decoding full epoch 26 2024-01-20T11:15:35.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.687+0000 7f5752ffd640 20 client.4377.objecter dump_active .. 0 homeless 2024-01-20T11:15:35.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.687+0000 7f5752ffd640 1 -- 172.21.15.110:0/309752555 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f57540548e0 con 0x7f575c1cbd60 2024-01-20T11:15:35.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.687+0000 7f5752ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:15:35.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.687+0000 7f5752ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:15:35.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.902+0000 7f5764415640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-20T11:15:35.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.902+0000 7f5764415640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-20T11:15:35.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.902+0000 7f5764415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:15:35.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.902+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f575c075490 con 0x7f575c1cbd60 2024-01-20T11:15:35.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.906+0000 7f5752ffd640 1 -- 172.21.15.110:0/309752555 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5754019020 con 0x7f575c1cbd60 2024-01-20T11:15:35.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.906+0000 7f5752ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-20T11:15:35.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.906+0000 7f5752ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:15:35.906 INFO:teuthology.orchestra.run.smithi110.stdout:90194313224 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 10 librados: watch_flush enter 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 10 librados: watch_flush exit 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 20 client.4377.objecter shutdown clearing up homeless session... 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 10 client.4377.objecter successfully canceled tick 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f574006c8e0 msgr2=0x7f574006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f574006c8e0 0x7f574006eda0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f574c009710 tx=0x7f574c006c90 comp rx=0 tx=0).stop 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 10 monclient: shutdown 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1cbd60 msgr2=0x7f575c1cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1cbd60 0x7f575c1cc1f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f575400b970 tx=0x7f575400be30 comp rx=0 tx=0).stop 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 shutdown_connections 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f574006c8e0 0x7f574006eda0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1cbd60 0x7f575c1cc1f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f575c0a9ad0 0x7f575c1d7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 --2- 172.21.15.110:0/309752555 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f575c0a96f0 0x7f575c1d7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:15:35.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 >> 172.21.15.110:0/309752555 conn(0x7f575c0b7bc0 msgr2=0x7f575c13b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:15:35.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 shutdown_connections 2024-01-20T11:15:35.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 -- 172.21.15.110:0/309752555 wait complete. 2024-01-20T11:15:35.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:35.908+0000 7f5764415640 1 librados: shutdown 2024-01-20T11:15:35.919 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313223 got 90194313224 for osd.7 2024-01-20T11:15:35.919 DEBUG:teuthology.parallel:result is None 2024-01-20T11:15:36.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:36.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:36.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:36.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.064+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:15:36.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.064+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:15:36.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.064+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:36.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.064+0000 7f12a4ff9640 10 monclient: _renew_subs 2024-01-20T11:15:36.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.064+0000 7f12a4ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:15:36.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.064+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=27}) v3 -- 0x7f1284004e60 con 0x7f12a80ad6b0 2024-01-20T11:15:36.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:36.064+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f12840051f0 con 0x7f1294072ca0 2024-01-20T11:15:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:37.007+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:37.007+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:37.007+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:38.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:38.007+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:38.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:38.007+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:38.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:38.007+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:39.007+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:39.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:39.007+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:39.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:39.007+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:40.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:40.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:40.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:40.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:40.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:40.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:41.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:41.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:41.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:41.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:41.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:41.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:41.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:41.065+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:15:41.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:41.065+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:15:41.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:41.065+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:41.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:41.065+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284005360 con 0x7f1294072ca0 2024-01-20T11:15:41.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:41.105+0000 7f12a7fff640 1 -- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 msgr2=0x7f12940750c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6818/1586889676, fault. 2024-01-20T11:15:41.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:41.105+0000 7f12a7fff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 0x7f12940750c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:15:42.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:42.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:42.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:42.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:42.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:42.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:43.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:43.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:43.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:43.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:43.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:43.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:44.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:44.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:44.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:44.008+0000 7f128effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:45.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:45.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:45.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:45.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:45.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:45.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:46.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:46.009+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:46.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:46.009+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:46.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:46.009+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:46.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:46.066+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:15:46.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:46.066+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:15:46.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:46.066+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:46.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:46.066+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f12840054d0 con 0x7f1294072ca0 2024-01-20T11:15:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:47.009+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:47.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:47.009+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:47.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:47.009+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:48.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:48.009+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:48.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:48.009+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:48.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:48.009+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:49.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:49.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:49.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:50.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:50.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:50.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:50.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:50.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:50.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:51.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:51.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:51.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:51.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:51.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:51.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:51.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:51.066+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:15:51.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:51.066+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:15:51.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:51.066+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:51.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:51.066+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284005640 con 0x7f1294072ca0 2024-01-20T11:15:51.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:51.314+0000 7f12a7fff640 1 -- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 msgr2=0x7f12940750c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6818/1586889676, fault. 2024-01-20T11:15:51.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:51.314+0000 7f12a7fff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 0x7f12940750c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:15:52.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:52.009+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:52.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:52.009+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:52.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:52.009+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:53.008 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:44.033 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:53.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:53.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:53.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:53.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:53.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:53.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:53.011 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:44.036 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:53.044 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:44.069 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:53.057 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:01:44.063 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:53.058 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:01:44.065 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:53.063 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:44.087 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:53.066 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:44.090 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:53.091 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:01:44.098 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:53.106 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:44.131 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:15:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:54.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:54.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:54.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:54.008+0000 7f128effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:15:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:55.008+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:55.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:55.008+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:55.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:55.008+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:56.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:56.009+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:56.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:56.009+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:56.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:56.009+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:56.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:56.067+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:15:56.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:56.067+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:15:56.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:56.067+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:15:56.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:56.067+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f12840057b0 con 0x7f1294072ca0 2024-01-20T11:15:57.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:57.009+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:57.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:57.009+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:57.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:57.009+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:58.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:58.009+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:58.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:58.009+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:58.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:58.009+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:15:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:59.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:15:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:59.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:15:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:15:59.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:00.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:00.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:00.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:00.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:00.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:00.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:01.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:01.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:01.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:01.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:01.068+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:01.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:01.068+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:01.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:01.068+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:01.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:01.068+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284005920 con 0x7f1294072ca0 2024-01-20T11:16:01.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:01.725+0000 7f12a7fff640 1 -- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 msgr2=0x7f12940750c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6818/1586889676, fault. 2024-01-20T11:16:01.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:01.725+0000 7f12a7fff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 0x7f12940750c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:16:02.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:02.009+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:02.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:02.009+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:02.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:02.009+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:03.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:03.009+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:03.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:03.009+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:03.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:03.009+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:04.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:04.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:04.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:04.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:04.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:04.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:04.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:04.010+0000 7f128effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:05.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:05.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:05.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:05.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:05.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:05.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:06.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:06.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:06.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:06.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:06.069+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:06.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:06.069+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:06.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:06.069+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:06.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:06.069+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284005a90 con 0x7f1294072ca0 2024-01-20T11:16:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:07.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:07.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:07.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:08.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:08.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:08.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:08.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:08.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:08.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:09.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:09.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:09.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:09.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:09.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:09.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:10.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:10.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:10.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:10.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:10.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:10.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:11.010+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:11.010+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:11.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:11.010+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:11.070+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:11.070+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:11.070+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:11.070+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284005c00 con 0x7f1294072ca0 2024-01-20T11:16:11.897 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:11.892+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:12.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:12.011+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:12.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:12.011+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:12.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:12.011+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:12.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:12.537+0000 7f12a7fff640 1 -- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 msgr2=0x7f12940750c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6818/1586889676, fault. 2024-01-20T11:16:12.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:12.537+0000 7f12a7fff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 0x7f12940750c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:16:12.850 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:12.850+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:13.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:13.011+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:13.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:13.011+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:13.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:13.011+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:13.081 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:04.105 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:13.088 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:04.112 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:13.105 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:04.130 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:13.130 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:04.154 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:13.133 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:04.157 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:13.135 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:02:04.141 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:13.136 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:02:04.142 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:13.150 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:02:04.156 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:13.152 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:04.176 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:13.807 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:13.806+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:14.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:14.011+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:14.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:14.011+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:14.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:14.011+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:14.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:14.011+0000 7f128effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:14.780 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:14.779+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:15.011+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:15.011+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:15.011+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:15.836 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:15.834+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:16.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:16.012+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:16.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:16.012+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:16.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:16.012+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:16.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:16.071+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:16.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:16.071+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:16.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:16.071+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:16.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:16.071+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284005d70 con 0x7f1294072ca0 2024-01-20T11:16:16.866 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:16.865+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:17.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:17.012+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:17.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:17.012+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:17.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:17.012+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:17.826 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:17.824+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:18.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:18.012+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:18.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:18.012+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:18.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:18.012+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:18.877 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:18.875+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:19.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:19.012+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:19.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:19.012+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:19.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:19.012+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:19.843 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:19.841+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:20.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:20.012+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:20.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:20.012+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:20.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:20.012+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:20.802 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:20.801+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:21.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:21.012+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:21.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:21.012+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:21.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:21.012+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:21.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:21.072+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:21.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:21.072+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:21.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:21.072+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:21.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:21.072+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284005ee0 con 0x7f1294072ca0 2024-01-20T11:16:21.843 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:21.841+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:22.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:22.013+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:22.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:22.013+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:22.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:22.013+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:22.857 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:22.856+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:23.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:23.013+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:23.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:23.013+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:23.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:23.013+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:23.872 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:23.871+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:24.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:24.013+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:24.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:24.013+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:24.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:24.013+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:24.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:24.013+0000 7f128effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:24.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:24.141+0000 7f12a7fff640 1 -- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 msgr2=0x7f12940750c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6818/1586889676, fault. 2024-01-20T11:16:24.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:24.141+0000 7f12a7fff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 0x7f12940750c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-20T11:16:24.872 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:24.871+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:25.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:25.013+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:25.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:25.013+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:25.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:25.013+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:25.864 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:25.863+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:26.013+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:26.013+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:26.013+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:26.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:26.073+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:26.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:26.073+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:26.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:26.073+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:26.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:26.073+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284006050 con 0x7f1294072ca0 2024-01-20T11:16:26.884 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:26.883+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:27.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:27.013+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:27.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:27.013+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:27.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:27.013+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:27.895 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:27.893+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:28.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:28.013+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:28.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:28.013+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:28.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:28.013+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:28.937 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:28.935+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:29.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:29.014+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:29.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:29.014+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:29.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:29.014+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:29.894 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:29.893+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:30.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:30.014+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:30.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:30.014+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:30.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:30.014+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:30.907 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:30.906+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:31.014+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:31.014+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:31.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:31.014+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:31.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:31.074+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:31.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:31.074+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:31.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:31.074+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:31.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:31.074+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f12840061c0 con 0x7f1294072ca0 2024-01-20T11:16:31.950 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:31.949+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:32.014+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:32.014+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:32.014+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:32.987 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:32.985+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:33.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:33.014+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:33.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:33.014+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:33.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:33.014+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:33.226 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:24.251 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:33.229 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:24.254 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:33.244 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:24.268 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:33.276 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:02:24.283 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:33.277 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:02:24.284 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:33.279 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:24.303 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:33.285 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:24.309 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:33.291 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:02:24.298 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:33.301 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:24.325 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:34.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:34.014+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:34.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:34.014+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:34.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:34.014+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:34.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:34.014+0000 7f128effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:34.037 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:34.036+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:35.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:35.014+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:35.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:35.014+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:35.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:35.014+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:35.015 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:35.013+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:36.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:36.015+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:36.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:36.015+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:36.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:36.015+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:36.028 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:36.026+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:36.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:36.075+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:36.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:36.075+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:36.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:36.075+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:36.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:36.075+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284006330 con 0x7f1294072ca0 2024-01-20T11:16:37.008 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:37.007+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:37.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:37.015+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:37.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:37.015+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:37.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:37.015+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:37.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:37.354+0000 7f12a7fff640 1 -- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 msgr2=0x7f12940750c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6818/1586889676, fault. 2024-01-20T11:16:37.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:37.354+0000 7f12a7fff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 0x7f12940750c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-01-20T11:16:38.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:38.015+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:38.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:38.015+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:38.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:38.015+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:38.031 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:38.030+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:39.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:39.015+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:39.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:39.015+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:39.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:39.015+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:39.057 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:39.056+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:40.011 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:40.009+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:40.015+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:40.015+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:40.015+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:40.978 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:40.976+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:41.016+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:41.016+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:41.016+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:41.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:41.076+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:41.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:41.076+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:41.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:41.076+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:41.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:41.076+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f12840064a0 con 0x7f1294072ca0 2024-01-20T11:16:41.996 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:41.995+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:42.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:42.016+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:42.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:42.016+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:42.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:42.016+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:42.948 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:42.946+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:43.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:43.016+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:43.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:43.016+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:43.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:43.016+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:43.924 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:43.923+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:44.015+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:44.015+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:44.015+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:44.016+0000 7f128effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:44.968 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:44.967+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:45.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:45.016+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:45.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:45.016+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:45.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:45.016+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:45.987 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:45.985+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:46.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:46.016+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:46.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:46.016+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:46.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:46.016+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:46.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:46.077+0000 7f12a4ff9640 10 client.4357.objecter tick 2024-01-20T11:16:46.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:46.077+0000 7f12a4ff9640 10 client.4357.objecter pinging osd that serves command tid 1 (osd.5) 2024-01-20T11:16:46.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:46.077+0000 7f12a4ff9640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:46.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:46.077+0000 7f12a4ff9640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] -- ping magic: 0 v1 -- 0x7f1284006610 con 0x7f1294072ca0 2024-01-20T11:16:47.005 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:47.003+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:47.016+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:47.016+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:47.016+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:47.992 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:47.992+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:48.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:48.016+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:48.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:48.016+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:48.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:48.016+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:48.313 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:16:48.309+0000 12a58640 -1 osd.3 26 heartbeat_check: no reply from 172.21.15.134:6823 osd.5 since back 2024-01-20T11:15:27.062054+0000 front 2024-01-20T11:15:27.055152+0000 (oldest deadline 2024-01-20T11:16:47.929641+0000) 2024-01-20T11:16:48.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-20T11:16:48.497+0000 ea50640 -1 osd.2 26 heartbeat_check: no reply from 172.21.15.134:6823 osd.5 since back 2024-01-20T11:15:23.722728+0000 front 2024-01-20T11:15:23.706436+0000 (oldest deadline 2024-01-20T11:16:48.399243+0000) 2024-01-20T11:16:48.627 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-20T11:16:48.621+0000 ea50640 -1 osd.0 26 heartbeat_check: no reply from 172.21.15.134:6823 osd.5 since back 2024-01-20T11:15:25.418850+0000 front 2024-01-20T11:15:25.417600+0000 (oldest deadline 2024-01-20T11:16:48.308675+0000) 2024-01-20T11:16:48.722 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-20T11:16:48.714+0000 ea50640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.134:6823 osd.5 since back 2024-01-20T11:15:24.710847+0000 front 2024-01-20T11:15:24.720311+0000 (oldest deadline 2024-01-20T11:16:48.205091+0000) 2024-01-20T11:16:48.853 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:16:48.849+0000 12a58640 -1 osd.4 26 heartbeat_check: no reply from 172.21.15.134:6823 osd.5 since back 2024-01-20T11:15:27.030346+0000 front 2024-01-20T11:15:27.038977+0000 (oldest deadline 2024-01-20T11:16:48.175183+0000) 2024-01-20T11:16:48.967 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:48.966+0000 12a58640 -1 osd.7 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:49.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.016+0000 7f128effd640 10 monclient: tick 2024-01-20T11:16:49.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.016+0000 7f128effd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:49.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.016+0000 7f128effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:49.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 <== mon.1 v2:172.21.15.134:3300/0 7 ==== osd_map(27..27 src has 1..27) v4 ==== 317+0+0 (secure 0 0 0) 0x7f129c07e150 con 0x7f12a80ad6b0 2024-01-20T11:16:49.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 10 client.4357.objecter ms_dispatch 0x7f12a8000b90 osd_map(27..27 src has 1..27) v4 2024-01-20T11:16:49.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 3 client.4357.objecter handle_osd_map got epochs [27,27] > 26 2024-01-20T11:16:49.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 3 client.4357.objecter handle_osd_map decoding incremental epoch 27 2024-01-20T11:16:49.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 10 client.4357.objecter checking command 1 2024-01-20T11:16:49.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 10 client.4357.objecter _check_command_map_dne tid 1 current 27 map_dne_bound 0 2024-01-20T11:16:49.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:49.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=2) v1 -- 0x7f12940759a0 con 0x7f12a80ad6b0 2024-01-20T11:16:49.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 10 client.4357.objecter close_session for osd.5 2024-01-20T11:16:49.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 msgr2=0x7f12940750c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:49.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 1 --2- 172.21.15.110:0/587004102 >> [v2:172.21.15.134:6818/1586889676,v1:172.21.15.134:6819/1586889676] conn(0x7f1294072ca0 0x7f12940750c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 10 client.4357.objecter command_op 1 2024-01-20T11:16:49.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 20 client.4357.objecter put_session s=0x7f12940725d0 osd=5 2 2024-01-20T11:16:49.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 15 client.4357.objecter _session_command_op_remove 5 1 2024-01-20T11:16:49.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 20 client.4357.objecter put_session s=0x7f12940725d0 osd=5 1 2024-01-20T11:16:49.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 15 client.4357.objecter _session_command_op_assign -1 1 2024-01-20T11:16:49.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.235+0000 7f128ffff640 20 client.4357.objecter dump_active .. 1 homeless 2024-01-20T11:16:49.299 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:16:49.294+0000 12a58640 -1 osd.3 26 heartbeat_check: no reply from 172.21.15.134:6823 osd.5 since back 2024-01-20T11:15:27.062054+0000 front 2024-01-20T11:15:27.055152+0000 (oldest deadline 2024-01-20T11:16:47.929641+0000) 2024-01-20T11:16:49.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.330+0000 7f128ffff640 1 -- 172.21.15.110:0/587004102 <== mon.1 v2:172.21.15.134:3300/0 8 ==== mon_get_version_reply(handle=2 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f129c079bf0 con 0x7f12a80ad6b0 2024-01-20T11:16:49.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.330+0000 7f128ffff640 10 monclient: handle_get_version_reply finishing 2 version 27 2024-01-20T11:16:49.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.330+0000 7f12a5ffb640 10 client.4357.objecter _check_command_map_dne tid 1 current 27 map_dne_bound 27 2024-01-20T11:16:49.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.330+0000 7f12a5ffb640 10 client.4357.objecter _finish_command 1 = osd:6 osd down 2024-01-20T11:16:49.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.330+0000 7f12a5ffb640 15 client.4357.objecter _session_command_op_remove -1 1 2024-01-20T11:16:49.331 INFO:teuthology.orchestra.run.smithi110.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-01-20T11:16:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-20T11:16:49.334 INFO:tasks.ceph:ignoring flush pg stats error, probably testing upgrade: Command failed on smithi110 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats' 2024-01-20T11:16:49.334 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-20T11:16:49.334 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:16:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.439+0000 7efd76b7c640 10 monclient: get_monmap_and_config 2024-01-20T11:16:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.439+0000 7efd76b7c640 10 monclient: build_initial_monmap 2024-01-20T11:16:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.439+0000 7efd76b7c640 10 monclient: monmap: 2024-01-20T11:16:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:16:49.440150+0000 2024-01-20T11:16:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:16:49.440150+0000 2024-01-20T11:16:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:16:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:16:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:16:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:16:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:16:49.440 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 1 Processor -- start 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 1 -- start start 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 10 monclient: init 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd700a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd700a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efd70142af0 0x7efd7014d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 10 monclient(hunting): picked mon.noname-c con 0x7efd70142af0 addr v2:172.21.15.134:3300/0 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd700b25c0 con 0x7efd700a96f0 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd70142f50 con 0x7efd700a9ad0 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd700aaea0 con 0x7efd70142af0 2024-01-20T11:16:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.440+0000 7efd76b7c640 10 monclient(hunting): authenticate will time out at 1452.791154s 2024-01-20T11:16:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.441+0000 7efd74918640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7efd700a96f0 0x7efd7013f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59794/0 (socket says 172.21.15.110:59794) 2024-01-20T11:16:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.441+0000 7efd74918640 1 -- 172.21.15.110:0/2587700243 learned_addr learned my addr 172.21.15.110:0/2587700243 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:16:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.441+0000 7efd75119640 1 --2- 172.21.15.110:0/2587700243 >> v2:172.21.15.134:3300/0 conn(0x7efd70142af0 0x7efd7014d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.442+0000 7efd75119640 10 monclient(hunting): get_auth_request con 0x7efd70142af0 auth_method 0 2024-01-20T11:16:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.442+0000 7efd75119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.442+0000 7efd75119640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.442+0000 7efd75119640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:49.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.442+0000 7efd75119640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:49.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.442+0000 7efd75119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.442+0000 7efd75119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd6f7fe640 1 -- 172.21.15.110:0/2587700243 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736311963 0 0) 0x7efd700b25c0 con 0x7efd700a96f0 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd6f7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd6f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd6f7fe640 10 monclient(hunting): my global_id is 4380 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd6f7fe640 1 -- 172.21.15.110:0/2587700243 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd60003520 con 0x7efd700a96f0 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd75119640 10 monclient(hunting): handle_auth_done global_id 4411 payload 274 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd75119640 1 -- 172.21.15.110:0/2587700243 >> v1:172.21.15.134:6789/0 conn(0x7efd700a9ad0 legacy=0x7efd701423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd75119640 1 -- 172.21.15.110:0/2587700243 >> v1:172.21.15.110:6789/0 conn(0x7efd700a96f0 legacy=0x7efd7013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd75119640 10 monclient: _finish_hunting 0 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd75119640 1 monclient: found mon.noname-c 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd75119640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd75119640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:16:49.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.444+0000 7efd75119640 1 -- 172.21.15.110:0/2587700243 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7013fbe0 con 0x7efd70142af0 2024-01-20T11:16:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.449+0000 7efd75119640 1 --2- 172.21.15.110:0/2587700243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd70142af0 0x7efd7014d860 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd68009d90 tx=0x7efd6802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b30b1642ed9b7d36 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 1 -- 172.21.15.110:0/2587700243 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6802cc60 con 0x7efd70142af0 2024-01-20T11:16:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 10 monclient: dump: 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:16:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 1 -- 172.21.15.110:0/2587700243 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6802cdc0 con 0x7efd70142af0 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd76b7c640 5 monclient: authenticate success, global_id 4411 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd76b7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 1 -- 172.21.15.110:0/2587700243 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68034660 con 0x7efd70142af0 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd6f7fe640 10 monclient: dump: 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:49.457 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:49.457 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:49.457 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:49.457 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:49.457 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:49.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd76b7c640 10 monclient: get_monmap_and_config success 2024-01-20T11:16:49.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.454+0000 7efd76b7c640 4 set_mon_vals no callback set 2024-01-20T11:16:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.455+0000 7efd76b7c640 10 monclient: shutdown 2024-01-20T11:16:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.455+0000 7efd76b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.455+0000 7efd76b7c640 1 -- 172.21.15.110:0/2587700243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd70142af0 msgr2=0x7efd7014d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.455+0000 7efd76b7c640 1 --2- 172.21.15.110:0/2587700243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd70142af0 0x7efd7014d860 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd68009d90 tx=0x7efd6802beb0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.455+0000 7efd76b7c640 1 -- 172.21.15.110:0/2587700243 shutdown_connections 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.455+0000 7efd76b7c640 1 --2- 172.21.15.110:0/2587700243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd70142af0 0x7efd7014d860 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.455+0000 7efd76b7c640 1 -- 172.21.15.110:0/2587700243 >> 172.21.15.110:0/2587700243 conn(0x7efd700b7bc0 msgr2=0x7efd700b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.455+0000 7efd76b7c640 1 -- 172.21.15.110:0/2587700243 shutdown_connections 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.455+0000 7efd76b7c640 1 -- 172.21.15.110:0/2587700243 wait complete. 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.456+0000 7efd76b7c640 10 monclient: build_initial_monmap 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.456+0000 7efd76b7c640 10 monclient: monmap: 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.456+0000 7efd76b7c640 1 librados: starting msgr at 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.456+0000 7efd76b7c640 1 librados: starting objecter 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.456+0000 7efd76b7c640 1 Processor -- start 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.456+0000 7efd76b7c640 1 -- start start 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.456+0000 7efd76b7c640 1 librados: setting wanted keys 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.456+0000 7efd76b7c640 1 librados: calling monclient init 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.456+0000 7efd76b7c640 10 monclient: init 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd700a96f0 0x7efd701d77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient(hunting): picked mon.noname-b con 0x7efd700a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:16:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd700a9ad0 0x7efd700ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient(hunting): picked mon.noname-a con 0x7efd700a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd70142af0 0x7efd700ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient(hunting): picked mon.noname-c con 0x7efd70142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd700afad0 con 0x7efd700a9ad0 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701d72b0 con 0x7efd700a96f0 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd700aeb90 con 0x7efd70142af0 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.457+0000 7efd76b7c640 10 monclient(hunting): authenticate will time out at 1452.807794s 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd74918640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd700a96f0 0x7efd701d77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd75119640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd70142af0 0x7efd700ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd74918640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd700a96f0 0x7efd701d77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46648/0 (socket says 172.21.15.110:46648) 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd74918640 1 -- 172.21.15.110:0/3268686887 learned_addr learned my addr 172.21.15.110:0/3268686887 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd74918640 10 monclient(hunting): get_auth_request con 0x7efd700a96f0 auth_method 0 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd74918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd74918640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd74918640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd75119640 10 monclient(hunting): get_auth_request con 0x7efd70142af0 auth_method 0 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd75119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd75119640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd75119640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.458+0000 7efd6ffff640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd700a9ad0 0x7efd700ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd6ffff640 10 monclient(hunting): get_auth_request con 0x7efd700a9ad0 auth_method 0 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd6ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd74918640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd74918640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd74918640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd75119640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd75119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd75119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.459+0000 7efd6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 10 monclient(hunting): handle_auth_done global_id 4417 payload 824 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 1 -- 172.21.15.110:0/3268686887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd70142af0 msgr2=0x7efd700ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd70142af0 0x7efd700ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 1 -- 172.21.15.110:0/3268686887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd700a9ad0 msgr2=0x7efd700ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd700a9ad0 0x7efd700ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 10 monclient: _finish_hunting 0 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 1 monclient: found mon.noname-b 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.462+0000 7efd74918640 1 -- 172.21.15.110:0/3268686887 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68009a70 con 0x7efd700a96f0 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.464+0000 7efd74918640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd700a96f0 0x7efd701d77d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd6400c500 tx=0x7efd6400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.464+0000 7efd6d7fa640 10 client.?.objecter ms_handle_connect 0x7efd700a96f0 2024-01-20T11:16:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.464+0000 7efd6d7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:16:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 1 -- 172.21.15.110:0/3268686887 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64005ce0 con 0x7efd700a96f0 2024-01-20T11:16:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 10 monclient: dump: 2024-01-20T11:16:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 10 monclient: _finish_auth 0 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 1 -- 172.21.15.110:0/3268686887 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd64005e40 con 0x7efd700a96f0 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd76b7c640 5 monclient: authenticate success, global_id 4417 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd76b7c640 10 monclient: _renew_subs 2024-01-20T11:16:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd76b7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd701d7d10 con 0x7efd700a96f0 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 1 -- 172.21.15.110:0/3268686887 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64007330 con 0x7efd700a96f0 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6d7fa640 10 monclient: dump: 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd6effd640 4 set_mon_vals no callback set 2024-01-20T11:16:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd76b7c640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd76b7c640 10 monclient: _renew_subs 2024-01-20T11:16:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd76b7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd700b0440 con 0x7efd700a96f0 2024-01-20T11:16:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.468+0000 7efd76b7c640 1 librados: init done 2024-01-20T11:16:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.469+0000 7efd76b7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.469+0000 7efd76b7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.469+0000 7efd76b7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.469+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd70075490 con 0x7efd700a96f0 2024-01-20T11:16:49.662 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-20T11:16:49.662+0000 ea50640 -1 osd.0 26 heartbeat_check: no reply from 172.21.15.134:6823 osd.5 since back 2024-01-20T11:15:25.418850+0000 front 2024-01-20T11:15:25.417600+0000 (oldest deadline 2024-01-20T11:16:48.308675+0000) 2024-01-20T11:16:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.726+0000 7efd6d7fa640 1 -- 172.21.15.110:0/3268686887 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7efd64007550 con 0x7efd700a96f0 2024-01-20T11:16:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.726+0000 7efd6d7fa640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.727+0000 7efd6ffff640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.727+0000 7efd6ffff640 10 monclient: get_auth_request con 0x7efd5406c8b0 auth_method 0 2024-01-20T11:16:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.727+0000 7efd6d7fa640 10 client.4417.objecter ms_handle_connect 0x7efd5406c8b0 2024-01-20T11:16:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.727+0000 7efd6ffff640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efd700af440 tx=0x7efd5c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:49.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.730+0000 7efd6d7fa640 1 -- 172.21.15.110:0/3268686887 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efd64057df0 con 0x7efd700a96f0 2024-01-20T11:16:49.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.730+0000 7efd6d7fa640 10 client.4417.objecter ms_dispatch 0x7efd70000b90 osd_map(27..27 src has 1..27) v4 2024-01-20T11:16:49.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.730+0000 7efd6d7fa640 3 client.4417.objecter handle_osd_map got epochs [27,27] > 0 2024-01-20T11:16:49.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.730+0000 7efd6d7fa640 3 client.4417.objecter handle_osd_map decoding full epoch 27 2024-01-20T11:16:49.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.730+0000 7efd6d7fa640 20 client.4417.objecter dump_active .. 0 homeless 2024-01-20T11:16:49.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.742+0000 7efd6d7fa640 1 -- 172.21.15.110:0/3268686887 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7efd64054130 con 0x7efd700a96f0 2024-01-20T11:16:49.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.742+0000 7efd6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:49.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.742+0000 7efd6d7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:16:49.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-20T11:16:49.751+0000 ea50640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.134:6823 osd.5 since back 2024-01-20T11:15:24.710847+0000 front 2024-01-20T11:15:24.720311+0000 (oldest deadline 2024-01-20T11:16:48.205091+0000) 2024-01-20T11:16:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.910+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd700afc40 con 0x7efd5406c8b0 2024-01-20T11:16:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.913+0000 7efd6d7fa640 1 -- 172.21.15.110:0/3268686887 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64236 (secure 0 0 0) 0x7efd700afc40 con 0x7efd5406c8b0 2024-01-20T11:16:49.913 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:16:49.914 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 10 librados: watch_flush enter 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 10 librados: watch_flush exit 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 20 client.4417.objecter shutdown clearing up homeless session... 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 10 client.4417.objecter successfully canceled tick 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 msgr2=0x7efd5406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efd700af440 tx=0x7efd5c009040 comp rx=0 tx=0).stop 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 10 monclient: shutdown 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd700a96f0 msgr2=0x7efd701d77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd700a96f0 0x7efd701d77d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd6400c500 tx=0x7efd6400c9c0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 shutdown_connections 2024-01-20T11:16:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd70142af0 0x7efd700ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd700a9ad0 0x7efd700ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 --2- 172.21.15.110:0/3268686887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd700a96f0 0x7efd701d77d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 >> 172.21.15.110:0/3268686887 conn(0x7efd700b7bc0 msgr2=0x7efd700b7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 shutdown_connections 2024-01-20T11:16:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 -- 172.21.15.110:0/3268686887 wait complete. 2024-01-20T11:16:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:49.915+0000 7efd76b7c640 1 librados: shutdown 2024-01-20T11:16:49.926 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-01-20T11:16:49.248569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219040,"kb_used_data":2664,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755680,"statfs":{"total":773094113280,"available":772869816320,"internally_reserved":0,"allocated":2727936,"data_stored":1449689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":131072},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.519683"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.877271+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:15:29.877271+0000","last_peered":"2024-01-20T11:15:29.877271+0000","last_clean":"2024-01-20T11:15:29.877271+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:15:29.877271+0000","last_undegraded":"2024-01-20T11:15:29.877271+0000","last_fullsized":"2024-01-20T11:15:29.877271+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T22:26:49.902059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.356575+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:15:29.356575+0000","last_peered":"2024-01-20T11:15:29.356575+0000","last_clean":"2024-01-20T11:15:29.356575+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:15:29.356575+0000","last_undegraded":"2024-01-20T11:15:29.356575+0000","last_fullsized":"2024-01-20T11:15:29.356575+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T17:21:09.255806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.872586+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:15:29.872586+0000","last_peered":"2024-01-20T11:15:29.872586+0000","last_clean":"2024-01-20T11:15:29.872586+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:15:29.872586+0000","last_undegraded":"2024-01-20T11:15:29.872586+0000","last_fullsized":"2024-01-20T11:15:29.872586+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T14:37:21.843002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-20T11:15:27.266904+0000","last_change":"2024-01-20T11:15:27.266904+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:15:27.266904+0000","last_undegraded":"2024-01-20T11:15:27.266904+0000","last_fullsized":"2024-01-20T11:15:27.266904+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.864214+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:15:29.864214+0000","last_peered":"2024-01-20T11:15:29.864214+0000","last_clean":"2024-01-20T11:15:29.864214+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:15:29.864214+0000","last_undegraded":"2024-01-20T11:15:29.864214+0000","last_fullsized":"2024-01-20T11:15:29.864214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:00:34.021727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3,"reported_epoch":26,"state":"creating+peering","last_fresh":"2024-01-20T11:15:31.002344+0000","last_change":"2024-01-20T11:15:27.887479+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:15:31.002344+0000","last_undegraded":"2024-01-20T11:15:31.002344+0000","last_fullsized":"2024-01-20T11:15:31.002344+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":4,"reported_epoch":26,"state":"creating+activating","last_fresh":"2024-01-20T11:15:29.868562+0000","last_change":"2024-01-20T11:15:28.693467+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:15:29.868562+0000","last_undegraded":"2024-01-20T11:15:29.868562+0000","last_fullsized":"2024-01-20T11:15:29.868562+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.874929+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:15:29.874929+0000","last_peered":"2024-01-20T11:15:29.874929+0000","last_clean":"2024-01-20T11:15:29.874929+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:15:29.874929+0000","last_undegraded":"2024-01-20T11:15:29.874929+0000","last_fullsized":"2024-01-20T11:15:29.874929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:25:31.758358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":120,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-20T11:15:28.772627+0000","last_change":"2024-01-20T11:15:15.736505+0000","last_active":"2024-01-20T11:15:28.772627+0000","last_peered":"2024-01-20T11:15:28.772627+0000","last_clean":"2024-01-20T11:15:28.772627+0000","last_became_active":"2024-01-20T11:15:15.702676+0000","last_became_peered":"2024-01-20T11:15:15.702676+0000","last_unstale":"2024-01-20T11:15:28.772627+0000","last_undegraded":"2024-01-20T11:15:28.772627+0000","last_fullsized":"2024-01-20T11:15:28.772627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:50:44.769291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.1189999999999998},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":9.8699999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.839},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.1890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.265000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":17.045999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.7240000000000002},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":13.534000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.125},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":15.468999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.369},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":14.724}]}]},{"osd":7,"up_from":21,"seq":90194313239,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.5570000000000004},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":13.426}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.6929999999999996},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.5190000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.768000000000001},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.362}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.324},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.132}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.917},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":14.986000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.905000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":17.827999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313240,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":10.226000000000001},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.7869999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9900000000000002},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.561}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":6.875},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.885999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.5969999999999995},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":14.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":14.199},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.163}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":12.888999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.350000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444054,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.597},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":4.8780000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7489999999999997},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.584}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.475},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.412000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.272},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.041}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.169},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.2930000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":7.8680000000000003},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":16.018999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444055,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":643072,"data_stored":480471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":11.217000000000001},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":7.1639999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.0470000000000006},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.026999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.789999999999999},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.5999999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":10.256},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.495999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":12.337999999999999},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":13.667999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.7010000000000005},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.571999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444055,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":17.498999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":9.8089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.431},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.721}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":16.687999999999999},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.9000000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.481999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.8300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":18.321000000000002},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":12.472}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.552},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.647}]}]},{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":8.0670000000000002},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.032}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":11.747999999999999},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":14.138}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.231999999999999},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.129}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.855},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.029}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.673999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.9529999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.401999999999999},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":17.678000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:16:49.927 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:16:50.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.029+0000 7fd89c1f0640 10 monclient: get_monmap_and_config 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.029+0000 7fd89c1f0640 10 monclient: build_initial_monmap 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.029+0000 7fd89c1f0640 10 monclient: monmap: 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:16:50.030492+0000 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:16:50.030492+0000 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:16:50.030 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:16:50.031 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.030+0000 7fd89c1f0640 1 Processor -- start 2024-01-20T11:16:50.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.030+0000 7fd89c1f0640 1 -- start start 2024-01-20T11:16:50.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.030+0000 7fd89c1f0640 10 monclient: init 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd894145760 addr v1:172.21.15.134:6789/0 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd8940a9d80 0x7fd894145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 10 monclient(hunting): picked mon.noname-a con 0x7fd8940a9d80 addr v2:172.21.15.110:3300/0 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd89413f8b0 0x7fd8940b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 10 monclient(hunting): picked mon.noname-b con 0x7fd89413f8b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd894146250 con 0x7fd894145760 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8940ab050 con 0x7fd8940a9d80 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940b20c0 con 0x7fd89413f8b0 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.031+0000 7fd89c1f0640 10 monclient(hunting): authenticate will time out at 1453.381473s 2024-01-20T11:16:50.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.032+0000 7fd89a78d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd89413f8b0 0x7fd8940b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.032+0000 7fd89978b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd8940a9d80 0x7fd894145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.032+0000 7fd89978b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd8940a9d80 0x7fd894145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34234/0 (socket says 172.21.15.110:34234) 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.032+0000 7fd89978b640 1 -- 172.21.15.110:0/3366554481 learned_addr learned my addr 172.21.15.110:0/3366554481 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.032+0000 7fd89978b640 10 monclient(hunting): get_auth_request con 0x7fd8940a9d80 auth_method 0 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.032+0000 7fd89978b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.032+0000 7fd89978b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.032+0000 7fd89978b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.033+0000 7fd89978b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.033+0000 7fd89978b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:50.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.033+0000 7fd89978b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:50.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.033+0000 7fd89a78d640 10 monclient(hunting): get_auth_request con 0x7fd89413f8b0 auth_method 0 2024-01-20T11:16:50.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.033+0000 7fd89a78d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:50.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.033+0000 7fd89a78d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.033+0000 7fd89a78d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.035+0000 7fd89a78d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:50.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.035+0000 7fd89a78d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:50.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.035+0000 7fd89a78d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:50.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.039+0000 7fd89a78d640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-01-20T11:16:50.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.040+0000 7fd89a78d640 1 -- 172.21.15.110:0/3366554481 >> v2:172.21.15.110:3300/0 conn(0x7fd8940a9d80 msgr2=0x7fd894145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:16:50.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.040+0000 7fd89a78d640 1 --2- 172.21.15.110:0/3366554481 >> v2:172.21.15.110:3300/0 conn(0x7fd8940a9d80 0x7fd894145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.040+0000 7fd89a78d640 1 -- 172.21.15.110:0/3366554481 >> v1:172.21.15.134:6789/0 conn(0x7fd894145760 legacy=0x7fd89413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.040+0000 7fd89a78d640 10 monclient: _finish_hunting 0 2024-01-20T11:16:50.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.040+0000 7fd89a78d640 1 monclient: found mon.noname-b 2024-01-20T11:16:50.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.040+0000 7fd89a78d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:50.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.040+0000 7fd89a78d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:16:50.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.040+0000 7fd89a78d640 1 -- 172.21.15.110:0/3366554481 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8940b2bb0 con 0x7fd89413f8b0 2024-01-20T11:16:50.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.042+0000 7fd89a78d640 1 --2- 172.21.15.110:0/3366554481 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd89413f8b0 0x7fd8940b2670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8900097a0 tx=0x7fd89002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb55434e48196d8a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:50.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 1 -- 172.21.15.110:0/3366554481 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd890031830 con 0x7fd89413f8b0 2024-01-20T11:16:50.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:50.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 10 monclient: dump: 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 10 monclient: _finish_auth 0 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 10 monclient: _check_auth_tickets 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 1 -- 172.21.15.110:0/3366554481 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd890031990 con 0x7fd89413f8b0 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd89c1f0640 5 monclient: authenticate success, global_id 4295 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd89c1f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd89c1f0640 10 monclient: get_monmap_and_config success 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 1 -- 172.21.15.110:0/3366554481 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd890031d20 con 0x7fd89413f8b0 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd89c1f0640 4 set_mon_vals no callback set 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.096+0000 7fd898f8a640 10 monclient: dump: 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:50.099 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 10 monclient: shutdown 2024-01-20T11:16:50.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:50.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3366554481 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd89413f8b0 msgr2=0x7fd8940b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 1 --2- 172.21.15.110:0/3366554481 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd89413f8b0 0x7fd8940b2670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8900097a0 tx=0x7fd89002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3366554481 shutdown_connections 2024-01-20T11:16:50.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 1 --2- 172.21.15.110:0/3366554481 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd89413f8b0 0x7fd8940b2670 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 1 --2- 172.21.15.110:0/3366554481 >> v2:172.21.15.110:3300/0 conn(0x7fd8940a9d80 0x7fd894145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3366554481 >> 172.21.15.110:0/3366554481 conn(0x7fd8940663b0 msgr2=0x7fd8940667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3366554481 shutdown_connections 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.097+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3366554481 wait complete. 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.098+0000 7fd89c1f0640 10 monclient: build_initial_monmap 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.098+0000 7fd89c1f0640 10 monclient: monmap: 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:16:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.098+0000 7fd89c1f0640 1 librados: starting msgr at 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.098+0000 7fd89c1f0640 1 librados: starting objecter 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.098+0000 7fd89c1f0640 1 Processor -- start 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.098+0000 7fd89c1f0640 1 -- start start 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.098+0000 7fd89c1f0640 1 librados: setting wanted keys 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.098+0000 7fd89c1f0640 1 librados: calling monclient init 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.098+0000 7fd89c1f0640 10 monclient: init 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8940a9d80 0x7fd894141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient(hunting): picked mon.noname-c con 0x7fd8940a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd89413f8b0 0x7fd8941daf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient(hunting): picked mon.noname-a con 0x7fd89413f8b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd894145760 0x7fd8941db460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient(hunting): picked mon.noname-b con 0x7fd894145760 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89413a1c0 con 0x7fd89413f8b0 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd894142d80 con 0x7fd894145760 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8941db9a0 con 0x7fd8940a9d80 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd89c1f0640 10 monclient(hunting): authenticate will time out at 1453.449669s 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.099+0000 7fd899f8c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8940a9d80 0x7fd894141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.100+0000 7fd899f8c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8940a9d80 0x7fd894141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:60584/0 (socket says 172.21.15.110:60584) 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.100+0000 7fd899f8c640 1 -- 172.21.15.110:0/3703686357 learned_addr learned my addr 172.21.15.110:0/3703686357 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.100+0000 7fd899f8c640 10 monclient(hunting): get_auth_request con 0x7fd8940a9d80 auth_method 0 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.100+0000 7fd899f8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.100+0000 7fd899f8c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.100+0000 7fd899f8c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.100+0000 7fd899f8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.100+0000 7fd899f8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.100+0000 7fd899f8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.101+0000 7fd89978b640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd89413f8b0 0x7fd8941daf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.101+0000 7fd89978b640 10 monclient(hunting): get_auth_request con 0x7fd89413f8b0 auth_method 0 2024-01-20T11:16:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.101+0000 7fd89978b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.101+0000 7fd89978b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.101+0000 7fd89978b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.105 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:50.098+0000 12a58640 -1 osd.7 27 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.102+0000 7fd89978b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.102+0000 7fd89978b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.102+0000 7fd89978b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.102+0000 7fd89a78d640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd894145760 0x7fd8941db460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd89a78d640 10 monclient(hunting): get_auth_request con 0x7fd894145760 auth_method 0 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd89a78d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd89a78d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd89a78d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 1 -- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd894145760 msgr2=0x7fd8941db460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd894145760 0x7fd8941db460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 1 -- 172.21.15.110:0/3703686357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd89413f8b0 msgr2=0x7fd8941daf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd89413f8b0 0x7fd8941daf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 10 monclient: _finish_hunting 0 2024-01-20T11:16:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 1 monclient: found mon.noname-c 2024-01-20T11:16:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:16:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.103+0000 7fd899f8c640 1 -- 172.21.15.110:0/3703686357 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd890009400 con 0x7fd8940a9d80 2024-01-20T11:16:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.105+0000 7fd899f8c640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8940a9d80 0x7fd894141a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00c730 tx=0x7fd88c00cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.105+0000 7fd88a7fc640 10 client.?.objecter ms_handle_connect 0x7fd8940a9d80 2024-01-20T11:16:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.105+0000 7fd88a7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:16:50.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 1 -- 172.21.15.110:0/3703686357 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c006d50 con 0x7fd8940a9d80 2024-01-20T11:16:50.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:50.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 10 monclient: dump: 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 1 -- 172.21.15.110:0/3703686357 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c00ce40 con 0x7fd8940a9d80 2024-01-20T11:16:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 1 -- 172.21.15.110:0/3703686357 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c003bb0 con 0x7fd8940a9d80 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd89c1f0640 5 monclient: authenticate success, global_id 4298 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd89c1f0640 10 monclient: _renew_subs 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd89c1f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd894139ce0 con 0x7fd8940a9d80 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88bfff640 4 set_mon_vals no callback set 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd88a7fc640 10 monclient: dump: 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:50.112 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd89c1f0640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd89c1f0640 10 monclient: _renew_subs 2024-01-20T11:16:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd89c1f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:16:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8941393b0 con 0x7fd8940a9d80 2024-01-20T11:16:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.109+0000 7fd89c1f0640 1 librados: init done 2024-01-20T11:16:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.110+0000 7fd89c1f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.110+0000 7fd89c1f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.110+0000 7fd89c1f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:16:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.110+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd894075490 con 0x7fd8940a9d80 2024-01-20T11:16:50.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.351+0000 7fd88a7fc640 1 -- 172.21.15.110:0/3703686357 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd88c003d10 con 0x7fd8940a9d80 2024-01-20T11:16:50.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.351+0000 7fd88a7fc640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd87806c890 0x7fd87806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.352+0000 7fd89978b640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd87806c890 0x7fd87806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.352+0000 7fd89978b640 10 monclient: get_auth_request con 0x7fd87806c890 auth_method 0 2024-01-20T11:16:50.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.352+0000 7fd89978b640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd87806c890 0x7fd87806ed50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd89413b210 tx=0x7fd884009240 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:50.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.352+0000 7fd88a7fc640 10 client.4298.objecter ms_handle_connect 0x7fd87806c890 2024-01-20T11:16:50.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.364+0000 7fd88a7fc640 1 -- 172.21.15.110:0/3703686357 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd88c0591c0 con 0x7fd8940a9d80 2024-01-20T11:16:50.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.364+0000 7fd88a7fc640 10 client.4298.objecter ms_dispatch 0x7fd894000b90 osd_map(27..27 src has 1..27) v4 2024-01-20T11:16:50.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.364+0000 7fd88a7fc640 3 client.4298.objecter handle_osd_map got epochs [27,27] > 0 2024-01-20T11:16:50.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.364+0000 7fd88a7fc640 3 client.4298.objecter handle_osd_map decoding full epoch 27 2024-01-20T11:16:50.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.365+0000 7fd88a7fc640 20 client.4298.objecter dump_active .. 0 homeless 2024-01-20T11:16:50.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.365+0000 7fd88a7fc640 1 -- 172.21.15.110:0/3703686357 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd88c054e00 con 0x7fd8940a9d80 2024-01-20T11:16:50.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.365+0000 7fd88a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:50.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.365+0000 7fd88a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:16:50.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.594+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8940b5450 con 0x7fd87806c890 2024-01-20T11:16:50.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.597+0000 7fd88a7fc640 1 -- 172.21.15.110:0/3703686357 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64235 (secure 0 0 0) 0x7fd8940b5450 con 0x7fd87806c890 2024-01-20T11:16:50.597 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:16:50.598 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:16:50.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 10 librados: watch_flush enter 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 10 librados: watch_flush exit 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 20 client.4298.objecter shutdown clearing up homeless session... 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 10 client.4298.objecter successfully canceled tick 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd87806c890 msgr2=0x7fd87806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd87806c890 0x7fd87806ed50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd89413b210 tx=0x7fd884009240 comp rx=0 tx=0).stop 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 10 monclient: shutdown 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8940a9d80 msgr2=0x7fd894141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8940a9d80 0x7fd894141a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00c730 tx=0x7fd88c00cbf0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 shutdown_connections 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd87806c890 0x7fd87806ed50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd894145760 0x7fd8941db460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd89413f8b0 0x7fd8941daf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 --2- 172.21.15.110:0/3703686357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8940a9d80 0x7fd894141a80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 >> 172.21.15.110:0/3703686357 conn(0x7fd8940663b0 msgr2=0x7fd89413cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:50.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 shutdown_connections 2024-01-20T11:16:50.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 -- 172.21.15.110:0/3703686357 wait complete. 2024-01-20T11:16:50.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.599+0000 7fd89c1f0640 1 librados: shutdown 2024-01-20T11:16:50.613 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-01-20T11:16:50.391088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219040,"kb_used_data":2664,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755680,"statfs":{"total":773094113280,"available":772869816320,"internally_reserved":0,"allocated":2727936,"data_stored":1449689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":131072},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.661840"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.877271+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:15:29.877271+0000","last_peered":"2024-01-20T11:15:29.877271+0000","last_clean":"2024-01-20T11:15:29.877271+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:15:29.877271+0000","last_undegraded":"2024-01-20T11:15:29.877271+0000","last_fullsized":"2024-01-20T11:15:29.877271+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T22:26:49.902059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.356575+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:15:29.356575+0000","last_peered":"2024-01-20T11:15:29.356575+0000","last_clean":"2024-01-20T11:15:29.356575+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:15:29.356575+0000","last_undegraded":"2024-01-20T11:15:29.356575+0000","last_fullsized":"2024-01-20T11:15:29.356575+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T17:21:09.255806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.872586+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:15:29.872586+0000","last_peered":"2024-01-20T11:15:29.872586+0000","last_clean":"2024-01-20T11:15:29.872586+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:15:29.872586+0000","last_undegraded":"2024-01-20T11:15:29.872586+0000","last_fullsized":"2024-01-20T11:15:29.872586+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T14:37:21.843002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-20T11:15:27.266904+0000","last_change":"2024-01-20T11:15:27.266904+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:15:27.266904+0000","last_undegraded":"2024-01-20T11:15:27.266904+0000","last_fullsized":"2024-01-20T11:15:27.266904+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.864214+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:15:29.864214+0000","last_peered":"2024-01-20T11:15:29.864214+0000","last_clean":"2024-01-20T11:15:29.864214+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:15:29.864214+0000","last_undegraded":"2024-01-20T11:15:29.864214+0000","last_fullsized":"2024-01-20T11:15:29.864214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:00:34.021727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3,"reported_epoch":26,"state":"creating+peering","last_fresh":"2024-01-20T11:15:31.002344+0000","last_change":"2024-01-20T11:15:27.887479+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:15:31.002344+0000","last_undegraded":"2024-01-20T11:15:31.002344+0000","last_fullsized":"2024-01-20T11:15:31.002344+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":4,"reported_epoch":26,"state":"creating+activating","last_fresh":"2024-01-20T11:15:29.868562+0000","last_change":"2024-01-20T11:15:28.693467+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:15:29.868562+0000","last_undegraded":"2024-01-20T11:15:29.868562+0000","last_fullsized":"2024-01-20T11:15:29.868562+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.874929+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:15:29.874929+0000","last_peered":"2024-01-20T11:15:29.874929+0000","last_clean":"2024-01-20T11:15:29.874929+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:15:29.874929+0000","last_undegraded":"2024-01-20T11:15:29.874929+0000","last_fullsized":"2024-01-20T11:15:29.874929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:25:31.758358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":120,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-20T11:15:28.772627+0000","last_change":"2024-01-20T11:15:15.736505+0000","last_active":"2024-01-20T11:15:28.772627+0000","last_peered":"2024-01-20T11:15:28.772627+0000","last_clean":"2024-01-20T11:15:28.772627+0000","last_became_active":"2024-01-20T11:15:15.702676+0000","last_became_peered":"2024-01-20T11:15:15.702676+0000","last_unstale":"2024-01-20T11:15:28.772627+0000","last_undegraded":"2024-01-20T11:15:28.772627+0000","last_fullsized":"2024-01-20T11:15:28.772627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:50:44.769291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.1189999999999998},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":9.8699999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.839},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.1890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.265000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":17.045999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.7240000000000002},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":13.534000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.125},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":15.468999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.369},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":14.724}]}]},{"osd":7,"up_from":21,"seq":90194313239,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.5570000000000004},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":13.426}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.6929999999999996},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.5190000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.768000000000001},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.362}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.324},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.132}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.917},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":14.986000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.905000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":17.827999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313240,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":10.226000000000001},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.7869999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9900000000000002},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.561}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":6.875},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.885999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.5969999999999995},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":14.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":14.199},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.163}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":12.888999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.350000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444054,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.597},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":4.8780000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7489999999999997},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.584}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.475},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.412000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.272},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.041}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.169},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.2930000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":7.8680000000000003},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":16.018999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444055,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":643072,"data_stored":480471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":11.217000000000001},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":7.1639999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.0470000000000006},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.026999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.789999999999999},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.5999999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":10.256},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.495999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":12.337999999999999},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":13.667999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.7010000000000005},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.571999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444055,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":17.498999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":9.8089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.431},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.721}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":16.687999999999999},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.9000000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.481999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.8300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":18.321000000000002},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":12.472}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.552},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.647}]}]},{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":8.0670000000000002},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.032}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":11.747999999999999},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":14.138}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.231999999999999},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.129}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.855},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.029}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.673999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.9529999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.401999999999999},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":17.678000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:16:50.615 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:16:50.659 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-20T11:16:50.658+0000 ea50640 -1 osd.0 26 heartbeat_check: no reply from 172.21.15.134:6823 osd.5 since back 2024-01-20T11:15:25.418850+0000 front 2024-01-20T11:15:25.417600+0000 (oldest deadline 2024-01-20T11:16:48.308675+0000) 2024-01-20T11:16:50.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.728+0000 7f2a88407640 10 monclient: get_monmap_and_config 2024-01-20T11:16:50.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.728+0000 7f2a88407640 10 monclient: build_initial_monmap 2024-01-20T11:16:50.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.728+0000 7f2a88407640 10 monclient: monmap: 2024-01-20T11:16:50.728 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:50.728 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:16:50.728706+0000 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:16:50.728706+0000 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.728+0000 7f2a88407640 1 Processor -- start 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.728+0000 7f2a88407640 1 -- start start 2024-01-20T11:16:50.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.728+0000 7f2a88407640 10 monclient: init 2024-01-20T11:16:50.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:50.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-01-20T11:16:50.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2a800a96f0 0x7f2a8013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 10 monclient(hunting): picked mon.noname-c con 0x7f2a800a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:16:50.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a8013fa10 0x7f2a801423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 10 monclient(hunting): picked mon.noname-b con 0x7f2a8013fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a800a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a800b25c0 con 0x7f2a800a9ad0 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a800aaea0 con 0x7f2a800a96f0 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a800ab0a0 con 0x7f2a8013fa10 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.729+0000 7f2a88407640 10 monclient(hunting): authenticate will time out at 1454.079711s 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a859a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a8013fa10 0x7f2a801423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a861a3640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2a800a96f0 0x7f2a8013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a869a4640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2a800a9ad0 0x7f2a80142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59806/0 (socket says 172.21.15.110:59806) 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a869a4640 1 -- 172.21.15.110:0/1153843522 learned_addr learned my addr 172.21.15.110:0/1153843522 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:16:50.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a861a3640 10 monclient(hunting): get_auth_request con 0x7f2a800a96f0 auth_method 0 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a861a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a861a3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a861a3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a859a2640 10 monclient(hunting): get_auth_request con 0x7f2a8013fa10 auth_method 0 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a859a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a859a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.730+0000 7f2a859a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.731+0000 7f2a861a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.731+0000 7f2a861a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.731+0000 7f2a861a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.731+0000 7f2a859a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.731+0000 7f2a859a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:50.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.731+0000 7f2a859a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:50.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a851a1640 1 -- 172.21.15.110:0/1153843522 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185861756 0 0) 0x7f2a800b25c0 con 0x7f2a800a9ad0 2024-01-20T11:16:50.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a851a1640 10 monclient(hunting): _negotiate 2024-01-20T11:16:50.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a851a1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a851a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a851a1640 10 monclient(hunting): my global_id is 4392 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a851a1640 1 -- 172.21.15.110:0/1153843522 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a6c003520 con 0x7f2a800a9ad0 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a859a2640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a859a2640 1 -- 172.21.15.110:0/1153843522 >> v2:172.21.15.134:3300/0 conn(0x7f2a800a96f0 msgr2=0x7f2a8013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a859a2640 1 --2- 172.21.15.110:0/1153843522 >> v2:172.21.15.134:3300/0 conn(0x7f2a800a96f0 0x7f2a8013f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a859a2640 1 -- 172.21.15.110:0/1153843522 >> v1:172.21.15.110:6789/0 conn(0x7f2a800a9ad0 legacy=0x7f2a80142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a859a2640 10 monclient: _finish_hunting 0 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a859a2640 1 monclient: found mon.noname-b 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a859a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a859a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.732+0000 7f2a859a2640 1 -- 172.21.15.110:0/1153843522 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a800b3fb0 con 0x7f2a8013fa10 2024-01-20T11:16:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.734+0000 7f2a859a2640 1 --2- 172.21.15.110:0/1153843522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a8013fa10 0x7f2a801423e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2a700097a0 tx=0x7f2a7002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7c116042c18189a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 1 -- 172.21.15.110:0/1153843522 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7002e070 con 0x7f2a8013fa10 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 10 monclient: dump: 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 10 monclient: _finish_auth 0 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 10 monclient: _check_auth_tickets 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 1 -- 172.21.15.110:0/1153843522 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a700315a0 con 0x7f2a8013fa10 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 1 -- 172.21.15.110:0/1153843522 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a70031930 con 0x7f2a8013fa10 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a88407640 5 monclient: authenticate success, global_id 4301 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a88407640 10 monclient: get_monmap_and_config success 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a88407640 4 set_mon_vals no callback set 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.740+0000 7f2a851a1640 10 monclient: dump: 2024-01-20T11:16:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:50.742 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 10 monclient: shutdown 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 1 -- 172.21.15.110:0/1153843522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a8013fa10 msgr2=0x7f2a801423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 1 --2- 172.21.15.110:0/1153843522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a8013fa10 0x7f2a801423e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2a700097a0 tx=0x7f2a7002b840 comp rx=0 tx=0).stop 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 1 -- 172.21.15.110:0/1153843522 shutdown_connections 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 1 --2- 172.21.15.110:0/1153843522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a8013fa10 0x7f2a801423e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 1 --2- 172.21.15.110:0/1153843522 >> v2:172.21.15.134:3300/0 conn(0x7f2a800a96f0 0x7f2a8013f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 1 -- 172.21.15.110:0/1153843522 >> 172.21.15.110:0/1153843522 conn(0x7f2a800b7bc0 msgr2=0x7f2a800b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 1 -- 172.21.15.110:0/1153843522 shutdown_connections 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 1 -- 172.21.15.110:0/1153843522 wait complete. 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 10 monclient: build_initial_monmap 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.741+0000 7f2a88407640 10 monclient: monmap: 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:16:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 librados: starting msgr at 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 librados: starting objecter 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 Processor -- start 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 -- start start 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 librados: setting wanted keys 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 librados: calling monclient init 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient: init 2024-01-20T11:16:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a800a96f0 0x7f2a80136da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient(hunting): picked mon.noname-b con 0x7f2a800a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a800a9ad0 0x7f2a801d7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient(hunting): picked mon.noname-c con 0x7f2a800a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a801d7c70 0x7f2a80138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient(hunting): picked mon.noname-a con 0x7f2a801d7c70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8013b1a0 con 0x7f2a801d7c70 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a801d8100 con 0x7f2a800a96f0 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a80137df0 con 0x7f2a800a9ad0 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.742+0000 7f2a88407640 10 monclient(hunting): authenticate will time out at 1454.093126s 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.743+0000 7f2a861a3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a800a96f0 0x7f2a80136da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.743+0000 7f2a859a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a800a9ad0 0x7f2a801d7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.743+0000 7f2a861a3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a800a96f0 0x7f2a80136da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46684/0 (socket says 172.21.15.110:46684) 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.743+0000 7f2a861a3640 1 -- 172.21.15.110:0/4062616799 learned_addr learned my addr 172.21.15.110:0/4062616799 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.743+0000 7f2a861a3640 10 monclient(hunting): get_auth_request con 0x7f2a800a96f0 auth_method 0 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.743+0000 7f2a861a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.743+0000 7f2a861a3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.743+0000 7f2a861a3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a861a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a861a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a861a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a859a2640 10 monclient(hunting): get_auth_request con 0x7f2a800a9ad0 auth_method 0 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a859a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a859a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a859a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a859a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a859a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.744+0000 7f2a859a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 10 monclient(hunting): handle_auth_done global_id 4426 payload 824 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 1 -- 172.21.15.110:0/4062616799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a800a9ad0 msgr2=0x7f2a801d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a800a9ad0 0x7f2a801d7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 1 -- 172.21.15.110:0/4062616799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a801d7c70 msgr2=0x7f2a80138280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a801d7c70 0x7f2a80138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 10 monclient: _finish_hunting 0 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 1 monclient: found mon.noname-b 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.747+0000 7f2a861a3640 1 -- 172.21.15.110:0/4062616799 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a70009400 con 0x7f2a800a96f0 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.749+0000 7f2a861a3640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a800a96f0 0x7f2a80136da0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009db0 tx=0x7f2a7c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.749+0000 7f2a76ffd640 10 client.?.objecter ms_handle_connect 0x7f2a800a96f0 2024-01-20T11:16:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.749+0000 7f2a76ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 1 -- 172.21.15.110:0/4062616799 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c014480 con 0x7f2a800a96f0 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 10 monclient: dump: 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 10 monclient: _finish_auth 0 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 1 -- 172.21.15.110:0/4062616799 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7c00f040 con 0x7f2a800a96f0 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a88407640 5 monclient: authenticate success, global_id 4426 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a88407640 10 monclient: _renew_subs 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a88407640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a80139830 con 0x7f2a800a96f0 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 1 -- 172.21.15.110:0/4062616799 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c014800 con 0x7f2a800a96f0 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a76ffd640 10 monclient: dump: 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a849a0640 4 set_mon_vals no callback set 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a88407640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a88407640 10 monclient: _renew_subs 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a88407640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a801389a0 con 0x7f2a800a96f0 2024-01-20T11:16:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.753+0000 7f2a88407640 1 librados: init done 2024-01-20T11:16:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.754+0000 7f2a88407640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.754+0000 7f2a88407640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.754+0000 7f2a88407640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:50.754+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a80075490 con 0x7f2a800a96f0 2024-01-20T11:16:51.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.006+0000 7f2a76ffd640 1 -- 172.21.15.110:0/4062616799 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2a7c014960 con 0x7f2a800a96f0 2024-01-20T11:16:51.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.007+0000 7f2a76ffd640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a6406c860 0x7f2a6406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:51.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.007+0000 7f2a859a2640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a6406c860 0x7f2a6406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:51.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.007+0000 7f2a859a2640 10 monclient: get_auth_request con 0x7f2a6406c860 auth_method 0 2024-01-20T11:16:51.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.008+0000 7f2a859a2640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a6406c860 0x7f2a6406ed20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2a70036000 tx=0x7f2a7003d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:51.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.008+0000 7f2a76ffd640 10 client.4426.objecter ms_handle_connect 0x7f2a6406c860 2024-01-20T11:16:51.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.010+0000 7f2a76ffd640 1 -- 172.21.15.110:0/4062616799 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2a7c0598b0 con 0x7f2a800a96f0 2024-01-20T11:16:51.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.010+0000 7f2a76ffd640 10 client.4426.objecter ms_dispatch 0x7f2a80000b90 osd_map(28..28 src has 1..28) v4 2024-01-20T11:16:51.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.010+0000 7f2a76ffd640 3 client.4426.objecter handle_osd_map got epochs [28,28] > 0 2024-01-20T11:16:51.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.010+0000 7f2a76ffd640 3 client.4426.objecter handle_osd_map decoding full epoch 28 2024-01-20T11:16:51.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.010+0000 7f2a76ffd640 20 client.4426.objecter dump_active .. 0 homeless 2024-01-20T11:16:51.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.021+0000 7f2a76ffd640 1 -- 172.21.15.110:0/4062616799 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2a7c0554f0 con 0x7f2a800a96f0 2024-01-20T11:16:51.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.022+0000 7f2a76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:51.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.022+0000 7f2a76ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:16:51.132 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:16:51.131+0000 12a58640 -1 osd.7 28 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4203.0:2 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2024-01-20T11:16:51.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.199+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a80142df0 con 0x7f2a6406c860 2024-01-20T11:16:51.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.202+0000 7f2a76ffd640 1 -- 172.21.15.110:0/4062616799 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64349 (secure 0 0 0) 0x7f2a80142df0 con 0x7f2a6406c860 2024-01-20T11:16:51.202 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:16:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:16:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.203+0000 7f2a88407640 10 librados: watch_flush enter 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 10 librados: watch_flush exit 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 20 client.4426.objecter shutdown clearing up homeless session... 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 10 client.4426.objecter successfully canceled tick 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a6406c860 msgr2=0x7f2a6406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a6406c860 0x7f2a6406ed20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2a70036000 tx=0x7f2a7003d040 comp rx=0 tx=0).stop 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 10 monclient: shutdown 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a800a96f0 msgr2=0x7f2a80136da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a800a96f0 0x7f2a80136da0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009db0 tx=0x7f2a7c00c490 comp rx=0 tx=0).stop 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 shutdown_connections 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a6406c860 0x7f2a6406ed20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a801d7c70 0x7f2a80138280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a800a9ad0 0x7f2a801d7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 --2- 172.21.15.110:0/4062616799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a800a96f0 0x7f2a80136da0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 >> 172.21.15.110:0/4062616799 conn(0x7f2a800b7bc0 msgr2=0x7f2a800b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 shutdown_connections 2024-01-20T11:16:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 -- 172.21.15.110:0/4062616799 wait complete. 2024-01-20T11:16:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.204+0000 7f2a88407640 1 librados: shutdown 2024-01-20T11:16:51.214 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-01-20T11:16:50.730969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":17,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219048,"kb_used_data":2676,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755672,"statfs":{"total":773094113280,"available":772869808128,"internally_reserved":0,"allocated":2740224,"data_stored":1455128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":131072},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001361"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.877271+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:15:29.877271+0000","last_peered":"2024-01-20T11:15:29.877271+0000","last_clean":"2024-01-20T11:15:29.877271+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:15:29.877271+0000","last_undegraded":"2024-01-20T11:15:29.877271+0000","last_fullsized":"2024-01-20T11:15:29.877271+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T22:26:49.902059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.356575+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:15:29.356575+0000","last_peered":"2024-01-20T11:15:29.356575+0000","last_clean":"2024-01-20T11:15:29.356575+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:15:29.356575+0000","last_undegraded":"2024-01-20T11:15:29.356575+0000","last_fullsized":"2024-01-20T11:15:29.356575+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T17:21:09.255806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.872586+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:15:29.872586+0000","last_peered":"2024-01-20T11:15:29.872586+0000","last_clean":"2024-01-20T11:15:29.872586+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:15:29.872586+0000","last_undegraded":"2024-01-20T11:15:29.872586+0000","last_fullsized":"2024-01-20T11:15:29.872586+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T14:37:21.843002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":6,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-01-20T11:16:50.513148+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:50.513148+0000","last_peered":"2024-01-20T11:16:50.513148+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:50.513148+0000","last_undegraded":"2024-01-20T11:16:50.513148+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:34:49.467555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.864214+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:15:29.864214+0000","last_peered":"2024-01-20T11:15:29.864214+0000","last_clean":"2024-01-20T11:15:29.864214+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:15:29.864214+0000","last_undegraded":"2024-01-20T11:15:29.864214+0000","last_fullsized":"2024-01-20T11:15:29.864214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:00:34.021727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3,"reported_epoch":26,"state":"creating+peering","last_fresh":"2024-01-20T11:15:31.002344+0000","last_change":"2024-01-20T11:15:27.887479+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:15:31.002344+0000","last_undegraded":"2024-01-20T11:15:31.002344+0000","last_fullsized":"2024-01-20T11:15:31.002344+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":4,"reported_epoch":26,"state":"creating+activating","last_fresh":"2024-01-20T11:15:29.868562+0000","last_change":"2024-01-20T11:15:28.693467+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:15:29.868562+0000","last_undegraded":"2024-01-20T11:15:29.868562+0000","last_fullsized":"2024-01-20T11:15:29.868562+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.874929+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:15:29.874929+0000","last_peered":"2024-01-20T11:15:29.874929+0000","last_clean":"2024-01-20T11:15:29.874929+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:15:29.874929+0000","last_undegraded":"2024-01-20T11:15:29.874929+0000","last_fullsized":"2024-01-20T11:15:29.874929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:25:31.758358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":120,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-20T11:15:28.772627+0000","last_change":"2024-01-20T11:15:15.736505+0000","last_active":"2024-01-20T11:15:28.772627+0000","last_peered":"2024-01-20T11:15:28.772627+0000","last_clean":"2024-01-20T11:15:28.772627+0000","last_became_active":"2024-01-20T11:15:15.702676+0000","last_became_peered":"2024-01-20T11:15:15.702676+0000","last_unstale":"2024-01-20T11:15:28.772627+0000","last_undegraded":"2024-01-20T11:15:28.772627+0000","last_fullsized":"2024-01-20T11:15:28.772627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:50:44.769291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.1189999999999998},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":9.8699999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.839},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.1890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.265000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":17.045999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.7240000000000002},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":13.534000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.125},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":15.468999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.369},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":14.724}]}]},{"osd":7,"up_from":21,"seq":90194313239,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.5570000000000004},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":13.426}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.6929999999999996},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.5190000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.768000000000001},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.362}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.324},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.132}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.917},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":14.986000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.905000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":17.827999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313240,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":10.226000000000001},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.7869999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9900000000000002},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.561}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":6.875},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.885999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.5969999999999995},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":14.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":14.199},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.163}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":12.888999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.350000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444054,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.597},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":4.8780000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7489999999999997},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.584}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.475},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.412000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.272},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.041}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.169},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.2930000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":7.8680000000000003},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":16.018999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444055,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":643072,"data_stored":480471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":11.217000000000001},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":7.1639999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.0470000000000006},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.026999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.789999999999999},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.5999999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":10.256},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.495999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":12.337999999999999},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":13.667999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.7010000000000005},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.571999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444055,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":17.498999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":9.8089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.431},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.721}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":16.687999999999999},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.9000000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.481999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.8300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":18.321000000000002},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":12.472}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.552},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.647}]}]},{"osd":6,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":89486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":6.8399999999999999},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":10.265000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":14.116},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":12.711}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6989999999999998},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.291}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.9990000000000006},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.868}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.489000000000001},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.452}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.218},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.413}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:16:51.216 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:16:51.216 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 6, 'reported_epoch': 28, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:50.513148+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:50.513148+0000', 'last_peered': '2024-01-20T11:16:50.513148+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:50.513148+0000', 'last_undegraded': '2024-01-20T11:16:50.513148+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:34:49.467555+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:16:51.216 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:16:51.216 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 26, 'state': 'creating+peering', 'last_fresh': '2024-01-20T11:15:31.002344+0000', 'last_change': '2024-01-20T11:15:27.887479+0000', 'last_active': '2024-01-20T11:15:27.266904+0000', 'last_peered': '2024-01-20T11:15:27.266904+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-20T11:15:31.002344+0000', 'last_undegraded': '2024-01-20T11:15:31.002344+0000', 'last_fullsized': '2024-01-20T11:15:31.002344+0000', 'mapping_epoch': 24, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:16:51.216 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:16:51.216 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 4, 'reported_epoch': 26, 'state': 'creating+activating', 'last_fresh': '2024-01-20T11:15:29.868562+0000', 'last_change': '2024-01-20T11:15:28.693467+0000', 'last_active': '2024-01-20T11:15:27.266904+0000', 'last_peered': '2024-01-20T11:15:27.266904+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-20T11:15:29.868562+0000', 'last_undegraded': '2024-01-20T11:15:29.868562+0000', 'last_fullsized': '2024-01-20T11:15:29.868562+0000', 'mapping_epoch': 24, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': ['3', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:16:51.216 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.316+0000 7fc2e239e640 10 monclient: get_monmap_and_config 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.316+0000 7fc2e239e640 10 monclient: build_initial_monmap 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.316+0000 7fc2e239e640 10 monclient: monmap: 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:16:51.316672+0000 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:16:51.316672+0000 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:51.316 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:16:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:16:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:16:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:16:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:16:51.317 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.316+0000 7fc2e239e640 1 Processor -- start 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 1 -- start start 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 10 monclient: init 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2dc0a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a9ad0 0x7fc2dc1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 10 monclient(hunting): picked mon.noname-b con 0x7fc2dc0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2dc142920 addr v1:172.21.15.134:6789/0 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc0b25c0 con 0x7fc2dc0a96f0 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc13fbe0 con 0x7fc2dc142920 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc0aaea0 con 0x7fc2dc0a9ad0 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.317+0000 7fc2e239e640 10 monclient(hunting): authenticate will time out at 1454.667851s 2024-01-20T11:16:51.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.318+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a9ad0 0x7fc2dc1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:51.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.318+0000 7fc2dbfff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc2dc0a96f0 0x7fc2dc13f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59820/0 (socket says 172.21.15.110:59820) 2024-01-20T11:16:51.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.318+0000 7fc2dbfff640 1 -- 172.21.15.110:0/2073254369 learned_addr learned my addr 172.21.15.110:0/2073254369 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:16:51.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.318+0000 7fc2d3fff640 10 monclient(hunting): get_auth_request con 0x7fc2dc0a9ad0 auth_method 0 2024-01-20T11:16:51.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.318+0000 7fc2d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:51.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.318+0000 7fc2d3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:51.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.318+0000 7fc2d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:51.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.319+0000 7fc2d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:51.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.319+0000 7fc2d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:51.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.319+0000 7fc2d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:51.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2d3fff640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2024-01-20T11:16:51.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2d3fff640 1 -- 172.21.15.110:0/2073254369 >> v1:172.21.15.134:6789/0 conn(0x7fc2dc142920 legacy=0x7fc2dc14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:51.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2d3fff640 1 -- 172.21.15.110:0/2073254369 >> v1:172.21.15.110:6789/0 conn(0x7fc2dc0a96f0 legacy=0x7fc2dc13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:51.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2d3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:16:51.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2d3fff640 1 monclient: found mon.noname-b 2024-01-20T11:16:51.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2db7fe640 1 -- 172.21.15.110:0/2073254369 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257346491 0 0) 0x7fc2dc0b25c0 con 0x7fc2dc0a96f0 2024-01-20T11:16:51.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:51.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2d3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:16:51.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2d3fff640 1 -- 172.21.15.110:0/2073254369 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc0ab0a0 con 0x7fc2dc0a9ad0 2024-01-20T11:16:51.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.320+0000 7fc2db7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:16:51.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.322+0000 7fc2d3fff640 1 --2- 172.21.15.110:0/2073254369 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a9ad0 0x7fc2dc1423e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40097d0 tx=0x7fc2c402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de44b8866a4d9edc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:51.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.325+0000 7fc2db7fe640 1 -- 172.21.15.110:0/2073254369 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4031800 con 0x7fc2dc0a9ad0 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 10 monclient: dump: 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:51.326 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 10 monclient: _finish_auth 0 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 1 -- 172.21.15.110:0/2073254369 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c4031960 con 0x7fc2dc0a9ad0 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 5 monclient: authenticate success, global_id 4307 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 10 monclient: get_monmap_and_config success 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 4 set_mon_vals no callback set 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 1 -- 172.21.15.110:0/2073254369 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4031cf0 con 0x7fc2dc0a9ad0 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:16:51.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2db7fe640 10 monclient: dump: 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:51.328 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:51.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 10 monclient: shutdown 2024-01-20T11:16:51.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:51.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 1 -- 172.21.15.110:0/2073254369 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a9ad0 msgr2=0x7fc2dc1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:51.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 1 --2- 172.21.15.110:0/2073254369 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a9ad0 0x7fc2dc1423e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40097d0 tx=0x7fc2c402b8a0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 1 -- 172.21.15.110:0/2073254369 shutdown_connections 2024-01-20T11:16:51.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 1 --2- 172.21.15.110:0/2073254369 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a9ad0 0x7fc2dc1423e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 1 -- 172.21.15.110:0/2073254369 >> 172.21.15.110:0/2073254369 conn(0x7fc2dc0b7bc0 msgr2=0x7fc2dc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:51.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 1 -- 172.21.15.110:0/2073254369 shutdown_connections 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.326+0000 7fc2e239e640 1 -- 172.21.15.110:0/2073254369 wait complete. 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.327+0000 7fc2e239e640 10 monclient: build_initial_monmap 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.327+0000 7fc2e239e640 10 monclient: monmap: 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:16:51.333 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 librados: starting msgr at 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 librados: starting objecter 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 Processor -- start 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 -- start start 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 librados: setting wanted keys 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 librados: calling monclient init 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient: init 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a96f0 0x7fc2dc140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient(hunting): picked mon.noname-c con 0x7fc2dc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2dc0a9ad0 0x7fc2dc145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient(hunting): picked mon.noname-a con 0x7fc2dc0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:16:51.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2dc142920 0x7fc2dc144270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient(hunting): picked mon.noname-b con 0x7fc2dc142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc136aa0 con 0x7fc2dc0a9ad0 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc145540 con 0x7fc2dc142920 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1d7440 con 0x7fc2dc0a96f0 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.328+0000 7fc2e239e640 10 monclient(hunting): authenticate will time out at 1454.678983s 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2e093b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2dc142920 0x7fc2dc144270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2dbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a96f0 0x7fc2dc140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2d3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2dc0a9ad0 0x7fc2dc145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:51.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2e093b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2dc142920 0x7fc2dc144270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46698/0 (socket says 172.21.15.110:46698) 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2e093b640 1 -- 172.21.15.110:0/458894380 learned_addr learned my addr 172.21.15.110:0/458894380 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2e093b640 10 monclient(hunting): get_auth_request con 0x7fc2dc142920 auth_method 0 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2e093b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2e093b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2e093b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2dbfff640 10 monclient(hunting): get_auth_request con 0x7fc2dc0a96f0 auth_method 0 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2dbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2d3fff640 10 monclient(hunting): get_auth_request con 0x7fc2dc0a9ad0 auth_method 0 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2d3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.329+0000 7fc2d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:51.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.330+0000 7fc2e093b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:51.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.330+0000 7fc2e093b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:51.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.330+0000 7fc2e093b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:51.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.330+0000 7fc2dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:51.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.330+0000 7fc2dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:51.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.330+0000 7fc2dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:51.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.330+0000 7fc2d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:51.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.330+0000 7fc2d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:51.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.330+0000 7fc2d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 10 monclient(hunting): handle_auth_done global_id 4432 payload 824 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 1 -- 172.21.15.110:0/458894380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a96f0 msgr2=0x7fc2dc140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a96f0 0x7fc2dc140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 1 -- 172.21.15.110:0/458894380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2dc0a9ad0 msgr2=0x7fc2dc145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2dc0a9ad0 0x7fc2dc145000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 10 monclient: _finish_hunting 0 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 1 monclient: found mon.noname-b 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:16:51.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.333+0000 7fc2e093b640 1 -- 172.21.15.110:0/458894380 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4009400 con 0x7fc2dc142920 2024-01-20T11:16:51.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.336+0000 7fc2e093b640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2dc142920 0x7fc2dc144270 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400c6b0 tx=0x7fc2d400cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:51.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.336+0000 7fc2d97fa640 10 client.?.objecter ms_handle_connect 0x7fc2dc142920 2024-01-20T11:16:51.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.336+0000 7fc2d97fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:16:51.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 1 -- 172.21.15.110:0/458894380 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4003b70 con 0x7fc2dc142920 2024-01-20T11:16:51.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 10 monclient: dump: 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:51.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 10 monclient: _finish_auth 0 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 10 monclient: _check_auth_tickets 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 1 -- 172.21.15.110:0/458894380 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d4003cd0 con 0x7fc2dc142920 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2e239e640 5 monclient: authenticate success, global_id 4432 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2e239e640 10 monclient: _renew_subs 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2e239e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc1447b0 con 0x7fc2dc142920 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 1 -- 172.21.15.110:0/458894380 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4005310 con 0x7fc2dc142920 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:51.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2d97fa640 10 monclient: dump: 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2da7fc640 4 set_mon_vals no callback set 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2e239e640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:51.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2e239e640 10 monclient: _renew_subs 2024-01-20T11:16:51.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2e239e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:51.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc1d7960 con 0x7fc2dc142920 2024-01-20T11:16:51.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.340+0000 7fc2e239e640 1 librados: init done 2024-01-20T11:16:51.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.341+0000 7fc2e239e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:51.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.341+0000 7fc2e239e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:51.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.341+0000 7fc2e239e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:51.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.341+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2dc1d72b0 con 0x7fc2dc142920 2024-01-20T11:16:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.593+0000 7fc2d97fa640 1 -- 172.21.15.110:0/458894380 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc2d4005470 con 0x7fc2dc142920 2024-01-20T11:16:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.593+0000 7fc2d97fa640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2bc06c8b0 0x7fc2bc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.594+0000 7fc2dbfff640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2bc06c8b0 0x7fc2bc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.594+0000 7fc2dbfff640 10 monclient: get_auth_request con 0x7fc2bc06c8b0 auth_method 0 2024-01-20T11:16:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.594+0000 7fc2d97fa640 10 client.4432.objecter ms_handle_connect 0x7fc2bc06c8b0 2024-01-20T11:16:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.594+0000 7fc2dbfff640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2bc06c8b0 0x7fc2bc06ed70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc007f70 tx=0x7fc2cc007f00 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.596+0000 7fc2d97fa640 1 -- 172.21.15.110:0/458894380 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc2d4058970 con 0x7fc2dc142920 2024-01-20T11:16:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.596+0000 7fc2d97fa640 10 client.4432.objecter ms_dispatch 0x7fc2dc000b90 osd_map(28..28 src has 1..28) v4 2024-01-20T11:16:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.596+0000 7fc2d97fa640 3 client.4432.objecter handle_osd_map got epochs [28,28] > 0 2024-01-20T11:16:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.596+0000 7fc2d97fa640 3 client.4432.objecter handle_osd_map decoding full epoch 28 2024-01-20T11:16:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.597+0000 7fc2d97fa640 20 client.4432.objecter dump_active .. 0 homeless 2024-01-20T11:16:51.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.608+0000 7fc2d97fa640 1 -- 172.21.15.110:0/458894380 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc2d4054cb0 con 0x7fc2dc142920 2024-01-20T11:16:51.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.608+0000 7fc2d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:51.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.608+0000 7fc2d97fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:16:51.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.875+0000 7fc2e239e640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:16:51.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.875+0000 7fc2e239e640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:16:51.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.875+0000 7fc2e239e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:51.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.875+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc2dc1d7650 con 0x7fc2dc142920 2024-01-20T11:16:51.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.906+0000 7fc2d97fa640 1 -- 172.21.15.110:0/458894380 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1417 (secure 0 0 0) 0x7fc2d40582d0 con 0x7fc2dc142920 2024-01-20T11:16:51.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.906+0000 7fc2d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:16:51.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.906+0000 7fc2d97fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:16:51.907 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:16:51.907 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs inactive, 1 pg peering","count":3},"muted":false},"SLOW_OPS":{"severity":"HEALTH_WARN","summary":{"message":"1 slow ops, oldest one blocked for 64 sec, osd.7 has slow ops","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":146,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":29,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"creating+activating","count":1},{"state_name":"creating+peering","count":1},{"state_name":"active+undersized","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224305152,"bytes_avail":772869808128,"bytes_total":773094113280,"inactive_pgs_ratio":0.2222222238779068},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:16:51.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 10 librados: watch_flush enter 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 10 librados: watch_flush exit 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 20 client.4432.objecter shutdown clearing up homeless session... 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 10 client.4432.objecter successfully canceled tick 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2bc06c8b0 msgr2=0x7fc2bc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2bc06c8b0 0x7fc2bc06ed70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc007f70 tx=0x7fc2cc007f00 comp rx=0 tx=0).stop 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 10 monclient: shutdown 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2dc142920 msgr2=0x7fc2dc144270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2dc142920 0x7fc2dc144270 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400c6b0 tx=0x7fc2d400cb70 comp rx=0 tx=0).stop 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 shutdown_connections 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2bc06c8b0 0x7fc2bc06ed70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2dc142920 0x7fc2dc144270 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2dc0a9ad0 0x7fc2dc145000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 --2- 172.21.15.110:0/458894380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2dc0a96f0 0x7fc2dc140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 >> 172.21.15.110:0/458894380 conn(0x7fc2dc0b7bc0 msgr2=0x7fc2dc13be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 shutdown_connections 2024-01-20T11:16:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 -- 172.21.15.110:0/458894380 wait complete. 2024-01-20T11:16:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:51.908+0000 7fc2e239e640 1 librados: shutdown 2024-01-20T11:16:51.920 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:16:51.920 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:16:52.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.020+0000 7fb2e0f9e640 10 monclient: get_monmap_and_config 2024-01-20T11:16:52.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.020+0000 7fb2e0f9e640 10 monclient: build_initial_monmap 2024-01-20T11:16:52.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.020+0000 7fb2e0f9e640 10 monclient: monmap: 2024-01-20T11:16:52.021 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:52.021 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:16:52.021652+0000 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:16:52.021652+0000 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.020+0000 7fb2e0f9e640 1 Processor -- start 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.020+0000 7fb2e0f9e640 1 -- start start 2024-01-20T11:16:52.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.020+0000 7fb2e0f9e640 10 monclient: init 2024-01-20T11:16:52.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:52.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-01-20T11:16:52.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb2dc0a96f0 0x7fb2dc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 10 monclient(hunting): picked mon.noname-c con 0x7fb2dc0a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2dc13fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0a9ad0 0x7fb2dc142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 10 monclient(hunting): picked mon.noname-b con 0x7fb2dc0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0b25c0 con 0x7fb2dc13fa10 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc0aaea0 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0ab0a0 con 0x7fb2dc0a9ad0 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2e0f9e640 10 monclient(hunting): authenticate will time out at 1455.372664s 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.021+0000 7fb2dad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0a9ad0 0x7fb2dc142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2da59c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb2dc0a96f0 0x7fb2dc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2d9d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb2dc13fa10 0x7fb2dc1423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59826/0 (socket says 172.21.15.110:59826) 2024-01-20T11:16:52.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2d9d9b640 1 -- 172.21.15.110:0/3817060696 learned_addr learned my addr 172.21.15.110:0/3817060696 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2dad9d640 10 monclient(hunting): get_auth_request con 0x7fb2dc0a9ad0 auth_method 0 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2dad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2da59c640 10 monclient(hunting): get_auth_request con 0x7fb2dc0a96f0 auth_method 0 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2da59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2da59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:52.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.022+0000 7fb2dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:52.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.024+0000 7fb2da59c640 10 monclient(hunting): handle_auth_done global_id 4435 payload 274 2024-01-20T11:16:52.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.024+0000 7fb2da59c640 1 -- 172.21.15.110:0/3817060696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0a9ad0 msgr2=0x7fb2dc142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:16:52.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.024+0000 7fb2da59c640 1 --2- 172.21.15.110:0/3817060696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0a9ad0 0x7fb2dc142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:52.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.024+0000 7fb2da59c640 1 -- 172.21.15.110:0/3817060696 >> v1:172.21.15.110:6789/0 conn(0x7fb2dc13fa10 legacy=0x7fb2dc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:52.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.024+0000 7fb2da59c640 10 monclient: _finish_hunting 0 2024-01-20T11:16:52.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.024+0000 7fb2da59c640 1 monclient: found mon.noname-c 2024-01-20T11:16:52.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.024+0000 7fb2da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:52.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.024+0000 7fb2da59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:16:52.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.024+0000 7fb2da59c640 1 -- 172.21.15.110:0/3817060696 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0b3fb0 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.026+0000 7fb2da59c640 1 --2- 172.21.15.110:0/3817060696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 0x7fb2dc13f4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009d90 tx=0x7fb2cc02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54ef7797176fcb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 1 -- 172.21.15.110:0/3817060696 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc02cbc0 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 10 monclient: dump: 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:52.031 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 10 monclient: _finish_auth 0 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 10 monclient: _check_auth_tickets 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 1 -- 172.21.15.110:0/3817060696 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc02cd20 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 1 -- 172.21.15.110:0/3817060696 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc034660 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 5 monclient: authenticate success, global_id 4435 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 10 monclient: get_monmap_and_config success 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 4 set_mon_vals no callback set 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:52.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2d959a640 10 monclient: dump: 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:52.033 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:52.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 10 monclient: shutdown 2024-01-20T11:16:52.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:52.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/3817060696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 msgr2=0x7fb2dc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:52.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 1 --2- 172.21.15.110:0/3817060696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 0x7fb2dc13f4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009d90 tx=0x7fb2cc02be10 comp rx=0 tx=0).stop 2024-01-20T11:16:52.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/3817060696 shutdown_connections 2024-01-20T11:16:52.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 1 --2- 172.21.15.110:0/3817060696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0a9ad0 0x7fb2dc142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 1 --2- 172.21.15.110:0/3817060696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 0x7fb2dc13f4d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/3817060696 >> 172.21.15.110:0/3817060696 conn(0x7fb2dc0b7bc0 msgr2=0x7fb2dc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/3817060696 shutdown_connections 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.030+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/3817060696 wait complete. 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.031+0000 7fb2e0f9e640 10 monclient: build_initial_monmap 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.031+0000 7fb2e0f9e640 10 monclient: monmap: 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:16:52.037 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.031+0000 7fb2e0f9e640 1 librados: starting msgr at 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.031+0000 7fb2e0f9e640 1 librados: starting objecter 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 Processor -- start 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 -- start start 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 librados: setting wanted keys 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 librados: calling monclient init 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient: init 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 0x7fb2dc13ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient(hunting): picked mon.noname-b con 0x7fb2dc0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2dc0a9ad0 0x7fb2dc13b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:52.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient(hunting): picked mon.noname-a con 0x7fb2dc0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0b0eb0 0x7fb2dc0b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient(hunting): picked mon.noname-c con 0x7fb2dc0b0eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0ac040 con 0x7fb2dc0a9ad0 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc13be00 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0ac2b0 con 0x7fb2dc0b0eb0 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.032+0000 7fb2e0f9e640 10 monclient(hunting): authenticate will time out at 1455.383861s 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2dad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0b0eb0 0x7fb2dc0b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2d9d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2dc0a9ad0 0x7fb2dc13b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 0x7fb2dc13ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 0x7fb2dc13ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46722/0 (socket says 172.21.15.110:46722) 2024-01-20T11:16:52.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 1 -- 172.21.15.110:0/2086954489 learned_addr learned my addr 172.21.15.110:0/2086954489 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 10 monclient(hunting): get_auth_request con 0x7fb2dc0a96f0 auth_method 0 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2dad9d640 10 monclient(hunting): get_auth_request con 0x7fb2dc0b0eb0 auth_method 0 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2dad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2d9d9b640 10 monclient(hunting): get_auth_request con 0x7fb2dc0a9ad0 auth_method 0 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2d9d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:52.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:52.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.033+0000 7fb2da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:52.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.034+0000 7fb2dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:52.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.034+0000 7fb2dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:52.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.034+0000 7fb2dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:52.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.034+0000 7fb2d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:52.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.034+0000 7fb2d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:52.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.034+0000 7fb2d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 1 -- 172.21.15.110:0/2086954489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0b0eb0 msgr2=0x7fb2dc0b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0b0eb0 0x7fb2dc0b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 1 -- 172.21.15.110:0/2086954489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2dc0a9ad0 msgr2=0x7fb2dc13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2dc0a9ad0 0x7fb2dc13b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 10 monclient: _finish_hunting 0 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 1 monclient: found mon.noname-b 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:16:52.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.036+0000 7fb2da59c640 1 -- 172.21.15.110:0/2086954489 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc009a70 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.038+0000 7fb2da59c640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 0x7fb2dc13ac00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc037040 tx=0x7fb2cc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:52.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.038+0000 7fb2d2ffd640 10 client.?.objecter ms_handle_connect 0x7fb2dc0a96f0 2024-01-20T11:16:52.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.038+0000 7fb2d2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:16:52.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2d2ffd640 1 -- 172.21.15.110:0/2086954489 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc040650 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2d2ffd640 10 monclient: dump: 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2d2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2d2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:52.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2d2ffd640 1 -- 172.21.15.110:0/2086954489 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc02ce90 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2e0f9e640 5 monclient: authenticate success, global_id 4438 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2e0f9e640 10 monclient: _renew_subs 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2e0f9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.041+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc0ac590 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2d2ffd640 1 -- 172.21.15.110:0/2086954489 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc040a30 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2d2ffd640 10 monclient: dump: 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:52.046 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2d8d99640 4 set_mon_vals no callback set 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2e0f9e640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2e0f9e640 10 monclient: _renew_subs 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2e0f9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc0b1880 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.042+0000 7fb2e0f9e640 1 librados: init done 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.043+0000 7fb2e0f9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:52.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.043+0000 7fb2e0f9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:52.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.043+0000 7fb2e0f9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:52.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.043+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc13bb20 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.062+0000 7fb2d2ffd640 1 -- 172.21.15.110:0/2086954489 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb2cc040b90 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.062+0000 7fb2d2ffd640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb2bc06c8e0 0x7fb2bc06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.062+0000 7fb2d9d9b640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb2bc06c8e0 0x7fb2bc06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.062+0000 7fb2d9d9b640 10 monclient: get_auth_request con 0x7fb2bc06c8e0 auth_method 0 2024-01-20T11:16:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.063+0000 7fb2d2ffd640 10 client.4438.objecter ms_handle_connect 0x7fb2bc06c8e0 2024-01-20T11:16:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.063+0000 7fb2d9d9b640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb2bc06c8e0 0x7fb2bc06eda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc13d030 tx=0x7fb2c4007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:52.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.066+0000 7fb2d2ffd640 1 -- 172.21.15.110:0/2086954489 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb2cc07b380 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.066+0000 7fb2d2ffd640 10 client.4438.objecter ms_dispatch 0x7fb2dc000b90 osd_map(29..29 src has 1..29) v4 2024-01-20T11:16:52.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.066+0000 7fb2d2ffd640 3 client.4438.objecter handle_osd_map got epochs [29,29] > 0 2024-01-20T11:16:52.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.066+0000 7fb2d2ffd640 3 client.4438.objecter handle_osd_map decoding full epoch 29 2024-01-20T11:16:52.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.066+0000 7fb2d2ffd640 20 client.4438.objecter dump_active .. 0 homeless 2024-01-20T11:16:52.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.301+0000 7fb2d2ffd640 1 -- 172.21.15.110:0/2086954489 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb2cc0776c0 con 0x7fb2dc0a96f0 2024-01-20T11:16:52.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.301+0000 7fb2d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:52.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.301+0000 7fb2d2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:16:52.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.467+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2dc0b3760 con 0x7fb2bc06c8e0 2024-01-20T11:16:52.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.470+0000 7fb2d2ffd640 1 -- 172.21.15.110:0/2086954489 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64460 (secure 0 0 0) 0x7fb2dc0b3760 con 0x7fb2bc06c8e0 2024-01-20T11:16:52.472 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:16:52.472 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 10 librados: watch_flush enter 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 10 librados: watch_flush exit 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 20 client.4438.objecter shutdown clearing up homeless session... 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 10 client.4438.objecter successfully canceled tick 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb2bc06c8e0 msgr2=0x7fb2bc06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb2bc06c8e0 0x7fb2bc06eda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc13d030 tx=0x7fb2c4007e80 comp rx=0 tx=0).stop 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 10 monclient: shutdown 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 msgr2=0x7fb2dc13ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:52.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 0x7fb2dc13ac00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc037040 tx=0x7fb2cc004a40 comp rx=0 tx=0).stop 2024-01-20T11:16:52.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 shutdown_connections 2024-01-20T11:16:52.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb2bc06c8e0 0x7fb2bc06eda0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:52.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2dc0b0eb0 0x7fb2dc0b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:52.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2dc0a9ad0 0x7fb2dc13b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:52.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 --2- 172.21.15.110:0/2086954489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2dc0a96f0 0x7fb2dc13ac00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:52.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 >> 172.21.15.110:0/2086954489 conn(0x7fb2dc0b7bc0 msgr2=0x7fb2dc13c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:52.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 shutdown_connections 2024-01-20T11:16:52.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 -- 172.21.15.110:0/2086954489 wait complete. 2024-01-20T11:16:52.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:52.472+0000 7fb2e0f9e640 1 librados: shutdown 2024-01-20T11:16:52.483 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-01-20T11:16:51.763392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":15,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219096,"kb_used_data":2724,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755624,"statfs":{"total":773094113280,"available":772869758976,"internally_reserved":0,"allocated":2789376,"data_stored":1476858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":262144},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.033432"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.382168+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:49.382168+0000","last_peered":"2024-01-20T11:16:49.382168+0000","last_clean":"2024-01-20T11:16:49.382168+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:49.382168+0000","last_undegraded":"2024-01-20T11:16:49.382168+0000","last_fullsized":"2024-01-20T11:16:49.382168+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T22:26:49.902059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-20T11:15:29.356575+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:15:29.356575+0000","last_peered":"2024-01-20T11:15:29.356575+0000","last_clean":"2024-01-20T11:15:29.356575+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:15:29.356575+0000","last_undegraded":"2024-01-20T11:15:29.356575+0000","last_fullsized":"2024-01-20T11:15:29.356575+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T17:21:09.255806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.379296+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:49.379296+0000","last_peered":"2024-01-20T11:16:49.379296+0000","last_clean":"2024-01-20T11:16:49.379296+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:49.379296+0000","last_undegraded":"2024-01-20T11:16:49.379296+0000","last_fullsized":"2024-01-20T11:16:49.379296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T14:37:21.843002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":6,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-01-20T11:16:50.513148+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:50.513148+0000","last_peered":"2024-01-20T11:16:50.513148+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:50.513148+0000","last_undegraded":"2024-01-20T11:16:50.513148+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:34:49.467555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.375559+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:49.375559+0000","last_peered":"2024-01-20T11:16:49.375559+0000","last_clean":"2024-01-20T11:16:49.375559+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:49.375559+0000","last_undegraded":"2024-01-20T11:16:49.375559+0000","last_fullsized":"2024-01-20T11:16:49.375559+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:00:34.021727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":5,"reported_epoch":27,"state":"creating+peering","last_fresh":"2024-01-20T11:16:50.027385+0000","last_change":"2024-01-20T11:15:27.266904+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:16:50.027385+0000","last_undegraded":"2024-01-20T11:16:50.027385+0000","last_fullsized":"2024-01-20T11:16:50.027385+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-01-20T11:16:50.514558+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:50.514558+0000","last_peered":"2024-01-20T11:16:50.514558+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:50.514558+0000","last_undegraded":"2024-01-20T11:16:50.514558+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.377350+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:49.377350+0000","last_peered":"2024-01-20T11:16:49.377350+0000","last_clean":"2024-01-20T11:16:49.377350+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:49.377350+0000","last_undegraded":"2024-01-20T11:16:49.377350+0000","last_fullsized":"2024-01-20T11:16:49.377350+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:25:31.758358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":120,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-20T11:15:28.772627+0000","last_change":"2024-01-20T11:15:15.736505+0000","last_active":"2024-01-20T11:15:28.772627+0000","last_peered":"2024-01-20T11:15:28.772627+0000","last_clean":"2024-01-20T11:15:28.772627+0000","last_became_active":"2024-01-20T11:15:15.702676+0000","last_became_peered":"2024-01-20T11:15:15.702676+0000","last_unstale":"2024-01-20T11:15:28.772627+0000","last_undegraded":"2024-01-20T11:15:28.772627+0000","last_fullsized":"2024-01-20T11:15:28.772627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:50:44.769291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.1189999999999998},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":9.8699999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.839},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.1890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.265000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":17.045999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.7240000000000002},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":13.534000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.125},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":15.468999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.369},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":14.724}]}]},{"osd":7,"up_from":21,"seq":90194313240,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":262144},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.7859999999999996},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.9420000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.5350000000000001},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.417999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.367000000000001},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.845000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.231},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":32.773000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":27.498000000000001},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":13.026999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":29.834},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":25.001999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313241,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":3.8700000000000001},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.8689999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9930000000000003},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.010000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":9.6959999999999997},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.663}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.007},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":17.876000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":12.305999999999999},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.894}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":17.431000000000001},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.526}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444054,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.597},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":4.8780000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7489999999999997},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.584}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.475},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.412000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.272},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.041}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.169},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.2930000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":7.8680000000000003},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":16.018999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444055,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":643072,"data_stored":480471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":11.217000000000001},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":7.1639999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.0470000000000006},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.026999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.789999999999999},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.5999999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":10.256},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.495999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":12.337999999999999},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":13.667999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.7010000000000005},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.571999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444055,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":17.498999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":9.8089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.431},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.721}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":16.687999999999999},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.9000000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.481999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.8300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":18.321000000000002},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":12.472}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.552},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.647}]}]},{"osd":6,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":89486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":6.8399999999999999},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":10.265000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":14.116},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":12.711}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6989999999999998},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.291}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.9990000000000006},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.868}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.489000000000001},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.452}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.218},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.413}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:16:55.486 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:16:55.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.583+0000 7fdf79446640 10 monclient: get_monmap_and_config 2024-01-20T11:16:55.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.583+0000 7fdf79446640 10 monclient: build_initial_monmap 2024-01-20T11:16:55.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.583+0000 7fdf79446640 10 monclient: monmap: 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:16:55.585529+0000 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:16:55.585529+0000 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.584+0000 7fdf79446640 1 Processor -- start 2024-01-20T11:16:55.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.584+0000 7fdf79446640 1 -- start start 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.584+0000 7fdf79446640 10 monclient: init 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf740a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf740a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdf74142af0 0x7fdf7414d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 10 monclient(hunting): picked mon.noname-c con 0x7fdf74142af0 addr v2:172.21.15.134:3300/0 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b25c0 con 0x7fdf740a9ad0 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf74142f50 con 0x7fdf740a96f0 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf740aaea0 con 0x7fdf74142af0 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf79446640 10 monclient(hunting): authenticate will time out at 1458.936359s 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf727fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdf740a9ad0 0x7fdf741423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59842/0 (socket says 172.21.15.110:59842) 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf737fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fdf74142af0 0x7fdf7414d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.585+0000 7fdf727fc640 1 -- 172.21.15.110:0/2975255741 learned_addr learned my addr 172.21.15.110:0/2975255741 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.586+0000 7fdf737fe640 10 monclient(hunting): get_auth_request con 0x7fdf74142af0 auth_method 0 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.586+0000 7fdf737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.586+0000 7fdf737fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.586+0000 7fdf737fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.587+0000 7fdf737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.587+0000 7fdf737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.587+0000 7fdf737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.588+0000 7fdf71ffb640 1 -- 172.21.15.110:0/2975255741 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256851923 0 0) 0x7fdf740b25c0 con 0x7fdf740a9ad0 2024-01-20T11:16:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.588+0000 7fdf71ffb640 10 monclient(hunting): _negotiate 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.588+0000 7fdf71ffb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.588+0000 7fdf71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.588+0000 7fdf71ffb640 10 monclient(hunting): my global_id is 4407 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.588+0000 7fdf71ffb640 1 -- 172.21.15.110:0/2975255741 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c003520 con 0x7fdf740a9ad0 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.589+0000 7fdf737fe640 10 monclient(hunting): handle_auth_done global_id 4441 payload 274 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.589+0000 7fdf737fe640 1 -- 172.21.15.110:0/2975255741 >> v1:172.21.15.134:6789/0 conn(0x7fdf740a96f0 legacy=0x7fdf7413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.589+0000 7fdf737fe640 1 -- 172.21.15.110:0/2975255741 >> v1:172.21.15.110:6789/0 conn(0x7fdf740a9ad0 legacy=0x7fdf741423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.589+0000 7fdf737fe640 10 monclient: _finish_hunting 0 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.589+0000 7fdf737fe640 1 monclient: found mon.noname-c 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.589+0000 7fdf737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.589+0000 7fdf737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:16:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.589+0000 7fdf737fe640 1 -- 172.21.15.110:0/2975255741 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7413fbe0 con 0x7fdf74142af0 2024-01-20T11:16:55.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.594+0000 7fdf737fe640 1 --2- 172.21.15.110:0/2975255741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf74142af0 0x7fdf7414d860 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009d90 tx=0x7fdf6c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f00bbbd66486d7de server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 1 -- 172.21.15.110:0/2975255741 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c02cc60 con 0x7fdf74142af0 2024-01-20T11:16:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:16:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 10 monclient: dump: 2024-01-20T11:16:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 10 monclient: _finish_auth 0 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 1 -- 172.21.15.110:0/2975255741 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c02cdc0 con 0x7fdf74142af0 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 1 -- 172.21.15.110:0/2975255741 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c034660 con 0x7fdf74142af0 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf79446640 5 monclient: authenticate success, global_id 4441 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf79446640 10 monclient: get_monmap_and_config success 2024-01-20T11:16:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf79446640 4 set_mon_vals no callback set 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.598+0000 7fdf71ffb640 10 monclient: dump: 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:55.601 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:55.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 10 monclient: shutdown 2024-01-20T11:16:55.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 1 -- 172.21.15.110:0/2975255741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf74142af0 msgr2=0x7fdf7414d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 1 --2- 172.21.15.110:0/2975255741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf74142af0 0x7fdf7414d860 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009d90 tx=0x7fdf6c02beb0 comp rx=0 tx=0).stop 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 1 -- 172.21.15.110:0/2975255741 shutdown_connections 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 1 --2- 172.21.15.110:0/2975255741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf74142af0 0x7fdf7414d860 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 1 -- 172.21.15.110:0/2975255741 >> 172.21.15.110:0/2975255741 conn(0x7fdf740b7bc0 msgr2=0x7fdf740b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 1 -- 172.21.15.110:0/2975255741 shutdown_connections 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 1 -- 172.21.15.110:0/2975255741 wait complete. 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 10 monclient: build_initial_monmap 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.599+0000 7fdf79446640 10 monclient: monmap: 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.600+0000 7fdf79446640 1 librados: starting msgr at 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.600+0000 7fdf79446640 1 librados: starting objecter 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.600+0000 7fdf79446640 1 Processor -- start 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.600+0000 7fdf79446640 1 -- start start 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.600+0000 7fdf79446640 1 librados: setting wanted keys 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.600+0000 7fdf79446640 1 librados: calling monclient init 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.600+0000 7fdf79446640 10 monclient: init 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf740a96f0 0x7fdf741403b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient(hunting): picked mon.noname-c con 0x7fdf740a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf740a9ad0 0x7fdf741295b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient(hunting): picked mon.noname-b con 0x7fdf740a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf74142af0 0x7fdf7413cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient(hunting): picked mon.noname-a con 0x7fdf74142af0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74140e80 con 0x7fdf74142af0 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74128eb0 con 0x7fdf740a9ad0 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74129190 con 0x7fdf740a96f0 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf79446640 10 monclient(hunting): authenticate will time out at 1458.952340s 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf727fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf740a9ad0 0x7fdf741295b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf72ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf740a96f0 0x7fdf741403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.601+0000 7fdf737fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf74142af0 0x7fdf7413cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf727fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf740a9ad0 0x7fdf741295b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46744/0 (socket says 172.21.15.110:46744) 2024-01-20T11:16:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf727fc640 1 -- 172.21.15.110:0/1505906107 learned_addr learned my addr 172.21.15.110:0/1505906107 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf727fc640 10 monclient(hunting): get_auth_request con 0x7fdf740a9ad0 auth_method 0 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf727fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf727fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf72ffd640 10 monclient(hunting): get_auth_request con 0x7fdf740a96f0 auth_method 0 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf72ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf737fe640 10 monclient(hunting): get_auth_request con 0x7fdf74142af0 auth_method 0 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf737fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf737fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.602+0000 7fdf72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 10 monclient(hunting): handle_auth_done global_id 4447 payload 824 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 1 -- 172.21.15.110:0/1505906107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf740a96f0 msgr2=0x7fdf741403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf740a96f0 0x7fdf741403b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 1 -- 172.21.15.110:0/1505906107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf74142af0 msgr2=0x7fdf7413cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf74142af0 0x7fdf7413cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 10 monclient: _finish_hunting 0 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 1 monclient: found mon.noname-b 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.605+0000 7fdf727fc640 1 -- 172.21.15.110:0/1505906107 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c009a70 con 0x7fdf740a9ad0 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.607+0000 7fdf727fc640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf740a9ad0 0x7fdf741295b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800c500 tx=0x7fdf6800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.607+0000 7fdf5b7fe640 10 client.?.objecter ms_handle_connect 0x7fdf740a9ad0 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.607+0000 7fdf5b7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:16:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf5b7fe640 1 -- 172.21.15.110:0/1505906107 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68005ce0 con 0x7fdf740a9ad0 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf5b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf5b7fe640 10 monclient: dump: 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf5b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:16:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf5b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf79446640 5 monclient: authenticate success, global_id 4447 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf79446640 10 monclient: _renew_subs 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf79446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7413fe00 con 0x7fdf740a9ad0 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf79446640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf79446640 10 monclient: _renew_subs 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf79446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7413e610 con 0x7fdf740a9ad0 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.609+0000 7fdf79446640 1 librados: init done 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.610+0000 7fdf79446640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.610+0000 7fdf79446640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.610+0000 7fdf79446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.610+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf74075490 con 0x7fdf740a9ad0 2024-01-20T11:16:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.818+0000 7fdf5b7fe640 1 -- 172.21.15.110:0/1505906107 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf68005e40 con 0x7fdf740a9ad0 2024-01-20T11:16:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.818+0000 7fdf5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.818+0000 7fdf717fa640 4 set_mon_vals no callback set 2024-01-20T11:16:55.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.853+0000 7fdf5b7fe640 1 -- 172.21.15.110:0/1505906107 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf680072f0 con 0x7fdf740a9ad0 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.853+0000 7fdf5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.853+0000 7fdf5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.853+0000 7fdf5b7fe640 10 monclient: dump: 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:55.854 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:55.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.853+0000 7fdf5b7fe640 1 -- 172.21.15.110:0/1505906107 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdf68007510 con 0x7fdf740a9ad0 2024-01-20T11:16:55.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.854+0000 7fdf5b7fe640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdf6006c860 0x7fdf6006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:55.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.854+0000 7fdf72ffd640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdf6006c860 0x7fdf6006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:55.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.854+0000 7fdf72ffd640 10 monclient: get_auth_request con 0x7fdf6006c860 auth_method 0 2024-01-20T11:16:55.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.854+0000 7fdf72ffd640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdf6006c860 0x7fdf6006ed20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf64001780 tx=0x7fdf64009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:55.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.854+0000 7fdf5b7fe640 10 client.4447.objecter ms_handle_connect 0x7fdf6006c860 2024-01-20T11:16:55.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.856+0000 7fdf5b7fe640 1 -- 172.21.15.110:0/1505906107 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdf68057d50 con 0x7fdf740a9ad0 2024-01-20T11:16:55.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.856+0000 7fdf5b7fe640 10 client.4447.objecter ms_dispatch 0x7fdf74000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:16:55.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.856+0000 7fdf5b7fe640 3 client.4447.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:16:55.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.856+0000 7fdf5b7fe640 3 client.4447.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:16:55.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.857+0000 7fdf5b7fe640 20 client.4447.objecter dump_active .. 0 homeless 2024-01-20T11:16:55.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.867+0000 7fdf5b7fe640 1 -- 172.21.15.110:0/1505906107 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fdf68054090 con 0x7fdf740a9ad0 2024-01-20T11:16:55.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.867+0000 7fdf5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:55.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:55.867+0000 7fdf5b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:16:56.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.040+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf74129300 con 0x7fdf6006c860 2024-01-20T11:16:56.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.043+0000 7fdf5b7fe640 1 -- 172.21.15.110:0/1505906107 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64552 (secure 0 0 0) 0x7fdf74129300 con 0x7fdf6006c860 2024-01-20T11:16:56.045 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:16:56.045 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:16:56.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 10 librados: watch_flush enter 2024-01-20T11:16:56.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 10 librados: watch_flush exit 2024-01-20T11:16:56.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 20 client.4447.objecter shutdown clearing up homeless session... 2024-01-20T11:16:56.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 10 client.4447.objecter successfully canceled tick 2024-01-20T11:16:56.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdf6006c860 msgr2=0x7fdf6006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdf6006c860 0x7fdf6006ed20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf64001780 tx=0x7fdf64009040 comp rx=0 tx=0).stop 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 10 monclient: shutdown 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf740a9ad0 msgr2=0x7fdf741295b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf740a9ad0 0x7fdf741295b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800c500 tx=0x7fdf6800c9c0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 shutdown_connections 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdf6006c860 0x7fdf6006ed20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf74142af0 0x7fdf7413cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf740a9ad0 0x7fdf741295b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 --2- 172.21.15.110:0/1505906107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf740a96f0 0x7fdf741403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 >> 172.21.15.110:0/1505906107 conn(0x7fdf740b7bc0 msgr2=0x7fdf740b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 shutdown_connections 2024-01-20T11:16:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 -- 172.21.15.110:0/1505906107 wait complete. 2024-01-20T11:16:56.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.045+0000 7fdf79446640 1 librados: shutdown 2024-01-20T11:16:56.057 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-01-20T11:16:54.731571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219220,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755500,"statfs":{"total":773094113280,"available":772869632000,"internally_reserved":0,"allocated":2924544,"data_stored":1536918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":262144},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.483001"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.382168+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:49.382168+0000","last_peered":"2024-01-20T11:16:49.382168+0000","last_clean":"2024-01-20T11:16:49.382168+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:49.382168+0000","last_undegraded":"2024-01-20T11:16:49.382168+0000","last_fullsized":"2024-01-20T11:16:49.382168+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T22:26:49.902059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.379296+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:49.379296+0000","last_peered":"2024-01-20T11:16:49.379296+0000","last_clean":"2024-01-20T11:16:49.379296+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:49.379296+0000","last_undegraded":"2024-01-20T11:16:49.379296+0000","last_fullsized":"2024-01-20T11:16:49.379296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T14:37:21.843002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":6,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-01-20T11:16:50.513148+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:50.513148+0000","last_peered":"2024-01-20T11:16:50.513148+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:50.513148+0000","last_undegraded":"2024-01-20T11:16:50.513148+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:34:49.467555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.375559+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:49.375559+0000","last_peered":"2024-01-20T11:16:49.375559+0000","last_clean":"2024-01-20T11:16:49.375559+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:49.375559+0000","last_undegraded":"2024-01-20T11:16:49.375559+0000","last_fullsized":"2024-01-20T11:16:49.375559+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:00:34.021727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":5,"reported_epoch":27,"state":"creating+peering","last_fresh":"2024-01-20T11:16:50.027385+0000","last_change":"2024-01-20T11:15:27.266904+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:16:50.027385+0000","last_undegraded":"2024-01-20T11:16:50.027385+0000","last_fullsized":"2024-01-20T11:16:50.027385+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-01-20T11:16:50.514558+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:50.514558+0000","last_peered":"2024-01-20T11:16:50.514558+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:50.514558+0000","last_undegraded":"2024-01-20T11:16:50.514558+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.377350+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:49.377350+0000","last_peered":"2024-01-20T11:16:49.377350+0000","last_clean":"2024-01-20T11:16:49.377350+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:49.377350+0000","last_undegraded":"2024-01-20T11:16:49.377350+0000","last_fullsized":"2024-01-20T11:16:49.377350+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:25:31.758358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280538,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.2520000000000007},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.1909999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6639999999999997},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":14.948}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.518000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":15.635999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":16.326000000000001},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":12.098000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":16.981999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":23.085000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":18.131},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":21.393000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313240,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":262144},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.7859999999999996},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.9420000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.5350000000000001},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.417999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.367000000000001},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.845000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.231},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":32.773000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":27.498000000000001},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":13.026999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":29.834},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":25.001999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313241,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":3.8700000000000001},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.8689999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9930000000000003},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.010000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":9.6959999999999997},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.663}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.007},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":17.876000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":12.305999999999999},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.894}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":17.431000000000001},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.526}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444055,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.9349999999999996},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.7609999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":13.702999999999999},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.177}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.728999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.874000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.715},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.507999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.417999999999999},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.8870000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":16.811},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":18.111999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444056,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":496762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":11.746},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":8.2460000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":18.129999999999999},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":6.1100000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":26.484999999999999},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":31.634}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":25.541},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":12.371}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":7.5279999999999996},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":13.222}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.962},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.096}]}]},{"osd":0,"up_from":17,"seq":73014444056,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":5.4509999999999996},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.6900000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.4770000000000003},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.898999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":17.065999999999999},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.4299999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.147},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.32}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":17.611999999999998},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":9.6110000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":11.207000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.449}]}]},{"osd":6,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":89486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":6.8399999999999999},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":10.265000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":14.116},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":12.711}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6989999999999998},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.291}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.9990000000000006},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.868}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.489000000000001},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.452}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.218},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.413}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:16:56.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:16:56.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.158+0000 7f6b6e626640 10 monclient: get_monmap_and_config 2024-01-20T11:16:56.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.158+0000 7f6b6e626640 10 monclient: build_initial_monmap 2024-01-20T11:16:56.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.158+0000 7f6b6e626640 10 monclient: monmap: 2024-01-20T11:16:56.159 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:56.159 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.159 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:16:56.160010+0000 2024-01-20T11:16:56.160 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:16:56.160010+0000 2024-01-20T11:16:56.160 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:56.160 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.160 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:16:56.160 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:16:56.160 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:16:56.160 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:16:56.160 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:16:56.160 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 1 Processor -- start 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 1 -- start start 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 10 monclient: init 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b6806bb00 addr v1:172.21.15.134:6789/0 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b680af790 addr v1:172.21.15.110:6789/0 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6b680aeb20 0x7f6b680aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 10 monclient(hunting): picked mon.noname-c con 0x7f6b680aeb20 addr v2:172.21.15.134:3300/0 2024-01-20T11:16:56.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6806bee0 con 0x7f6b680af790 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b680ada30 con 0x7f6b6806bb00 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b680add10 con 0x7f6b680aeb20 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.159+0000 7f6b6e626640 10 monclient(hunting): authenticate will time out at 1459.510977s 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.160+0000 7f6b6cbc3640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6b680aeb20 0x7f6b680aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.160+0000 7f6b677fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6b680af790 0x7f6b680ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59844/0 (socket says 172.21.15.110:59844) 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.160+0000 7f6b677fe640 1 -- 172.21.15.110:0/2931017041 learned_addr learned my addr 172.21.15.110:0/2931017041 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.160+0000 7f6b6cbc3640 10 monclient(hunting): get_auth_request con 0x7f6b680aeb20 auth_method 0 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.160+0000 7f6b6cbc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.160+0000 7f6b6cbc3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.160+0000 7f6b6cbc3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.161+0000 7f6b6cbc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:56.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.161+0000 7f6b6cbc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:56.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.161+0000 7f6b6cbc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:56.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.162+0000 7f6b66ffd640 1 -- 172.21.15.110:0/2931017041 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198407681 0 0) 0x7f6b6806bee0 con 0x7f6b680af790 2024-01-20T11:16:56.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.162+0000 7f6b66ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:16:56.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.162+0000 7f6b66ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.162+0000 7f6b66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.162+0000 7f6b66ffd640 10 monclient(hunting): my global_id is 4413 2024-01-20T11:16:56.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.162+0000 7f6b66ffd640 1 -- 172.21.15.110:0/2931017041 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b50003520 con 0x7f6b680af790 2024-01-20T11:16:56.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.163+0000 7f6b6cbc3640 10 monclient(hunting): handle_auth_done global_id 4450 payload 274 2024-01-20T11:16:56.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.163+0000 7f6b6cbc3640 1 -- 172.21.15.110:0/2931017041 >> v1:172.21.15.134:6789/0 conn(0x7f6b6806bb00 legacy=0x7f6b680b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.163+0000 7f6b6cbc3640 1 -- 172.21.15.110:0/2931017041 >> v1:172.21.15.110:6789/0 conn(0x7f6b680af790 legacy=0x7f6b680ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.163+0000 7f6b6cbc3640 10 monclient: _finish_hunting 0 2024-01-20T11:16:56.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.163+0000 7f6b6cbc3640 1 monclient: found mon.noname-c 2024-01-20T11:16:56.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.163+0000 7f6b6cbc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:56.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.163+0000 7f6b6cbc3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:16:56.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.163+0000 7f6b6cbc3640 1 -- 172.21.15.110:0/2931017041 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b680adff0 con 0x7f6b680aeb20 2024-01-20T11:16:56.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.166+0000 7f6b6cbc3640 1 --2- 172.21.15.110:0/2931017041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680aeb20 0x7f6b680aef90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009d90 tx=0x7f6b6002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41e1de6ca01d1b0a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:56.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.168+0000 7f6b66ffd640 1 -- 172.21.15.110:0/2931017041 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6002cc60 con 0x7f6b680aeb20 2024-01-20T11:16:56.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.168+0000 7f6b66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:56.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.168+0000 7f6b66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:16:56.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.168+0000 7f6b66ffd640 10 monclient: dump: 2024-01-20T11:16:56.169 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.168+0000 7f6b66ffd640 10 monclient: _finish_auth 0 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.168+0000 7f6b66ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.168+0000 7f6b66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:56.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.168+0000 7f6b6e626640 5 monclient: authenticate success, global_id 4450 2024-01-20T11:16:56.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.168+0000 7f6b6e626640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:16:56.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b66ffd640 1 -- 172.21.15.110:0/2931017041 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6002cdc0 con 0x7f6b680aeb20 2024-01-20T11:16:56.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b66ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:56.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b66ffd640 1 -- 172.21.15.110:0/2931017041 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60034640 con 0x7f6b680aeb20 2024-01-20T11:16:56.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:56.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b66ffd640 10 monclient: dump: 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b6e626640 10 monclient: get_monmap_and_config success 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b6e626640 4 set_mon_vals no callback set 2024-01-20T11:16:56.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b6e626640 10 monclient: shutdown 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b6e626640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b6e626640 1 -- 172.21.15.110:0/2931017041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680aeb20 msgr2=0x7f6b680aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b6e626640 1 --2- 172.21.15.110:0/2931017041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680aeb20 0x7f6b680aef90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009d90 tx=0x7f6b6002beb0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b6e626640 1 -- 172.21.15.110:0/2931017041 shutdown_connections 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b6e626640 1 --2- 172.21.15.110:0/2931017041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680aeb20 0x7f6b680aef90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.170+0000 7f6b6e626640 1 -- 172.21.15.110:0/2931017041 >> 172.21.15.110:0/2931017041 conn(0x7f6b68066d60 msgr2=0x7f6b680b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.171+0000 7f6b6e626640 1 -- 172.21.15.110:0/2931017041 shutdown_connections 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.171+0000 7f6b6e626640 1 -- 172.21.15.110:0/2931017041 wait complete. 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.171+0000 7f6b6e626640 10 monclient: build_initial_monmap 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.171+0000 7f6b6e626640 10 monclient: monmap: 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:16:56.174 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:16:56.175 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:56.175 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.175 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:16:56.175 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:16:56.175 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:16:56.175 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.171+0000 7f6b6e626640 1 librados: starting msgr at 2024-01-20T11:16:56.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.171+0000 7f6b6e626640 1 librados: starting objecter 2024-01-20T11:16:56.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.171+0000 7f6b6e626640 1 Processor -- start 2024-01-20T11:16:56.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 1 -- start start 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 1 librados: setting wanted keys 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 1 librados: calling monclient init 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient: init 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b6806bb00 0x7f6b680b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient(hunting): picked mon.noname-c con 0x7f6b6806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b680aeb20 0x7f6b681da4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient(hunting): picked mon.noname-a con 0x7f6b680aeb20 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680af790 0x7f6b681d6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient(hunting): picked mon.noname-b con 0x7f6b680af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:56.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b681dac30 con 0x7f6b680aeb20 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b68120f70 con 0x7f6b680af790 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b681d4a90 con 0x7f6b6806bb00 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.172+0000 7f6b6e626640 10 monclient(hunting): authenticate will time out at 1459.523873s 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b680aeb20 0x7f6b681da4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b6cbc3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680af790 0x7f6b681d6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b67fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b6806bb00 0x7f6b680b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b677fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b680aeb20 0x7f6b681da4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34312/0 (socket says 172.21.15.110:34312) 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b677fe640 1 -- 172.21.15.110:0/3294782727 learned_addr learned my addr 172.21.15.110:0/3294782727 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b677fe640 10 monclient(hunting): get_auth_request con 0x7f6b680aeb20 auth_method 0 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b6cbc3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680af790 0x7f6b681d6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46764/0 (socket says 172.21.15.110:46764) 2024-01-20T11:16:56.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b677fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b677fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b6cbc3640 10 monclient(hunting): get_auth_request con 0x7f6b680af790 auth_method 0 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b6cbc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b6cbc3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b6cbc3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b67fff640 10 monclient(hunting): get_auth_request con 0x7f6b6806bb00 auth_method 0 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b67fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.173+0000 7f6b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.174+0000 7f6b6cbc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.174+0000 7f6b6cbc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.174+0000 7f6b6cbc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.174+0000 7f6b677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:56.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.174+0000 7f6b677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.174+0000 7f6b677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.174+0000 7f6b67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.174+0000 7f6b67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.174+0000 7f6b67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 10 monclient(hunting): handle_auth_done global_id 4453 payload 824 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 1 -- 172.21.15.110:0/3294782727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b6806bb00 msgr2=0x7f6b680b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b6806bb00 0x7f6b680b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 1 -- 172.21.15.110:0/3294782727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b680aeb20 msgr2=0x7f6b681da4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b680aeb20 0x7f6b681da4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 10 monclient: _finish_hunting 0 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 1 monclient: found mon.noname-b 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:16:56.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.176+0000 7f6b6cbc3640 1 -- 172.21.15.110:0/3294782727 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b60009a70 con 0x7f6b680af790 2024-01-20T11:16:56.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.179+0000 7f6b6cbc3640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680af790 0x7f6b681d6f50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b60033000 tx=0x7f6b60004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:56.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.179+0000 7f6b64ff9640 10 client.?.objecter ms_handle_connect 0x7f6b680af790 2024-01-20T11:16:56.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.179+0000 7f6b64ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:16:56.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 1 -- 172.21.15.110:0/3294782727 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60004800 con 0x7f6b680af790 2024-01-20T11:16:56.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:56.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 10 monclient: dump: 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 10 monclient: _finish_auth 0 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:56.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 1 -- 172.21.15.110:0/3294782727 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b60003e90 con 0x7f6b680af790 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 1 -- 172.21.15.110:0/3294782727 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60034640 con 0x7f6b680af790 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b64ff9640 10 monclient: dump: 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b6e626640 5 monclient: authenticate success, global_id 4453 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b6e626640 10 monclient: _renew_subs 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b6e626640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b681db0c0 con 0x7f6b680af790 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b65ffb640 4 set_mon_vals no callback set 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b6e626640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b6e626640 10 monclient: _renew_subs 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b6e626640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b681d7490 con 0x7f6b680af790 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.182+0000 7f6b6e626640 1 librados: init done 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.183+0000 7f6b6e626640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.183+0000 7f6b6e626640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.183+0000 7f6b6e626640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:56.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.183+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b681daa10 con 0x7f6b680af790 2024-01-20T11:16:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.436+0000 7f6b64ff9640 1 -- 172.21.15.110:0/3294782727 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6b600347a0 con 0x7f6b680af790 2024-01-20T11:16:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.436+0000 7f6b64ff9640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b5406c8b0 0x7f6b5406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.436+0000 7f6b67fff640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b5406c8b0 0x7f6b5406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.436+0000 7f6b67fff640 10 monclient: get_auth_request con 0x7f6b5406c8b0 auth_method 0 2024-01-20T11:16:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.437+0000 7f6b64ff9640 10 client.4453.objecter ms_handle_connect 0x7f6b5406c8b0 2024-01-20T11:16:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.437+0000 7f6b67fff640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b5406c8b0 0x7f6b5406ed70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6b58000f60 tx=0x7f6b58009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.439+0000 7f6b64ff9640 1 -- 172.21.15.110:0/3294782727 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6b6007b050 con 0x7f6b680af790 2024-01-20T11:16:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.439+0000 7f6b64ff9640 10 client.4453.objecter ms_dispatch 0x7f6b68000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:16:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.439+0000 7f6b64ff9640 3 client.4453.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:16:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.439+0000 7f6b64ff9640 3 client.4453.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:16:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.439+0000 7f6b64ff9640 20 client.4453.objecter dump_active .. 0 homeless 2024-01-20T11:16:56.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.450+0000 7f6b64ff9640 1 -- 172.21.15.110:0/3294782727 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6b60077390 con 0x7f6b680af790 2024-01-20T11:16:56.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.450+0000 7f6b64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:56.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.450+0000 7f6b64ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:16:56.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.618+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b6806d380 con 0x7f6b5406c8b0 2024-01-20T11:16:56.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.621+0000 7f6b64ff9640 1 -- 172.21.15.110:0/3294782727 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64552 (secure 0 0 0) 0x7f6b580014a0 con 0x7f6b5406c8b0 2024-01-20T11:16:56.622 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:16:56.623 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:16:56.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 10 librados: watch_flush enter 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 10 librados: watch_flush exit 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 20 client.4453.objecter shutdown clearing up homeless session... 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 10 client.4453.objecter successfully canceled tick 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b5406c8b0 msgr2=0x7f6b5406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b5406c8b0 0x7f6b5406ed70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6b58000f60 tx=0x7f6b58009040 comp rx=0 tx=0).stop 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 10 monclient: shutdown 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680af790 msgr2=0x7f6b681d6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680af790 0x7f6b681d6f50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b60033000 tx=0x7f6b60004a40 comp rx=0 tx=0).stop 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 shutdown_connections 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b5406c8b0 0x7f6b5406ed70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b680af790 0x7f6b681d6f50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b680aeb20 0x7f6b681da4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 --2- 172.21.15.110:0/3294782727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b6806bb00 0x7f6b680b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 >> 172.21.15.110:0/3294782727 conn(0x7f6b68066d60 msgr2=0x7f6b680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:56.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 shutdown_connections 2024-01-20T11:16:56.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 -- 172.21.15.110:0/3294782727 wait complete. 2024-01-20T11:16:56.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.623+0000 7f6b6e626640 1 librados: shutdown 2024-01-20T11:16:56.634 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-01-20T11:16:54.731571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219220,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755500,"statfs":{"total":773094113280,"available":772869632000,"internally_reserved":0,"allocated":2924544,"data_stored":1536918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":262144},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.483001"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.382168+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:49.382168+0000","last_peered":"2024-01-20T11:16:49.382168+0000","last_clean":"2024-01-20T11:16:49.382168+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:49.382168+0000","last_undegraded":"2024-01-20T11:16:49.382168+0000","last_fullsized":"2024-01-20T11:16:49.382168+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T22:26:49.902059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.379296+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:49.379296+0000","last_peered":"2024-01-20T11:16:49.379296+0000","last_clean":"2024-01-20T11:16:49.379296+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:49.379296+0000","last_undegraded":"2024-01-20T11:16:49.379296+0000","last_fullsized":"2024-01-20T11:16:49.379296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T14:37:21.843002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":6,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-01-20T11:16:50.513148+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:50.513148+0000","last_peered":"2024-01-20T11:16:50.513148+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:50.513148+0000","last_undegraded":"2024-01-20T11:16:50.513148+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:34:49.467555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.375559+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:49.375559+0000","last_peered":"2024-01-20T11:16:49.375559+0000","last_clean":"2024-01-20T11:16:49.375559+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:49.375559+0000","last_undegraded":"2024-01-20T11:16:49.375559+0000","last_fullsized":"2024-01-20T11:16:49.375559+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:00:34.021727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":5,"reported_epoch":27,"state":"creating+peering","last_fresh":"2024-01-20T11:16:50.027385+0000","last_change":"2024-01-20T11:15:27.266904+0000","last_active":"2024-01-20T11:15:27.266904+0000","last_peered":"2024-01-20T11:15:27.266904+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-20T11:16:50.027385+0000","last_undegraded":"2024-01-20T11:16:50.027385+0000","last_fullsized":"2024-01-20T11:16:50.027385+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+undersized","last_fresh":"2024-01-20T11:16:50.514558+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:50.514558+0000","last_peered":"2024-01-20T11:16:50.514558+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:50.514558+0000","last_undegraded":"2024-01-20T11:16:50.514558+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-20T11:16:49.377350+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:49.377350+0000","last_peered":"2024-01-20T11:16:49.377350+0000","last_clean":"2024-01-20T11:16:49.377350+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:49.377350+0000","last_undegraded":"2024-01-20T11:16:49.377350+0000","last_fullsized":"2024-01-20T11:16:49.377350+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:25:31.758358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280538,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.2520000000000007},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.1909999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6639999999999997},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":14.948}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.518000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":15.635999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":16.326000000000001},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":12.098000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":16.981999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":23.085000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":18.131},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":21.393000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313240,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":262144},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.7859999999999996},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.9420000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.5350000000000001},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.417999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.367000000000001},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.845000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.231},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":32.773000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":27.498000000000001},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":13.026999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":29.834},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":25.001999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313241,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":3.8700000000000001},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.8689999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9930000000000003},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.010000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":9.6959999999999997},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.663}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.007},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":17.876000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":12.305999999999999},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.894}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":17.431000000000001},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.526}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444055,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.9349999999999996},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.7609999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":13.702999999999999},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.177}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.728999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.874000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.715},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.507999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.417999999999999},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.8870000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":16.811},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":18.111999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444056,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":496762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":11.746},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":8.2460000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":18.129999999999999},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":6.1100000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":26.484999999999999},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":31.634}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":25.541},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":12.371}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":7.5279999999999996},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":13.222}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.962},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.096}]}]},{"osd":0,"up_from":17,"seq":73014444056,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":5.4509999999999996},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.6900000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.4770000000000003},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.898999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":17.065999999999999},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.4299999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.147},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.32}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":17.611999999999998},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":9.6110000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":11.207000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.449}]}]},{"osd":6,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":89486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":6.8399999999999999},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":10.265000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":14.116},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":12.711}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6989999999999998},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.291}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.9990000000000006},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.868}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.489000000000001},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.452}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.218},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.413}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:16:56.636 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:16:56.636 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 6, 'reported_epoch': 28, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:50.513148+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:50.513148+0000', 'last_peered': '2024-01-20T11:16:50.513148+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:50.513148+0000', 'last_undegraded': '2024-01-20T11:16:50.513148+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:34:49.467555+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:16:56.636 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:16:56.636 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 5, 'reported_epoch': 27, 'state': 'creating+peering', 'last_fresh': '2024-01-20T11:16:50.027385+0000', 'last_change': '2024-01-20T11:15:27.266904+0000', 'last_active': '2024-01-20T11:15:27.266904+0000', 'last_peered': '2024-01-20T11:15:27.266904+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-20T11:16:50.027385+0000', 'last_undegraded': '2024-01-20T11:16:50.027385+0000', 'last_fullsized': '2024-01-20T11:16:50.027385+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:16:56.636 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:16:56.636 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 28, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:50.514558+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:50.514558+0000', 'last_peered': '2024-01-20T11:16:50.514558+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:50.514558+0000', 'last_undegraded': '2024-01-20T11:16:50.514558+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:16:56.637 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:16:56.637 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 127, 'reported_epoch': 29, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:51.809778+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:51.809778+0000', 'last_peered': '2024-01-20T11:16:51.809778+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:51.809778+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:16:56.637 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:16:56.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.736+0000 7fe02b90f640 10 monclient: get_monmap_and_config 2024-01-20T11:16:56.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.736+0000 7fe02b90f640 10 monclient: build_initial_monmap 2024-01-20T11:16:56.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.736+0000 7fe02b90f640 10 monclient: monmap: 2024-01-20T11:16:56.737 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:16:56.737781+0000 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:16:56.737781+0000 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.736+0000 7fe02b90f640 1 Processor -- start 2024-01-20T11:16:56.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.736+0000 7fe02b90f640 1 -- start start 2024-01-20T11:16:56.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.736+0000 7fe02b90f640 10 monclient: init 2024-01-20T11:16:56.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:56.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-01-20T11:16:56.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0240a96f0 0x7fe02413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0240a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe02413fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a9ad0 0x7fe024142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0240a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0240b25c0 con 0x7fe02413fa10 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0240aaea0 con 0x7fe0240a96f0 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240ab0a0 con 0x7fe0240a9ad0 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.737+0000 7fe02b90f640 10 monclient(hunting): authenticate will time out at 1460.088739s 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.738+0000 7fe029eac640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a9ad0 0x7fe024142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.738+0000 7fe028eaa640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fe02413fa10 0x7fe0241423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:34064/0 (socket says 172.21.15.110:34064) 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.738+0000 7fe028eaa640 1 -- 172.21.15.110:0/3521734767 learned_addr learned my addr 172.21.15.110:0/3521734767 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.738+0000 7fe029eac640 10 monclient(hunting): get_auth_request con 0x7fe0240a9ad0 auth_method 0 2024-01-20T11:16:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.738+0000 7fe029eac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.738+0000 7fe029eac640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.738+0000 7fe029eac640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.739+0000 7fe029eac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.739+0000 7fe029eac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.739+0000 7fe029eac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe01bfff640 1 -- 172.21.15.110:0/3521734767 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363424771 0 0) 0x7fe0240b25c0 con 0x7fe02413fa10 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe01bfff640 10 monclient(hunting): _negotiate 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe01bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe01bfff640 10 monclient(hunting): my global_id is 4456 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe01bfff640 1 -- 172.21.15.110:0/3521734767 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe010003520 con 0x7fe02413fa10 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe029eac640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe029eac640 1 -- 172.21.15.110:0/3521734767 >> v2:172.21.15.110:3300/0 conn(0x7fe0240a96f0 msgr2=0x7fe02413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe029eac640 1 --2- 172.21.15.110:0/3521734767 >> v2:172.21.15.110:3300/0 conn(0x7fe0240a96f0 0x7fe02413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe029eac640 1 -- 172.21.15.110:0/3521734767 >> v1:172.21.15.134:6789/0 conn(0x7fe02413fa10 legacy=0x7fe0241423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe029eac640 10 monclient: _finish_hunting 0 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe029eac640 1 monclient: found mon.noname-b 2024-01-20T11:16:56.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe029eac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:56.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe029eac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:16:56.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.740+0000 7fe029eac640 1 -- 172.21.15.110:0/3521734767 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0240b3fb0 con 0x7fe0240a9ad0 2024-01-20T11:16:56.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.742+0000 7fe029eac640 1 --2- 172.21.15.110:0/3521734767 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a9ad0 0x7fe024142af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe01c009d90 tx=0x7fe01c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37d9ac218801a2c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:56.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 1 -- 172.21.15.110:0/3521734767 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c02cbc0 con 0x7fe0240a9ad0 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 10 monclient: dump: 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 10 monclient: _finish_auth 0 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 1 -- 172.21.15.110:0/3521734767 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c02cd20 con 0x7fe0240a9ad0 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 1 -- 172.21.15.110:0/3521734767 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c034660 con 0x7fe0240a9ad0 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 5 monclient: authenticate success, global_id 4325 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 10 monclient: get_monmap_and_config success 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 4 set_mon_vals no callback set 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe01bfff640 10 monclient: dump: 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 10 monclient: shutdown 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 1 -- 172.21.15.110:0/3521734767 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a9ad0 msgr2=0x7fe024142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 1 --2- 172.21.15.110:0/3521734767 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a9ad0 0x7fe024142af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe01c009d90 tx=0x7fe01c02be10 comp rx=0 tx=0).stop 2024-01-20T11:16:56.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 1 -- 172.21.15.110:0/3521734767 shutdown_connections 2024-01-20T11:16:56.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 1 --2- 172.21.15.110:0/3521734767 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a9ad0 0x7fe024142af0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 1 --2- 172.21.15.110:0/3521734767 >> v2:172.21.15.110:3300/0 conn(0x7fe0240a96f0 0x7fe02413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.746+0000 7fe02b90f640 1 -- 172.21.15.110:0/3521734767 >> 172.21.15.110:0/3521734767 conn(0x7fe0240b7bc0 msgr2=0x7fe0240b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:56.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.747+0000 7fe02b90f640 1 -- 172.21.15.110:0/3521734767 shutdown_connections 2024-01-20T11:16:56.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.747+0000 7fe02b90f640 1 -- 172.21.15.110:0/3521734767 wait complete. 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.747+0000 7fe02b90f640 10 monclient: build_initial_monmap 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.747+0000 7fe02b90f640 10 monclient: monmap: 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:16:56.754 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 librados: starting msgr at 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 librados: starting objecter 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 Processor -- start 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 -- start start 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 librados: setting wanted keys 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 librados: calling monclient init 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient: init 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a96f0 0x7fe024133140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient(hunting): picked mon.noname-c con 0x7fe0240a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0240a9ad0 0x7fe024133680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0240a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe02413d2f0 0x7fe02413d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient(hunting): picked mon.noname-a con 0x7fe02413d2f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02413cda0 con 0x7fe02413d2f0 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe024134280 con 0x7fe0240a9ad0 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0241344d0 con 0x7fe0240a96f0 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.748+0000 7fe02b90f640 10 monclient(hunting): authenticate will time out at 1460.100161s 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.749+0000 7fe028eaa640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0240a9ad0 0x7fe024133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.749+0000 7fe028eaa640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0240a9ad0 0x7fe024133680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46772/0 (socket says 172.21.15.110:46772) 2024-01-20T11:16:56.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.749+0000 7fe028eaa640 1 -- 172.21.15.110:0/4231413468 learned_addr learned my addr 172.21.15.110:0/4231413468 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.749+0000 7fe028eaa640 10 monclient(hunting): get_auth_request con 0x7fe0240a9ad0 auth_method 0 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.749+0000 7fe028eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.749+0000 7fe028eaa640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.749+0000 7fe028eaa640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.750+0000 7fe028eaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.750+0000 7fe028eaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.750+0000 7fe028eaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.750+0000 7fe029eac640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe02413d2f0 0x7fe02413d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.750+0000 7fe0296ab640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a96f0 0x7fe024133140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.751+0000 7fe0296ab640 10 monclient(hunting): get_auth_request con 0x7fe0240a96f0 auth_method 0 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.751+0000 7fe0296ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.751+0000 7fe0296ab640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.751+0000 7fe0296ab640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.751+0000 7fe0296ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.751+0000 7fe0296ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.751+0000 7fe0296ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.751+0000 7fe029eac640 10 monclient(hunting): get_auth_request con 0x7fe02413d2f0 auth_method 0 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.752+0000 7fe029eac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.752+0000 7fe029eac640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.752+0000 7fe029eac640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.752+0000 7fe029eac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.752+0000 7fe029eac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.752+0000 7fe029eac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 10 monclient(hunting): handle_auth_done global_id 4459 payload 824 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 1 -- 172.21.15.110:0/4231413468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a96f0 msgr2=0x7fe024133140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a96f0 0x7fe024133140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 1 -- 172.21.15.110:0/4231413468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe02413d2f0 msgr2=0x7fe02413d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe02413d2f0 0x7fe02413d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 10 monclient: _finish_hunting 0 2024-01-20T11:16:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 1 monclient: found mon.noname-b 2024-01-20T11:16:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:16:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.753+0000 7fe028eaa640 1 -- 172.21.15.110:0/4231413468 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c009a70 con 0x7fe0240a9ad0 2024-01-20T11:16:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.755+0000 7fe028eaa640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0240a9ad0 0x7fe024133680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe01400c3f0 tx=0x7fe014001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.755+0000 7fe019ffb640 10 client.?.objecter ms_handle_connect 0x7fe0240a9ad0 2024-01-20T11:16:56.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.755+0000 7fe019ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:16:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 1 -- 172.21.15.110:0/4231413468 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe014005ce0 con 0x7fe0240a9ad0 2024-01-20T11:16:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 10 monclient: dump: 2024-01-20T11:16:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 10 monclient: _finish_auth 0 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 1 -- 172.21.15.110:0/4231413468 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe014005e40 con 0x7fe0240a9ad0 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe02b90f640 5 monclient: authenticate success, global_id 4459 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe02b90f640 10 monclient: _renew_subs 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe02b90f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02413c9c0 con 0x7fe0240a9ad0 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 1 -- 172.21.15.110:0/4231413468 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0140072f0 con 0x7fe0240a9ad0 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe019ffb640 10 monclient: dump: 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe01b7fe640 4 set_mon_vals no callback set 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe02b90f640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe02b90f640 10 monclient: _renew_subs 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe02b90f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02413dea0 con 0x7fe0240a9ad0 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.759+0000 7fe02b90f640 1 librados: init done 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.760+0000 7fe02b90f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.760+0000 7fe02b90f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.760+0000 7fe02b90f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:56.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.760+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0240b2230 con 0x7fe0240a9ad0 2024-01-20T11:16:56.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.778+0000 7fe019ffb640 1 -- 172.21.15.110:0/4231413468 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe014007450 con 0x7fe0240a9ad0 2024-01-20T11:16:56.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.778+0000 7fe019ffb640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe00806c8b0 0x7fe00806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:56.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.779+0000 7fe0296ab640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe00806c8b0 0x7fe00806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:56.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.779+0000 7fe0296ab640 10 monclient: get_auth_request con 0x7fe00806c8b0 auth_method 0 2024-01-20T11:16:56.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.779+0000 7fe019ffb640 10 client.4459.objecter ms_handle_connect 0x7fe00806c8b0 2024-01-20T11:16:56.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.779+0000 7fe0296ab640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe00806c8b0 0x7fe00806ed70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe020009990 tx=0x7fe020009110 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.782+0000 7fe019ffb640 1 -- 172.21.15.110:0/4231413468 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe014007e20 con 0x7fe0240a9ad0 2024-01-20T11:16:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.782+0000 7fe019ffb640 10 client.4459.objecter ms_dispatch 0x7fe024000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:16:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.782+0000 7fe019ffb640 3 client.4459.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:16:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.782+0000 7fe019ffb640 3 client.4459.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:16:56.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:56.782+0000 7fe019ffb640 20 client.4459.objecter dump_active .. 0 homeless 2024-01-20T11:16:57.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.023+0000 7fe019ffb640 1 -- 172.21.15.110:0/4231413468 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe0140578a0 con 0x7fe0240a9ad0 2024-01-20T11:16:57.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.023+0000 7fe019ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:57.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.023+0000 7fe019ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:16:57.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.269+0000 7fe02b90f640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:16:57.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.269+0000 7fe02b90f640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:16:57.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.269+0000 7fe02b90f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:16:57.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.269+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fe02413d010 con 0x7fe0240a9ad0 2024-01-20T11:16:57.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.275+0000 7fe019ffb640 1 -- 172.21.15.110:0/4231413468 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7fe02413d010 con 0x7fe0240a9ad0 2024-01-20T11:16:57.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.275+0000 7fe019ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:16:57.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.275+0000 7fe019ffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:16:57.276 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:16:57.276 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":151,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224567296,"bytes_avail":772869545984,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:16:57.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.276+0000 7fe02b90f640 10 librados: watch_flush enter 2024-01-20T11:16:57.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 10 librados: watch_flush exit 2024-01-20T11:16:57.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 20 client.4459.objecter shutdown clearing up homeless session... 2024-01-20T11:16:57.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 10 client.4459.objecter successfully canceled tick 2024-01-20T11:16:57.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe00806c8b0 msgr2=0x7fe00806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe00806c8b0 0x7fe00806ed70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe020009990 tx=0x7fe020009110 comp rx=0 tx=0).stop 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 10 monclient: shutdown 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0240a9ad0 msgr2=0x7fe024133680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0240a9ad0 0x7fe024133680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe01400c3f0 tx=0x7fe014001e70 comp rx=0 tx=0).stop 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 shutdown_connections 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe00806c8b0 0x7fe00806ed70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe02413d2f0 0x7fe02413d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0240a9ad0 0x7fe024133680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 --2- 172.21.15.110:0/4231413468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0240a96f0 0x7fe024133140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 >> 172.21.15.110:0/4231413468 conn(0x7fe0240b7bc0 msgr2=0x7fe0240b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 shutdown_connections 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 -- 172.21.15.110:0/4231413468 wait complete. 2024-01-20T11:16:57.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.277+0000 7fe02b90f640 1 librados: shutdown 2024-01-20T11:16:57.288 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:16:57.288 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:16:57.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.386+0000 7fd2bd4f2640 10 monclient: get_monmap_and_config 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.386+0000 7fd2bd4f2640 10 monclient: build_initial_monmap 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.386+0000 7fd2bd4f2640 10 monclient: monmap: 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:16:57.388479+0000 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:16:57.388479+0000 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:16:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:16:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:16:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:16:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:16:57.389 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.387+0000 7fd2bd4f2640 1 Processor -- start 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.387+0000 7fd2bd4f2640 1 -- start start 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.387+0000 7fd2bd4f2640 10 monclient: init 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b8145760 0x7fd2b8145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 10 monclient(hunting): picked mon.noname-b con 0x7fd2b8145760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2b80a9d80 addr v1:172.21.15.110:6789/0 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2b813f8b0 0x7fd2b80b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 10 monclient(hunting): picked mon.noname-a con 0x7fd2b813f8b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b8146250 con 0x7fd2b80a9d80 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b80ab050 con 0x7fd2b813f8b0 2024-01-20T11:16:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80b20c0 con 0x7fd2b8145760 2024-01-20T11:16:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2bd4f2640 10 monclient(hunting): authenticate will time out at 1460.739340s 2024-01-20T11:16:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2b6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b8145760 0x7fd2b8145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.388+0000 7fd2b77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2b813f8b0 0x7fd2b80b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b8145760 0x7fd2b8145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:60688/0 (socket says 172.21.15.110:60688) 2024-01-20T11:16:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b6ffd640 1 -- 172.21.15.110:0/2367384186 learned_addr learned my addr 172.21.15.110:0/2367384186 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b6ffd640 10 monclient(hunting): get_auth_request con 0x7fd2b8145760 auth_method 0 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b77fe640 10 monclient(hunting): get_auth_request con 0x7fd2b813f8b0 auth_method 0 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.389+0000 7fd2b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.390+0000 7fd2b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.390+0000 7fd2b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.390+0000 7fd2b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.391+0000 7fd2b6ffd640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-01-20T11:16:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.391+0000 7fd2b6ffd640 1 -- 172.21.15.110:0/2367384186 >> v2:172.21.15.110:3300/0 conn(0x7fd2b813f8b0 msgr2=0x7fd2b80b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:16:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.391+0000 7fd2b6ffd640 1 --2- 172.21.15.110:0/2367384186 >> v2:172.21.15.110:3300/0 conn(0x7fd2b813f8b0 0x7fd2b80b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.391+0000 7fd2b6ffd640 1 -- 172.21.15.110:0/2367384186 >> v1:172.21.15.110:6789/0 conn(0x7fd2b80a9d80 legacy=0x7fd2b813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.391+0000 7fd2b6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:16:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.391+0000 7fd2b6ffd640 1 monclient: found mon.noname-b 2024-01-20T11:16:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.391+0000 7fd2b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.391+0000 7fd2b6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:16:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.391+0000 7fd2b6ffd640 1 -- 172.21.15.110:0/2367384186 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b80b2bb0 con 0x7fd2b8145760 2024-01-20T11:16:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.395+0000 7fd2b6ffd640 1 --2- 172.21.15.110:0/2367384186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b8145760 0x7fd2b8145b40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009870 tx=0x7fd2a802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=625f4974322344ae server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.398+0000 7fd2b5ffb640 1 -- 172.21.15.110:0/2367384186 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8030840 con 0x7fd2b8145760 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.398+0000 7fd2b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 10 monclient: dump: 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 10 monclient: _finish_auth 0 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 1 -- 172.21.15.110:0/2367384186 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a80309a0 con 0x7fd2b8145760 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 1 -- 172.21.15.110:0/2367384186 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8030d30 con 0x7fd2b8145760 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 5 monclient: authenticate success, global_id 4331 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 10 monclient: get_monmap_and_config success 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 4 set_mon_vals no callback set 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2b5ffb640 10 monclient: dump: 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 10 monclient: shutdown 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2367384186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b8145760 msgr2=0x7fd2b8145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 1 --2- 172.21.15.110:0/2367384186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b8145760 0x7fd2b8145b40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009870 tx=0x7fd2a802b980 comp rx=0 tx=0).stop 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2367384186 shutdown_connections 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 1 --2- 172.21.15.110:0/2367384186 >> v2:172.21.15.110:3300/0 conn(0x7fd2b813f8b0 0x7fd2b80b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 1 --2- 172.21.15.110:0/2367384186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b8145760 0x7fd2b8145b40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2367384186 >> 172.21.15.110:0/2367384186 conn(0x7fd2b80663b0 msgr2=0x7fd2b80667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2367384186 shutdown_connections 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.399+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2367384186 wait complete. 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.400+0000 7fd2bd4f2640 10 monclient: build_initial_monmap 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.400+0000 7fd2bd4f2640 10 monclient: monmap: 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:16:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:16:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:16:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:16:57.405 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.400+0000 7fd2bd4f2640 1 librados: starting msgr at 2024-01-20T11:16:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.400+0000 7fd2bd4f2640 1 librados: starting objecter 2024-01-20T11:16:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.400+0000 7fd2bd4f2640 1 Processor -- start 2024-01-20T11:16:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 1 -- start start 2024-01-20T11:16:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 1 librados: setting wanted keys 2024-01-20T11:16:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 1 librados: calling monclient init 2024-01-20T11:16:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient: init 2024-01-20T11:16:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient: _reopen_session rank -1 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b80a9d80 0x7fd2b813a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient(hunting): picked mon.noname-c con 0x7fd2b80a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2b813f8b0 0x7fd2b80b5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient(hunting): picked mon.noname-b con 0x7fd2b813f8b0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2b8145760 0x7fd2b813fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient(hunting): picked mon.noname-a con 0x7fd2b8145760 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80b6960 con 0x7fd2b8145760 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b8142b30 con 0x7fd2b813f8b0 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:16:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80b6bd0 con 0x7fd2b80a9d80 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient(hunting): _renew_subs 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.401+0000 7fd2bd4f2640 10 monclient(hunting): authenticate will time out at 1460.752746s 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2b8145760 0x7fd2b813fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b80a9d80 0x7fd2b813a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b67fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2b813f8b0 0x7fd2b80b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b80a9d80 0x7fd2b813a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:60698/0 (socket says 172.21.15.110:60698) 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b6ffd640 1 -- 172.21.15.110:0/2813843375 learned_addr learned my addr 172.21.15.110:0/2813843375 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b6ffd640 10 monclient(hunting): get_auth_request con 0x7fd2b80a9d80 auth_method 0 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b67fc640 10 monclient(hunting): get_auth_request con 0x7fd2b813f8b0 auth_method 0 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b67fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b77fe640 10 monclient(hunting): get_auth_request con 0x7fd2b8145760 auth_method 0 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.402+0000 7fd2b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.403+0000 7fd2b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.403+0000 7fd2b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.403+0000 7fd2b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.403+0000 7fd2b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.403+0000 7fd2b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.403+0000 7fd2b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.403+0000 7fd2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.403+0000 7fd2b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.403+0000 7fd2b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:16:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2024-01-20T11:16:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 1 -- 172.21.15.110:0/2813843375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b80a9d80 msgr2=0x7fd2b813a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b80a9d80 0x7fd2b813a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 1 -- 172.21.15.110:0/2813843375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2b813f8b0 msgr2=0x7fd2b80b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2b813f8b0 0x7fd2b80b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:16:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 1 monclient: found mon.noname-a 2024-01-20T11:16:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:16:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:16:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.406+0000 7fd2b77fe640 1 -- 172.21.15.110:0/2813843375 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a80094d0 con 0x7fd2b8145760 2024-01-20T11:16:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.409+0000 7fd2b77fe640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2b8145760 0x7fd2b813fd70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000ef90 tx=0x7fd2b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.409+0000 7fd29f7fe640 10 client.?.objecter ms_handle_connect 0x7fd2b8145760 2024-01-20T11:16:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.409+0000 7fd29f7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 1 -- 172.21.15.110:0/2813843375 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0009c00 con 0x7fd2b8145760 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 10 monclient: dump: 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 1 -- 172.21.15.110:0/2813843375 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b000f040 con 0x7fd2b8145760 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 1 -- 172.21.15.110:0/2813843375 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b00146c0 con 0x7fd2b8145760 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd29f7fe640 10 monclient: dump: 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:16:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2bd4f2640 5 monclient: authenticate success, global_id 4428 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2bd4f2640 10 monclient: _renew_subs 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2bd4f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b81423a0 con 0x7fd2b8145760 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2b57fa640 4 set_mon_vals no callback set 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2bd4f2640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2bd4f2640 10 monclient: _renew_subs 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2bd4f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b8140490 con 0x7fd2b8145760 2024-01-20T11:16:57.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.412+0000 7fd2bd4f2640 1 librados: init done 2024-01-20T11:16:57.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.413+0000 7fd2bd4f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:57.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.413+0000 7fd2bd4f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:57.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.413+0000 7fd2bd4f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:16:57.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.413+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b80b2970 con 0x7fd2b8145760 2024-01-20T11:16:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.431+0000 7fd29f7fe640 1 -- 172.21.15.110:0/2813843375 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd2b0014820 con 0x7fd2b8145760 2024-01-20T11:16:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.431+0000 7fd29f7fe640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2a406c860 0x7fd2a406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:16:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.432+0000 7fd2b6ffd640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2a406c860 0x7fd2a406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:16:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.432+0000 7fd2b6ffd640 10 monclient: get_auth_request con 0x7fd2a406c860 auth_method 0 2024-01-20T11:16:57.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.432+0000 7fd2b6ffd640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2a406c860 0x7fd2a406ed20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009870 tx=0x7fd2a8008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:16:57.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.432+0000 7fd29f7fe640 10 client.4428.objecter ms_handle_connect 0x7fd2a406c860 2024-01-20T11:16:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.669+0000 7fd29f7fe640 1 -- 172.21.15.110:0/2813843375 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd2b0059610 con 0x7fd2b8145760 2024-01-20T11:16:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.669+0000 7fd29f7fe640 10 client.4428.objecter ms_dispatch 0x7fd2b8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:16:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.669+0000 7fd29f7fe640 3 client.4428.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:16:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.669+0000 7fd29f7fe640 3 client.4428.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:16:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.670+0000 7fd29f7fe640 20 client.4428.objecter dump_active .. 0 homeless 2024-01-20T11:16:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.670+0000 7fd29f7fe640 1 -- 172.21.15.110:0/2813843375 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd2b0055950 con 0x7fd2b8145760 2024-01-20T11:16:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.670+0000 7fd29f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:16:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.670+0000 7fd29f7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:16:57.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.836+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2b80b2200 con 0x7fd2a406c860 2024-01-20T11:16:57.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.839+0000 7fd29f7fe640 1 -- 172.21.15.110:0/2813843375 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64614 (secure 0 0 0) 0x7fd2a8039450 con 0x7fd2a406c860 2024-01-20T11:16:57.840 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:16:57.840 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:16:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.840+0000 7fd2bd4f2640 10 librados: watch_flush enter 2024-01-20T11:16:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.840+0000 7fd2bd4f2640 10 librados: watch_flush exit 2024-01-20T11:16:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 20 client.4428.objecter shutdown clearing up homeless session... 2024-01-20T11:16:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 10 client.4428.objecter successfully canceled tick 2024-01-20T11:16:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2a406c860 msgr2=0x7fd2a406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2a406c860 0x7fd2a406ed20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009870 tx=0x7fd2a8008030 comp rx=0 tx=0).stop 2024-01-20T11:16:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 10 monclient: shutdown 2024-01-20T11:16:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2b8145760 msgr2=0x7fd2b813fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2b8145760 0x7fd2b813fd70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000ef90 tx=0x7fd2b000c490 comp rx=0 tx=0).stop 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 shutdown_connections 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2a406c860 0x7fd2a406ed20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2b8145760 0x7fd2b813fd70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2b813f8b0 0x7fd2b80b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 --2- 172.21.15.110:0/2813843375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2b80a9d80 0x7fd2b813a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 >> 172.21.15.110:0/2813843375 conn(0x7fd2b80663b0 msgr2=0x7fd2b8136a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 shutdown_connections 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 -- 172.21.15.110:0/2813843375 wait complete. 2024-01-20T11:16:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:16:57.841+0000 7fd2bd4f2640 1 librados: shutdown 2024-01-20T11:16:57.852 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-01-20T11:16:56.731930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219304,"kb_used_data":2940,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755416,"statfs":{"total":773094113280,"available":772869545984,"internally_reserved":0,"allocated":3010560,"data_stored":1575866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":86,"apply_latency_ms":86,"commit_latency_ns":86000000,"apply_latency_ns":86000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.340842"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280538,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.2520000000000007},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.1909999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6639999999999997},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":14.948}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.518000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":15.635999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":16.326000000000001},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":12.098000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":16.981999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":23.085000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":18.131},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":21.393000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313241,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.3600000000000003},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.468999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.4},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":5.8140000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.0139999999999993},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.4320000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.476000000000001},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.718999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.172000000000001},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":12.255000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.297000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.105}]}]},{"osd":3,"up_from":21,"seq":90194313242,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.7240000000000002},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":13.547000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":16.838000000000001},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":19.035}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":10.958},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.686999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.500999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.326000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":7.8879999999999999},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":15.092000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":16.282},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":18.225999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444055,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.9349999999999996},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.7609999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":13.702999999999999},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.177}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.728999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.874000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.715},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.507999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.417999999999999},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.8870000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":16.811},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":18.111999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444056,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":496762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":11.746},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":8.2460000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":18.129999999999999},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":6.1100000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":26.484999999999999},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":31.634}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":25.541},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":12.371}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":7.5279999999999996},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":13.222}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.962},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.096}]}]},{"osd":0,"up_from":17,"seq":73014444056,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":94912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":5.4509999999999996},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.6900000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.4770000000000003},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.898999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":17.065999999999999},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.4299999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.147},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.32}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":17.611999999999998},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":9.6110000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":11.207000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.449}]}]},{"osd":6,"up_from":15,"seq":64424509465,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":12.673999999999999},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.3579999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.593},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":14.427}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.803000000000001},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3300000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.019},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":8.4849999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.622},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":10.513}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.9700000000000006},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.369}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:16:58.125 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:49.149 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:58.171 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:02:49.177 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:58.173 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:49.197 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:58.174 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:02:49.180 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:58.195 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:02:49.201 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:58.200 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:49.225 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:58.224 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:49.247 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:58.225 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:49.250 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:16:58.247 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:49.272 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:00.854 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:00.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.953+0000 7fb36893e640 10 monclient: get_monmap_and_config 2024-01-20T11:17:00.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.953+0000 7fb36893e640 10 monclient: build_initial_monmap 2024-01-20T11:17:00.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.953+0000 7fb36893e640 10 monclient: monmap: 2024-01-20T11:17:00.955 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:00.955 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:00.954879+0000 2024-01-20T11:17:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:00.954879+0000 2024-01-20T11:17:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:00.957 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.953+0000 7fb36893e640 1 Processor -- start 2024-01-20T11:17:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.953+0000 7fb36893e640 1 -- start start 2024-01-20T11:17:00.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.953+0000 7fb36893e640 10 monclient: init 2024-01-20T11:17:00.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:00.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-01-20T11:17:00.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3640a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:00.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a9ad0 0x7fb3641423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:00.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3640a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:00.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb364142920 0x7fb36414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:00.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 10 monclient(hunting): picked mon.noname-c con 0x7fb364142920 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:00.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3640b25c0 con 0x7fb3640a96f0 2024-01-20T11:17:00.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:00.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3640aaea0 con 0x7fb364142920 2024-01-20T11:17:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36413fbe0 con 0x7fb3640a9ad0 2024-01-20T11:17:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.954+0000 7fb36893e640 10 monclient(hunting): authenticate will time out at 1464.305876s 2024-01-20T11:17:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb362d9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb364142920 0x7fb36414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb361d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a9ad0 0x7fb3641423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb361d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a9ad0 0x7fb3641423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:60996/0 (socket says 172.21.15.110:60996) 2024-01-20T11:17:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb361d9b640 1 -- 172.21.15.110:0/603322507 learned_addr learned my addr 172.21.15.110:0/603322507 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb361d9b640 10 monclient(hunting): get_auth_request con 0x7fb3640a9ad0 auth_method 0 2024-01-20T11:17:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb361d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb361d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb361d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb362d9d640 10 monclient(hunting): get_auth_request con 0x7fb364142920 auth_method 0 2024-01-20T11:17:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb362d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb362d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.955+0000 7fb362d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.956+0000 7fb361d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:00.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.956+0000 7fb361d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:00.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.956+0000 7fb361d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:00.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.956+0000 7fb362d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:00.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.956+0000 7fb362d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:00.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.956+0000 7fb362d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:00.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.957+0000 7fb361d9b640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-01-20T11:17:00.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.957+0000 7fb361d9b640 1 -- 172.21.15.110:0/603322507 >> v2:172.21.15.134:3300/0 conn(0x7fb364142920 msgr2=0x7fb36414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:00.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.957+0000 7fb361d9b640 1 --2- 172.21.15.110:0/603322507 >> v2:172.21.15.134:3300/0 conn(0x7fb364142920 0x7fb36414c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:00.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.957+0000 7fb361d9b640 1 -- 172.21.15.110:0/603322507 >> v1:172.21.15.134:6789/0 conn(0x7fb3640a96f0 legacy=0x7fb36413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:00.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.957+0000 7fb361d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:17:00.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.957+0000 7fb361d9b640 1 monclient: found mon.noname-b 2024-01-20T11:17:00.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.957+0000 7fb361d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:00.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.957+0000 7fb361d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:00.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.957+0000 7fb361d9b640 1 -- 172.21.15.110:0/603322507 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3640ab0a0 con 0x7fb3640a9ad0 2024-01-20T11:17:00.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.959+0000 7fb361d9b640 1 --2- 172.21.15.110:0/603322507 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a9ad0 0x7fb3641423e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb358009e50 tx=0x7fb35802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5552b948febb4838 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 1 -- 172.21.15.110:0/603322507 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35802c9e0 con 0x7fb3640a9ad0 2024-01-20T11:17:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 10 monclient: dump: 2024-01-20T11:17:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 10 monclient: _finish_auth 0 2024-01-20T11:17:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 10 monclient: _check_auth_tickets 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36893e640 5 monclient: authenticate success, global_id 4337 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 1 -- 172.21.15.110:0/603322507 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35802cb40 con 0x7fb3640a9ad0 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36893e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36893e640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36893e640 4 set_mon_vals no callback set 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 1 -- 172.21.15.110:0/603322507 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3580343f0 con 0x7fb3640a9ad0 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.963+0000 7fb36159a640 10 monclient: dump: 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 10 monclient: shutdown 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 1 -- 172.21.15.110:0/603322507 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a9ad0 msgr2=0x7fb3641423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 1 --2- 172.21.15.110:0/603322507 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a9ad0 0x7fb3641423e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb358009e50 tx=0x7fb35802be40 comp rx=0 tx=0).stop 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 1 -- 172.21.15.110:0/603322507 shutdown_connections 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 1 --2- 172.21.15.110:0/603322507 >> v2:172.21.15.134:3300/0 conn(0x7fb364142920 0x7fb36414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:00.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 1 --2- 172.21.15.110:0/603322507 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a9ad0 0x7fb3641423e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 1 -- 172.21.15.110:0/603322507 >> 172.21.15.110:0/603322507 conn(0x7fb3640b7bc0 msgr2=0x7fb3640b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 1 -- 172.21.15.110:0/603322507 shutdown_connections 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.964+0000 7fb36893e640 1 -- 172.21.15.110:0/603322507 wait complete. 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.965+0000 7fb36893e640 10 monclient: build_initial_monmap 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.965+0000 7fb36893e640 10 monclient: monmap: 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:00.985 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.965+0000 7fb36893e640 1 librados: starting msgr at 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.965+0000 7fb36893e640 1 librados: starting objecter 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.965+0000 7fb36893e640 1 Processor -- start 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.965+0000 7fb36893e640 1 -- start start 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.965+0000 7fb36893e640 1 librados: setting wanted keys 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.965+0000 7fb36893e640 1 librados: calling monclient init 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.965+0000 7fb36893e640 10 monclient: init 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a96f0 0x7fb3640af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3640a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3640a9ad0 0x7fb3640afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3640a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:00.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb364142920 0x7fb3641da910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient(hunting): picked mon.noname-a con 0x7fb364142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3640b1720 con 0x7fb364142920 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3641da3f0 con 0x7fb3640a9ad0 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3640ac7d0 con 0x7fb3640a96f0 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36893e640 10 monclient(hunting): authenticate will time out at 1464.317379s 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb361d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3640a9ad0 0x7fb3640afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb362d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb364142920 0x7fb3641da910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.966+0000 7fb36259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a96f0 0x7fb3640af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb361d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3640a9ad0 0x7fb3640afef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:45588/0 (socket says 172.21.15.110:45588) 2024-01-20T11:17:00.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb361d9b640 1 -- 172.21.15.110:0/4246845723 learned_addr learned my addr 172.21.15.110:0/4246845723 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb361d9b640 10 monclient(hunting): get_auth_request con 0x7fb3640a9ad0 auth_method 0 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb361d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb361d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb361d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb362d9d640 10 monclient(hunting): get_auth_request con 0x7fb364142920 auth_method 0 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb362d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb362d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb362d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb36259c640 10 monclient(hunting): get_auth_request con 0x7fb3640a96f0 auth_method 0 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb36259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb36259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb36259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:00.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb361d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb361d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb361d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb362d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb362d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb362d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb36259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb36259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.967+0000 7fb36259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 10 monclient(hunting): handle_auth_done global_id 4468 payload 824 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 1 -- 172.21.15.110:0/4246845723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a96f0 msgr2=0x7fb3640af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a96f0 0x7fb3640af2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 1 -- 172.21.15.110:0/4246845723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb364142920 msgr2=0x7fb3641da910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:00.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb364142920 0x7fb3641da910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 1 monclient: found mon.noname-b 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.970+0000 7fb361d9b640 1 -- 172.21.15.110:0/4246845723 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358009b30 con 0x7fb3640a9ad0 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.972+0000 7fb361d9b640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3640a9ad0 0x7fb3640afef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb358009e50 tx=0x7fb358004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.972+0000 7fb352ffd640 10 client.?.objecter ms_handle_connect 0x7fb3640a9ad0 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.972+0000 7fb352ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 1 -- 172.21.15.110:0/4246845723 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358009510 con 0x7fb3640a9ad0 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 10 monclient: dump: 2024-01-20T11:17:00.990 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 1 -- 172.21.15.110:0/4246845723 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35803e450 con 0x7fb3640a9ad0 2024-01-20T11:17:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 1 -- 172.21.15.110:0/4246845723 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35803e800 con 0x7fb3640a9ad0 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb36893e640 5 monclient: authenticate success, global_id 4468 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb36893e640 10 monclient: _renew_subs 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb36893e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3640b0430 con 0x7fb3640a9ad0 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb352ffd640 10 monclient: dump: 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:00.992 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb360d99640 4 set_mon_vals no callback set 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb36893e640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb36893e640 10 monclient: _renew_subs 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb36893e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3640b0730 con 0x7fb3640a9ad0 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.976+0000 7fb36893e640 1 librados: init done 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.977+0000 7fb36893e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.977+0000 7fb36893e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.977+0000 7fb36893e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:00.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.977+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3640b1590 con 0x7fb3640a9ad0 2024-01-20T11:17:00.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.998+0000 7fb352ffd640 1 -- 172.21.15.110:0/4246845723 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb35803e960 con 0x7fb3640a9ad0 2024-01-20T11:17:00.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.999+0000 7fb352ffd640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb34406c8b0 0x7fb34406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:01.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.999+0000 7fb36259c640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb34406c8b0 0x7fb34406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:01.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.999+0000 7fb36259c640 10 monclient: get_auth_request con 0x7fb34406c8b0 auth_method 0 2024-01-20T11:17:01.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.999+0000 7fb352ffd640 10 client.4468.objecter ms_handle_connect 0x7fb34406c8b0 2024-01-20T11:17:01.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:00.999+0000 7fb36259c640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb34406c8b0 0x7fb34406ed70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb35c004f60 tx=0x7fb35c007e30 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.234+0000 7fb352ffd640 1 -- 172.21.15.110:0/4246845723 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb35807b440 con 0x7fb3640a9ad0 2024-01-20T11:17:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.234+0000 7fb352ffd640 10 client.4468.objecter ms_dispatch 0x7fb364000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.234+0000 7fb352ffd640 3 client.4468.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.234+0000 7fb352ffd640 3 client.4468.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.234+0000 7fb352ffd640 20 client.4468.objecter dump_active .. 0 homeless 2024-01-20T11:17:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.245+0000 7fb352ffd640 1 -- 172.21.15.110:0/4246845723 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb358077780 con 0x7fb3640a9ad0 2024-01-20T11:17:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.245+0000 7fb352ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.245+0000 7fb352ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:01.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.414+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3640b4400 con 0x7fb34406c8b0 2024-01-20T11:17:01.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.417+0000 7fb352ffd640 1 -- 172.21.15.110:0/4246845723 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64538 (secure 0 0 0) 0x7fb35c003940 con 0x7fb34406c8b0 2024-01-20T11:17:01.418 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:01.419 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:01.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 10 librados: watch_flush enter 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 10 librados: watch_flush exit 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 20 client.4468.objecter shutdown clearing up homeless session... 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 10 client.4468.objecter successfully canceled tick 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb34406c8b0 msgr2=0x7fb34406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb34406c8b0 0x7fb34406ed70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb35c004f60 tx=0x7fb35c007e30 comp rx=0 tx=0).stop 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 10 monclient: shutdown 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3640a9ad0 msgr2=0x7fb3640afef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3640a9ad0 0x7fb3640afef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb358009e50 tx=0x7fb358004a40 comp rx=0 tx=0).stop 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 shutdown_connections 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb34406c8b0 0x7fb34406ed70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb364142920 0x7fb3641da910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3640a9ad0 0x7fb3640afef0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 --2- 172.21.15.110:0/4246845723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3640a96f0 0x7fb3640af2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 >> 172.21.15.110:0/4246845723 conn(0x7fb3640b7bc0 msgr2=0x7fb36413b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:01.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 shutdown_connections 2024-01-20T11:17:01.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 -- 172.21.15.110:0/4246845723 wait complete. 2024-01-20T11:17:01.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.419+0000 7fb36893e640 1 librados: shutdown 2024-01-20T11:17:01.430 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-01-20T11:17:00.732559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":69,"apply_latency_ms":69,"commit_latency_ns":69000000,"apply_latency_ns":69000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.969166"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.5700000000000003},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.6890000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":6.0250000000000004},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.968}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.585000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":15.198}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.214},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":13.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.923999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":17.033999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":14.531000000000001},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":18.027000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313241,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.3600000000000003},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.468999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.4},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":5.8140000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.0139999999999993},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.4320000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.476000000000001},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.718999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.172000000000001},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":12.255000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.297000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.105}]}]},{"osd":3,"up_from":21,"seq":90194313242,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.7240000000000002},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":13.547000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":16.838000000000001},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":19.035}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":10.958},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.686999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.500999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.326000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":7.8879999999999999},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":15.092000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":16.282},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":18.225999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444056,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.4180000000000001},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":7.056}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.436999999999999},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.8019999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.811999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.8770000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.976000000000001},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.4849999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.304},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":11.615}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.593},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":12.948}]}]},{"osd":1,"up_from":17,"seq":73014444057,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":9.1920000000000002},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":5.6299999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.297000000000001},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":15.875999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":17.068000000000001},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.599}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":12.849},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.463999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":10.209},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":14.31}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.714},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":18.382000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444057,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":13.994999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.8949999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.984},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":9}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":12.616},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.7690000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.066000000000001},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.627000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":13.416},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":10.755000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.887},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":9.6069999999999993}]}]},{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":4.3399999999999999},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.5730000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":11.385},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":5.6749999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.6760000000000002},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.574}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.538},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.242000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.282999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.019}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.821},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":9.0980000000000008}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:01.432 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:01.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.531+0000 7f1a1b26f640 10 monclient: get_monmap_and_config 2024-01-20T11:17:01.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.531+0000 7f1a1b26f640 10 monclient: build_initial_monmap 2024-01-20T11:17:01.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.531+0000 7f1a1b26f640 10 monclient: monmap: 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:01.532775+0000 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:01.532775+0000 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.531+0000 7f1a1b26f640 1 Processor -- start 2024-01-20T11:17:01.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.531+0000 7f1a1b26f640 1 -- start start 2024-01-20T11:17:01.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.531+0000 7f1a1b26f640 10 monclient: init 2024-01-20T11:17:01.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:01.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-01-20T11:17:01.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1a140a96f0 0x7f1a1413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:01.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 10 monclient(hunting): picked mon.noname-c con 0x7f1a140a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:01.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a1413fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:01.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a9ad0 0x7f1a14142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:01.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 10 monclient(hunting): picked mon.noname-b con 0x7f1a140a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:01.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a140b25c0 con 0x7f1a1413fa10 2024-01-20T11:17:01.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a140aaea0 con 0x7f1a140a96f0 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a140ab0a0 con 0x7f1a140a9ad0 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.532+0000 7f1a1b26f640 10 monclient(hunting): authenticate will time out at 1464.883757s 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1980c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a9ad0 0x7f1a14142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1900b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1a140a96f0 0x7f1a1413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1980c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a9ad0 0x7f1a14142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:32784/0 (socket says 172.21.15.110:32784) 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1980c640 1 -- 172.21.15.110:0/3166097653 learned_addr learned my addr 172.21.15.110:0/3166097653 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1980c640 10 monclient(hunting): get_auth_request con 0x7f1a140a9ad0 auth_method 0 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1980c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:01.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1980c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1980c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1900b640 10 monclient(hunting): get_auth_request con 0x7f1a140a96f0 auth_method 0 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1900b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1900b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.533+0000 7f1a1900b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.534+0000 7f1a1980c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.534+0000 7f1a1980c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.534+0000 7f1a1980c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.534+0000 7f1a1900b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.534+0000 7f1a1900b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.534+0000 7f1a1900b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:01.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.535+0000 7f1a1980c640 10 monclient(hunting): handle_auth_done global_id 4343 payload 274 2024-01-20T11:17:01.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.535+0000 7f1a1980c640 1 -- 172.21.15.110:0/3166097653 >> v2:172.21.15.134:3300/0 conn(0x7f1a140a96f0 msgr2=0x7f1a1413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:01.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.535+0000 7f1a1980c640 1 --2- 172.21.15.110:0/3166097653 >> v2:172.21.15.134:3300/0 conn(0x7f1a140a96f0 0x7f1a1413f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.535+0000 7f1a1980c640 1 -- 172.21.15.110:0/3166097653 >> v1:172.21.15.134:6789/0 conn(0x7f1a1413fa10 legacy=0x7f1a141423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:01.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.535+0000 7f1a1980c640 10 monclient: _finish_hunting 0 2024-01-20T11:17:01.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.535+0000 7f1a1980c640 1 monclient: found mon.noname-b 2024-01-20T11:17:01.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.535+0000 7f1a1980c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:01.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.535+0000 7f1a1980c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:01.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.535+0000 7f1a1980c640 1 -- 172.21.15.110:0/3166097653 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a140b3fb0 con 0x7f1a140a9ad0 2024-01-20T11:17:01.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.537+0000 7f1a1980c640 1 --2- 172.21.15.110:0/3166097653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a9ad0 0x7f1a14142af0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009870 tx=0x7f1a1002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6a4b9bd76ad67ff server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:01.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 1 -- 172.21.15.110:0/3166097653 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a10031640 con 0x7f1a140a9ad0 2024-01-20T11:17:01.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:01.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:01.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 10 monclient: dump: 2024-01-20T11:17:01.542 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:01.542 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:01.542 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 10 monclient: _finish_auth 0 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 1 -- 172.21.15.110:0/3166097653 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a100317a0 con 0x7f1a140a9ad0 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a1b26f640 5 monclient: authenticate success, global_id 4343 2024-01-20T11:17:01.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a1b26f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:01.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:01.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a1b26f640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:01.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 1 -- 172.21.15.110:0/3166097653 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a10031b30 con 0x7f1a140a9ad0 2024-01-20T11:17:01.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:01.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a1b26f640 4 set_mon_vals no callback set 2024-01-20T11:17:01.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.541+0000 7f1a0bfff640 10 monclient: dump: 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:01.545 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:01.546 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:01.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 10 monclient: shutdown 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3166097653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a9ad0 msgr2=0x7f1a14142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 1 --2- 172.21.15.110:0/3166097653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a9ad0 0x7f1a14142af0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009870 tx=0x7f1a1002b8f0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3166097653 shutdown_connections 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 1 --2- 172.21.15.110:0/3166097653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a9ad0 0x7f1a14142af0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 1 --2- 172.21.15.110:0/3166097653 >> v2:172.21.15.134:3300/0 conn(0x7f1a140a96f0 0x7f1a1413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3166097653 >> 172.21.15.110:0/3166097653 conn(0x7f1a140b7bc0 msgr2=0x7f1a140b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3166097653 shutdown_connections 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3166097653 wait complete. 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 10 monclient: build_initial_monmap 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.542+0000 7f1a1b26f640 10 monclient: monmap: 2024-01-20T11:17:01.550 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 librados: starting msgr at 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 librados: starting objecter 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 Processor -- start 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 -- start start 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 librados: setting wanted keys 2024-01-20T11:17:01.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 librados: calling monclient init 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient: init 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a96f0 0x7f1a14138150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient(hunting): picked mon.noname-c con 0x7f1a140a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a140a9ad0 0x7f1a14138690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient(hunting): picked mon.noname-b con 0x7f1a140a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a1413b4f0 0x7f1a1413b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient(hunting): picked mon.noname-a con 0x7f1a1413b4f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a14133970 con 0x7f1a1413b4f0 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1413aaa0 con 0x7f1a140a9ad0 2024-01-20T11:17:01.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1413bec0 con 0x7f1a140a96f0 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.543+0000 7f1a1b26f640 10 monclient(hunting): authenticate will time out at 1464.895125s 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1900b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a96f0 0x7f1a14138150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1880a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a140a9ad0 0x7f1a14138690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1980c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a1413b4f0 0x7f1a1413b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1900b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a96f0 0x7f1a14138150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:32788/0 (socket says 172.21.15.110:32788) 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1880a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a140a9ad0 0x7f1a14138690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:45594/0 (socket says 172.21.15.110:45594) 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1900b640 1 -- 172.21.15.110:0/3998450657 learned_addr learned my addr 172.21.15.110:0/3998450657 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1900b640 10 monclient(hunting): get_auth_request con 0x7f1a140a96f0 auth_method 0 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1900b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1900b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1900b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1880a640 10 monclient(hunting): get_auth_request con 0x7f1a140a9ad0 auth_method 0 2024-01-20T11:17:01.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1880a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1880a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1880a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1980c640 10 monclient(hunting): get_auth_request con 0x7f1a1413b4f0 auth_method 0 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1980c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1980c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.544+0000 7f1a1980c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.545+0000 7f1a1880a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.545+0000 7f1a1880a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.545+0000 7f1a1880a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.545+0000 7f1a1900b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.545+0000 7f1a1900b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.545+0000 7f1a1900b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.545+0000 7f1a1980c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.545+0000 7f1a1980c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:01.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.545+0000 7f1a1980c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 10 monclient(hunting): handle_auth_done global_id 4474 payload 824 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 1 -- 172.21.15.110:0/3998450657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a96f0 msgr2=0x7f1a14138150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a96f0 0x7f1a14138150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 1 -- 172.21.15.110:0/3998450657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a1413b4f0 msgr2=0x7f1a1413b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a1413b4f0 0x7f1a1413b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 10 monclient: _finish_hunting 0 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 1 monclient: found mon.noname-b 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:01.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.547+0000 7f1a1880a640 1 -- 172.21.15.110:0/3998450657 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a100094d0 con 0x7f1a140a9ad0 2024-01-20T11:17:01.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.549+0000 7f1a1880a640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a140a9ad0 0x7f1a14138690 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1a140aaa10 tx=0x7f1a0400c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.549+0000 7f1a09ffb640 10 client.?.objecter ms_handle_connect 0x7f1a140a9ad0 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.549+0000 7f1a09ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 1 -- 172.21.15.110:0/3998450657 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a040052d0 con 0x7f1a140a9ad0 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 10 monclient: dump: 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:01.557 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 1 -- 172.21.15.110:0/3998450657 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a04005430 con 0x7f1a140a9ad0 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a1b26f640 5 monclient: authenticate success, global_id 4474 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a1b26f640 10 monclient: _renew_subs 2024-01-20T11:17:01.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a1b26f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a140ac0d0 con 0x7f1a140a9ad0 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 1 -- 172.21.15.110:0/3998450657 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04003830 con 0x7f1a140a9ad0 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a09ffb640 10 monclient: dump: 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:01.559 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:01.560 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:01.560 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:01.560 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:01.560 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:01.560 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:01.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a0b7fe640 4 set_mon_vals no callback set 2024-01-20T11:17:01.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a1b26f640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:01.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a1b26f640 10 monclient: _renew_subs 2024-01-20T11:17:01.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a1b26f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:01.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1413af10 con 0x7f1a140a9ad0 2024-01-20T11:17:01.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.553+0000 7f1a1b26f640 1 librados: init done 2024-01-20T11:17:01.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.554+0000 7f1a1b26f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:01.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.554+0000 7f1a1b26f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:01.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.554+0000 7f1a1b26f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:01.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.554+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1413b0d0 con 0x7f1a140a9ad0 2024-01-20T11:17:01.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.572+0000 7f1a09ffb640 1 -- 172.21.15.110:0/3998450657 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1a04003990 con 0x7f1a140a9ad0 2024-01-20T11:17:01.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.573+0000 7f1a09ffb640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f19f806c8b0 0x7f19f806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:01.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.573+0000 7f1a1900b640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f19f806c8b0 0x7f19f806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:01.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.573+0000 7f1a1900b640 10 monclient: get_auth_request con 0x7f19f806c8b0 auth_method 0 2024-01-20T11:17:01.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.573+0000 7f1a09ffb640 10 client.4474.objecter ms_handle_connect 0x7f19f806c8b0 2024-01-20T11:17:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.573+0000 7f1a1900b640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f19f806c8b0 0x7f19f806ed70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009a40 tx=0x7f1a0c00b040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.575+0000 7f1a09ffb640 1 -- 172.21.15.110:0/3998450657 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1a04057a90 con 0x7f1a140a9ad0 2024-01-20T11:17:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.575+0000 7f1a09ffb640 10 client.4474.objecter ms_dispatch 0x7f1a14000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.575+0000 7f1a09ffb640 3 client.4474.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.575+0000 7f1a09ffb640 3 client.4474.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.575+0000 7f1a09ffb640 20 client.4474.objecter dump_active .. 0 homeless 2024-01-20T11:17:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.816+0000 7f1a09ffb640 1 -- 172.21.15.110:0/3998450657 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1a04053dd0 con 0x7f1a140a9ad0 2024-01-20T11:17:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.816+0000 7f1a09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:01.816+0000 7f1a09ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:02.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.058+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1a1413abe0 con 0x7f19f806c8b0 2024-01-20T11:17:02.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.061+0000 7f1a09ffb640 1 -- 172.21.15.110:0/3998450657 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64538 (secure 0 0 0) 0x7f1a0c00f040 con 0x7f19f806c8b0 2024-01-20T11:17:02.062 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:02.063 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:02.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 10 librados: watch_flush enter 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 10 librados: watch_flush exit 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 20 client.4474.objecter shutdown clearing up homeless session... 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 10 client.4474.objecter successfully canceled tick 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f19f806c8b0 msgr2=0x7f19f806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f19f806c8b0 0x7f19f806ed70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009a40 tx=0x7f1a0c00b040 comp rx=0 tx=0).stop 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 10 monclient: shutdown 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a140a9ad0 msgr2=0x7f1a14138690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a140a9ad0 0x7f1a14138690 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1a140aaa10 tx=0x7f1a0400c820 comp rx=0 tx=0).stop 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 shutdown_connections 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f19f806c8b0 0x7f19f806ed70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a1413b4f0 0x7f1a1413b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a140a9ad0 0x7f1a14138690 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 --2- 172.21.15.110:0/3998450657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a140a96f0 0x7f1a14138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 >> 172.21.15.110:0/3998450657 conn(0x7f1a140b7bc0 msgr2=0x7f1a14137440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:02.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 shutdown_connections 2024-01-20T11:17:02.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 -- 172.21.15.110:0/3998450657 wait complete. 2024-01-20T11:17:02.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.063+0000 7f1a1b26f640 1 librados: shutdown 2024-01-20T11:17:02.074 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-01-20T11:17:00.732559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":69,"apply_latency_ms":69,"commit_latency_ns":69000000,"apply_latency_ns":69000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.969166"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.5700000000000003},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.6890000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":6.0250000000000004},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.968}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.585000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":15.198}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.214},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":13.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.923999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":17.033999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":14.531000000000001},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":18.027000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313241,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.3600000000000003},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.468999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.4},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":5.8140000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.0139999999999993},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.4320000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.476000000000001},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.718999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.172000000000001},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":12.255000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.297000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.105}]}]},{"osd":3,"up_from":21,"seq":90194313242,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.7240000000000002},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":13.547000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":16.838000000000001},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":19.035}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":10.958},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.686999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.500999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.326000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":7.8879999999999999},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":15.092000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":16.282},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":18.225999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444056,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.4180000000000001},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":7.056}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.436999999999999},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.8019999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.811999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.8770000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.976000000000001},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.4849999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.304},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":11.615}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.593},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":12.948}]}]},{"osd":1,"up_from":17,"seq":73014444057,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":9.1920000000000002},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":5.6299999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.297000000000001},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":15.875999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":17.068000000000001},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.599}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":12.849},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.463999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":10.209},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":14.31}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.714},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":18.382000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444057,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":13.994999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.8949999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.984},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":9}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":12.616},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.7690000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.066000000000001},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.627000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":13.416},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":10.755000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.887},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":9.6069999999999993}]}]},{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":4.3399999999999999},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.5730000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":11.385},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":5.6749999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.6760000000000002},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.574}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.538},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.242000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.282999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.019}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.821},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":9.0980000000000008}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:02.076 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:02.076 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:02.076 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:02.076 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.002387+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.002387+0000', 'last_peered': '2024-01-20T11:16:53.002387+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.002387+0000', 'last_undegraded': '2024-01-20T11:16:53.002387+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:02.076 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:02.076 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.858298+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.858298+0000', 'last_peered': '2024-01-20T11:16:52.858298+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.858298+0000', 'last_undegraded': '2024-01-20T11:16:52.858298+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:02.077 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:02.077 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 127, 'reported_epoch': 29, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:51.809778+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:51.809778+0000', 'last_peered': '2024-01-20T11:16:51.809778+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:51.809778+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:02.077 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:02.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.174+0000 7f32d8d41640 10 monclient: get_monmap_and_config 2024-01-20T11:17:02.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.174+0000 7f32d8d41640 10 monclient: build_initial_monmap 2024-01-20T11:17:02.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.174+0000 7f32d8d41640 10 monclient: monmap: 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:02.176305+0000 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:02.176305+0000 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:02.176 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.175+0000 7f32d8d41640 1 Processor -- start 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.175+0000 7f32d8d41640 1 -- start start 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.175+0000 7f32d8d41640 10 monclient: init 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32d4071c60 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32d40ac540 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f32d40b0930 0x7f32d414a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 10 monclient(hunting): picked mon.noname-c con 0x7f32d40b0930 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40ad610 con 0x7f32d4071c60 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40721b0 con 0x7f32d40ac540 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d4072040 con 0x7f32d40b0930 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d8d41640 10 monclient(hunting): authenticate will time out at 1465.527278s 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d2d9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f32d40b0930 0x7f32d414a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d259c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f32d4071c60 0x7f32d4142180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58236/0 (socket says 172.21.15.110:58236) 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.176+0000 7f32d259c640 1 -- 172.21.15.110:0/265218756 learned_addr learned my addr 172.21.15.110:0/265218756 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.177+0000 7f32d2d9d640 10 monclient(hunting): get_auth_request con 0x7f32d40b0930 auth_method 0 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.177+0000 7f32d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.177+0000 7f32d2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.177+0000 7f32d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.177+0000 7f32d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.177+0000 7f32d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.177+0000 7f32d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.178+0000 7f32d1d9b640 1 -- 172.21.15.110:0/265218756 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154446414 0 0) 0x7f32d40ad610 con 0x7f32d4071c60 2024-01-20T11:17:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.178+0000 7f32d1d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:17:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.178+0000 7f32d1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.178+0000 7f32d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.178+0000 7f32d1d9b640 10 monclient(hunting): my global_id is 4437 2024-01-20T11:17:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.179+0000 7f32d1d9b640 1 -- 172.21.15.110:0/265218756 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c0003520 con 0x7f32d4071c60 2024-01-20T11:17:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.181+0000 7f32d2d9d640 10 monclient(hunting): handle_auth_done global_id 4477 payload 274 2024-01-20T11:17:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.181+0000 7f32d2d9d640 1 -- 172.21.15.110:0/265218756 >> v1:172.21.15.134:6789/0 conn(0x7f32d40ac540 legacy=0x7f32d40ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.181+0000 7f32d2d9d640 1 -- 172.21.15.110:0/265218756 >> v1:172.21.15.110:6789/0 conn(0x7f32d4071c60 legacy=0x7f32d4142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.181+0000 7f32d2d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:17:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.181+0000 7f32d2d9d640 1 monclient: found mon.noname-c 2024-01-20T11:17:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.181+0000 7f32d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.181+0000 7f32d2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.181+0000 7f32d2d9d640 1 -- 172.21.15.110:0/265218756 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4072340 con 0x7f32d40b0930 2024-01-20T11:17:02.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.185+0000 7f32d2d9d640 1 --2- 172.21.15.110:0/265218756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 0x7f32d414a520 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009d90 tx=0x7f32cc02e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=221b8b32f431ba61 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:02.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 1 -- 172.21.15.110:0/265218756 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc036440 con 0x7f32d40b0930 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 10 monclient: dump: 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:02.190 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 10 monclient: _finish_auth 0 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 1 -- 172.21.15.110:0/265218756 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32cc0365a0 con 0x7f32d40b0930 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 5 monclient: authenticate success, global_id 4477 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 1 -- 172.21.15.110:0/265218756 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc036930 con 0x7f32d40b0930 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 4 set_mon_vals no callback set 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:02.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d1d9b640 10 monclient: dump: 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:02.192 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 10 monclient: shutdown 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 1 -- 172.21.15.110:0/265218756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 msgr2=0x7f32d414a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 1 --2- 172.21.15.110:0/265218756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 0x7f32d414a520 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009d90 tx=0x7f32cc02e160 comp rx=0 tx=0).stop 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 1 -- 172.21.15.110:0/265218756 shutdown_connections 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 1 --2- 172.21.15.110:0/265218756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 0x7f32d414a520 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 1 -- 172.21.15.110:0/265218756 >> 172.21.15.110:0/265218756 conn(0x7f32d40b3e60 msgr2=0x7f32d40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 1 -- 172.21.15.110:0/265218756 shutdown_connections 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.189+0000 7f32d8d41640 1 -- 172.21.15.110:0/265218756 wait complete. 2024-01-20T11:17:02.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.190+0000 7f32d8d41640 10 monclient: build_initial_monmap 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.190+0000 7f32d8d41640 10 monclient: monmap: 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.190+0000 7f32d8d41640 1 librados: starting msgr at 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.190+0000 7f32d8d41640 1 librados: starting objecter 2024-01-20T11:17:02.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.190+0000 7f32d8d41640 1 Processor -- start 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 1 -- start start 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 1 librados: setting wanted keys 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 1 librados: calling monclient init 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient: init 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32d4071c60 0x7f32d41da6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient(hunting): picked mon.noname-c con 0x7f32d4071c60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32d40ac540 0x7f32d406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient(hunting): picked mon.noname-a con 0x7f32d40ac540 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 0x7f32d41dad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient(hunting): picked mon.noname-b con 0x7f32d40b0930 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:02.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4070f60 con 0x7f32d40ac540 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d41c8970 con 0x7f32d40b0930 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d406d710 con 0x7f32d4071c60 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.191+0000 7f32d8d41640 10 monclient(hunting): authenticate will time out at 1465.542802s 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32cbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32d40ac540 0x7f32d406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 0x7f32d41dad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32d4071c60 0x7f32d41da6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 0x7f32d41dad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:45606/0 (socket says 172.21.15.110:45606) 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 1 -- 172.21.15.110:0/460939156 learned_addr learned my addr 172.21.15.110:0/460939156 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 10 monclient(hunting): get_auth_request con 0x7f32d40b0930 auth_method 0 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:02.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32cbfff640 10 monclient(hunting): get_auth_request con 0x7f32d40ac540 auth_method 0 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d259c640 10 monclient(hunting): get_auth_request con 0x7f32d4071c60 auth_method 0 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.192+0000 7f32d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.193+0000 7f32cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.193+0000 7f32cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:02.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.193+0000 7f32cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.193+0000 7f32d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.193+0000 7f32d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.193+0000 7f32d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 10 monclient(hunting): handle_auth_done global_id 4483 payload 824 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 1 -- 172.21.15.110:0/460939156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32d4071c60 msgr2=0x7f32d41da6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32d4071c60 0x7f32d41da6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 1 -- 172.21.15.110:0/460939156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32d40ac540 msgr2=0x7f32d406d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32d40ac540 0x7f32d406d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 1 monclient: found mon.noname-b 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.195+0000 7f32d2d9d640 1 -- 172.21.15.110:0/460939156 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc009a70 con 0x7f32d40b0930 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.197+0000 7f32d2d9d640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 0x7f32d41dad70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009eb0 tx=0x7f32cc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.197+0000 7f32caffd640 10 client.?.objecter ms_handle_connect 0x7f32d40b0930 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.197+0000 7f32caffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.200+0000 7f32caffd640 1 -- 172.21.15.110:0/460939156 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc03e7b0 con 0x7f32d40b0930 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.200+0000 7f32caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.200+0000 7f32caffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.200+0000 7f32caffd640 10 monclient: dump: 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.204 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.200+0000 7f32caffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.200+0000 7f32caffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.200+0000 7f32caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.200+0000 7f32caffd640 1 -- 172.21.15.110:0/460939156 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32cc03e910 con 0x7f32d40b0930 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.200+0000 7f32d8d41640 5 monclient: authenticate success, global_id 4483 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 10 monclient: _renew_subs 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d41c8be0 con 0x7f32d40b0930 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32caffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:02.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32caffd640 1 -- 172.21.15.110:0/460939156 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc03ecc0 con 0x7f32d40b0930 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32caffd640 10 monclient: dump: 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d159a640 4 set_mon_vals no callback set 2024-01-20T11:17:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 10 monclient: _renew_subs 2024-01-20T11:17:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d413a950 con 0x7f32d40b0930 2024-01-20T11:17:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 1 librados: init done 2024-01-20T11:17:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.201+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d413ab70 con 0x7f32d40b0930 2024-01-20T11:17:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.219+0000 7f32caffd640 1 -- 172.21.15.110:0/460939156 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f32cc03d070 con 0x7f32d40b0930 2024-01-20T11:17:02.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.220+0000 7f32caffd640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f32b406c860 0x7f32b406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.220+0000 7f32d259c640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f32b406c860 0x7f32b406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.220+0000 7f32d259c640 10 monclient: get_auth_request con 0x7f32b406c860 auth_method 0 2024-01-20T11:17:02.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.221+0000 7f32caffd640 10 client.4483.objecter ms_handle_connect 0x7f32b406c860 2024-01-20T11:17:02.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.221+0000 7f32d259c640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f32b406c860 0x7f32b406ed20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f32c40011e0 tx=0x7f32c4007d30 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.224+0000 7f32caffd640 1 -- 172.21.15.110:0/460939156 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f32cc07b3e0 con 0x7f32d40b0930 2024-01-20T11:17:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.224+0000 7f32caffd640 10 client.4483.objecter ms_dispatch 0x7f32d4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.224+0000 7f32caffd640 3 client.4483.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.224+0000 7f32caffd640 3 client.4483.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.224+0000 7f32caffd640 20 client.4483.objecter dump_active .. 0 homeless 2024-01-20T11:17:02.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.462+0000 7f32caffd640 1 -- 172.21.15.110:0/460939156 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f32cc077720 con 0x7f32d40b0930 2024-01-20T11:17:02.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.462+0000 7f32caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:02.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.462+0000 7f32caffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:02.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.694+0000 7f32d8d41640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:02.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.694+0000 7f32d8d41640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:02.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.694+0000 7f32d8d41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.694+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f32d4075490 con 0x7f32d40b0930 2024-01-20T11:17:02.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.701+0000 7f32caffd640 1 -- 172.21.15.110:0/460939156 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7f32cc03b080 con 0x7f32d40b0930 2024-01-20T11:17:02.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.701+0000 7f32caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:02.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.701+0000 7f32caffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:02.702 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:02.702 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":157,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 10 librados: watch_flush enter 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 10 librados: watch_flush exit 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 20 client.4483.objecter shutdown clearing up homeless session... 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 10 client.4483.objecter successfully canceled tick 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f32b406c860 msgr2=0x7f32b406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f32b406c860 0x7f32b406ed20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f32c40011e0 tx=0x7f32c4007d30 comp rx=0 tx=0).stop 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 10 monclient: shutdown 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 msgr2=0x7f32d41dad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 0x7f32d41dad70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009eb0 tx=0x7f32cc004a40 comp rx=0 tx=0).stop 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 shutdown_connections 2024-01-20T11:17:02.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f32b406c860 0x7f32b406ed20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32d40b0930 0x7f32d41dad70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32d40ac540 0x7f32d406d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 --2- 172.21.15.110:0/460939156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32d4071c60 0x7f32d41da6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 >> 172.21.15.110:0/460939156 conn(0x7f32d40b3e60 msgr2=0x7f32d4073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:02.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 shutdown_connections 2024-01-20T11:17:02.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 -- 172.21.15.110:0/460939156 wait complete. 2024-01-20T11:17:02.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.703+0000 7f32d8d41640 1 librados: shutdown 2024-01-20T11:17:02.715 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:02.715 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:02.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.815+0000 7fe5ed2c9640 10 monclient: get_monmap_and_config 2024-01-20T11:17:02.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.815+0000 7fe5ed2c9640 10 monclient: build_initial_monmap 2024-01-20T11:17:02.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.815+0000 7fe5ed2c9640 10 monclient: monmap: 2024-01-20T11:17:02.816 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:02.816 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.816 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:02.816999+0000 2024-01-20T11:17:02.817 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:02.816999+0000 2024-01-20T11:17:02.817 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:02.817 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.817 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:02.817 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:02.817 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:02.817 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:02.817 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:02.817 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 1 Processor -- start 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 1 -- start start 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient: init 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe5e80b4d10 0x7fe5e80b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e80b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5e80b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe5e8145980 0x7fe5e8145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e8145980 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e8146370 con 0x7fe5e80b17c0 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:02.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e80b50f0 con 0x7fe5e80b4d10 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e80aade0 con 0x7fe5e8145980 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.816+0000 7fe5ed2c9640 10 monclient(hunting): authenticate will time out at 1466.167948s 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.817+0000 7fe5e6ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe5e80b4d10 0x7fe5e80b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.817+0000 7fe5e77fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe5e8145980 0x7fe5e8145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.817+0000 7fe5e6ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe5e80b4d10 0x7fe5e80b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48826/0 (socket says 172.21.15.110:48826) 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.817+0000 7fe5e6ffd640 1 -- 172.21.15.110:0/881198232 learned_addr learned my addr 172.21.15.110:0/881198232 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.817+0000 7fe5e6ffd640 10 monclient(hunting): get_auth_request con 0x7fe5e80b4d10 auth_method 0 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.817+0000 7fe5e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.817+0000 7fe5e6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.817+0000 7fe5e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e77fe640 10 monclient(hunting): get_auth_request con 0x7fe5e8145980 auth_method 0 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:02.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:02.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:02.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:02.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:02.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:02.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.818+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.819+0000 7fe5e6ffd640 10 monclient(hunting): handle_auth_done global_id 4443 payload 274 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.819+0000 7fe5e6ffd640 1 -- 172.21.15.110:0/881198232 >> v2:172.21.15.134:3300/0 conn(0x7fe5e8145980 msgr2=0x7fe5e8145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.819+0000 7fe5e6ffd640 1 --2- 172.21.15.110:0/881198232 >> v2:172.21.15.134:3300/0 conn(0x7fe5e8145980 0x7fe5e8145e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.819+0000 7fe5e6ffd640 1 -- 172.21.15.110:0/881198232 >> v1:172.21.15.134:6789/0 conn(0x7fe5e80b17c0 legacy=0x7fe5e80b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.819+0000 7fe5e6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.819+0000 7fe5e6ffd640 1 monclient: found mon.noname-a 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.819+0000 7fe5e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.819+0000 7fe5e6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.819+0000 7fe5e6ffd640 1 -- 172.21.15.110:0/881198232 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8145460 con 0x7fe5e80b4d10 2024-01-20T11:17:02.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.821+0000 7fe5e6ffd640 1 --2- 172.21.15.110:0/881198232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e80b4d10 0x7fe5e80b3da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80aa1c0 tx=0x7fe5d802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42c1fa748f6f4fc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:02.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 1 -- 172.21.15.110:0/881198232 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d80317e0 con 0x7fe5e80b4d10 2024-01-20T11:17:02.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:02.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:02.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 10 monclient: dump: 2024-01-20T11:17:02.826 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:02.826 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.826 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.826 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 10 monclient: _finish_auth 0 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 1 -- 172.21.15.110:0/881198232 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8031940 con 0x7fe5e80b4d10 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5ed2c9640 5 monclient: authenticate success, global_id 4443 2024-01-20T11:17:02.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5ed2c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5ed2c9640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 1 -- 172.21.15.110:0/881198232 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8031cd0 con 0x7fe5e80b4d10 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5ed2c9640 4 set_mon_vals no callback set 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5e67fc640 10 monclient: dump: 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:02.828 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.829 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:02.829 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:02.829 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:02.829 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5ed2c9640 10 monclient: shutdown 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5ed2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/881198232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e80b4d10 msgr2=0x7fe5e80b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.825+0000 7fe5ed2c9640 1 --2- 172.21.15.110:0/881198232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e80b4d10 0x7fe5e80b3da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80aa1c0 tx=0x7fe5d802b880 comp rx=0 tx=0).stop 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/881198232 shutdown_connections 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 1 --2- 172.21.15.110:0/881198232 >> v2:172.21.15.134:3300/0 conn(0x7fe5e8145980 0x7fe5e8145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 1 --2- 172.21.15.110:0/881198232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e80b4d10 0x7fe5e80b3da0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/881198232 >> 172.21.15.110:0/881198232 conn(0x7fe5e8066590 msgr2=0x7fe5e80669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/881198232 shutdown_connections 2024-01-20T11:17:02.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/881198232 wait complete. 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 10 monclient: build_initial_monmap 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 10 monclient: monmap: 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 1 librados: starting msgr at 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.826+0000 7fe5ed2c9640 1 librados: starting objecter 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 Processor -- start 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 -- start start 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 librados: setting wanted keys 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 librados: calling monclient init 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient: init 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e80b17c0 0x7fe5e8141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e80b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e80b4d10 0x7fe5e81dac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e80b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e8145980 0x7fe5e81db1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e8145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:02.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8142cc0 con 0x7fe5e8145980 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e81429e0 con 0x7fe5e80b4d10 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8141890 con 0x7fe5e80b17c0 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.827+0000 7fe5ed2c9640 10 monclient(hunting): authenticate will time out at 1466.178807s 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e80b4d10 0x7fe5e81dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e8145980 0x7fe5e81db1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e80b17c0 0x7fe5e8141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e80b4d10 0x7fe5e81dac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:45626/0 (socket says 172.21.15.110:45626) 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 1 -- 172.21.15.110:0/493979283 learned_addr learned my addr 172.21.15.110:0/493979283 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 10 monclient(hunting): get_auth_request con 0x7fe5e80b4d10 auth_method 0 2024-01-20T11:17:02.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e77fe640 10 monclient(hunting): get_auth_request con 0x7fe5e8145980 auth_method 0 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e6ffd640 10 monclient(hunting): get_auth_request con 0x7fe5e80b17c0 auth_method 0 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:02.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.828+0000 7fe5e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.829+0000 7fe5e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.829+0000 7fe5e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.829+0000 7fe5e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.829+0000 7fe5e77fe640 1 -- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e8145980 msgr2=0x7fe5e81db1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.829+0000 7fe5e77fe640 1 -- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e8145980 msgr2=0x7fe5e81db1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.829+0000 7fe5e77fe640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e8145980 0x7fe5e81db1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.829+0000 7fe5e77fe640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e8145980 0x7fe5e81db1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 10 monclient(hunting): handle_auth_done global_id 4489 payload 824 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 1 -- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e80b17c0 msgr2=0x7fe5e8141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e80b17c0 0x7fe5e8141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 1 -- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e8145980 msgr2=0x7fe5e81db1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-20T11:17:02.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e8145980 0x7fe5e81db1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:02.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:02.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 1 monclient: found mon.noname-b 2024-01-20T11:17:02.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:02.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.831+0000 7fe5dffff640 1 -- 172.21.15.110:0/493979283 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d80094d0 con 0x7fe5e80b4d10 2024-01-20T11:17:02.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.833+0000 7fe5dffff640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e80b4d10 0x7fe5e81dac90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000c440 tx=0x7fe5d0001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:02.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.833+0000 7fe5df7fe640 10 client.?.objecter ms_handle_connect 0x7fe5e80b4d10 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.833+0000 7fe5df7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 1 -- 172.21.15.110:0/493979283 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d00052d0 con 0x7fe5e80b4d10 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 10 monclient: dump: 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:02.841 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 10 monclient: _finish_auth 0 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 1 -- 172.21.15.110:0/493979283 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d0005430 con 0x7fe5e80b4d10 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 1 -- 172.21.15.110:0/493979283 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0007260 con 0x7fe5e80b4d10 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5ed2c9640 5 monclient: authenticate success, global_id 4489 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5df7fe640 10 monclient: dump: 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.842 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5ed2c9640 10 monclient: _renew_subs 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5e5ffb640 4 set_mon_vals no callback set 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5ed2c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e8147f70 con 0x7fe5e80b4d10 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5ed2c9640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5ed2c9640 10 monclient: _renew_subs 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5ed2c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e8148530 con 0x7fe5e80b4d10 2024-01-20T11:17:02.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.837+0000 7fe5ed2c9640 1 librados: init done 2024-01-20T11:17:02.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.838+0000 7fe5ed2c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:02.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.838+0000 7fe5ed2c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:02.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.838+0000 7fe5ed2c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:02.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.838+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e8142e70 con 0x7fe5e80b4d10 2024-01-20T11:17:02.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.856+0000 7fe5df7fe640 1 -- 172.21.15.110:0/493979283 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe5d00073c0 con 0x7fe5e80b4d10 2024-01-20T11:17:02.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.857+0000 7fe5df7fe640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5c806c8b0 0x7fe5c806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:02.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.857+0000 7fe5e6ffd640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5c806c8b0 0x7fe5c806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:02.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.857+0000 7fe5e6ffd640 10 monclient: get_auth_request con 0x7fe5c806c8b0 auth_method 0 2024-01-20T11:17:02.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.857+0000 7fe5df7fe640 10 client.4489.objecter ms_handle_connect 0x7fe5c806c8b0 2024-01-20T11:17:02.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.857+0000 7fe5e6ffd640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5c806c8b0 0x7fe5c806ed70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8036000 tx=0x7fe5d803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:02.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.860+0000 7fe5df7fe640 1 -- 172.21.15.110:0/493979283 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe5d0057940 con 0x7fe5e80b4d10 2024-01-20T11:17:02.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.860+0000 7fe5df7fe640 10 client.4489.objecter ms_dispatch 0x7fe5e8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:02.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.860+0000 7fe5df7fe640 3 client.4489.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:02.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.860+0000 7fe5df7fe640 3 client.4489.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:02.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:02.860+0000 7fe5df7fe640 20 client.4489.objecter dump_active .. 0 homeless 2024-01-20T11:17:03.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.105+0000 7fe5df7fe640 1 -- 172.21.15.110:0/493979283 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe5d0053c80 con 0x7fe5e80b4d10 2024-01-20T11:17:03.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.105+0000 7fe5df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:03.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.105+0000 7fe5df7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:03.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.294+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5e814c7f0 con 0x7fe5c806c8b0 2024-01-20T11:17:03.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.299+0000 7fe5df7fe640 1 -- 172.21.15.110:0/493979283 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64429 (secure 0 0 0) 0x7fe5d8031bb0 con 0x7fe5c806c8b0 2024-01-20T11:17:03.300 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:03.301 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:03.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 10 librados: watch_flush enter 2024-01-20T11:17:03.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 10 librados: watch_flush exit 2024-01-20T11:17:03.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 20 client.4489.objecter shutdown clearing up homeless session... 2024-01-20T11:17:03.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 10 client.4489.objecter successfully canceled tick 2024-01-20T11:17:03.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5c806c8b0 msgr2=0x7fe5c806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:03.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5c806c8b0 0x7fe5c806ed70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8036000 tx=0x7fe5d803d040 comp rx=0 tx=0).stop 2024-01-20T11:17:03.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 10 monclient: shutdown 2024-01-20T11:17:03.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:03.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e80b4d10 msgr2=0x7fe5e81dac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e80b4d10 0x7fe5e81dac90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000c440 tx=0x7fe5d0001e70 comp rx=0 tx=0).stop 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 shutdown_connections 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5c806c8b0 0x7fe5c806ed70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e8145980 0x7fe5e81db1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e80b4d10 0x7fe5e81dac90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 --2- 172.21.15.110:0/493979283 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e80b17c0 0x7fe5e8141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 >> 172.21.15.110:0/493979283 conn(0x7fe5e8066590 msgr2=0x7fe5e80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 shutdown_connections 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 -- 172.21.15.110:0/493979283 wait complete. 2024-01-20T11:17:03.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:03.301+0000 7fe5ed2c9640 1 librados: shutdown 2024-01-20T11:17:03.313 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-01-20T11:17:02.732861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.5700000000000003},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.6890000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":6.0250000000000004},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.968}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.585000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":15.198}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.214},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":13.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.923999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":17.033999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":14.531000000000001},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":18.027000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313242,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6870000000000003},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.1099999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.7450000000000001},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.848000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.468},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":12.147}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.173},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.081}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.185},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":12.755000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.800000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.497}]}]},{"osd":3,"up_from":21,"seq":90194313243,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4180000000000001},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":4.9610000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.5440000000000005},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.4619999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":8.8529999999999998},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.462}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.394},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.288}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":11.042},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.129}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":14.555},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.718999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444056,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.4180000000000001},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":7.056}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.436999999999999},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.8019999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.811999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.8770000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.976000000000001},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.4849999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.304},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":11.615}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.593},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":12.948}]}]},{"osd":1,"up_from":17,"seq":73014444057,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":9.1920000000000002},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":5.6299999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.297000000000001},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":15.875999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":17.068000000000001},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.599}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":12.849},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.463999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":10.209},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":14.31}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.714},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":18.382000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444057,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":13.994999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.8949999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.984},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":9}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":12.616},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.7690000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.066000000000001},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.627000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":13.416},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":10.755000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.887},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":9.6069999999999993}]}]},{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":4.3399999999999999},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.5730000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":11.385},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":5.6749999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.6760000000000002},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.574}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.538},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.242000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.282999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.019}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.821},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":9.0980000000000008}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:06.315 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:06.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.414+0000 7f2ec6a75640 10 monclient: get_monmap_and_config 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.415+0000 7f2ec6a75640 10 monclient: build_initial_monmap 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.415+0000 7f2ec6a75640 10 monclient: monmap: 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:06.416619+0000 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:06.416619+0000 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:06.417 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.415+0000 7f2ec6a75640 1 Processor -- start 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.415+0000 7f2ec6a75640 1 -- start start 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.415+0000 7f2ec6a75640 10 monclient: init 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2ec00b4d10 0x7f2ec00b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec00b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ec00b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2ec0145980 0x7f2ec0145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec0145980 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec0146370 con 0x7f2ec00b17c0 2024-01-20T11:17:06.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec00b50f0 con 0x7f2ec0145980 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec00aade0 con 0x7f2ec00b4d10 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec6a75640 10 monclient(hunting): authenticate will time out at 1469.767588s 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec5012640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2ec0145980 0x7f2ec0145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.416+0000 7f2ec4811640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2ec00b4d10 0x7f2ec00b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec5012640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2ec0145980 0x7f2ec0145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48850/0 (socket says 172.21.15.110:48850) 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec5012640 1 -- 172.21.15.110:0/3811409152 learned_addr learned my addr 172.21.15.110:0/3811409152 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec5012640 10 monclient(hunting): get_auth_request con 0x7f2ec0145980 auth_method 0 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec5012640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec5012640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec5012640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:06.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec4811640 10 monclient(hunting): get_auth_request con 0x7f2ec00b4d10 auth_method 0 2024-01-20T11:17:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec4811640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec4811640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec4811640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec5012640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec5012640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec5012640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec4811640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec4811640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.417+0000 7f2ec4811640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.419+0000 7f2ec5012640 10 monclient(hunting): handle_auth_done global_id 4449 payload 274 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.419+0000 7f2ec5012640 1 -- 172.21.15.110:0/3811409152 >> v2:172.21.15.134:3300/0 conn(0x7f2ec00b4d10 msgr2=0x7f2ec00b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.419+0000 7f2ec5012640 1 --2- 172.21.15.110:0/3811409152 >> v2:172.21.15.134:3300/0 conn(0x7f2ec00b4d10 0x7f2ec00b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.419+0000 7f2ec5012640 1 -- 172.21.15.110:0/3811409152 >> v1:172.21.15.134:6789/0 conn(0x7f2ec00b17c0 legacy=0x7f2ec00b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.419+0000 7f2ec5012640 10 monclient: _finish_hunting 0 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.419+0000 7f2ec5012640 1 monclient: found mon.noname-a 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.419+0000 7f2ec5012640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.419+0000 7f2ec5012640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.419+0000 7f2ec5012640 1 -- 172.21.15.110:0/3811409152 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec0145460 con 0x7f2ec0145980 2024-01-20T11:17:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.421+0000 7f2ec5012640 1 --2- 172.21.15.110:0/3811409152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec0145980 0x7f2ec0145e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00b8cd0 tx=0x7f2eb802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1783b8f733ebb46 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 1 -- 172.21.15.110:0/3811409152 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb802c960 con 0x7f2ec0145980 2024-01-20T11:17:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 10 monclient: dump: 2024-01-20T11:17:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 10 monclient: _finish_auth 0 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 1 -- 172.21.15.110:0/3811409152 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb802cac0 con 0x7f2ec0145980 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ec6a75640 5 monclient: authenticate success, global_id 4449 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ec6a75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:06.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ec6a75640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 1 -- 172.21.15.110:0/3811409152 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb802ce50 con 0x7f2ec0145980 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ec6a75640 4 set_mon_vals no callback set 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.424+0000 7f2ebf7fe640 10 monclient: dump: 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:06.428 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:06.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 10 monclient: shutdown 2024-01-20T11:17:06.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:06.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 1 -- 172.21.15.110:0/3811409152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec0145980 msgr2=0x7f2ec0145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:06.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 1 --2- 172.21.15.110:0/3811409152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec0145980 0x7f2ec0145e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00b8cd0 tx=0x7f2eb802bdc0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 1 -- 172.21.15.110:0/3811409152 shutdown_connections 2024-01-20T11:17:06.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 1 --2- 172.21.15.110:0/3811409152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec0145980 0x7f2ec0145e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 1 --2- 172.21.15.110:0/3811409152 >> v2:172.21.15.134:3300/0 conn(0x7f2ec00b4d10 0x7f2ec00b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 1 -- 172.21.15.110:0/3811409152 >> 172.21.15.110:0/3811409152 conn(0x7f2ec0066590 msgr2=0x7f2ec00669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:06.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 1 -- 172.21.15.110:0/3811409152 shutdown_connections 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.425+0000 7f2ec6a75640 1 -- 172.21.15.110:0/3811409152 wait complete. 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.426+0000 7f2ec6a75640 10 monclient: build_initial_monmap 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.426+0000 7f2ec6a75640 10 monclient: monmap: 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:06.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.426+0000 7f2ec6a75640 1 librados: starting msgr at 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.426+0000 7f2ec6a75640 1 librados: starting objecter 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.426+0000 7f2ec6a75640 1 Processor -- start 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.426+0000 7f2ec6a75640 1 -- start start 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.426+0000 7f2ec6a75640 1 librados: setting wanted keys 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.426+0000 7f2ec6a75640 1 librados: calling monclient init 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.426+0000 7f2ec6a75640 10 monclient: init 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec00b17c0 0x7f2ec0146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec00b17c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ec00b4d10 0x7f2ec0137e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec00b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ec0145980 0x7f2ec0138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:06.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec0145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec01432d0 con 0x7f2ec00b17c0 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec0147f50 con 0x7f2ec00b4d10 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec01481c0 con 0x7f2ec0145980 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec6a75640 10 monclient(hunting): authenticate will time out at 1469.778360s 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec4811640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec00b17c0 0x7f2ec0146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ebffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ec00b4d10 0x7f2ec0137e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec5012640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ec0145980 0x7f2ec0138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec4811640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec00b17c0 0x7f2ec0146800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48858/0 (socket says 172.21.15.110:48858) 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec4811640 1 -- 172.21.15.110:0/101452297 learned_addr learned my addr 172.21.15.110:0/101452297 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec4811640 10 monclient(hunting): get_auth_request con 0x7f2ec00b17c0 auth_method 0 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec4811640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:06.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec4811640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.427+0000 7f2ec4811640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ebffff640 10 monclient(hunting): get_auth_request con 0x7f2ec00b4d10 auth_method 0 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ebffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ebffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec5012640 10 monclient(hunting): get_auth_request con 0x7f2ec0145980 auth_method 0 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec5012640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec5012640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec5012640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ebffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ebffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ebffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec4811640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec4811640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:06.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec4811640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:06.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec5012640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:06.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec5012640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:06.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.428+0000 7f2ec5012640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 10 monclient(hunting): handle_auth_done global_id 4495 payload 824 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 1 -- 172.21.15.110:0/101452297 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ec0145980 msgr2=0x7f2ec0138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ec0145980 0x7f2ec0138350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 1 -- 172.21.15.110:0/101452297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec00b17c0 msgr2=0x7f2ec0146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec00b17c0 0x7f2ec0146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 1 monclient: found mon.noname-b 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.431+0000 7f2ebffff640 1 -- 172.21.15.110:0/101452297 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8009b30 con 0x7f2ec00b4d10 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.433+0000 7f2ebffff640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ec00b4d10 0x7f2ec0137e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00b3e0 tx=0x7f2eac00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:06.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.433+0000 7f2ebd7fa640 10 client.?.objecter ms_handle_connect 0x7f2ec00b4d10 2024-01-20T11:17:06.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.433+0000 7f2ebd7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:06.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 1 -- 172.21.15.110:0/101452297 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac004020 con 0x7f2ec00b4d10 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 10 monclient: dump: 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 10 monclient: _finish_auth 0 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 1 -- 172.21.15.110:0/101452297 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eac004180 con 0x7f2ec00b4d10 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 1 -- 172.21.15.110:0/101452297 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac011630 con 0x7f2ec00b4d10 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ec6a75640 5 monclient: authenticate success, global_id 4495 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ec6a75640 10 monclient: _renew_subs 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ec6a75640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec0142db0 con 0x7f2ec00b4d10 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebeffd640 4 set_mon_vals no callback set 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ebd7fa640 10 monclient: dump: 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ec6a75640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ec6a75640 10 monclient: _renew_subs 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ec6a75640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec01472c0 con 0x7f2ec00b4d10 2024-01-20T11:17:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.437+0000 7f2ec6a75640 1 librados: init done 2024-01-20T11:17:06.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.438+0000 7f2ec6a75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:06.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.438+0000 7f2ec6a75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:06.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.438+0000 7f2ec6a75640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:06.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.438+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec0142ff0 con 0x7f2ec00b4d10 2024-01-20T11:17:06.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.685+0000 7f2ebd7fa640 1 -- 172.21.15.110:0/101452297 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2eac011790 con 0x7f2ec00b4d10 2024-01-20T11:17:06.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.686+0000 7f2ebd7fa640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ea406c890 0x7f2ea406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:06.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.686+0000 7f2ec4811640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ea406c890 0x7f2ea406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:06.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.686+0000 7f2ec4811640 10 monclient: get_auth_request con 0x7f2ea406c890 auth_method 0 2024-01-20T11:17:06.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.686+0000 7f2ec4811640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ea406c890 0x7f2ea406ed50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009c00 tx=0x7f2eb4009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:06.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.686+0000 7f2ebd7fa640 10 client.4495.objecter ms_handle_connect 0x7f2ea406c890 2024-01-20T11:17:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.690+0000 7f2ebd7fa640 1 -- 172.21.15.110:0/101452297 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2eac058400 con 0x7f2ec00b4d10 2024-01-20T11:17:06.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.690+0000 7f2ebd7fa640 10 client.4495.objecter ms_dispatch 0x7f2ec0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:06.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.690+0000 7f2ebd7fa640 3 client.4495.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:06.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.690+0000 7f2ebd7fa640 3 client.4495.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:06.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.690+0000 7f2ebd7fa640 20 client.4495.objecter dump_active .. 0 homeless 2024-01-20T11:17:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.700+0000 7f2ebd7fa640 1 -- 172.21.15.110:0/101452297 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2eac0581d0 con 0x7f2ec00b4d10 2024-01-20T11:17:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.700+0000 7f2ebd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.700+0000 7f2ebd7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:06.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.869+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ec014c7f0 con 0x7f2ea406c890 2024-01-20T11:17:06.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.872+0000 7f2ebd7fa640 1 -- 172.21.15.110:0/101452297 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64464 (secure 0 0 0) 0x7f2eb4011b20 con 0x7f2ea406c890 2024-01-20T11:17:06.873 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:06.874 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.873+0000 7f2ec6a75640 10 librados: watch_flush enter 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 10 librados: watch_flush exit 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 20 client.4495.objecter shutdown clearing up homeless session... 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 10 client.4495.objecter successfully canceled tick 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ea406c890 msgr2=0x7f2ea406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ea406c890 0x7f2ea406ed50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009c00 tx=0x7f2eb4009190 comp rx=0 tx=0).stop 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 10 monclient: shutdown 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ec00b4d10 msgr2=0x7f2ec0137e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ec00b4d10 0x7f2ec0137e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00b3e0 tx=0x7f2eac00b8a0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 shutdown_connections 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ea406c890 0x7f2ea406ed50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ec0145980 0x7f2ec0138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ec00b4d10 0x7f2ec0137e10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 --2- 172.21.15.110:0/101452297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec00b17c0 0x7f2ec0146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 >> 172.21.15.110:0/101452297 conn(0x7f2ec0066590 msgr2=0x7f2ec0144c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 shutdown_connections 2024-01-20T11:17:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 -- 172.21.15.110:0/101452297 wait complete. 2024-01-20T11:17:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.874+0000 7f2ec6a75640 1 librados: shutdown 2024-01-20T11:17:06.888 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-01-20T11:17:06.733532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001961"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280540,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.0289999999999999},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.6260000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":10.452999999999999},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.164999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.792},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.864999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.218},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":15.914}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":18.635999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.6400000000000006},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":17.695}]}]},{"osd":7,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.411},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.7569999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.9260000000000002},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":15.935}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":17.446999999999999},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.6839999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.573},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":18.225999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.215999999999999},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":11.816000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.159000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.471}]}]},{"osd":3,"up_from":21,"seq":90194313244,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.3390000000000004},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.5739999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.342000000000001},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.3469999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":12.205},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.535}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.699999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.672000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":11.59},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":16.385999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":17.184999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.478}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444057,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.3670000000000009},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":15.317}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.7519999999999998},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.9339999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.266999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.6929999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.942},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.555}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.012},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":5.9640000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.5920000000000005},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":13.518000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444058,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":8.1690000000000005},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":7.867}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.323},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.005000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.068},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":10.964}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":14.398999999999999},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":17.004999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":11.657999999999999},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":15.593}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.3420000000000005},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.989000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444058,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":11.718},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.181}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9480000000000004},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.974}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":9.8219999999999992},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.3680000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.464},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.1289999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":15.760999999999999},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":12.381}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.956},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":13.577}]}]},{"osd":6,"up_from":15,"seq":64424509467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":7.7370000000000001},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.8220000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.556},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":13.648999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.2949999999999999},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.811999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.468},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.8610000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.6750000000000007},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.217000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.568},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.266999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:06.889 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.987+0000 7fbd4984b640 10 monclient: get_monmap_and_config 2024-01-20T11:17:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.987+0000 7fbd4984b640 10 monclient: build_initial_monmap 2024-01-20T11:17:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.987+0000 7fbd4984b640 10 monclient: monmap: 2024-01-20T11:17:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:06.988749+0000 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:06.988749+0000 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.987+0000 7fbd4984b640 1 Processor -- start 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.987+0000 7fbd4984b640 1 -- start start 2024-01-20T11:17:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.987+0000 7fbd4984b640 10 monclient: init 2024-01-20T11:17:06.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:06.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-01-20T11:17:06.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd440a96f0 0x7fbd4413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 10 monclient(hunting): picked mon.noname-b con 0x7fbd440a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbd4413fa10 0x7fbd441423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 10 monclient(hunting): picked mon.noname-a con 0x7fbd4413fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd440a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd440b25c0 con 0x7fbd440a9ad0 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd440aaea0 con 0x7fbd4413fa10 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440ab0a0 con 0x7fbd440a96f0 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.988+0000 7fbd4984b640 10 monclient(hunting): authenticate will time out at 1470.339692s 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd42ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd440a96f0 0x7fbd4413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbd4413fa10 0x7fbd441423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbd4413fa10 0x7fbd441423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48860/0 (socket says 172.21.15.110:48860) 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 1 -- 172.21.15.110:0/550374807 learned_addr learned my addr 172.21.15.110:0/550374807 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 10 monclient(hunting): get_auth_request con 0x7fbd4413fa10 auth_method 0 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd42ffd640 10 monclient(hunting): get_auth_request con 0x7fbd440a96f0 auth_method 0 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd42ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.989+0000 7fbd427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.990+0000 7fbd42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.990+0000 7fbd42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.990+0000 7fbd42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.991+0000 7fbd427fc640 10 monclient(hunting): handle_auth_done global_id 4455 payload 274 2024-01-20T11:17:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.991+0000 7fbd427fc640 1 -- 172.21.15.110:0/550374807 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd440a96f0 msgr2=0x7fbd4413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.991+0000 7fbd427fc640 1 --2- 172.21.15.110:0/550374807 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd440a96f0 0x7fbd4413f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.991+0000 7fbd427fc640 1 -- 172.21.15.110:0/550374807 >> v1:172.21.15.110:6789/0 conn(0x7fbd440a9ad0 legacy=0x7fbd44142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.991+0000 7fbd427fc640 10 monclient: _finish_hunting 0 2024-01-20T11:17:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.991+0000 7fbd427fc640 1 monclient: found mon.noname-a 2024-01-20T11:17:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.991+0000 7fbd427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.991+0000 7fbd427fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.991+0000 7fbd427fc640 1 -- 172.21.15.110:0/550374807 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd440b3fb0 con 0x7fbd4413fa10 2024-01-20T11:17:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.993+0000 7fbd427fc640 1 --2- 172.21.15.110:0/550374807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd4413fa10 0x7fbd441423e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009870 tx=0x7fbd3802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0a369552e60f0c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 1 -- 172.21.15.110:0/550374807 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380317c0 con 0x7fbd4413fa10 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 10 monclient: dump: 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:06.999 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 1 -- 172.21.15.110:0/550374807 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd38031920 con 0x7fbd4413fa10 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd4984b640 5 monclient: authenticate success, global_id 4455 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 1 -- 172.21.15.110:0/550374807 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38031cb0 con 0x7fbd4413fa10 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd4984b640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd4984b640 4 set_mon_vals no callback set 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.997+0000 7fbd41ffb640 10 monclient: dump: 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.001 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.001 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:07.001 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.001 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:07.001 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:07.001 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:07.001 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 10 monclient: shutdown 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 1 -- 172.21.15.110:0/550374807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd4413fa10 msgr2=0x7fbd441423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 1 --2- 172.21.15.110:0/550374807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd4413fa10 0x7fbd441423e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009870 tx=0x7fbd3802b860 comp rx=0 tx=0).stop 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 1 -- 172.21.15.110:0/550374807 shutdown_connections 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 1 --2- 172.21.15.110:0/550374807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd4413fa10 0x7fbd441423e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 1 --2- 172.21.15.110:0/550374807 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd440a96f0 0x7fbd4413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 1 -- 172.21.15.110:0/550374807 >> 172.21.15.110:0/550374807 conn(0x7fbd440b7bc0 msgr2=0x7fbd440b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 1 -- 172.21.15.110:0/550374807 shutdown_connections 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.998+0000 7fbd4984b640 1 -- 172.21.15.110:0/550374807 wait complete. 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.999+0000 7fbd4984b640 10 monclient: build_initial_monmap 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.999+0000 7fbd4984b640 10 monclient: monmap: 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.999+0000 7fbd4984b640 1 librados: starting msgr at 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.999+0000 7fbd4984b640 1 librados: starting objecter 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.999+0000 7fbd4984b640 1 Processor -- start 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.999+0000 7fbd4984b640 1 -- start start 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.999+0000 7fbd4984b640 1 librados: setting wanted keys 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.999+0000 7fbd4984b640 1 librados: calling monclient init 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:06.999+0000 7fbd4984b640 10 monclient: init 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd440a96f0 0x7fbd441386a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient(hunting): picked mon.noname-b con 0x7fbd440a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd440a9ad0 0x7fbd4413a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient(hunting): picked mon.noname-a con 0x7fbd440a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd44138e80 0x7fbd4413ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient(hunting): picked mon.noname-c con 0x7fbd44138e80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd44137fa0 con 0x7fbd440a9ad0 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd44139ec0 con 0x7fbd440a96f0 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4413a1a0 con 0x7fbd44138e80 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.000+0000 7fbd4984b640 10 monclient(hunting): authenticate will time out at 1470.351731s 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd440a9ad0 0x7fbd4413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd437fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd44138e80 0x7fbd4413ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd42ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd440a96f0 0x7fbd441386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd440a9ad0 0x7fbd4413a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48872/0 (socket says 172.21.15.110:48872) 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 1 -- 172.21.15.110:0/3325153031 learned_addr learned my addr 172.21.15.110:0/3325153031 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 10 monclient(hunting): get_auth_request con 0x7fbd440a9ad0 auth_method 0 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd437fe640 10 monclient(hunting): get_auth_request con 0x7fbd44138e80 auth_method 0 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd437fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd42ffd640 10 monclient(hunting): get_auth_request con 0x7fbd440a96f0 auth_method 0 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd42ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.001+0000 7fbd437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.002+0000 7fbd437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.002+0000 7fbd437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.002+0000 7fbd42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.002+0000 7fbd42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.002+0000 7fbd42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 10 monclient(hunting): handle_auth_done global_id 4458 payload 824 2024-01-20T11:17:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 1 -- 172.21.15.110:0/3325153031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd44138e80 msgr2=0x7fbd4413ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd44138e80 0x7fbd4413ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 1 -- 172.21.15.110:0/3325153031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd440a96f0 msgr2=0x7fbd441386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd440a96f0 0x7fbd441386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 10 monclient: _finish_hunting 0 2024-01-20T11:17:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 1 monclient: found mon.noname-a 2024-01-20T11:17:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.004+0000 7fbd427fc640 1 -- 172.21.15.110:0/3325153031 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd380094d0 con 0x7fbd440a9ad0 2024-01-20T11:17:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.007+0000 7fbd427fc640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd440a9ad0 0x7fbd4413a930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbd380090b0 tx=0x7fbd380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.007+0000 7fbd2bfff640 10 client.?.objecter ms_handle_connect 0x7fbd440a9ad0 2024-01-20T11:17:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.007+0000 7fbd2bfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 1 -- 172.21.15.110:0/3325153031 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38032860 con 0x7fbd440a9ad0 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 10 monclient: dump: 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 10 monclient: _finish_auth 0 2024-01-20T11:17:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 1 -- 172.21.15.110:0/3325153031 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd380329e0 con 0x7fbd440a9ad0 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd4984b640 5 monclient: authenticate success, global_id 4458 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 1 -- 172.21.15.110:0/3325153031 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38032d90 con 0x7fbd440a9ad0 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd4984b640 10 monclient: _renew_subs 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd4984b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd44138be0 con 0x7fbd440a9ad0 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd417fa640 4 set_mon_vals no callback set 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd2bfff640 10 monclient: dump: 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd4984b640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd4984b640 10 monclient: _renew_subs 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd4984b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd441daed0 con 0x7fbd440a9ad0 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.010+0000 7fbd4984b640 1 librados: init done 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.011+0000 7fbd4984b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.011+0000 7fbd4984b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:07.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.011+0000 7fbd4984b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:07.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.011+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd44138110 con 0x7fbd440a9ad0 2024-01-20T11:17:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.030+0000 7fbd2bfff640 1 -- 172.21.15.110:0/3325153031 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fbd3804e450 con 0x7fbd440a9ad0 2024-01-20T11:17:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.030+0000 7fbd2bfff640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd3006c8e0 0x7fbd3006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.031+0000 7fbd42ffd640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd3006c8e0 0x7fbd3006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.031+0000 7fbd42ffd640 10 monclient: get_auth_request con 0x7fbd3006c8e0 auth_method 0 2024-01-20T11:17:07.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.031+0000 7fbd2bfff640 10 client.4458.objecter ms_handle_connect 0x7fbd3006c8e0 2024-01-20T11:17:07.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.031+0000 7fbd42ffd640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd3006c8e0 0x7fbd3006eda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c0113e0 tx=0x7fbd3c00b040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:07.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.264+0000 7fbd2bfff640 1 -- 172.21.15.110:0/3325153031 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbd3807a410 con 0x7fbd440a9ad0 2024-01-20T11:17:07.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.264+0000 7fbd2bfff640 10 client.4458.objecter ms_dispatch 0x7fbd44000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:07.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.264+0000 7fbd2bfff640 3 client.4458.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:07.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.264+0000 7fbd2bfff640 3 client.4458.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:07.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.264+0000 7fbd2bfff640 20 client.4458.objecter dump_active .. 0 homeless 2024-01-20T11:17:07.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.264+0000 7fbd2bfff640 1 -- 172.21.15.110:0/3325153031 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fbd38076750 con 0x7fbd440a9ad0 2024-01-20T11:17:07.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.264+0000 7fbd2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:07.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.264+0000 7fbd2bfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.451+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd440b5cb0 con 0x7fbd3006c8e0 2024-01-20T11:17:07.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.454+0000 7fbd2bfff640 1 -- 172.21.15.110:0/3325153031 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64464 (secure 0 0 0) 0x7fbd440b5cb0 con 0x7fbd3006c8e0 2024-01-20T11:17:07.455 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:07.455 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 10 librados: watch_flush enter 2024-01-20T11:17:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 10 librados: watch_flush exit 2024-01-20T11:17:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 20 client.4458.objecter shutdown clearing up homeless session... 2024-01-20T11:17:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 10 client.4458.objecter successfully canceled tick 2024-01-20T11:17:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd3006c8e0 msgr2=0x7fbd3006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd3006c8e0 0x7fbd3006eda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c0113e0 tx=0x7fbd3c00b040 comp rx=0 tx=0).stop 2024-01-20T11:17:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 10 monclient: shutdown 2024-01-20T11:17:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd440a9ad0 msgr2=0x7fbd4413a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd440a9ad0 0x7fbd4413a930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbd380090b0 tx=0x7fbd380046c0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.456+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 shutdown_connections 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.457+0000 7fbd4984b640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd3006c8e0 0x7fbd3006eda0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.457+0000 7fbd4984b640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd44138e80 0x7fbd4413ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.457+0000 7fbd4984b640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd440a9ad0 0x7fbd4413a930 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.457+0000 7fbd4984b640 1 --2- 172.21.15.110:0/3325153031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd440a96f0 0x7fbd441386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.457+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 >> 172.21.15.110:0/3325153031 conn(0x7fbd440b7bc0 msgr2=0x7fbd440b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.457+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 shutdown_connections 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.457+0000 7fbd4984b640 1 -- 172.21.15.110:0/3325153031 wait complete. 2024-01-20T11:17:07.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.457+0000 7fbd4984b640 1 librados: shutdown 2024-01-20T11:17:07.471 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-01-20T11:17:06.733532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001961"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280540,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.0289999999999999},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.6260000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":10.452999999999999},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.164999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.792},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.864999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.218},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":15.914}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":18.635999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.6400000000000006},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":17.695}]}]},{"osd":7,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.411},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.7569999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.9260000000000002},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":15.935}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":17.446999999999999},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.6839999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.573},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":18.225999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.215999999999999},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":11.816000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.159000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.471}]}]},{"osd":3,"up_from":21,"seq":90194313244,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.3390000000000004},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.5739999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.342000000000001},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.3469999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":12.205},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.535}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.699999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.672000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":11.59},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":16.385999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":17.184999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.478}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444057,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.3670000000000009},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":15.317}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.7519999999999998},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.9339999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.266999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.6929999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.942},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.555}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.012},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":5.9640000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.5920000000000005},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":13.518000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444058,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":8.1690000000000005},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":7.867}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.323},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.005000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.068},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":10.964}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":14.398999999999999},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":17.004999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":11.657999999999999},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":15.593}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.3420000000000005},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.989000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444058,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":11.718},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.181}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9480000000000004},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.974}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":9.8219999999999992},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.3680000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.464},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.1289999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":15.760999999999999},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":12.381}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.956},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":13.577}]}]},{"osd":6,"up_from":15,"seq":64424509467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":7.7370000000000001},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.8220000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.556},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":13.648999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.2949999999999999},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.811999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.468},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.8610000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.6750000000000007},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.217000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.568},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.266999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:07.473 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:07.473 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:07.473 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:07.473 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.002387+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.002387+0000', 'last_peered': '2024-01-20T11:16:53.002387+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.002387+0000', 'last_undegraded': '2024-01-20T11:16:53.002387+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:07.473 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:07.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.858298+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.858298+0000', 'last_peered': '2024-01-20T11:16:52.858298+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.858298+0000', 'last_undegraded': '2024-01-20T11:16:52.858298+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:07.474 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:07.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 127, 'reported_epoch': 29, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:51.809778+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:51.809778+0000', 'last_peered': '2024-01-20T11:16:51.809778+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:51.809778+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:07.474 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.571+0000 7fe468ca2640 10 monclient: get_monmap_and_config 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.571+0000 7fe468ca2640 10 monclient: build_initial_monmap 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.571+0000 7fe468ca2640 10 monclient: monmap: 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:07.572751+0000 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:07.572751+0000 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:07.572 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:07.573 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:07.573 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:07.573 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:07.573 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.571+0000 7fe468ca2640 1 Processor -- start 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.571+0000 7fe468ca2640 1 -- start start 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.571+0000 7fe468ca2640 10 monclient: init 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4640a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe4640a9ad0 0x7fe4641423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 10 monclient(hunting): picked mon.noname-c con 0x7fe4640a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe464142920 0x7fe46414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 10 monclient(hunting): picked mon.noname-b con 0x7fe464142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4640b25c0 con 0x7fe4640a96f0 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4640aaea0 con 0x7fe4640a9ad0 2024-01-20T11:17:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46413fbe0 con 0x7fe464142920 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe468ca2640 10 monclient(hunting): authenticate will time out at 1470.923602s 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.572+0000 7fe462d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe464142920 0x7fe46414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe459d9b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe4640a9ad0 0x7fe4641423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe462d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe464142920 0x7fe46414c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:32834/0 (socket says 172.21.15.110:32834) 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe462d9d640 1 -- 172.21.15.110:0/1315209788 learned_addr learned my addr 172.21.15.110:0/1315209788 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe462d9d640 10 monclient(hunting): get_auth_request con 0x7fe464142920 auth_method 0 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe462d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe462d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe462d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe459d9b640 10 monclient(hunting): get_auth_request con 0x7fe4640a9ad0 auth_method 0 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe459d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe459d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe459d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe462d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe462d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.573+0000 7fe462d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.574+0000 7fe459d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.574+0000 7fe459d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.574+0000 7fe459d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.575+0000 7fe462d9d640 10 monclient(hunting): handle_auth_done global_id 4364 payload 274 2024-01-20T11:17:07.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.575+0000 7fe462d9d640 1 -- 172.21.15.110:0/1315209788 >> v2:172.21.15.134:3300/0 conn(0x7fe4640a9ad0 msgr2=0x7fe4641423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:07.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.575+0000 7fe462d9d640 1 --2- 172.21.15.110:0/1315209788 >> v2:172.21.15.134:3300/0 conn(0x7fe4640a9ad0 0x7fe4641423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.575+0000 7fe462d9d640 1 -- 172.21.15.110:0/1315209788 >> v1:172.21.15.134:6789/0 conn(0x7fe4640a96f0 legacy=0x7fe46413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:07.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.575+0000 7fe462d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:17:07.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.575+0000 7fe462d9d640 1 monclient: found mon.noname-b 2024-01-20T11:17:07.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.575+0000 7fe462d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:07.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.575+0000 7fe462d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:07.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.575+0000 7fe462d9d640 1 -- 172.21.15.110:0/1315209788 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640ab0a0 con 0x7fe464142920 2024-01-20T11:17:07.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.577+0000 7fe462d9d640 1 --2- 172.21.15.110:0/1315209788 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe464142920 0x7fe46414c850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009e50 tx=0x7fe45c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c04f61fc36199c98 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:07.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.579+0000 7fe461d9b640 1 -- 172.21.15.110:0/1315209788 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c02ca70 con 0x7fe464142920 2024-01-20T11:17:07.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.579+0000 7fe461d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:07.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.579+0000 7fe461d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:07.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.579+0000 7fe461d9b640 10 monclient: dump: 2024-01-20T11:17:07.580 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:07.580 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.579+0000 7fe461d9b640 10 monclient: _finish_auth 0 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.579+0000 7fe461d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.579+0000 7fe461d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.579+0000 7fe468ca2640 5 monclient: authenticate success, global_id 4364 2024-01-20T11:17:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.579+0000 7fe468ca2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe461d9b640 1 -- 172.21.15.110:0/1315209788 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45c02cbd0 con 0x7fe464142920 2024-01-20T11:17:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe461d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe461d9b640 1 -- 172.21.15.110:0/1315209788 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c0343f0 con 0x7fe464142920 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe461d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe461d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe461d9b640 10 monclient: dump: 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 4 set_mon_vals no callback set 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 10 monclient: shutdown 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 1 -- 172.21.15.110:0/1315209788 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe464142920 msgr2=0x7fe46414c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 1 --2- 172.21.15.110:0/1315209788 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe464142920 0x7fe46414c850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009e50 tx=0x7fe45c02bed0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 1 -- 172.21.15.110:0/1315209788 shutdown_connections 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 1 --2- 172.21.15.110:0/1315209788 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe464142920 0x7fe46414c850 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 1 --2- 172.21.15.110:0/1315209788 >> v2:172.21.15.134:3300/0 conn(0x7fe4640a9ad0 0x7fe4641423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 1 -- 172.21.15.110:0/1315209788 >> 172.21.15.110:0/1315209788 conn(0x7fe4640b7bc0 msgr2=0x7fe4640b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 1 -- 172.21.15.110:0/1315209788 shutdown_connections 2024-01-20T11:17:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.581+0000 7fe468ca2640 1 -- 172.21.15.110:0/1315209788 wait complete. 2024-01-20T11:17:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.582+0000 7fe468ca2640 10 monclient: build_initial_monmap 2024-01-20T11:17:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.582+0000 7fe468ca2640 10 monclient: monmap: 2024-01-20T11:17:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.582+0000 7fe468ca2640 1 librados: starting msgr at 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.582+0000 7fe468ca2640 1 librados: starting objecter 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 Processor -- start 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 -- start start 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 librados: setting wanted keys 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 librados: calling monclient init 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient: init 2024-01-20T11:17:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4640a96f0 0x7fe4641dabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient(hunting): picked mon.noname-c con 0x7fe4640a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4640a9ad0 0x7fe4640af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient(hunting): picked mon.noname-a con 0x7fe4640a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe464142920 0x7fe4640afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient(hunting): picked mon.noname-b con 0x7fe464142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641db790 con 0x7fe4640a9ad0 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46411fa60 con 0x7fe464142920 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640b0170 con 0x7fe4640a96f0 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.583+0000 7fe468ca2640 10 monclient(hunting): authenticate will time out at 1470.934908s 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe464142920 0x7fe4640afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe46259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4640a96f0 0x7fe4641dabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe459d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4640a9ad0 0x7fe4640af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe464142920 0x7fe4640afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:45652/0 (socket says 172.21.15.110:45652) 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 1 -- 172.21.15.110:0/3719409841 learned_addr learned my addr 172.21.15.110:0/3719409841 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 10 monclient(hunting): get_auth_request con 0x7fe464142920 auth_method 0 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe46259c640 10 monclient(hunting): get_auth_request con 0x7fe4640a96f0 auth_method 0 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe46259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe46259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe46259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe459d9b640 10 monclient(hunting): get_auth_request con 0x7fe4640a9ad0 auth_method 0 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe459d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe459d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe459d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.584+0000 7fe462d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.585+0000 7fe46259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.585+0000 7fe46259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.585+0000 7fe46259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.585+0000 7fe459d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.585+0000 7fe459d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.585+0000 7fe459d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:07.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 10 monclient(hunting): handle_auth_done global_id 4504 payload 824 2024-01-20T11:17:07.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 1 -- 172.21.15.110:0/3719409841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4640a96f0 msgr2=0x7fe4641dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4640a96f0 0x7fe4641dabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 1 -- 172.21.15.110:0/3719409841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4640a9ad0 msgr2=0x7fe4640af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4640a9ad0 0x7fe4640af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 1 monclient: found mon.noname-b 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.587+0000 7fe462d9d640 1 -- 172.21.15.110:0/3719409841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c009b30 con 0x7fe464142920 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.589+0000 7fe462d9d640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe464142920 0x7fe4640afc30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009f70 tx=0x7fe45c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.589+0000 7fe45b7fe640 10 client.?.objecter ms_handle_connect 0x7fe464142920 2024-01-20T11:17:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.589+0000 7fe45b7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 1 -- 172.21.15.110:0/3719409841 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c009510 con 0x7fe464142920 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 10 monclient: dump: 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:07.595 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 1 -- 172.21.15.110:0/3719409841 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45c03e450 con 0x7fe464142920 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe468ca2640 5 monclient: authenticate success, global_id 4504 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe468ca2640 10 monclient: _renew_subs 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe468ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4640b03f0 con 0x7fe464142920 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 1 -- 172.21.15.110:0/3719409841 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c03e800 con 0x7fe464142920 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe45b7fe640 10 monclient: dump: 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe46159a640 4 set_mon_vals no callback set 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe468ca2640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe468ca2640 10 monclient: _renew_subs 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe468ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4641393a0 con 0x7fe464142920 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.593+0000 7fe468ca2640 1 librados: init done 2024-01-20T11:17:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.594+0000 7fe468ca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:07.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.594+0000 7fe468ca2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:07.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.594+0000 7fe468ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:07.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.594+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe464075490 con 0x7fe464142920 2024-01-20T11:17:07.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.836+0000 7fe45b7fe640 1 -- 172.21.15.110:0/3719409841 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe45c03e960 con 0x7fe464142920 2024-01-20T11:17:07.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.836+0000 7fe45b7fe640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe44406c8b0 0x7fe44406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:07.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.837+0000 7fe46259c640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe44406c8b0 0x7fe44406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:07.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.837+0000 7fe46259c640 10 monclient: get_auth_request con 0x7fe44406c8b0 auth_method 0 2024-01-20T11:17:07.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.837+0000 7fe45b7fe640 10 client.4504.objecter ms_handle_connect 0x7fe44406c8b0 2024-01-20T11:17:07.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.837+0000 7fe46259c640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe44406c8b0 0x7fe44406ed70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe454001780 tx=0x7fe454009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.840+0000 7fe45b7fe640 1 -- 172.21.15.110:0/3719409841 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe45c07b3d0 con 0x7fe464142920 2024-01-20T11:17:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.840+0000 7fe45b7fe640 10 client.4504.objecter ms_dispatch 0x7fe464000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.840+0000 7fe45b7fe640 3 client.4504.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.840+0000 7fe45b7fe640 3 client.4504.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.840+0000 7fe45b7fe640 20 client.4504.objecter dump_active .. 0 homeless 2024-01-20T11:17:07.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.851+0000 7fe45b7fe640 1 -- 172.21.15.110:0/3719409841 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe45c077710 con 0x7fe464142920 2024-01-20T11:17:07.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.851+0000 7fe45b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:07.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:07.851+0000 7fe45b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:08.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.091+0000 7fe468ca2640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:08.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.091+0000 7fe468ca2640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:08.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.091+0000 7fe468ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:08.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.091+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fe4640b63c0 con 0x7fe464142920 2024-01-20T11:17:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.097+0000 7fe45b7fe640 1 -- 172.21.15.110:0/3719409841 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7fe45c07ad30 con 0x7fe464142920 2024-01-20T11:17:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.097+0000 7fe45b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.097+0000 7fe45b7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:08.098 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:08.098 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":162,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 10 librados: watch_flush enter 2024-01-20T11:17:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 10 librados: watch_flush exit 2024-01-20T11:17:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 20 client.4504.objecter shutdown clearing up homeless session... 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 10 client.4504.objecter successfully canceled tick 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe44406c8b0 msgr2=0x7fe44406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe44406c8b0 0x7fe44406ed70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe454001780 tx=0x7fe454009040 comp rx=0 tx=0).stop 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 10 monclient: shutdown 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe464142920 msgr2=0x7fe4640afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe464142920 0x7fe4640afc30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009f70 tx=0x7fe45c004a40 comp rx=0 tx=0).stop 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 shutdown_connections 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe44406c8b0 0x7fe44406ed70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe464142920 0x7fe4640afc30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4640a9ad0 0x7fe4640af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 --2- 172.21.15.110:0/3719409841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4640a96f0 0x7fe4641dabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 >> 172.21.15.110:0/3719409841 conn(0x7fe4640b7bc0 msgr2=0x7fe46413b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 shutdown_connections 2024-01-20T11:17:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 -- 172.21.15.110:0/3719409841 wait complete. 2024-01-20T11:17:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.099+0000 7fe468ca2640 1 librados: shutdown 2024-01-20T11:17:08.111 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:08.111 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.210+0000 7f4878cc2640 10 monclient: get_monmap_and_config 2024-01-20T11:17:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.210+0000 7f4878cc2640 10 monclient: build_initial_monmap 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.210+0000 7f4878cc2640 10 monclient: monmap: 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:08.212334+0000 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:08.212334+0000 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:08.212 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.211+0000 7f4878cc2640 1 Processor -- start 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.211+0000 7f4878cc2640 1 -- start start 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.211+0000 7f4878cc2640 10 monclient: init 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48740a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f48740a9ad0 0x7f48741423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 10 monclient(hunting): picked mon.noname-c con 0x7f48740a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4874142920 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48740b25c0 con 0x7f4874142920 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487413fbe0 con 0x7f48740a96f0 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48740aaea0 con 0x7f48740a9ad0 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.212+0000 7f4878cc2640 10 monclient(hunting): authenticate will time out at 1471.563329s 2024-01-20T11:17:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.243+0000 7f486bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f48740a9ad0 0x7f48741423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.244+0000 7f4872d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4874142920 0x7f487414c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58248/0 (socket says 172.21.15.110:58248) 2024-01-20T11:17:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.244+0000 7f4872d9d640 1 -- 172.21.15.110:0/3838268271 learned_addr learned my addr 172.21.15.110:0/3838268271 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:08.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.245+0000 7f486bfff640 10 monclient(hunting): get_auth_request con 0x7f48740a9ad0 auth_method 0 2024-01-20T11:17:08.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.245+0000 7f486bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:08.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.245+0000 7f486bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:08.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.245+0000 7f486bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:08.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.245+0000 7f486bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:08.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.245+0000 7f486bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:08.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.245+0000 7f486bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.247+0000 7f486bfff640 10 monclient(hunting): handle_auth_done global_id 4507 payload 274 2024-01-20T11:17:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.247+0000 7f486bfff640 1 -- 172.21.15.110:0/3838268271 >> v1:172.21.15.134:6789/0 conn(0x7f48740a96f0 legacy=0x7f487413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.247+0000 7f486bfff640 1 -- 172.21.15.110:0/3838268271 >> v1:172.21.15.110:6789/0 conn(0x7f4874142920 legacy=0x7f487414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.247+0000 7f486bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.247+0000 7f486bfff640 1 monclient: found mon.noname-c 2024-01-20T11:17:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.247+0000 7f486bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.247+0000 7f486bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.247+0000 7f486bfff640 1 -- 172.21.15.110:0/3838268271 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48740ab0a0 con 0x7f48740a9ad0 2024-01-20T11:17:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.251+0000 7f486bfff640 1 --2- 172.21.15.110:0/3838268271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48740a9ad0 0x7f48741423e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f485c0097a0 tx=0x7f485c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5ec23524c58e9ae server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 1 -- 172.21.15.110:0/3838268271 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c031830 con 0x7f48740a9ad0 2024-01-20T11:17:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 10 monclient: dump: 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 10 monclient: _finish_auth 0 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 1 -- 172.21.15.110:0/3838268271 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485c031990 con 0x7f48740a9ad0 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 1 -- 172.21.15.110:0/3838268271 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c031d20 con 0x7f48740a9ad0 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4878cc2640 5 monclient: authenticate success, global_id 4507 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4878cc2640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4878cc2640 4 set_mon_vals no callback set 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.255+0000 7f4871d9b640 10 monclient: dump: 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:08.259 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.256+0000 7f4878cc2640 10 monclient: shutdown 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.256+0000 7f4878cc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.256+0000 7f4878cc2640 1 -- 172.21.15.110:0/3838268271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48740a9ad0 msgr2=0x7f48741423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.256+0000 7f4878cc2640 1 --2- 172.21.15.110:0/3838268271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48740a9ad0 0x7f48741423e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f485c0097a0 tx=0x7f485c02b8d0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.256+0000 7f4878cc2640 1 -- 172.21.15.110:0/3838268271 shutdown_connections 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.256+0000 7f4878cc2640 1 --2- 172.21.15.110:0/3838268271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48740a9ad0 0x7f48741423e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.256+0000 7f4878cc2640 1 -- 172.21.15.110:0/3838268271 >> 172.21.15.110:0/3838268271 conn(0x7f48740b7bc0 msgr2=0x7f48740b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.256+0000 7f4878cc2640 1 -- 172.21.15.110:0/3838268271 shutdown_connections 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.256+0000 7f4878cc2640 1 -- 172.21.15.110:0/3838268271 wait complete. 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.257+0000 7f4878cc2640 10 monclient: build_initial_monmap 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.257+0000 7f4878cc2640 10 monclient: monmap: 2024-01-20T11:17:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.257+0000 7f4878cc2640 1 librados: starting msgr at 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.257+0000 7f4878cc2640 1 librados: starting objecter 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.257+0000 7f4878cc2640 1 Processor -- start 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.257+0000 7f4878cc2640 1 -- start start 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.257+0000 7f4878cc2640 1 librados: setting wanted keys 2024-01-20T11:17:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.257+0000 7f4878cc2640 1 librados: calling monclient init 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.257+0000 7f4878cc2640 10 monclient: init 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48740a96f0 0x7f48741dabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient(hunting): picked mon.noname-a con 0x7f48740a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48740a9ad0 0x7f48740afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient(hunting): picked mon.noname-c con 0x7f48740a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4874142920 0x7f48740aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient(hunting): picked mon.noname-b con 0x7f4874142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48740b1920 con 0x7f48740a96f0 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48741db5d0 con 0x7f4874142920 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48740b04a0 con 0x7f48740a9ad0 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f4878cc2640 10 monclient(hunting): authenticate will time out at 1471.609578s 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.258+0000 7f487259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48740a96f0 0x7f48741dabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f487259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48740a96f0 0x7f48741dabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48886/0 (socket says 172.21.15.110:48886) 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f487259c640 1 -- 172.21.15.110:0/3200272387 learned_addr learned my addr 172.21.15.110:0/3200272387 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f487259c640 10 monclient(hunting): get_auth_request con 0x7f48740a96f0 auth_method 0 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f487259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f487259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f487259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f4872d9d640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4874142920 0x7f48740aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f4872d9d640 10 monclient(hunting): get_auth_request con 0x7f4874142920 auth_method 0 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f4872d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f4872d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f4872d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f487259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f487259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f487259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f4872d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f4872d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.259+0000 7f4872d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 10 monclient(hunting): handle_auth_done global_id 4510 payload 824 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 1 -- 172.21.15.110:0/3200272387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48740a9ad0 msgr2=0x7f48740afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48740a9ad0 0x7f48740afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 1 -- 172.21.15.110:0/3200272387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48740a96f0 msgr2=0x7f48741dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48740a96f0 0x7f48741dabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 1 monclient: found mon.noname-b 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.262+0000 7f4872d9d640 1 -- 172.21.15.110:0/3200272387 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c009400 con 0x7f4874142920 2024-01-20T11:17:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.264+0000 7f4872d9d640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4874142920 0x7f48740aff60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f486c00caa0 tx=0x7f486c00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.264+0000 7f486affd640 10 client.?.objecter ms_handle_connect 0x7f4874142920 2024-01-20T11:17:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.264+0000 7f486affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 1 -- 172.21.15.110:0/3200272387 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c00ac00 con 0x7f4874142920 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 10 monclient: dump: 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 10 monclient: _finish_auth 0 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 1 -- 172.21.15.110:0/3200272387 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c00ad60 con 0x7f4874142920 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f4878cc2640 5 monclient: authenticate success, global_id 4510 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f4878cc2640 10 monclient: _renew_subs 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f4878cc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48740b0720 con 0x7f4874142920 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 1 -- 172.21.15.110:0/3200272387 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c012600 con 0x7f4874142920 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f486affd640 10 monclient: dump: 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f487159a640 4 set_mon_vals no callback set 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f4878cc2640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f4878cc2640 10 monclient: _renew_subs 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f4878cc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48741db3c0 con 0x7f4874142920 2024-01-20T11:17:08.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.268+0000 7f4878cc2640 1 librados: init done 2024-01-20T11:17:08.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.269+0000 7f4878cc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:08.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.269+0000 7f4878cc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:08.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.269+0000 7f4878cc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:08.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.269+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48740af740 con 0x7f4874142920 2024-01-20T11:17:08.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.509+0000 7f486affd640 1 -- 172.21.15.110:0/3200272387 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f486c007070 con 0x7f4874142920 2024-01-20T11:17:08.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.509+0000 7f486affd640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f485406c860 0x7f485406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:08.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.510+0000 7f487259c640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f485406c860 0x7f485406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:08.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.510+0000 7f487259c640 10 monclient: get_auth_request con 0x7f485406c860 auth_method 0 2024-01-20T11:17:08.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.510+0000 7f487259c640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f485406c860 0x7f485406ed20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4864007fd0 tx=0x7f4864009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:08.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.510+0000 7f486affd640 10 client.4510.objecter ms_handle_connect 0x7f485406c860 2024-01-20T11:17:08.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.514+0000 7f486affd640 1 -- 172.21.15.110:0/3200272387 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f486c058350 con 0x7f4874142920 2024-01-20T11:17:08.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.514+0000 7f486affd640 10 client.4510.objecter ms_dispatch 0x7f4874000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:08.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.514+0000 7f486affd640 3 client.4510.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:08.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.514+0000 7f486affd640 3 client.4510.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:08.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.514+0000 7f486affd640 20 client.4510.objecter dump_active .. 0 homeless 2024-01-20T11:17:08.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.526+0000 7f486affd640 1 -- 172.21.15.110:0/3200272387 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f486c054690 con 0x7f4874142920 2024-01-20T11:17:08.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.526+0000 7f486affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:08.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.526+0000 7f486affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:08.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.697+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48740b4400 con 0x7f485406c860 2024-01-20T11:17:08.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.700+0000 7f486affd640 1 -- 172.21.15.110:0/3200272387 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64464 (secure 0 0 0) 0x7f48640014c0 con 0x7f485406c860 2024-01-20T11:17:08.701 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:08.702 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:08.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 10 librados: watch_flush enter 2024-01-20T11:17:08.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 10 librados: watch_flush exit 2024-01-20T11:17:08.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 20 client.4510.objecter shutdown clearing up homeless session... 2024-01-20T11:17:08.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 10 client.4510.objecter successfully canceled tick 2024-01-20T11:17:08.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f485406c860 msgr2=0x7f485406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:08.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f485406c860 0x7f485406ed20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4864007fd0 tx=0x7f4864009040 comp rx=0 tx=0).stop 2024-01-20T11:17:08.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 10 monclient: shutdown 2024-01-20T11:17:08.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:08.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4874142920 msgr2=0x7f48740aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:08.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4874142920 0x7f48740aff60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f486c00caa0 tx=0x7f486c00cf60 comp rx=0 tx=0).stop 2024-01-20T11:17:08.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 shutdown_connections 2024-01-20T11:17:08.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f485406c860 0x7f485406ed20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4874142920 0x7f48740aff60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48740a9ad0 0x7f48740afa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 --2- 172.21.15.110:0/3200272387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48740a96f0 0x7f48741dabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:08.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 >> 172.21.15.110:0/3200272387 conn(0x7f48740b7bc0 msgr2=0x7f487413b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:08.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 shutdown_connections 2024-01-20T11:17:08.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 -- 172.21.15.110:0/3200272387 wait complete. 2024-01-20T11:17:08.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:08.702+0000 7f4878cc2640 1 librados: shutdown 2024-01-20T11:17:08.714 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-01-20T11:17:06.733532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001961"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280540,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.0289999999999999},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.6260000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":10.452999999999999},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.164999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.792},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.864999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.218},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":15.914}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":18.635999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.6400000000000006},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":17.695}]}]},{"osd":7,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.411},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.7569999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.9260000000000002},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":15.935}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":17.446999999999999},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.6839999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.573},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":18.225999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.215999999999999},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":11.816000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.159000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.471}]}]},{"osd":3,"up_from":21,"seq":90194313244,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.3390000000000004},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.5739999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.342000000000001},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.3469999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":12.205},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.535}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.699999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.672000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":11.59},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":16.385999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":17.184999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.478}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444057,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.3670000000000009},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":15.317}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.7519999999999998},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.9339999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.266999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.6929999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.942},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.555}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.012},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":5.9640000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.5920000000000005},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":13.518000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444058,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":8.1690000000000005},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":7.867}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.323},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.005000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.068},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":10.964}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":14.398999999999999},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":17.004999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":11.657999999999999},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":15.593}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.3420000000000005},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.989000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444058,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":11.718},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.181}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9480000000000004},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.974}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":9.8219999999999992},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.3680000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.464},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.1289999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":15.760999999999999},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":12.381}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.956},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":13.577}]}]},{"osd":6,"up_from":15,"seq":64424509467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":7.7370000000000001},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.8220000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.556},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":13.648999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.2949999999999999},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.811999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.468},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.8610000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.6750000000000007},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.217000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.568},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.266999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:11.717 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.816+0000 7fcf85268640 10 monclient: get_monmap_and_config 2024-01-20T11:17:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.816+0000 7fcf85268640 10 monclient: build_initial_monmap 2024-01-20T11:17:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.816+0000 7fcf85268640 10 monclient: monmap: 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:11.817864+0000 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:11.817864+0000 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:11.818 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:11.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 1 Processor -- start 2024-01-20T11:17:11.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 1 -- start start 2024-01-20T11:17:11.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 10 monclient: init 2024-01-20T11:17:11.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:11.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf800a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf800a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fcf80142af0 0x7fcf8014d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 10 monclient(hunting): picked mon.noname-c con 0x7fcf80142af0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf800b25c0 con 0x7fcf800a9ad0 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf80142f50 con 0x7fcf800a96f0 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf800aaea0 con 0x7fcf80142af0 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.817+0000 7fcf85268640 10 monclient(hunting): authenticate will time out at 1475.169108s 2024-01-20T11:17:11.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.818+0000 7fcf7f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fcf80142af0 0x7fcf8014d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:11.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.818+0000 7fcf7f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fcf80142af0 0x7fcf8014d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:40148/0 (socket says 172.21.15.110:40148) 2024-01-20T11:17:11.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.818+0000 7fcf7f7fe640 1 -- 172.21.15.110:0/2042847780 learned_addr learned my addr 172.21.15.110:0/2042847780 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:11.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.818+0000 7fcf7f7fe640 10 monclient(hunting): get_auth_request con 0x7fcf80142af0 auth_method 0 2024-01-20T11:17:11.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.818+0000 7fcf7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:11.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.818+0000 7fcf7f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:11.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.818+0000 7fcf7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:11.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.819+0000 7fcf7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:11.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.819+0000 7fcf7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:11.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.819+0000 7fcf7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:11.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.821+0000 7fcf7f7fe640 10 monclient(hunting): handle_auth_done global_id 4513 payload 274 2024-01-20T11:17:11.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.821+0000 7fcf7f7fe640 1 -- 172.21.15.110:0/2042847780 >> v1:172.21.15.134:6789/0 conn(0x7fcf800a96f0 legacy=0x7fcf8013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:11.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.821+0000 7fcf7f7fe640 1 -- 172.21.15.110:0/2042847780 >> v1:172.21.15.110:6789/0 conn(0x7fcf800a9ad0 legacy=0x7fcf801423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:11.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.821+0000 7fcf7f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:11.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.821+0000 7fcf7f7fe640 1 monclient: found mon.noname-c 2024-01-20T11:17:11.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.821+0000 7fcf7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:11.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.821+0000 7fcf7f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:11.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.821+0000 7fcf7f7fe640 1 -- 172.21.15.110:0/2042847780 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8013fbe0 con 0x7fcf80142af0 2024-01-20T11:17:11.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.823+0000 7fcf7f7fe640 1 --2- 172.21.15.110:0/2042847780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf80142af0 0x7fcf8014d860 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009e50 tx=0x7fcf7802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab0ce047888027d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:11.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.835+0000 7fcf7e7fc640 1 -- 172.21.15.110:0/2042847780 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7802cb20 con 0x7fcf80142af0 2024-01-20T11:17:11.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.835+0000 7fcf7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:11.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:11.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 10 monclient: dump: 2024-01-20T11:17:11.837 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:11.837 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:11.837 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:11.838 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:11.838 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:11.838 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:11.838 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:11.838 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:11.838 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:11.838 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:11.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:17:11.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:17:11.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:11.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 1 -- 172.21.15.110:0/2042847780 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7802cc80 con 0x7fcf80142af0 2024-01-20T11:17:11.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:11.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 5 monclient: authenticate success, global_id 4513 2024-01-20T11:17:11.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 1 -- 172.21.15.110:0/2042847780 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf780343f0 con 0x7fcf80142af0 2024-01-20T11:17:11.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:11.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:11.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf7e7fc640 10 monclient: dump: 2024-01-20T11:17:11.840 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:11.840 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:11.840 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:11.840 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:11.840 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:11.840 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:11.841 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:11.841 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:11.841 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:11.841 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:11.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:11.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 4 set_mon_vals no callback set 2024-01-20T11:17:11.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 10 monclient: shutdown 2024-01-20T11:17:11.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 1 -- 172.21.15.110:0/2042847780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf80142af0 msgr2=0x7fcf8014d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 1 --2- 172.21.15.110:0/2042847780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf80142af0 0x7fcf8014d860 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009e50 tx=0x7fcf7802bf80 comp rx=0 tx=0).stop 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 1 -- 172.21.15.110:0/2042847780 shutdown_connections 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 1 --2- 172.21.15.110:0/2042847780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf80142af0 0x7fcf8014d860 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 1 -- 172.21.15.110:0/2042847780 >> 172.21.15.110:0/2042847780 conn(0x7fcf800b7bc0 msgr2=0x7fcf800b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 1 -- 172.21.15.110:0/2042847780 shutdown_connections 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.836+0000 7fcf85268640 1 -- 172.21.15.110:0/2042847780 wait complete. 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.837+0000 7fcf85268640 10 monclient: build_initial_monmap 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.837+0000 7fcf85268640 10 monclient: monmap: 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:11.844 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.837+0000 7fcf85268640 1 librados: starting msgr at 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.837+0000 7fcf85268640 1 librados: starting objecter 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.837+0000 7fcf85268640 1 Processor -- start 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.837+0000 7fcf85268640 1 -- start start 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.837+0000 7fcf85268640 1 librados: setting wanted keys 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.837+0000 7fcf85268640 1 librados: calling monclient init 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.837+0000 7fcf85268640 10 monclient: init 2024-01-20T11:17:11.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf800a96f0 0x7fcf801345f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient(hunting): picked mon.noname-a con 0x7fcf800a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf800a9ad0 0x7fcf80131fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient(hunting): picked mon.noname-b con 0x7fcf800a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf80142af0 0x7fcf801324f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient(hunting): picked mon.noname-c con 0x7fcf80142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80136140 con 0x7fcf800a96f0 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80136420 con 0x7fcf800a9ad0 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80136700 con 0x7fcf80142af0 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:11.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.838+0000 7fcf85268640 10 monclient(hunting): authenticate will time out at 1475.189726s 2024-01-20T11:17:11.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.880+0000 7fcf77fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf800a9ad0 0x7fcf80131fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.881+0000 7fcf77fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf800a9ad0 0x7fcf80131fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:40158/0 (socket says 172.21.15.110:40158) 2024-01-20T11:17:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.881+0000 7fcf77fff640 1 -- 172.21.15.110:0/3202107952 learned_addr learned my addr 172.21.15.110:0/3202107952 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.881+0000 7fcf77fff640 10 monclient(hunting): get_auth_request con 0x7fcf800a9ad0 auth_method 0 2024-01-20T11:17:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.881+0000 7fcf77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.881+0000 7fcf77fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.881+0000 7fcf77fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:11.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.882+0000 7fcf77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:11.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.882+0000 7fcf77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:11.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.882+0000 7fcf77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:11.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 10 monclient(hunting): handle_auth_done global_id 4516 payload 824 2024-01-20T11:17:11.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 1 -- 172.21.15.110:0/3202107952 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf80142af0 msgr2=0x7fcf801324f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:11.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf80142af0 0x7fcf801324f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:11.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 1 -- 172.21.15.110:0/3202107952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf800a96f0 msgr2=0x7fcf801345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:11.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf800a96f0 0x7fcf801345f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:11.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:11.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 1 monclient: found mon.noname-b 2024-01-20T11:17:11.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:11.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:11.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.884+0000 7fcf77fff640 1 -- 172.21.15.110:0/3202107952 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78009b30 con 0x7fcf800a9ad0 2024-01-20T11:17:11.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.887+0000 7fcf77fff640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf800a9ad0 0x7fcf80131fb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800c440 tx=0x7fcf68001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:11.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.887+0000 7fcf777fe640 10 client.?.objecter ms_handle_connect 0x7fcf800a9ad0 2024-01-20T11:17:11.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.887+0000 7fcf777fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:11.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 1 -- 172.21.15.110:0/3202107952 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf680052d0 con 0x7fcf800a9ad0 2024-01-20T11:17:11.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:11.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:11.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 10 monclient: dump: 2024-01-20T11:17:11.892 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:11.892 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:11.892 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:11.892 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 10 monclient: _finish_auth 0 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 10 monclient: _check_auth_tickets 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 1 -- 172.21.15.110:0/3202107952 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf68005430 con 0x7fcf800a9ad0 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf85268640 5 monclient: authenticate success, global_id 4516 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 1 -- 172.21.15.110:0/3202107952 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68007260 con 0x7fcf800a9ad0 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:11.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf777fe640 10 monclient: dump: 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf7dffb640 4 set_mon_vals no callback set 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf85268640 10 monclient: _renew_subs 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf85268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf80135080 con 0x7fcf800a9ad0 2024-01-20T11:17:11.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf85268640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:11.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf85268640 10 monclient: _renew_subs 2024-01-20T11:17:11.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf85268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:11.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf80134dd0 con 0x7fcf800a9ad0 2024-01-20T11:17:11.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.890+0000 7fcf85268640 1 librados: init done 2024-01-20T11:17:11.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.891+0000 7fcf85268640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:11.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.891+0000 7fcf85268640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:11.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.891+0000 7fcf85268640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:11.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.891+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8013fa00 con 0x7fcf800a9ad0 2024-01-20T11:17:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.909+0000 7fcf777fe640 1 -- 172.21.15.110:0/3202107952 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fcf680073c0 con 0x7fcf800a9ad0 2024-01-20T11:17:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.910+0000 7fcf777fe640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcf6006c8e0 0x7fcf6006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.910+0000 7fcf7effd640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcf6006c8e0 0x7fcf6006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.910+0000 7fcf7effd640 10 monclient: get_auth_request con 0x7fcf6006c8e0 auth_method 0 2024-01-20T11:17:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.910+0000 7fcf7effd640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcf6006c8e0 0x7fcf6006eda0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcf70000f60 tx=0x7fcf700079b0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:11.910+0000 7fcf777fe640 10 client.4516.objecter ms_handle_connect 0x7fcf6006c8e0 2024-01-20T11:17:12.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.159+0000 7fcf777fe640 1 -- 172.21.15.110:0/3202107952 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcf680581a0 con 0x7fcf800a9ad0 2024-01-20T11:17:12.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.159+0000 7fcf777fe640 10 client.4516.objecter ms_dispatch 0x7fcf80000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:12.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.159+0000 7fcf777fe640 3 client.4516.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:12.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.159+0000 7fcf777fe640 3 client.4516.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:12.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.159+0000 7fcf777fe640 20 client.4516.objecter dump_active .. 0 homeless 2024-01-20T11:17:12.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.171+0000 7fcf777fe640 1 -- 172.21.15.110:0/3202107952 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fcf68053de0 con 0x7fcf800a9ad0 2024-01-20T11:17:12.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.171+0000 7fcf777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:12.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.171+0000 7fcf777fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:12.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.339+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcf800b3fb0 con 0x7fcf6006c8e0 2024-01-20T11:17:12.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.342+0000 7fcf777fe640 1 -- 172.21.15.110:0/3202107952 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64502 (secure 0 0 0) 0x7fcf70005080 con 0x7fcf6006c8e0 2024-01-20T11:17:12.343 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 10 librados: watch_flush enter 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 10 librados: watch_flush exit 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 20 client.4516.objecter shutdown clearing up homeless session... 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 10 client.4516.objecter successfully canceled tick 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcf6006c8e0 msgr2=0x7fcf6006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcf6006c8e0 0x7fcf6006eda0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcf70000f60 tx=0x7fcf700079b0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 10 monclient: shutdown 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf800a9ad0 msgr2=0x7fcf80131fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf800a9ad0 0x7fcf80131fb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800c440 tx=0x7fcf68001e70 comp rx=0 tx=0).stop 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 shutdown_connections 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcf6006c8e0 0x7fcf6006eda0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf80142af0 0x7fcf801324f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf800a9ad0 0x7fcf80131fb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 --2- 172.21.15.110:0/3202107952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf800a96f0 0x7fcf801345f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.343+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 >> 172.21.15.110:0/3202107952 conn(0x7fcf800b7bc0 msgr2=0x7fcf800b72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.344+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 shutdown_connections 2024-01-20T11:17:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.344+0000 7fcf85268640 1 -- 172.21.15.110:0/3202107952 wait complete. 2024-01-20T11:17:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.344+0000 7fcf85268640 1 librados: shutdown 2024-01-20T11:17:12.355 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-01-20T11:17:10.734164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001931"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.0289999999999999},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.6260000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":10.452999999999999},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.164999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.792},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.864999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.218},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":15.914}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":18.635999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.6400000000000006},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":17.695}]}]},{"osd":7,"up_from":21,"seq":90194313243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.411},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.7569999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.9260000000000002},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":15.935}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":17.446999999999999},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.6839999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.573},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":18.225999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.215999999999999},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":11.816000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.159000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.471}]}]},{"osd":3,"up_from":21,"seq":90194313244,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.3390000000000004},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.5739999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.342000000000001},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.3469999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":12.205},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.535}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.699999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.672000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":11.59},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":16.385999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":17.184999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.478}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444058,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.0049999999999999},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.2599999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.3520000000000003},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.2319999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.685},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.8480000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.832000000000001},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.009},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.819000000000001},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":15.833}]}]},{"osd":1,"up_from":17,"seq":73014444059,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":7.5389999999999997},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":8.3520000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.3149999999999995},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.169}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.534000000000001},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.0579999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":12.695},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":18.088999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":14.276},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":14.824999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.178999999999998},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.898}]}]},{"osd":0,"up_from":17,"seq":73014444059,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":10.048999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.734}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.0490000000000004},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.346}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":12.113},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.4860000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.759},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.6639999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":15.478},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":11.500999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.717000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":8.9359999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":7.5259999999999998},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.7599999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.3840000000000003},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":9.5969999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.0009999999999994},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.422000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.612},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":16.007000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.202},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.920999999999999},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.776}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:12.357 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:12.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.454+0000 7f8efb361640 10 monclient: get_monmap_and_config 2024-01-20T11:17:12.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.454+0000 7f8efb361640 10 monclient: build_initial_monmap 2024-01-20T11:17:12.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.454+0000 7f8efb361640 10 monclient: monmap: 2024-01-20T11:17:12.456 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:12.456 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:12.456 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:12.456328+0000 2024-01-20T11:17:12.456 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:12.456328+0000 2024-01-20T11:17:12.456 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:12.456 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:12.456 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:12.456 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:12.457 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:12.457 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:12.457 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:12.457 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.455+0000 7f8efb361640 1 Processor -- start 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.455+0000 7f8efb361640 1 -- start start 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.455+0000 7f8efb361640 10 monclient: init 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ef406f920 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ef406fd00 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 0x7f8ef414b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef4144060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef40715b0 con 0x7f8ef406fd00 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef4071890 con 0x7f8ef406f920 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:12.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef4071b70 con 0x7f8ef4144060 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8efb361640 10 monclient(hunting): authenticate will time out at 1475.807437s 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8ef90fd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8ef406f920 0x7f8ef413d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:52394/0 (socket says 172.21.15.110:52394) 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8ef90fd640 1 -- 172.21.15.110:0/3667433172 learned_addr learned my addr 172.21.15.110:0/3667433172 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.456+0000 7f8ef98fe640 1 --2- 172.21.15.110:0/3667433172 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 0x7f8ef414b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.457+0000 7f8ef98fe640 10 monclient(hunting): get_auth_request con 0x7f8ef4144060 auth_method 0 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.457+0000 7f8ef98fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.457+0000 7f8ef98fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.457+0000 7f8ef98fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.457+0000 7f8ef98fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.457+0000 7f8ef98fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:12.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.457+0000 7f8ef98fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:12.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8eebfff640 1 -- 172.21.15.110:0/3667433172 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606896454 0 0) 0x7f8ef4071890 con 0x7f8ef406f920 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8eebfff640 10 monclient(hunting): _negotiate 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8eebfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8eebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8eebfff640 10 monclient(hunting): my global_id is 4519 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8eebfff640 1 -- 172.21.15.110:0/3667433172 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0003520 con 0x7f8ef406f920 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8ef98fe640 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8ef98fe640 1 -- 172.21.15.110:0/3667433172 >> v1:172.21.15.134:6789/0 conn(0x7f8ef406f920 legacy=0x7f8ef413d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8ef98fe640 1 -- 172.21.15.110:0/3667433172 >> v1:172.21.15.110:6789/0 conn(0x7f8ef406fd00 legacy=0x7f8ef4143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8ef98fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8ef98fe640 1 monclient: found mon.noname-b 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8ef98fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8ef98fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:12.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.459+0000 7f8ef98fe640 1 -- 172.21.15.110:0/3667433172 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef4143430 con 0x7f8ef4144060 2024-01-20T11:17:12.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.461+0000 7f8ef98fe640 1 --2- 172.21.15.110:0/3667433172 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 0x7f8ef414b730 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009dc0 tx=0x7f8eec02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c534a714f2bf804 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:12.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 1 -- 172.21.15.110:0/3667433172 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec02cbb0 con 0x7f8ef4144060 2024-01-20T11:17:12.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:12.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:12.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 10 monclient: dump: 2024-01-20T11:17:12.466 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 10 monclient: _finish_auth 0 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 1 -- 172.21.15.110:0/3667433172 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec02cd10 con 0x7f8ef4144060 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:12.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 1 -- 172.21.15.110:0/3667433172 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec034660 con 0x7f8ef4144060 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8eebfff640 10 monclient: dump: 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:12.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8efb361640 5 monclient: authenticate success, global_id 4370 2024-01-20T11:17:12.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8efb361640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:12.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.465+0000 7f8efb361640 4 set_mon_vals no callback set 2024-01-20T11:17:12.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.466+0000 7f8efb361640 10 monclient: shutdown 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.466+0000 7f8efb361640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.466+0000 7f8efb361640 1 -- 172.21.15.110:0/3667433172 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 msgr2=0x7f8ef414b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.466+0000 7f8efb361640 1 --2- 172.21.15.110:0/3667433172 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 0x7f8ef414b730 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009dc0 tx=0x7f8eec02be00 comp rx=0 tx=0).stop 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.466+0000 7f8efb361640 1 -- 172.21.15.110:0/3667433172 shutdown_connections 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.466+0000 7f8efb361640 1 --2- 172.21.15.110:0/3667433172 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 0x7f8ef414b730 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.466+0000 7f8efb361640 1 -- 172.21.15.110:0/3667433172 >> 172.21.15.110:0/3667433172 conn(0x7f8ef4066d60 msgr2=0x7f8ef40b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.466+0000 7f8efb361640 1 -- 172.21.15.110:0/3667433172 shutdown_connections 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.466+0000 7f8efb361640 1 -- 172.21.15.110:0/3667433172 wait complete. 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 10 monclient: build_initial_monmap 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 10 monclient: monmap: 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 1 librados: starting msgr at 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 1 librados: starting objecter 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 1 Processor -- start 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 1 -- start start 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 1 librados: setting wanted keys 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 1 librados: calling monclient init 2024-01-20T11:17:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 10 monclient: init 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ef406f920 0x7f8ef41391d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef406f920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef406fd00 0x7f8ef406bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 10 monclient(hunting): picked mon.noname-a con 0x7f8ef406fd00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.467+0000 7f8efb361640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 0x7f8ef4131cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8efb361640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef4144060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8efb361640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8efb361640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef406c8a0 con 0x7f8ef406fd00 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8efb361640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8efb361640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef406c4f0 con 0x7f8ef406f920 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8efb361640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:12.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8efb361640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef406b740 con 0x7f8ef4144060 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8efb361640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8efb361640 10 monclient(hunting): authenticate will time out at 1475.819206s 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef98fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 0x7f8ef4131cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef88fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef406fd00 0x7f8ef406bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef90fd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ef406f920 0x7f8ef41391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef88fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef406fd00 0x7f8ef406bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34312/0 (socket says 172.21.15.110:34312) 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef88fc640 1 -- 172.21.15.110:0/576761409 learned_addr learned my addr 172.21.15.110:0/576761409 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef88fc640 10 monclient(hunting): get_auth_request con 0x7f8ef406fd00 auth_method 0 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef88fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef88fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef88fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef98fe640 10 monclient(hunting): get_auth_request con 0x7f8ef4144060 auth_method 0 2024-01-20T11:17:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef98fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef98fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef98fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef90fd640 10 monclient(hunting): get_auth_request con 0x7f8ef406f920 auth_method 0 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef90fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef90fd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.468+0000 7f8ef90fd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.469+0000 7f8ef88fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.469+0000 7f8ef88fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.469+0000 7f8ef88fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.469+0000 7f8ef98fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.469+0000 7f8ef98fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.469+0000 7f8ef98fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.469+0000 7f8ef90fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.469+0000 7f8ef90fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.469+0000 7f8ef90fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 10 monclient(hunting): handle_auth_done global_id 4522 payload 824 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 1 -- 172.21.15.110:0/576761409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 msgr2=0x7f8ef4131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 0x7f8ef4131cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 1 -- 172.21.15.110:0/576761409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef406fd00 msgr2=0x7f8ef406bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef406fd00 0x7f8ef406bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 1 monclient: found mon.noname-b 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:12.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.472+0000 7f8ef90fd640 1 -- 172.21.15.110:0/576761409 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec009a70 con 0x7f8ef406f920 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.474+0000 7f8ef90fd640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ef406f920 0x7f8ef41391d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400c3c0 tx=0x7f8ee4001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.474+0000 7f8ee9ffb640 10 client.?.objecter ms_handle_connect 0x7f8ef406f920 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.474+0000 7f8ee9ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/576761409 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4005ce0 con 0x7f8ef406f920 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 10 monclient: dump: 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:12.480 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/576761409 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee4005e40 con 0x7f8ef406f920 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/576761409 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40072d0 con 0x7f8ef406f920 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8efb361640 5 monclient: authenticate success, global_id 4522 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8efb361640 10 monclient: _renew_subs 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8efb361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:12.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8eeb7fe640 4 set_mon_vals no callback set 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef4132230 con 0x7f8ef406f920 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8ee9ffb640 10 monclient: dump: 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:12.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8efb361640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:12.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8efb361640 10 monclient: _renew_subs 2024-01-20T11:17:12.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8efb361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:12.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef41da880 con 0x7f8ef406f920 2024-01-20T11:17:12.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.478+0000 7f8efb361640 1 librados: init done 2024-01-20T11:17:12.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.479+0000 7f8efb361640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:12.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.479+0000 7f8efb361640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:12.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.479+0000 7f8efb361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:12.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.479+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef4132530 con 0x7f8ef406f920 2024-01-20T11:17:12.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.497+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/576761409 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8ee40074f0 con 0x7f8ef406f920 2024-01-20T11:17:12.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.497+0000 7f8ee9ffb640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ed806c8b0 0x7f8ed806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:12.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.498+0000 7f8ef88fc640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ed806c8b0 0x7f8ed806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:12.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.498+0000 7f8ef88fc640 10 monclient: get_auth_request con 0x7f8ed806c8b0 auth_method 0 2024-01-20T11:17:12.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.498+0000 7f8ef88fc640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ed806c8b0 0x7f8ed806ed70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ef406d8e0 tx=0x7f8ef0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:12.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.498+0000 7f8ee9ffb640 10 client.4522.objecter ms_handle_connect 0x7f8ed806c8b0 2024-01-20T11:17:12.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.500+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/576761409 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8ee4057c60 con 0x7f8ef406f920 2024-01-20T11:17:12.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.500+0000 7f8ee9ffb640 10 client.4522.objecter ms_dispatch 0x7f8ef4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.500+0000 7f8ee9ffb640 3 client.4522.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.500+0000 7f8ee9ffb640 3 client.4522.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.501+0000 7f8ee9ffb640 20 client.4522.objecter dump_active .. 0 homeless 2024-01-20T11:17:12.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.746+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/576761409 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8ee4053fa0 con 0x7f8ef406f920 2024-01-20T11:17:12.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.746+0000 7f8ee9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:12.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.746+0000 7f8ee9ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.924+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ef414bc70 con 0x7f8ed806c8b0 2024-01-20T11:17:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.927+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/576761409 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64478 (secure 0 0 0) 0x7f8ef000f820 con 0x7f8ed806c8b0 2024-01-20T11:17:12.929 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:12.929 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 10 librados: watch_flush enter 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 10 librados: watch_flush exit 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 20 client.4522.objecter shutdown clearing up homeless session... 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 10 client.4522.objecter successfully canceled tick 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ed806c8b0 msgr2=0x7f8ed806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ed806c8b0 0x7f8ed806ed70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ef406d8e0 tx=0x7f8ef0009040 comp rx=0 tx=0).stop 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 10 monclient: shutdown 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ef406f920 msgr2=0x7f8ef41391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ef406f920 0x7f8ef41391d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400c3c0 tx=0x7f8ee4001e70 comp rx=0 tx=0).stop 2024-01-20T11:17:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 shutdown_connections 2024-01-20T11:17:12.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ed806c8b0 0x7f8ed806ed70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef4144060 0x7f8ef4131cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef406fd00 0x7f8ef406bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 --2- 172.21.15.110:0/576761409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ef406f920 0x7f8ef41391d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:12.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 >> 172.21.15.110:0/576761409 conn(0x7f8ef4066d60 msgr2=0x7f8ef413a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:12.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.929+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 shutdown_connections 2024-01-20T11:17:12.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.930+0000 7f8efb361640 1 -- 172.21.15.110:0/576761409 wait complete. 2024-01-20T11:17:12.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:12.930+0000 7f8efb361640 1 librados: shutdown 2024-01-20T11:17:12.941 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-01-20T11:17:12.734446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001886"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.0289999999999999},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.6260000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":10.452999999999999},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.164999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.792},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.864999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.218},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":15.914}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":18.635999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.6400000000000006},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":17.695}]}]},{"osd":7,"up_from":21,"seq":90194313244,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.5970000000000004},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.5109999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.6999999999999993},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":15.692}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.555999999999999},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.109}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.397},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":16.370999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.689},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":14.359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.015000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.949}]}]},{"osd":3,"up_from":21,"seq":90194313245,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":11.489000000000001},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4860000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":18.771000000000001},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":19.655000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":10.148},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":7.2640000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.1549999999999994},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.339}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":6.2000000000000002},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.728}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":14.961},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.365}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444058,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.0049999999999999},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.2599999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.3520000000000003},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.2319999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.685},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.8480000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.832000000000001},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.009},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.819000000000001},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":15.833}]}]},{"osd":1,"up_from":17,"seq":73014444059,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":7.5389999999999997},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":8.3520000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.3149999999999995},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.169}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.534000000000001},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.0579999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":12.695},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":18.088999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":14.276},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":14.824999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.178999999999998},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.898}]}]},{"osd":0,"up_from":17,"seq":73014444059,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":10.048999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.734}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.0490000000000004},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.346}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":12.113},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.4860000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.759},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.6639999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":15.478},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":11.500999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.717000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":8.9359999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":7.5259999999999998},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.7599999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.3840000000000003},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":9.5969999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.0009999999999994},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.422000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.612},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":16.007000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.202},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.920999999999999},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.776}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:12.942 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:12.942 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:12.942 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:12.943 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.002387+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.002387+0000', 'last_peered': '2024-01-20T11:16:53.002387+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.002387+0000', 'last_undegraded': '2024-01-20T11:16:53.002387+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:12.943 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:12.943 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.858298+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.858298+0000', 'last_peered': '2024-01-20T11:16:52.858298+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.858298+0000', 'last_undegraded': '2024-01-20T11:16:52.858298+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:12.943 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:12.943 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 127, 'reported_epoch': 29, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:51.809778+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:51.809778+0000', 'last_peered': '2024-01-20T11:16:51.809778+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:51.809778+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:12.943 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.042+0000 7fbfd931d640 10 monclient: get_monmap_and_config 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.042+0000 7fbfd931d640 10 monclient: build_initial_monmap 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.042+0000 7fbfd931d640 10 monclient: monmap: 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:13.043845+0000 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:13.043845+0000 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:13.044 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.042+0000 7fbfd931d640 1 Processor -- start 2024-01-20T11:17:13.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.042+0000 7fbfd931d640 1 -- start start 2024-01-20T11:17:13.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.042+0000 7fbfd931d640 10 monclient: init 2024-01-20T11:17:13.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:13.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-01-20T11:17:13.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 0x7fbfd413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd40a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbfd413fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbfd40a9ad0 0x7fbfd4142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd40a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd40b25c0 con 0x7fbfd413fa10 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd40aaea0 con 0x7fbfd40a9ad0 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40ab0a0 con 0x7fbfd40a96f0 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.043+0000 7fbfd931d640 10 monclient(hunting): authenticate will time out at 1476.394678s 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 0x7fbfd413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd37fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbfd40a9ad0 0x7fbfd4142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd27fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbfd413fa10 0x7fbfd41423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:52402/0 (socket says 172.21.15.110:52402) 2024-01-20T11:17:13.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd27fc640 1 -- 172.21.15.110:0/2350713684 learned_addr learned my addr 172.21.15.110:0/2350713684 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd37fe640 10 monclient(hunting): get_auth_request con 0x7fbfd40a9ad0 auth_method 0 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd2ffd640 10 monclient(hunting): get_auth_request con 0x7fbfd40a96f0 auth_method 0 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.044+0000 7fbfd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.045+0000 7fbfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.045+0000 7fbfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.045+0000 7fbfd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.046+0000 7fbfd37fe640 10 monclient(hunting): handle_auth_done global_id 4473 payload 274 2024-01-20T11:17:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.046+0000 7fbfd37fe640 1 -- 172.21.15.110:0/2350713684 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 msgr2=0x7fbfd413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.046+0000 7fbfd37fe640 1 --2- 172.21.15.110:0/2350713684 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 0x7fbfd413f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.046+0000 7fbfd37fe640 1 -- 172.21.15.110:0/2350713684 >> v1:172.21.15.134:6789/0 conn(0x7fbfd413fa10 legacy=0x7fbfd41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.046+0000 7fbfd37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.046+0000 7fbfd37fe640 1 monclient: found mon.noname-a 2024-01-20T11:17:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.046+0000 7fbfd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.046+0000 7fbfd37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.046+0000 7fbfd37fe640 1 -- 172.21.15.110:0/2350713684 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd40b3fb0 con 0x7fbfd40a9ad0 2024-01-20T11:17:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.048+0000 7fbfd37fe640 1 --2- 172.21.15.110:0/2350713684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 0x7fbfd4142af0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0097a0 tx=0x7fbfbc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff65783a3fde8885 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:13.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.051+0000 7fbfd1ffb640 1 -- 172.21.15.110:0/2350713684 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc02e070 con 0x7fbfd40a9ad0 2024-01-20T11:17:13.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.052+0000 7fbfd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.052+0000 7fbfd1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.052+0000 7fbfd1ffb640 10 monclient: dump: 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.052+0000 7fbfd1ffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.052+0000 7fbfd1ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:13.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.052+0000 7fbfd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:13.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.052+0000 7fbfd931d640 5 monclient: authenticate success, global_id 4473 2024-01-20T11:17:13.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.052+0000 7fbfd931d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:13.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.053+0000 7fbfd1ffb640 1 -- 172.21.15.110:0/2350713684 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc0315a0 con 0x7fbfd40a9ad0 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.053+0000 7fbfd1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.053+0000 7fbfd1ffb640 1 -- 172.21.15.110:0/2350713684 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc031930 con 0x7fbfd40a9ad0 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.053+0000 7fbfd931d640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.053+0000 7fbfd931d640 4 set_mon_vals no callback set 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.053+0000 7fbfd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.053+0000 7fbfd1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.053+0000 7fbfd1ffb640 10 monclient: dump: 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.055 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.056 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:13.056 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.056 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:13.056 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:13.056 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:13.056 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 10 monclient: shutdown 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 1 -- 172.21.15.110:0/2350713684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 msgr2=0x7fbfd4142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 1 --2- 172.21.15.110:0/2350713684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 0x7fbfd4142af0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0097a0 tx=0x7fbfbc02b840 comp rx=0 tx=0).stop 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 1 -- 172.21.15.110:0/2350713684 shutdown_connections 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 1 --2- 172.21.15.110:0/2350713684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 0x7fbfd4142af0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 1 --2- 172.21.15.110:0/2350713684 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 0x7fbfd413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 1 -- 172.21.15.110:0/2350713684 >> 172.21.15.110:0/2350713684 conn(0x7fbfd40b7bc0 msgr2=0x7fbfd40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 1 -- 172.21.15.110:0/2350713684 shutdown_connections 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.054+0000 7fbfd931d640 1 -- 172.21.15.110:0/2350713684 wait complete. 2024-01-20T11:17:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.055+0000 7fbfd931d640 10 monclient: build_initial_monmap 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.055+0000 7fbfd931d640 10 monclient: monmap: 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.055+0000 7fbfd931d640 1 librados: starting msgr at 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.055+0000 7fbfd931d640 1 librados: starting objecter 2024-01-20T11:17:13.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 Processor -- start 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 -- start start 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 librados: setting wanted keys 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 librados: calling monclient init 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient: init 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 0x7fbfd4133040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient(hunting): picked mon.noname-c con 0x7fbfd40a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 0x7fbfd413d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd40a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfd413caf0 0x7fbfd413da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd413caf0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd413d100 con 0x7fbfd40a9ad0 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd413ebe0 con 0x7fbfd413caf0 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd41345f0 con 0x7fbfd40a96f0 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.056+0000 7fbfd931d640 10 monclient(hunting): authenticate will time out at 1476.408017s 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd27fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 0x7fbfd413d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 0x7fbfd4133040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd37fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfd413caf0 0x7fbfd413da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd27fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 0x7fbfd413d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34324/0 (socket says 172.21.15.110:34324) 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd27fc640 1 -- 172.21.15.110:0/18509195 learned_addr learned my addr 172.21.15.110:0/18509195 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd27fc640 10 monclient(hunting): get_auth_request con 0x7fbfd40a9ad0 auth_method 0 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd37fe640 10 monclient(hunting): get_auth_request con 0x7fbfd413caf0 auth_method 0 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd2ffd640 10 monclient(hunting): get_auth_request con 0x7fbfd40a96f0 auth_method 0 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.057+0000 7fbfd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.058+0000 7fbfd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.058+0000 7fbfd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.058+0000 7fbfd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.058+0000 7fbfd27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.058+0000 7fbfd27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.058+0000 7fbfd27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.058+0000 7fbfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.058+0000 7fbfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.058+0000 7fbfd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 10 monclient(hunting): handle_auth_done global_id 4528 payload 824 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 1 -- 172.21.15.110:0/18509195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 msgr2=0x7fbfd4133040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 0x7fbfd4133040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 1 -- 172.21.15.110:0/18509195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 msgr2=0x7fbfd413d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 0x7fbfd413d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 1 monclient: found mon.noname-b 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.061+0000 7fbfd37fe640 1 -- 172.21.15.110:0/18509195 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc009400 con 0x7fbfd413caf0 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.063+0000 7fbfd37fe640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfd413caf0 0x7fbfd413da90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbfd41430c0 tx=0x7fbfbc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.063+0000 7fbfcb7fe640 10 client.?.objecter ms_handle_connect 0x7fbfd413caf0 2024-01-20T11:17:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.063+0000 7fbfcb7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:13.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 1 -- 172.21.15.110:0/18509195 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc032630 con 0x7fbfd413caf0 2024-01-20T11:17:13.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 10 monclient: dump: 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 10 monclient: _finish_auth 0 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 1 -- 172.21.15.110:0/18509195 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc0327b0 con 0x7fbfd413caf0 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd931d640 5 monclient: authenticate success, global_id 4528 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd931d640 10 monclient: _renew_subs 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd931d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd4132590 con 0x7fbfd413caf0 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 1 -- 172.21.15.110:0/18509195 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc032b60 con 0x7fbfd413caf0 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfcb7fe640 10 monclient: dump: 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd17fa640 4 set_mon_vals no callback set 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd931d640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd931d640 10 monclient: _renew_subs 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd931d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd413dfd0 con 0x7fbfd413caf0 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.067+0000 7fbfd931d640 1 librados: init done 2024-01-20T11:17:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.068+0000 7fbfd931d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:13.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.068+0000 7fbfd931d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:13.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.068+0000 7fbfd931d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:13.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.068+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd413d270 con 0x7fbfd413caf0 2024-01-20T11:17:13.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.314+0000 7fbfcb7fe640 1 -- 172.21.15.110:0/18509195 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fbfbc032cc0 con 0x7fbfd413caf0 2024-01-20T11:17:13.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.315+0000 7fbfcb7fe640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbfb406c8b0 0x7fbfb406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.315+0000 7fbfd2ffd640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbfb406c8b0 0x7fbfb406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.315+0000 7fbfd2ffd640 10 monclient: get_auth_request con 0x7fbfb406c8b0 auth_method 0 2024-01-20T11:17:13.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.315+0000 7fbfcb7fe640 10 client.4528.objecter ms_handle_connect 0x7fbfb406c8b0 2024-01-20T11:17:13.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.315+0000 7fbfd2ffd640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbfb406c8b0 0x7fbfb406ed70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0113e0 tx=0x7fbfcc00b040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:13.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.317+0000 7fbfcb7fe640 1 -- 172.21.15.110:0/18509195 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbfbc079d00 con 0x7fbfd413caf0 2024-01-20T11:17:13.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.317+0000 7fbfcb7fe640 10 client.4528.objecter ms_dispatch 0x7fbfd4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:13.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.317+0000 7fbfcb7fe640 3 client.4528.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:13.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.317+0000 7fbfcb7fe640 3 client.4528.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:13.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.317+0000 7fbfcb7fe640 20 client.4528.objecter dump_active .. 0 homeless 2024-01-20T11:17:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.329+0000 7fbfcb7fe640 1 -- 172.21.15.110:0/18509195 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fbfbc076040 con 0x7fbfd413caf0 2024-01-20T11:17:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.329+0000 7fbfcb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.329+0000 7fbfcb7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:13.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.571+0000 7fbfd931d640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:13.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.571+0000 7fbfd931d640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:13.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.571+0000 7fbfd931d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:13.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.571+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbfd413d270 con 0x7fbfd413caf0 2024-01-20T11:17:13.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.576+0000 7fbfcb7fe640 1 -- 172.21.15.110:0/18509195 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7fbfd413d270 con 0x7fbfd413caf0 2024-01-20T11:17:13.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.576+0000 7fbfcb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:13.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.576+0000 7fbfcb7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:13.577 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:13.578 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":168,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:13.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.578+0000 7fbfd931d640 10 librados: watch_flush enter 2024-01-20T11:17:13.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.578+0000 7fbfd931d640 10 librados: watch_flush exit 2024-01-20T11:17:13.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.578+0000 7fbfd931d640 20 client.4528.objecter shutdown clearing up homeless session... 2024-01-20T11:17:13.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.578+0000 7fbfd931d640 10 client.4528.objecter successfully canceled tick 2024-01-20T11:17:13.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbfb406c8b0 msgr2=0x7fbfb406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbfb406c8b0 0x7fbfb406ed70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0113e0 tx=0x7fbfcc00b040 comp rx=0 tx=0).stop 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 10 monclient: shutdown 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfd413caf0 msgr2=0x7fbfd413da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfd413caf0 0x7fbfd413da90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbfd41430c0 tx=0x7fbfbc0046c0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 shutdown_connections 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbfb406c8b0 0x7fbfb406ed70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfd413caf0 0x7fbfd413da90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd40a9ad0 0x7fbfd413d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 --2- 172.21.15.110:0/18509195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfd40a96f0 0x7fbfd4133040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 >> 172.21.15.110:0/18509195 conn(0x7fbfd40b7bc0 msgr2=0x7fbfd40b6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 shutdown_connections 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 -- 172.21.15.110:0/18509195 wait complete. 2024-01-20T11:17:13.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.579+0000 7fbfd931d640 1 librados: shutdown 2024-01-20T11:17:13.591 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:13.591 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.689+0000 7ff364747640 10 monclient: get_monmap_and_config 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.689+0000 7ff364747640 10 monclient: build_initial_monmap 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.689+0000 7ff364747640 10 monclient: monmap: 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:13.690649+0000 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:13.690649+0000 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:13.690 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:13.691 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:13.691 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.689+0000 7ff364747640 1 Processor -- start 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.689+0000 7ff364747640 1 -- start start 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.689+0000 7ff364747640 10 monclient: init 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff35c0a96f0 0x7ff35c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 10 monclient(hunting): picked mon.noname-a con 0x7ff35c0a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff35c13fa10 0x7ff35c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 10 monclient(hunting): picked mon.noname-c con 0x7ff35c13fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff35c0a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0b25c0 con 0x7ff35c0a9ad0 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c0aaea0 con 0x7ff35c0a96f0 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c0ab0a0 con 0x7ff35c13fa10 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff364747640 10 monclient(hunting): authenticate will time out at 1477.041650s 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.690+0000 7ff361ce2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff35c13fa10 0x7ff35c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff3624e3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff35c0a96f0 0x7ff35c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff361ce2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff35c13fa10 0x7ff35c1423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:40178/0 (socket says 172.21.15.110:40178) 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff361ce2640 1 -- 172.21.15.110:0/1170667489 learned_addr learned my addr 172.21.15.110:0/1170667489 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff361ce2640 10 monclient(hunting): get_auth_request con 0x7ff35c13fa10 auth_method 0 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff361ce2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff361ce2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff361ce2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff3624e3640 10 monclient(hunting): get_auth_request con 0x7ff35c0a96f0 auth_method 0 2024-01-20T11:17:13.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff3624e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff3624e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff3624e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff361ce2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff361ce2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff361ce2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff3624e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff3624e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.691+0000 7ff3624e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.693+0000 7ff3624e3640 10 monclient(hunting): handle_auth_done global_id 4479 payload 274 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.693+0000 7ff3624e3640 1 -- 172.21.15.110:0/1170667489 >> v2:172.21.15.134:3300/0 conn(0x7ff35c13fa10 msgr2=0x7ff35c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.693+0000 7ff3624e3640 1 --2- 172.21.15.110:0/1170667489 >> v2:172.21.15.134:3300/0 conn(0x7ff35c13fa10 0x7ff35c1423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.693+0000 7ff3624e3640 1 -- 172.21.15.110:0/1170667489 >> v1:172.21.15.110:6789/0 conn(0x7ff35c0a9ad0 legacy=0x7ff35c142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.693+0000 7ff3624e3640 10 monclient: _finish_hunting 0 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.693+0000 7ff3624e3640 1 monclient: found mon.noname-a 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.693+0000 7ff3624e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.693+0000 7ff3624e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.693+0000 7ff3624e3640 1 -- 172.21.15.110:0/1170667489 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0b3fb0 con 0x7ff35c0a96f0 2024-01-20T11:17:13.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.695+0000 7ff3624e3640 1 --2- 172.21.15.110:0/1170667489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a96f0 0x7ff35c13f4d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff35c06d0f0 tx=0x7ff35802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0b0c25ac6a6b6f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:13.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 1 -- 172.21.15.110:0/1170667489 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff358031710 con 0x7ff35c0a96f0 2024-01-20T11:17:13.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:13.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 10 monclient: dump: 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 10 monclient: _finish_auth 0 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 10 monclient: _check_auth_tickets 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:13.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff364747640 5 monclient: authenticate success, global_id 4479 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 1 -- 172.21.15.110:0/1170667489 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35802f050 con 0x7ff35c0a96f0 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff364747640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 1 -- 172.21.15.110:0/1170667489 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff358031a90 con 0x7ff35c0a96f0 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff364747640 4 set_mon_vals no callback set 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.699+0000 7ff3614e1640 10 monclient: dump: 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:13.702 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:13.703 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 10 monclient: shutdown 2024-01-20T11:17:13.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:13.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 1 -- 172.21.15.110:0/1170667489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a96f0 msgr2=0x7ff35c13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:13.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 1 --2- 172.21.15.110:0/1170667489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a96f0 0x7ff35c13f4d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff35c06d0f0 tx=0x7ff35802b7b0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 1 -- 172.21.15.110:0/1170667489 shutdown_connections 2024-01-20T11:17:13.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 1 --2- 172.21.15.110:0/1170667489 >> v2:172.21.15.134:3300/0 conn(0x7ff35c13fa10 0x7ff35c1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 1 --2- 172.21.15.110:0/1170667489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a96f0 0x7ff35c13f4d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 1 -- 172.21.15.110:0/1170667489 >> 172.21.15.110:0/1170667489 conn(0x7ff35c0b7bc0 msgr2=0x7ff35c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 1 -- 172.21.15.110:0/1170667489 shutdown_connections 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.700+0000 7ff364747640 1 -- 172.21.15.110:0/1170667489 wait complete. 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.701+0000 7ff364747640 10 monclient: build_initial_monmap 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.701+0000 7ff364747640 10 monclient: monmap: 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:13.707 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.701+0000 7ff364747640 1 librados: starting msgr at 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.701+0000 7ff364747640 1 librados: starting objecter 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.701+0000 7ff364747640 1 Processor -- start 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.701+0000 7ff364747640 1 -- start start 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.701+0000 7ff364747640 1 librados: setting wanted keys 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.701+0000 7ff364747640 1 librados: calling monclient init 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.701+0000 7ff364747640 10 monclient: init 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c0a96f0 0x7ff35c1d71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient(hunting): picked mon.noname-c con 0x7ff35c0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a9ad0 0x7ff35c13b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient(hunting): picked mon.noname-a con 0x7ff35c0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:13.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff35c1d7c20 0x7ff35c133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient(hunting): picked mon.noname-b con 0x7ff35c1d7c20 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c13c130 con 0x7ff35c0a9ad0 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c13afc0 con 0x7ff35c1d7c20 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c13a8c0 con 0x7ff35c0a96f0 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff364747640 10 monclient(hunting): authenticate will time out at 1477.053407s 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff361ce2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a9ad0 0x7ff35c13b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff362ce4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff35c1d7c20 0x7ff35c133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff3624e3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c0a96f0 0x7ff35c1d71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff361ce2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a9ad0 0x7ff35c13b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34346/0 (socket says 172.21.15.110:34346) 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff361ce2640 1 -- 172.21.15.110:0/4122864223 learned_addr learned my addr 172.21.15.110:0/4122864223 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff361ce2640 10 monclient(hunting): get_auth_request con 0x7ff35c0a9ad0 auth_method 0 2024-01-20T11:17:13.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff361ce2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff361ce2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.702+0000 7ff361ce2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff362ce4640 10 monclient(hunting): get_auth_request con 0x7ff35c1d7c20 auth_method 0 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff362ce4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff362ce4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff362ce4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff3624e3640 10 monclient(hunting): get_auth_request con 0x7ff35c0a96f0 auth_method 0 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff3624e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff3624e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff3624e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff361ce2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff361ce2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff361ce2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff362ce4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff362ce4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff362ce4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff3624e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:13.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff3624e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:13.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.703+0000 7ff3624e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 10 monclient(hunting): handle_auth_done global_id 4534 payload 824 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 1 -- 172.21.15.110:0/4122864223 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c0a96f0 msgr2=0x7ff35c1d71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c0a96f0 0x7ff35c1d71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 1 -- 172.21.15.110:0/4122864223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a9ad0 msgr2=0x7ff35c13b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a9ad0 0x7ff35c13b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 10 monclient: _finish_hunting 0 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 1 monclient: found mon.noname-b 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.706+0000 7ff362ce4640 1 -- 172.21.15.110:0/4122864223 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff358009400 con 0x7ff35c1d7c20 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.708+0000 7ff362ce4640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff35c1d7c20 0x7ff35c133d30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff35400c750 tx=0x7ff35400cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:13.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.708+0000 7ff352ffd640 10 client.?.objecter ms_handle_connect 0x7ff35c1d7c20 2024-01-20T11:17:13.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.708+0000 7ff352ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 1 -- 172.21.15.110:0/4122864223 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354003b70 con 0x7ff35c1d7c20 2024-01-20T11:17:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 10 monclient: dump: 2024-01-20T11:17:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 1 -- 172.21.15.110:0/4122864223 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff354003cf0 con 0x7ff35c1d7c20 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff364747640 5 monclient: authenticate success, global_id 4534 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff364747640 10 monclient: _renew_subs 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff364747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35c13abe0 con 0x7ff35c1d7c20 2024-01-20T11:17:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 1 -- 172.21.15.110:0/4122864223 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354003ea0 con 0x7ff35c1d7c20 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff352ffd640 10 monclient: dump: 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff364747640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff360ce0640 4 set_mon_vals no callback set 2024-01-20T11:17:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff364747640 10 monclient: _renew_subs 2024-01-20T11:17:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff364747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c13bc00 con 0x7ff35c1d7c20 2024-01-20T11:17:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.712+0000 7ff364747640 1 librados: init done 2024-01-20T11:17:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.713+0000 7ff364747640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.713+0000 7ff364747640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.713+0000 7ff364747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.713+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35c075490 con 0x7ff35c1d7c20 2024-01-20T11:17:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.972+0000 7ff352ffd640 1 -- 172.21.15.110:0/4122864223 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff3540052b0 con 0x7ff35c1d7c20 2024-01-20T11:17:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.972+0000 7ff352ffd640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff34006c8b0 0x7ff34006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.972+0000 7ff3624e3640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff34006c8b0 0x7ff34006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.972+0000 7ff3624e3640 10 monclient: get_auth_request con 0x7ff34006c8b0 auth_method 0 2024-01-20T11:17:13.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.973+0000 7ff3624e3640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff34006c8b0 0x7ff34006ed70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff358035000 tx=0x7ff358008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:13.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.973+0000 7ff352ffd640 10 client.4534.objecter ms_handle_connect 0x7ff34006c8b0 2024-01-20T11:17:13.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.976+0000 7ff352ffd640 1 -- 172.21.15.110:0/4122864223 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff354057ea0 con 0x7ff35c1d7c20 2024-01-20T11:17:13.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.976+0000 7ff352ffd640 10 client.4534.objecter ms_dispatch 0x7ff35c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:13.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.976+0000 7ff352ffd640 3 client.4534.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:13.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.976+0000 7ff352ffd640 3 client.4534.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:13.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.976+0000 7ff352ffd640 20 client.4534.objecter dump_active .. 0 homeless 2024-01-20T11:17:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.986+0000 7ff352ffd640 1 -- 172.21.15.110:0/4122864223 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff3540541e0 con 0x7ff35c1d7c20 2024-01-20T11:17:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.986+0000 7ff352ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:13.986+0000 7ff352ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:14.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.155+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff35c13b100 con 0x7ff34006c8b0 2024-01-20T11:17:14.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.160+0000 7ff352ffd640 1 -- 172.21.15.110:0/4122864223 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64478 (secure 0 0 0) 0x7ff35c13b100 con 0x7ff34006c8b0 2024-01-20T11:17:14.161 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:14.162 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:14.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.161+0000 7ff364747640 10 librados: watch_flush enter 2024-01-20T11:17:14.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.161+0000 7ff364747640 10 librados: watch_flush exit 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.161+0000 7ff364747640 20 client.4534.objecter shutdown clearing up homeless session... 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.161+0000 7ff364747640 10 client.4534.objecter successfully canceled tick 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff34006c8b0 msgr2=0x7ff34006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff34006c8b0 0x7ff34006ed70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff358035000 tx=0x7ff358008030 comp rx=0 tx=0).stop 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 10 monclient: shutdown 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff35c1d7c20 msgr2=0x7ff35c133d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff35c1d7c20 0x7ff35c133d30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff35400c750 tx=0x7ff35400cc10 comp rx=0 tx=0).stop 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 shutdown_connections 2024-01-20T11:17:14.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff34006c8b0 0x7ff34006ed70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:14.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff35c1d7c20 0x7ff35c133d30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:14.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0a9ad0 0x7ff35c13b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:14.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 --2- 172.21.15.110:0/4122864223 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c0a96f0 0x7ff35c1d71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:14.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 >> 172.21.15.110:0/4122864223 conn(0x7ff35c0b7bc0 msgr2=0x7ff35c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:14.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 shutdown_connections 2024-01-20T11:17:14.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 -- 172.21.15.110:0/4122864223 wait complete. 2024-01-20T11:17:14.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:14.162+0000 7ff364747640 1 librados: shutdown 2024-01-20T11:17:14.173 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-01-20T11:17:12.734446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001886"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.0289999999999999},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.6260000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":10.452999999999999},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.164999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.792},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.864999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.218},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":15.914}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":18.635999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.6400000000000006},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":17.695}]}]},{"osd":7,"up_from":21,"seq":90194313244,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.5970000000000004},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.5109999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.6999999999999993},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":15.692}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.555999999999999},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.109}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.397},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":16.370999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.689},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":14.359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.015000000000001},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.949}]}]},{"osd":3,"up_from":21,"seq":90194313245,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":11.489000000000001},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4860000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":18.771000000000001},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":19.655000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":10.148},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":7.2640000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.1549999999999994},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.339}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":6.2000000000000002},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.728}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":14.961},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.365}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444058,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.0049999999999999},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.2599999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.3520000000000003},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.2319999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.685},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.8480000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.832000000000001},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.009},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.819000000000001},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":15.833}]}]},{"osd":1,"up_from":17,"seq":73014444059,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":7.5389999999999997},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":8.3520000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.3149999999999995},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.169}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":14.534000000000001},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.0579999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":12.695},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":18.088999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":14.276},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":14.824999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.178999999999998},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.898}]}]},{"osd":0,"up_from":17,"seq":73014444059,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":10.048999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":6.734}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.0490000000000004},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.346}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":12.113},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.4860000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.759},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.6639999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":15.478},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":11.500999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.717000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":8.9359999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509468,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":11.781000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":106.059,"5min":106.059,"15min":106.059},"last":7.5259999999999998},{"interface":"front","average":{"1min":14.972,"5min":14.972,"15min":14.972},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":157.67699999999999,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.7599999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.562999999999999},"min":{"1min":4.6319999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":189.37700000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.3840000000000003},{"interface":"front","average":{"1min":31.734000000000002,"5min":31.734000000000002,"15min":31.734000000000002},"min":{"1min":7.0499999999999998,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":190.304,"5min":190.304,"15min":190.304},"last":9.5969999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:16:12 2024","interfaces":[{"interface":"back","average":{"1min":15.147,"5min":15.147,"15min":15.147},"min":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":128.67099999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.0009999999999994},{"interface":"front","average":{"1min":16.167999999999999,"5min":16.167999999999999,"15min":16.167999999999999},"min":{"1min":7.9960000000000004,"5min":7.9960000000000004,"15min":7.9960000000000004},"max":{"1min":101.75700000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.422000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.612},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":16.007000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.202},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.920999999999999},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.776}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:17.176 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:17.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.281+0000 7f87704cf640 10 monclient: get_monmap_and_config 2024-01-20T11:17:17.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.281+0000 7f87704cf640 10 monclient: build_initial_monmap 2024-01-20T11:17:17.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.281+0000 7f87704cf640 10 monclient: monmap: 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:17.282769+0000 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:17.282769+0000 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.281+0000 7f87704cf640 1 Processor -- start 2024-01-20T11:17:17.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.281+0000 7f87704cf640 1 -- start start 2024-01-20T11:17:17.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.281+0000 7f87704cf640 10 monclient: init 2024-01-20T11:17:17.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:17.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-01-20T11:17:17.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a96f0 0x7f876813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 10 monclient(hunting): picked mon.noname-b con 0x7f87680a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:17.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f876813fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:17.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f87680a9ad0 0x7f8768142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 10 monclient(hunting): picked mon.noname-a con 0x7f87680a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:17.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87680b25c0 con 0x7f876813fa10 2024-01-20T11:17:17.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87680aaea0 con 0x7f87680a9ad0 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87680ab0a0 con 0x7f87680a96f0 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.282+0000 7f87704cf640 10 monclient(hunting): authenticate will time out at 1480.633853s 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876e26b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a96f0 0x7f876813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876ea6c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f87680a9ad0 0x7f8768142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876ea6c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f87680a9ad0 0x7f8768142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34352/0 (socket says 172.21.15.110:34352) 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876ea6c640 1 -- 172.21.15.110:0/3628481096 learned_addr learned my addr 172.21.15.110:0/3628481096 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876ea6c640 10 monclient(hunting): get_auth_request con 0x7f87680a9ad0 auth_method 0 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876ea6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876ea6c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876ea6c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876e26b640 10 monclient(hunting): get_auth_request con 0x7f87680a96f0 auth_method 0 2024-01-20T11:17:17.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876e26b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876e26b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.283+0000 7f876e26b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.284+0000 7f876ea6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.284+0000 7f876ea6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.284+0000 7f876ea6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.284+0000 7f876e26b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.284+0000 7f876e26b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.284+0000 7f876e26b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876e26b640 10 monclient(hunting): handle_auth_done global_id 4385 payload 274 2024-01-20T11:17:17.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876e26b640 1 -- 172.21.15.110:0/3628481096 >> v2:172.21.15.110:3300/0 conn(0x7f87680a9ad0 msgr2=0x7f8768142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:17.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876e26b640 1 --2- 172.21.15.110:0/3628481096 >> v2:172.21.15.110:3300/0 conn(0x7f87680a9ad0 0x7f8768142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876e26b640 1 -- 172.21.15.110:0/3628481096 >> v1:172.21.15.110:6789/0 conn(0x7f876813fa10 legacy=0x7f87681423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:17.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876e26b640 10 monclient: _finish_hunting 0 2024-01-20T11:17:17.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876e26b640 1 monclient: found mon.noname-b 2024-01-20T11:17:17.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876e26b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:17.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876e26b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:17.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876e26b640 1 -- 172.21.15.110:0/3628481096 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680b3fb0 con 0x7f87680a96f0 2024-01-20T11:17:17.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.285+0000 7f876ea6c640 1 --2- 172.21.15.110:0/3628481096 >> v2:172.21.15.110:3300/0 conn(0x7f87680a9ad0 0x7f8768142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:17.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.287+0000 7f876e26b640 1 --2- 172.21.15.110:0/3628481096 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a96f0 0x7f876813f4d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f87640097a0 tx=0x7f876402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35efc49248e68479 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:17.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 1 -- 172.21.15.110:0/3628481096 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876402f050 con 0x7f87680a96f0 2024-01-20T11:17:17.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:17.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:17.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 10 monclient: dump: 2024-01-20T11:17:17.293 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:17.293 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.293 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.293 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.293 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 10 monclient: _finish_auth 0 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 10 monclient: _check_auth_tickets 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 1 -- 172.21.15.110:0/3628481096 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8764031700 con 0x7f87680a96f0 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f87704cf640 5 monclient: authenticate success, global_id 4385 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f87704cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f87704cf640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 1 -- 172.21.15.110:0/3628481096 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764031a90 con 0x7f87680a96f0 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f87704cf640 4 set_mon_vals no callback set 2024-01-20T11:17:17.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f876d269640 10 monclient: dump: 2024-01-20T11:17:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.291+0000 7f87704cf640 10 monclient: shutdown 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.292+0000 7f87704cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.292+0000 7f87704cf640 1 -- 172.21.15.110:0/3628481096 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a96f0 msgr2=0x7f876813f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.292+0000 7f87704cf640 1 --2- 172.21.15.110:0/3628481096 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a96f0 0x7f876813f4d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f87640097a0 tx=0x7f876402b7a0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.292+0000 7f87704cf640 1 -- 172.21.15.110:0/3628481096 shutdown_connections 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.292+0000 7f87704cf640 1 --2- 172.21.15.110:0/3628481096 >> v2:172.21.15.110:3300/0 conn(0x7f87680a9ad0 0x7f8768142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.292+0000 7f87704cf640 1 --2- 172.21.15.110:0/3628481096 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a96f0 0x7f876813f4d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.292+0000 7f87704cf640 1 -- 172.21.15.110:0/3628481096 >> 172.21.15.110:0/3628481096 conn(0x7f87680b7bc0 msgr2=0x7f87680b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.292+0000 7f87704cf640 1 -- 172.21.15.110:0/3628481096 shutdown_connections 2024-01-20T11:17:17.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.292+0000 7f87704cf640 1 -- 172.21.15.110:0/3628481096 wait complete. 2024-01-20T11:17:17.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.293+0000 7f87704cf640 10 monclient: build_initial_monmap 2024-01-20T11:17:17.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.293+0000 7f87704cf640 10 monclient: monmap: 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.293+0000 7f87704cf640 1 librados: starting msgr at 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.293+0000 7f87704cf640 1 librados: starting objecter 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.293+0000 7f87704cf640 1 Processor -- start 2024-01-20T11:17:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.293+0000 7f87704cf640 1 -- start start 2024-01-20T11:17:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.293+0000 7f87704cf640 1 librados: setting wanted keys 2024-01-20T11:17:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.293+0000 7f87704cf640 1 librados: calling monclient init 2024-01-20T11:17:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.293+0000 7f87704cf640 10 monclient: init 2024-01-20T11:17:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:17:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87680a96f0 0x7f876813b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient(hunting): picked mon.noname-b con 0x7f87680a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a9ad0 0x7f87681dad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient(hunting): picked mon.noname-c con 0x7f87680a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:17.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f876813b660 0x7f87681db2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient(hunting): picked mon.noname-a con 0x7f876813b660 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:17.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87680abd20 con 0x7f876813b660 2024-01-20T11:17:17.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87681da840 con 0x7f87680a96f0 2024-01-20T11:17:17.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8768137ff0 con 0x7f87680a9ad0 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f87704cf640 10 monclient(hunting): authenticate will time out at 1480.645521s 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f876e26b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87680a96f0 0x7f876813b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.294+0000 7f876da6a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a9ad0 0x7f87681dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876ea6c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f876813b660 0x7f87681db2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876e26b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87680a96f0 0x7f876813b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:40204/0 (socket says 172.21.15.110:40204) 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876e26b640 1 -- 172.21.15.110:0/713888069 learned_addr learned my addr 172.21.15.110:0/713888069 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876e26b640 10 monclient(hunting): get_auth_request con 0x7f87680a96f0 auth_method 0 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876e26b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876e26b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876e26b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876ea6c640 10 monclient(hunting): get_auth_request con 0x7f876813b660 auth_method 0 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876ea6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876ea6c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876ea6c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876da6a640 10 monclient(hunting): get_auth_request con 0x7f87680a9ad0 auth_method 0 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876da6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876da6a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876da6a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876e26b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876e26b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.295+0000 7f876e26b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.296+0000 7f876ea6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.296+0000 7f876ea6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.296+0000 7f876ea6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.296+0000 7f876da6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.296+0000 7f876da6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.296+0000 7f876da6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 10 monclient(hunting): handle_auth_done global_id 4537 payload 824 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 1 -- 172.21.15.110:0/713888069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a9ad0 msgr2=0x7f87681dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a9ad0 0x7f87681dad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 1 -- 172.21.15.110:0/713888069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f876813b660 msgr2=0x7f87681db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f876813b660 0x7f87681db2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 10 monclient: _finish_hunting 0 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 1 monclient: found mon.noname-b 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:17.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.298+0000 7f876e26b640 1 -- 172.21.15.110:0/713888069 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8760009b30 con 0x7f87680a96f0 2024-01-20T11:17:17.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.300+0000 7f876e26b640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87680a96f0 0x7f876813b120 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8764000c00 tx=0x7f87640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:17.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.300+0000 7f875effd640 10 client.?.objecter ms_handle_connect 0x7f87680a96f0 2024-01-20T11:17:17.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.300+0000 7f875effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:17.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 1 -- 172.21.15.110:0/713888069 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640339b0 con 0x7f87680a96f0 2024-01-20T11:17:17.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:17.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:17.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 10 monclient: dump: 2024-01-20T11:17:17.309 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:17.310 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.310 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.310 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.310 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:17.310 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.310 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:17.310 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:17.310 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:17.311 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 10 monclient: _finish_auth 0 2024-01-20T11:17:17.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:17.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:17.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 1 -- 172.21.15.110:0/713888069 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8764033b10 con 0x7f87680a96f0 2024-01-20T11:17:17.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f87704cf640 5 monclient: authenticate success, global_id 4537 2024-01-20T11:17:17.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f87704cf640 10 monclient: _renew_subs 2024-01-20T11:17:17.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f87704cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:17.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8764009400 con 0x7f87680a96f0 2024-01-20T11:17:17.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:17.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 1 -- 172.21.15.110:0/713888069 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876403c400 con 0x7f87680a96f0 2024-01-20T11:17:17.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:17.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:17.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f875effd640 10 monclient: dump: 2024-01-20T11:17:17.312 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:17.312 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.312 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.312 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f876ca68640 4 set_mon_vals no callback set 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f87704cf640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f87704cf640 10 monclient: _renew_subs 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f87704cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876813a680 con 0x7f87680a96f0 2024-01-20T11:17:17.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.304+0000 7f87704cf640 1 librados: init done 2024-01-20T11:17:17.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.305+0000 7f87704cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:17.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.305+0000 7f87704cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:17.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.305+0000 7f87704cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:17.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.305+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87680aa800 con 0x7f87680a96f0 2024-01-20T11:17:17.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.555+0000 7f875effd640 1 -- 172.21.15.110:0/713888069 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f876403a040 con 0x7f87680a96f0 2024-01-20T11:17:17.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.556+0000 7f875effd640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f874c06c8b0 0x7f874c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.556+0000 7f876da6a640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f874c06c8b0 0x7f874c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.556+0000 7f876da6a640 10 monclient: get_auth_request con 0x7f874c06c8b0 auth_method 0 2024-01-20T11:17:17.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.556+0000 7f875effd640 10 client.4537.objecter ms_handle_connect 0x7f874c06c8b0 2024-01-20T11:17:17.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.556+0000 7f876da6a640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f874c06c8b0 0x7f874c06ed70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f87681d82b0 tx=0x7f8758006f90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:17.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.558+0000 7f875effd640 1 -- 172.21.15.110:0/713888069 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8764079ed0 con 0x7f87680a96f0 2024-01-20T11:17:17.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.558+0000 7f875effd640 10 client.4537.objecter ms_dispatch 0x7f8768000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:17.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.558+0000 7f875effd640 3 client.4537.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:17.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.558+0000 7f875effd640 3 client.4537.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:17.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.558+0000 7f875effd640 20 client.4537.objecter dump_active .. 0 homeless 2024-01-20T11:17:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.571+0000 7f875effd640 1 -- 172.21.15.110:0/713888069 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8764076210 con 0x7f87680a96f0 2024-01-20T11:17:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.571+0000 7f875effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.571+0000 7f875effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:17.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.755+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87680abe90 con 0x7f874c06c8b0 2024-01-20T11:17:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.761+0000 7f875effd640 1 -- 172.21.15.110:0/713888069 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64406 (secure 0 0 0) 0x7f87680abe90 con 0x7f874c06c8b0 2024-01-20T11:17:17.762 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:17.762 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:17.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 10 librados: watch_flush enter 2024-01-20T11:17:17.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 10 librados: watch_flush exit 2024-01-20T11:17:17.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 20 client.4537.objecter shutdown clearing up homeless session... 2024-01-20T11:17:17.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 10 client.4537.objecter successfully canceled tick 2024-01-20T11:17:17.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f874c06c8b0 msgr2=0x7f874c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:17.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f874c06c8b0 0x7f874c06ed70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f87681d82b0 tx=0x7f8758006f90 comp rx=0 tx=0).stop 2024-01-20T11:17:17.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 10 monclient: shutdown 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87680a96f0 msgr2=0x7f876813b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87680a96f0 0x7f876813b120 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8764000c00 tx=0x7f87640046c0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 shutdown_connections 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f874c06c8b0 0x7f874c06ed70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f876813b660 0x7f87681db2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87680a9ad0 0x7f87681dad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 --2- 172.21.15.110:0/713888069 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87680a96f0 0x7f876813b120 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 >> 172.21.15.110:0/713888069 conn(0x7f87680b7bc0 msgr2=0x7f87680b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 shutdown_connections 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 -- 172.21.15.110:0/713888069 wait complete. 2024-01-20T11:17:17.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.762+0000 7f87704cf640 1 librados: shutdown 2024-01-20T11:17:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-01-20T11:17:16.735080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001899"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.7319999999999993},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.4100000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":6.0010000000000003},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.819000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.022},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.625999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.5749999999999993},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":12.435}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.332000000000001},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":15.137}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.132999999999999},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":16.741}]}]},{"osd":7,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.989000000000001},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.0519999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.5350000000000001},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":15.191000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":17.376999999999999},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.6470000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.42},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":19.763000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.927},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":16.513000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.113},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":18.716000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313246,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.0899999999999999},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.1260000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.855},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.189}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":7.9779999999999998},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.493}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.143000000000001},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.426}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":8.7029999999999994},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.089}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":15.231999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.913}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444059,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.0049999999999999},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.2599999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.3520000000000003},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.2319999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.685},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.8480000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.832000000000001},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.009},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.819000000000001},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":15.833}]}]},{"osd":1,"up_from":17,"seq":73014444060,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":7.3499999999999996},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":6.6130000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.298},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.734}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":9.3330000000000002},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.1579999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":14.433},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":10.593}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":10.013},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":16.027000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.566000000000001},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.247}]}]},{"osd":0,"up_from":17,"seq":73014444060,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":9.1219999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":8.0510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.0579999999999998},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":10.416}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":15.4},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.6509999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.186999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.1849999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":9.8510000000000009},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":12.76}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.351000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":14.599}]}]},{"osd":6,"up_from":15,"seq":64424509469,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":10.471},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.718}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.0429999999999993},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.394}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.843},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3819999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.036},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.666}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.786999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.333},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.885999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:17.775 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:17.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.873+0000 7f560ae15640 10 monclient: get_monmap_and_config 2024-01-20T11:17:17.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.873+0000 7f560ae15640 10 monclient: build_initial_monmap 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.873+0000 7f560ae15640 10 monclient: monmap: 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:17.874315+0000 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:17.874315+0000 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:17.874 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.873+0000 7f560ae15640 1 Processor -- start 2024-01-20T11:17:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.873+0000 7f560ae15640 1 -- start start 2024-01-20T11:17:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.873+0000 7f560ae15640 10 monclient: init 2024-01-20T11:17:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-01-20T11:17:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f560406f920 0x7f560406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient(hunting): picked mon.noname-c con 0x7f560406f920 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560413d230 0x7f560413d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient(hunting): picked mon.noname-b con 0x7f560413d230 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5604143950 0x7f5604143da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient(hunting): picked mon.noname-a con 0x7f5604143950 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56041442e0 con 0x7f5604143950 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56040715b0 con 0x7f560406f920 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5604071890 con 0x7f560413d230 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f560ae15640 10 monclient(hunting): authenticate will time out at 1481.225286s 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f5608bb1640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f560406f920 0x7f560406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f56093b2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5604143950 0x7f5604143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f5603fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560413d230 0x7f560413d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f56093b2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5604143950 0x7f5604143da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34376/0 (socket says 172.21.15.110:34376) 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f56093b2640 1 -- 172.21.15.110:0/252440201 learned_addr learned my addr 172.21.15.110:0/252440201 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f56093b2640 10 monclient(hunting): get_auth_request con 0x7f5604143950 auth_method 0 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f56093b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f56093b2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f56093b2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f5608bb1640 10 monclient(hunting): get_auth_request con 0x7f560406f920 auth_method 0 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f5608bb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f5608bb1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.874+0000 7f5608bb1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5603fff640 10 monclient(hunting): get_auth_request con 0x7f560413d230 auth_method 0 2024-01-20T11:17:17.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5603fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5603fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5603fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5608bb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5608bb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5608bb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f56093b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f56093b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f56093b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5603fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5603fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.875+0000 7f5603fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 10 monclient(hunting): handle_auth_done global_id 4540 payload 274 2024-01-20T11:17:17.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 1 -- 172.21.15.110:0/252440201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560413d230 msgr2=0x7f560413d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:17.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 1 --2- 172.21.15.110:0/252440201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560413d230 0x7f560413d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 1 -- 172.21.15.110:0/252440201 >> v2:172.21.15.110:3300/0 conn(0x7f5604143950 msgr2=0x7f5604143da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:17.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 1 --2- 172.21.15.110:0/252440201 >> v2:172.21.15.110:3300/0 conn(0x7f5604143950 0x7f5604143da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 10 monclient: _finish_hunting 0 2024-01-20T11:17:17.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 1 monclient: found mon.noname-c 2024-01-20T11:17:17.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:17.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:17.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f5608bb1640 1 -- 172.21.15.110:0/252440201 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604071b70 con 0x7f560406f920 2024-01-20T11:17:17.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.877+0000 7f56093b2640 1 --2- 172.21.15.110:0/252440201 >> v2:172.21.15.110:3300/0 conn(0x7f5604143950 0x7f5604143da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:17.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.879+0000 7f5608bb1640 1 --2- 172.21.15.110:0/252440201 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560406f920 0x7f560406fd00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5604067350 tx=0x7f55f802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4531680248bcdc0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:17.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 1 -- 172.21.15.110:0/252440201 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f802f050 con 0x7f560406f920 2024-01-20T11:17:17.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:17.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 10 monclient: dump: 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 10 monclient: _finish_auth 0 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 10 monclient: _check_auth_tickets 2024-01-20T11:17:17.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 1 -- 172.21.15.110:0/252440201 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f8031700 con 0x7f560406f920 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 5 monclient: authenticate success, global_id 4540 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 1 -- 172.21.15.110:0/252440201 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8031a90 con 0x7f560406f920 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 4 set_mon_vals no callback set 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f56037fe640 10 monclient: dump: 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.885 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.886 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.886 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:17.886 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.886 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:17.886 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:17.887 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:17.887 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 10 monclient: shutdown 2024-01-20T11:17:17.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:17.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 1 -- 172.21.15.110:0/252440201 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560406f920 msgr2=0x7f560406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:17.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 1 --2- 172.21.15.110:0/252440201 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560406f920 0x7f560406fd00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5604067350 tx=0x7f55f802b7a0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 1 -- 172.21.15.110:0/252440201 shutdown_connections 2024-01-20T11:17:17.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 1 --2- 172.21.15.110:0/252440201 >> v2:172.21.15.110:3300/0 conn(0x7f5604143950 0x7f5604143da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 1 --2- 172.21.15.110:0/252440201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560413d230 0x7f560413d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 1 --2- 172.21.15.110:0/252440201 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560406f920 0x7f560406fd00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 1 -- 172.21.15.110:0/252440201 >> 172.21.15.110:0/252440201 conn(0x7f5604066d60 msgr2=0x7f56040b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 1 -- 172.21.15.110:0/252440201 shutdown_connections 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.883+0000 7f560ae15640 1 -- 172.21.15.110:0/252440201 wait complete. 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.884+0000 7f560ae15640 10 monclient: build_initial_monmap 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.884+0000 7f560ae15640 10 monclient: monmap: 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:17.892 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.884+0000 7f560ae15640 1 librados: starting msgr at 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.884+0000 7f560ae15640 1 librados: starting objecter 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.884+0000 7f560ae15640 1 Processor -- start 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.884+0000 7f560ae15640 1 -- start start 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.884+0000 7f560ae15640 1 librados: setting wanted keys 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.884+0000 7f560ae15640 1 librados: calling monclient init 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.884+0000 7f560ae15640 10 monclient: init 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560406f920 0x7f5604133e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient(hunting): picked mon.noname-c con 0x7f560406f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560413d230 0x7f5604137a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient(hunting): picked mon.noname-b con 0x7f560413d230 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5604143950 0x7f5604141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient(hunting): picked mon.noname-a con 0x7f5604143950 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5604138bc0 con 0x7f5604143950 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5604138ea0 con 0x7f560413d230 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5604143120 con 0x7f560406f920 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f560ae15640 10 monclient(hunting): authenticate will time out at 1481.236668s 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.885+0000 7f56093b2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5604143950 0x7f5604141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5603fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560413d230 0x7f5604137a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5608bb1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560406f920 0x7f5604133e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f56093b2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5604143950 0x7f5604141190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34386/0 (socket says 172.21.15.110:34386) 2024-01-20T11:17:17.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f56093b2640 1 -- 172.21.15.110:0/2232956497 learned_addr learned my addr 172.21.15.110:0/2232956497 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f56093b2640 10 monclient(hunting): get_auth_request con 0x7f5604143950 auth_method 0 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f56093b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f56093b2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f56093b2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5603fff640 10 monclient(hunting): get_auth_request con 0x7f560413d230 auth_method 0 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5603fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5603fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5603fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5608bb1640 10 monclient(hunting): get_auth_request con 0x7f560406f920 auth_method 0 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5608bb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5608bb1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5608bb1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5603fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5603fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5603fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f56093b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f56093b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f56093b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5608bb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5608bb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.886+0000 7f5608bb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 10 monclient(hunting): handle_auth_done global_id 4543 payload 824 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 1 -- 172.21.15.110:0/2232956497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560406f920 msgr2=0x7f5604133e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560406f920 0x7f5604133e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 1 -- 172.21.15.110:0/2232956497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5604143950 msgr2=0x7f5604141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5604143950 0x7f5604141190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:17.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 1 monclient: found mon.noname-b 2024-01-20T11:17:17.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:17.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:17.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f5603fff640 1 -- 172.21.15.110:0/2232956497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc009b30 con 0x7f560413d230 2024-01-20T11:17:17.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.889+0000 7f56093b2640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5604143950 0x7f5604141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:17.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.891+0000 7f5603fff640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560413d230 0x7f5604137a80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55f000e7c0 tx=0x7f55f000ebc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:17.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.892+0000 7f56017fa640 10 client.?.objecter ms_handle_connect 0x7f560413d230 2024-01-20T11:17:17.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.892+0000 7f56017fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:17.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 1 -- 172.21.15.110:0/2232956497 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0005990 con 0x7f560413d230 2024-01-20T11:17:17.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:17.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:17.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 10 monclient: dump: 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 10 monclient: _finish_auth 0 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 10 monclient: _check_auth_tickets 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 1 -- 172.21.15.110:0/2232956497 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f0005af0 con 0x7f560413d230 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 1 -- 172.21.15.110:0/2232956497 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0005e60 con 0x7f560413d230 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f56017fa640 10 monclient: dump: 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:17.900 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 5 monclient: authenticate success, global_id 4543 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 10 monclient: _renew_subs 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f5602ffd640 4 set_mon_vals no callback set 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f8009400 con 0x7f560413d230 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 10 monclient: _renew_subs 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5604142ed0 con 0x7f560413d230 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 1 librados: init done 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:17.896+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5604075490 con 0x7f560413d230 2024-01-20T11:17:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.146+0000 7f56017fa640 1 -- 172.21.15.110:0/2232956497 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f55f000c730 con 0x7f560413d230 2024-01-20T11:17:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.146+0000 7f56017fa640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f55e806c8b0 0x7f55e806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.146+0000 7f5608bb1640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f55e806c8b0 0x7f55e806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.146+0000 7f5608bb1640 10 monclient: get_auth_request con 0x7f55e806c8b0 auth_method 0 2024-01-20T11:17:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.146+0000 7f56017fa640 10 client.4543.objecter ms_handle_connect 0x7f55e806c8b0 2024-01-20T11:17:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.146+0000 7f5608bb1640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f55e806c8b0 0x7f55e806ed70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f55f8035000 tx=0x7f55f8008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:18.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.149+0000 7f56017fa640 1 -- 172.21.15.110:0/2232956497 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f55f00587b0 con 0x7f560413d230 2024-01-20T11:17:18.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.149+0000 7f56017fa640 10 client.4543.objecter ms_dispatch 0x7f5604000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:18.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.149+0000 7f56017fa640 3 client.4543.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:18.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.149+0000 7f56017fa640 3 client.4543.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:18.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.149+0000 7f56017fa640 20 client.4543.objecter dump_active .. 0 homeless 2024-01-20T11:17:18.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.160+0000 7f56017fa640 1 -- 172.21.15.110:0/2232956497 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f55f0054af0 con 0x7f560413d230 2024-01-20T11:17:18.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.160+0000 7f56017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:18.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.160+0000 7f56017fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:18.226 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:09.250 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:18.274 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:03:09.280 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:18.275 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:03:09.282 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:18.286 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:09.311 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:18.291 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:03:09.297 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:18.307 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:09.332 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:18.334 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:09.358 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:18.337 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:09.361 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:18.355 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:09.379 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:18.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.357+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f560413fe70 con 0x7f55e806c8b0 2024-01-20T11:17:18.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.361+0000 7f56017fa640 1 -- 172.21.15.110:0/2232956497 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64406 (secure 0 0 0) 0x7f55f8042070 con 0x7f55e806c8b0 2024-01-20T11:17:18.362 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:18.362 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:18.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 10 librados: watch_flush enter 2024-01-20T11:17:18.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 10 librados: watch_flush exit 2024-01-20T11:17:18.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 20 client.4543.objecter shutdown clearing up homeless session... 2024-01-20T11:17:18.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 10 client.4543.objecter successfully canceled tick 2024-01-20T11:17:18.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f55e806c8b0 msgr2=0x7f55e806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f55e806c8b0 0x7f55e806ed70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f55f8035000 tx=0x7f55f8008030 comp rx=0 tx=0).stop 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 10 monclient: shutdown 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560413d230 msgr2=0x7f5604137a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560413d230 0x7f5604137a80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55f000e7c0 tx=0x7f55f000ebc0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 shutdown_connections 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f55e806c8b0 0x7f55e806ed70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5604143950 0x7f5604141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f560413d230 0x7f5604137a80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 --2- 172.21.15.110:0/2232956497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f560406f920 0x7f5604133e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 >> 172.21.15.110:0/2232956497 conn(0x7f5604066d60 msgr2=0x7f56040b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 shutdown_connections 2024-01-20T11:17:18.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 -- 172.21.15.110:0/2232956497 wait complete. 2024-01-20T11:17:18.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.363+0000 7f560ae15640 1 librados: shutdown 2024-01-20T11:17:18.375 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-01-20T11:17:16.735080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001899"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":127,"reported_epoch":29,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:51.809778+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:51.809778+0000","last_peered":"2024-01-20T11:16:51.809778+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:51.809778+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.7319999999999993},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.4100000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":6.0010000000000003},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.819000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.022},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.625999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.5749999999999993},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":12.435}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.332000000000001},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":15.137}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.132999999999999},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":16.741}]}]},{"osd":7,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.989000000000001},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.0519999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.5350000000000001},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":15.191000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":17.376999999999999},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.6470000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.42},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":19.763000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.927},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":16.513000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.113},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":18.716000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313246,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.0899999999999999},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.1260000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.855},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.189}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":7.9779999999999998},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.493}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.143000000000001},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.426}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":8.7029999999999994},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.089}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":15.231999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.913}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444059,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"min":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":25.603999999999999,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.0049999999999999},{"interface":"front","average":{"1min":8.6170000000000009,"5min":8.6170000000000009,"15min":8.6170000000000009},"min":{"1min":4.3559999999999999,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.152000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.2599999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":14.291,"5min":14.291,"15min":14.291},"min":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"max":{"1min":65.064999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.3520000000000003},{"interface":"front","average":{"1min":16.827000000000002,"5min":16.827000000000002,"15min":16.827000000000002},"min":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":32.414999999999999,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.2319999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.685},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.8480000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.832000000000001},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":16.076000000000001,"5min":16.076000000000001,"15min":16.076000000000001},"min":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":35.186999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.009},{"interface":"front","average":{"1min":10.949,"5min":10.949,"15min":10.949},"min":{"1min":4.7850000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":32.984000000000002,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.819000000000001},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":15.833}]}]},{"osd":1,"up_from":17,"seq":73014444060,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7390000000000008,"5min":9.7390000000000008,"15min":9.7390000000000008},"min":{"1min":5.5759999999999996,"5min":5.5759999999999996,"15min":5.5759999999999996},"max":{"1min":15.821999999999999,"5min":15.821999999999999,"15min":15.821999999999999},"last":7.3499999999999996},{"interface":"front","average":{"1min":10.596,"5min":10.596,"15min":10.596},"min":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":30.439,"5min":30.439,"15min":30.439},"last":6.6130000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":13.443,"15min":13.443},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.298},{"interface":"front","average":{"1min":13.599,"5min":13.599,"15min":13.599},"min":{"1min":7.2649999999999997,"5min":7.2649999999999997,"15min":7.2649999999999997},"max":{"1min":23.981999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.734}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":9.3330000000000002},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.1579999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":14.433},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":10.593}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":12.282999999999999,"5min":12.282999999999999,"15min":12.282999999999999},"min":{"1min":7.1139999999999999,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":19.478999999999999,"5min":19.478999999999999,"15min":19.478999999999999},"last":10.013},{"interface":"front","average":{"1min":15.781000000000001,"5min":15.781000000000001,"15min":15.781000000000001},"min":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"max":{"1min":31.634,"5min":31.634,"15min":31.634},"last":16.027000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.566000000000001},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.247}]}]},{"osd":0,"up_from":17,"seq":73014444060,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":15.379,"5min":15.379,"15min":15.379},"min":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":114.843,"5min":114.843,"15min":114.843},"last":9.1219999999999999},{"interface":"front","average":{"1min":16.315999999999999,"5min":16.315999999999999,"15min":16.315999999999999},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":121.044,"5min":121.044,"15min":121.044},"last":8.0510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":32.008000000000003,"5min":32.008000000000003,"15min":32.008000000000003},"min":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":379.17099999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.0579999999999998},{"interface":"front","average":{"1min":33.503999999999998,"5min":33.503999999999998,"15min":33.503999999999998},"min":{"1min":7.5659999999999998,"5min":7.5659999999999998,"15min":7.5659999999999998},"max":{"1min":373.88299999999998,"5min":373.88299999999998,"15min":373.88299999999998},"last":10.416}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":15.4},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.6509999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.186999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.1849999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:13 2024","interfaces":[{"interface":"back","average":{"1min":36.143999999999998,"5min":36.143999999999998,"15min":36.143999999999998},"min":{"1min":10.125,"5min":10.125,"15min":10.125},"max":{"1min":375.733,"5min":375.733,"15min":375.733},"last":9.8510000000000009},{"interface":"front","average":{"1min":33.488999999999997,"5min":33.488999999999997,"15min":33.488999999999997},"min":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":374.654,"5min":374.654,"15min":374.654},"last":12.76}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.351000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":14.599}]}]},{"osd":6,"up_from":15,"seq":64424509469,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":10.471},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.718}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.0429999999999993},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.394}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.843},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3819999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.036},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.666}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.786999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.333},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.885999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:18.376 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:18.376 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:18.377 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:18.377 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.002387+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.002387+0000', 'last_peered': '2024-01-20T11:16:53.002387+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.002387+0000', 'last_undegraded': '2024-01-20T11:16:53.002387+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:18.377 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:18.377 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.858298+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.858298+0000', 'last_peered': '2024-01-20T11:16:52.858298+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.858298+0000', 'last_undegraded': '2024-01-20T11:16:52.858298+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:18.378 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:18.378 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 127, 'reported_epoch': 29, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:51.809778+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:51.809778+0000', 'last_peered': '2024-01-20T11:16:51.809778+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:51.809778+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:18.379 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:18.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.482+0000 7fa37b215640 10 monclient: get_monmap_and_config 2024-01-20T11:17:18.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.482+0000 7fa37b215640 10 monclient: build_initial_monmap 2024-01-20T11:17:18.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.482+0000 7fa37b215640 10 monclient: monmap: 2024-01-20T11:17:18.484 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:18.484 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:18.484 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:18.484364+0000 2024-01-20T11:17:18.484 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:18.484364+0000 2024-01-20T11:17:18.485 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:18.485 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:18.485 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:18.485 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:18.485 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:18.485 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:18.486 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:18.486 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:18.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.483+0000 7fa37b215640 1 Processor -- start 2024-01-20T11:17:18.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.483+0000 7fa37b215640 1 -- start start 2024-01-20T11:17:18.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.483+0000 7fa37b215640 10 monclient: init 2024-01-20T11:17:18.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:18.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-01-20T11:17:18.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 0x7fa3740b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:18.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 10 monclient(hunting): picked mon.noname-b con 0x7fa3740b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:18.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3740b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa374145980 0x7fa374145e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 10 monclient(hunting): picked mon.noname-c con 0x7fa374145980 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3740aade0 con 0x7fa3740b17c0 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3740b50f0 con 0x7fa374145980 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa374145460 con 0x7fa3740b4d10 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa37b215640 10 monclient(hunting): authenticate will time out at 1481.835327s 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.484+0000 7fa3797b2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa374145980 0x7fa374145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.485+0000 7fa3797b2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa374145980 0x7fa374145e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:40228/0 (socket says 172.21.15.110:40228) 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.485+0000 7fa3797b2640 1 -- 172.21.15.110:0/2790143348 learned_addr learned my addr 172.21.15.110:0/2790143348 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.485+0000 7fa3797b2640 10 monclient(hunting): get_auth_request con 0x7fa374145980 auth_method 0 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.485+0000 7fa3797b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.485+0000 7fa3797b2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.485+0000 7fa3797b2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.486+0000 7fa3797b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.486+0000 7fa3797b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.486+0000 7fa3797b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.486+0000 7fa378fb1640 1 --2- 172.21.15.110:0/2790143348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 0x7fa3740b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.487+0000 7fa3797b2640 10 monclient(hunting): handle_auth_done global_id 4546 payload 274 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.487+0000 7fa3797b2640 1 -- 172.21.15.110:0/2790143348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 msgr2=0x7fa3740b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.487+0000 7fa3797b2640 1 --2- 172.21.15.110:0/2790143348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 0x7fa3740b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.488+0000 7fa3797b2640 1 -- 172.21.15.110:0/2790143348 >> v1:172.21.15.134:6789/0 conn(0x7fa3740b17c0 legacy=0x7fa3740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.488+0000 7fa3797b2640 10 monclient: _finish_hunting 0 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.488+0000 7fa3797b2640 1 monclient: found mon.noname-c 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.488+0000 7fa3797b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.488+0000 7fa3797b2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:18.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.488+0000 7fa3797b2640 1 -- 172.21.15.110:0/2790143348 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa374146350 con 0x7fa374145980 2024-01-20T11:17:18.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.493+0000 7fa3797b2640 1 --2- 172.21.15.110:0/2790143348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa374145980 0x7fa374145e10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009e50 tx=0x7fa36c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1a0f6e816e2ca0b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.494+0000 7fa3737fe640 1 -- 172.21.15.110:0/2790143348 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c02ca70 con 0x7fa374145980 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.494+0000 7fa3737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.494+0000 7fa3737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.494+0000 7fa3737fe640 10 monclient: dump: 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:18.495 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:18.496 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:18.496 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:18.496 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:18.496 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:18.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.494+0000 7fa3737fe640 10 monclient: _finish_auth 0 2024-01-20T11:17:18.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.494+0000 7fa3737fe640 10 monclient: _check_auth_tickets 2024-01-20T11:17:18.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.494+0000 7fa3737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:18.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.494+0000 7fa37b215640 5 monclient: authenticate success, global_id 4546 2024-01-20T11:17:18.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.494+0000 7fa37b215640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa3737fe640 1 -- 172.21.15.110:0/2790143348 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c02cbd0 con 0x7fa374145980 2024-01-20T11:17:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa3737fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa3737fe640 1 -- 172.21.15.110:0/2790143348 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c0343f0 con 0x7fa374145980 2024-01-20T11:17:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa3737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa3737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa3737fe640 10 monclient: dump: 2024-01-20T11:17:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:18.504 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:18.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 4 set_mon_vals no callback set 2024-01-20T11:17:18.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 10 monclient: shutdown 2024-01-20T11:17:18.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:18.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 -- 172.21.15.110:0/2790143348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa374145980 msgr2=0x7fa374145e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:18.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 --2- 172.21.15.110:0/2790143348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa374145980 0x7fa374145e10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009e50 tx=0x7fa36c02bed0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 -- 172.21.15.110:0/2790143348 shutdown_connections 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 --2- 172.21.15.110:0/2790143348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa374145980 0x7fa374145e10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 --2- 172.21.15.110:0/2790143348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 0x7fa3740b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 -- 172.21.15.110:0/2790143348 >> 172.21.15.110:0/2790143348 conn(0x7fa374066590 msgr2=0x7fa3740669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 -- 172.21.15.110:0/2790143348 shutdown_connections 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 -- 172.21.15.110:0/2790143348 wait complete. 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 10 monclient: build_initial_monmap 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 10 monclient: monmap: 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 librados: starting msgr at 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 librados: starting objecter 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.497+0000 7fa37b215640 1 Processor -- start 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 1 -- start start 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 1 librados: setting wanted keys 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 1 librados: calling monclient init 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient: init 2024-01-20T11:17:18.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3740b17c0 0x7fa37413e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient(hunting): picked mon.noname-b con 0x7fa3740b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 0x7fa3741415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient(hunting): picked mon.noname-c con 0x7fa3740b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa374145980 0x7fa3741423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient(hunting): picked mon.noname-a con 0x7fa374145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa374146d20 con 0x7fa374145980 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa374136a40 con 0x7fa3740b17c0 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa374141f00 con 0x7fa3740b4d10 2024-01-20T11:17:18.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:18.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.498+0000 7fa37b215640 10 monclient(hunting): authenticate will time out at 1481.849936s 2024-01-20T11:17:18.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.499+0000 7fa378fb1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3740b17c0 0x7fa37413e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:18.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.499+0000 7fa378fb1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3740b17c0 0x7fa37413e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:40242/0 (socket says 172.21.15.110:40242) 2024-01-20T11:17:18.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.499+0000 7fa378fb1640 1 -- 172.21.15.110:0/3263348895 learned_addr learned my addr 172.21.15.110:0/3263348895 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:18.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.499+0000 7fa378fb1640 10 monclient(hunting): get_auth_request con 0x7fa3740b17c0 auth_method 0 2024-01-20T11:17:18.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.499+0000 7fa378fb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:18.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.499+0000 7fa378fb1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:18.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.499+0000 7fa378fb1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:18.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.500+0000 7fa378fb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:18.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.500+0000 7fa378fb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:18.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.500+0000 7fa378fb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:18.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa373fff640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 0x7fa3741415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:18.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 10 monclient(hunting): handle_auth_done global_id 4549 payload 824 2024-01-20T11:17:18.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 1 -- 172.21.15.110:0/3263348895 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 msgr2=0x7fa3741415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:18.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 0x7fa3741415f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 1 -- 172.21.15.110:0/3263348895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa374145980 msgr2=0x7fa3741423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:18.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa374145980 0x7fa3741423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:18.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 10 monclient: _finish_hunting 0 2024-01-20T11:17:18.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 1 monclient: found mon.noname-b 2024-01-20T11:17:18.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:18.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:18.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.502+0000 7fa378fb1640 1 -- 172.21.15.110:0/3263348895 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c009b30 con 0x7fa3740b17c0 2024-01-20T11:17:18.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.506+0000 7fa378fb1640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3740b17c0 0x7fa37413e750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa36800c7f0 tx=0x7fa36800ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:18.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.506+0000 7fa3717fa640 10 client.?.objecter ms_handle_connect 0x7fa3740b17c0 2024-01-20T11:17:18.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.506+0000 7fa3717fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:18.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 1 -- 172.21.15.110:0/3263348895 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368004220 con 0x7fa3740b17c0 2024-01-20T11:17:18.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:18.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:18.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 10 monclient: dump: 2024-01-20T11:17:18.517 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:18.517 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:18.517 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:18.518 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:18.518 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:18.518 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:18.518 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:18.518 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:18.518 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:18.518 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:18.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 10 monclient: _finish_auth 0 2024-01-20T11:17:18.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 10 monclient: _check_auth_tickets 2024-01-20T11:17:18.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:18.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 1 -- 172.21.15.110:0/3263348895 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3680043a0 con 0x7fa3740b17c0 2024-01-20T11:17:18.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:18.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 1 -- 172.21.15.110:0/3263348895 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368007860 con 0x7fa3740b17c0 2024-01-20T11:17:18.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa37b215640 5 monclient: authenticate success, global_id 4549 2024-01-20T11:17:18.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa37b215640 10 monclient: _renew_subs 2024-01-20T11:17:18.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa37b215640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:18.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa374146f20 con 0x7fa3740b17c0 2024-01-20T11:17:18.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa372ffd640 4 set_mon_vals no callback set 2024-01-20T11:17:18.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:18.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:18.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa3717fa640 10 monclient: dump: 2024-01-20T11:17:18.520 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:18.521 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:18.521 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:18.522 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:18.522 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:18.522 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:18.522 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:18.522 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:18.522 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:18.523 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:18.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa37b215640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:18.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa37b215640 10 monclient: _renew_subs 2024-01-20T11:17:18.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa37b215640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:18.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa374142920 con 0x7fa3740b17c0 2024-01-20T11:17:18.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.508+0000 7fa37b215640 1 librados: init done 2024-01-20T11:17:18.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.509+0000 7fa37b215640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:18.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.509+0000 7fa37b215640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:18.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.509+0000 7fa37b215640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:18.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.509+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3741420c0 con 0x7fa3740b17c0 2024-01-20T11:17:18.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.528+0000 7fa3717fa640 1 -- 172.21.15.110:0/3263348895 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa368007bd0 con 0x7fa3740b17c0 2024-01-20T11:17:18.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.529+0000 7fa3717fa640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa35806c860 0x7fa35806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:18.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.529+0000 7fa373fff640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa35806c860 0x7fa35806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:18.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.529+0000 7fa373fff640 10 monclient: get_auth_request con 0x7fa35806c860 auth_method 0 2024-01-20T11:17:18.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.530+0000 7fa3717fa640 10 client.4549.objecter ms_handle_connect 0x7fa35806c860 2024-01-20T11:17:18.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.530+0000 7fa373fff640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa35806c860 0x7fa35806ed20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa3740b3790 tx=0x7fa360009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:18.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.532+0000 7fa3717fa640 1 -- 172.21.15.110:0/3263348895 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa368058d30 con 0x7fa3740b17c0 2024-01-20T11:17:18.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.532+0000 7fa3717fa640 10 client.4549.objecter ms_dispatch 0x7fa374000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:18.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.532+0000 7fa3717fa640 3 client.4549.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:18.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.532+0000 7fa3717fa640 3 client.4549.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:18.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.532+0000 7fa3717fa640 20 client.4549.objecter dump_active .. 0 homeless 2024-01-20T11:17:18.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.776+0000 7fa3717fa640 1 -- 172.21.15.110:0/3263348895 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa368054970 con 0x7fa3740b17c0 2024-01-20T11:17:18.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.776+0000 7fa3717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:18.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:18.776+0000 7fa3717fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:19.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.010+0000 7fa37b215640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:19.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.010+0000 7fa37b215640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:19.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.010+0000 7fa37b215640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:19.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.010+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa374075490 con 0x7fa3740b17c0 2024-01-20T11:17:19.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.016+0000 7fa3717fa640 1 -- 172.21.15.110:0/3263348895 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7fa36801e020 con 0x7fa3740b17c0 2024-01-20T11:17:19.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.016+0000 7fa3717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:19.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.016+0000 7fa3717fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:19.016 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:19.016 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":173,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:19.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 10 librados: watch_flush enter 2024-01-20T11:17:19.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 10 librados: watch_flush exit 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 20 client.4549.objecter shutdown clearing up homeless session... 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 10 client.4549.objecter successfully canceled tick 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa35806c860 msgr2=0x7fa35806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa35806c860 0x7fa35806ed20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa3740b3790 tx=0x7fa360009040 comp rx=0 tx=0).stop 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 10 monclient: shutdown 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3740b17c0 msgr2=0x7fa37413e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3740b17c0 0x7fa37413e750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa36800c7f0 tx=0x7fa36800ccb0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 shutdown_connections 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa35806c860 0x7fa35806ed20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa374145980 0x7fa3741423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3740b4d10 0x7fa3741415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 --2- 172.21.15.110:0/3263348895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3740b17c0 0x7fa37413e750 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 >> 172.21.15.110:0/3263348895 conn(0x7fa374066590 msgr2=0x7fa3740b83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:19.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 shutdown_connections 2024-01-20T11:17:19.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 -- 172.21.15.110:0/3263348895 wait complete. 2024-01-20T11:17:19.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.018+0000 7fa37b215640 1 librados: shutdown 2024-01-20T11:17:19.029 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:19.029 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:19.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.128+0000 7fe12a9a4640 10 monclient: get_monmap_and_config 2024-01-20T11:17:19.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.128+0000 7fe12a9a4640 10 monclient: build_initial_monmap 2024-01-20T11:17:19.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.128+0000 7fe12a9a4640 10 monclient: monmap: 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:19.128846+0000 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:19.128846+0000 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.128+0000 7fe12a9a4640 1 Processor -- start 2024-01-20T11:17:19.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 1 -- start start 2024-01-20T11:17:19.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 10 monclient: init 2024-01-20T11:17:19.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:19.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-01-20T11:17:19.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1240a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:19.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe1240a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:19.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe124142af0 0x7fe12414d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:19.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 10 monclient(hunting): picked mon.noname-c con 0x7fe124142af0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:19.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1240b25c0 con 0x7fe1240a9ad0 2024-01-20T11:17:19.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe124142f50 con 0x7fe1240a96f0 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1240aaea0 con 0x7fe124142af0 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.129+0000 7fe12a9a4640 10 monclient(hunting): authenticate will time out at 1482.479767s 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.130+0000 7fe1237fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe1240a9ad0 0x7fe1241423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:47708/0 (socket says 172.21.15.110:47708) 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.130+0000 7fe1237fe640 1 -- 172.21.15.110:0/1099898513 learned_addr learned my addr 172.21.15.110:0/1099898513 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.130+0000 7fe128f41640 1 --2- 172.21.15.110:0/1099898513 >> v2:172.21.15.134:3300/0 conn(0x7fe124142af0 0x7fe12414d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.130+0000 7fe128f41640 10 monclient(hunting): get_auth_request con 0x7fe124142af0 auth_method 0 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.130+0000 7fe128f41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.130+0000 7fe128f41640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.130+0000 7fe128f41640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.131+0000 7fe128f41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.131+0000 7fe128f41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:19.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.131+0000 7fe128f41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:19.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.132+0000 7fe122ffd640 1 -- 172.21.15.110:0/1099898513 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434472691 0 0) 0x7fe1240b25c0 con 0x7fe1240a9ad0 2024-01-20T11:17:19.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.132+0000 7fe122ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.132+0000 7fe122ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.132+0000 7fe122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.132+0000 7fe122ffd640 10 monclient(hunting): my global_id is 4497 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.132+0000 7fe122ffd640 1 -- 172.21.15.110:0/1099898513 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe10c003520 con 0x7fe1240a9ad0 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.133+0000 7fe128f41640 10 monclient(hunting): handle_auth_done global_id 4552 payload 274 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.133+0000 7fe128f41640 1 -- 172.21.15.110:0/1099898513 >> v1:172.21.15.134:6789/0 conn(0x7fe1240a96f0 legacy=0x7fe12413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.133+0000 7fe128f41640 1 -- 172.21.15.110:0/1099898513 >> v1:172.21.15.110:6789/0 conn(0x7fe1240a9ad0 legacy=0x7fe1241423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.133+0000 7fe128f41640 10 monclient: _finish_hunting 0 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.133+0000 7fe128f41640 1 monclient: found mon.noname-c 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.133+0000 7fe128f41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.133+0000 7fe128f41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:19.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.133+0000 7fe128f41640 1 -- 172.21.15.110:0/1099898513 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12413fbe0 con 0x7fe124142af0 2024-01-20T11:17:19.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.138+0000 7fe128f41640 1 --2- 172.21.15.110:0/1099898513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe124142af0 0x7fe12414d860 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009d90 tx=0x7fe11c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e4f39146d1db6d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 1 -- 172.21.15.110:0/1099898513 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c02cc60 con 0x7fe124142af0 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 10 monclient: dump: 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:19.142 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 1 -- 172.21.15.110:0/1099898513 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c02cdc0 con 0x7fe124142af0 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 5 monclient: authenticate success, global_id 4552 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 4 set_mon_vals no callback set 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 1 -- 172.21.15.110:0/1099898513 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c034660 con 0x7fe124142af0 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe122ffd640 10 monclient: dump: 2024-01-20T11:17:19.143 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:19.144 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:19.144 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:19.144 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:19.144 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:19.144 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:19.144 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:19.144 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:19.144 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:19.144 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:19.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 10 monclient: shutdown 2024-01-20T11:17:19.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:19.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 1 -- 172.21.15.110:0/1099898513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe124142af0 msgr2=0x7fe12414d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:19.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 1 --2- 172.21.15.110:0/1099898513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe124142af0 0x7fe12414d860 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009d90 tx=0x7fe11c02beb0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 1 -- 172.21.15.110:0/1099898513 shutdown_connections 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 1 --2- 172.21.15.110:0/1099898513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe124142af0 0x7fe12414d860 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 1 -- 172.21.15.110:0/1099898513 >> 172.21.15.110:0/1099898513 conn(0x7fe1240b7bc0 msgr2=0x7fe1240b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 1 -- 172.21.15.110:0/1099898513 shutdown_connections 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.142+0000 7fe12a9a4640 1 -- 172.21.15.110:0/1099898513 wait complete. 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.143+0000 7fe12a9a4640 10 monclient: build_initial_monmap 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.143+0000 7fe12a9a4640 10 monclient: monmap: 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:19.148 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.143+0000 7fe12a9a4640 1 librados: starting msgr at 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.143+0000 7fe12a9a4640 1 librados: starting objecter 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.143+0000 7fe12a9a4640 1 Processor -- start 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 1 -- start start 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 1 librados: setting wanted keys 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 1 librados: calling monclient init 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient: init 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1240a96f0 0x7fe124139e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:19.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient(hunting): picked mon.noname-c con 0x7fe1240a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1240a9ad0 0x7fe124136b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient(hunting): picked mon.noname-b con 0x7fe1240a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe124142af0 0x7fe1241370d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient(hunting): picked mon.noname-a con 0x7fe124142af0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12413b3e0 con 0x7fe124142af0 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12413b6c0 con 0x7fe1240a9ad0 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12413b9a0 con 0x7fe1240a96f0 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.144+0000 7fe12a9a4640 10 monclient(hunting): authenticate will time out at 1482.494904s 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe128f41640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe124142af0 0x7fe1241370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe123fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1240a96f0 0x7fe124139e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:19.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe1237fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1240a9ad0 0x7fe124136b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe128f41640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe124142af0 0x7fe1241370d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34400/0 (socket says 172.21.15.110:34400) 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe128f41640 1 -- 172.21.15.110:0/3214256989 learned_addr learned my addr 172.21.15.110:0/3214256989 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe128f41640 10 monclient(hunting): get_auth_request con 0x7fe124142af0 auth_method 0 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe128f41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe128f41640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe128f41640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe1237fe640 10 monclient(hunting): get_auth_request con 0x7fe1240a9ad0 auth_method 0 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe1237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe1237fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe1237fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe123fff640 10 monclient(hunting): get_auth_request con 0x7fe1240a96f0 auth_method 0 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe123fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe123fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:19.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.145+0000 7fe123fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:19.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.146+0000 7fe1237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:19.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.146+0000 7fe1237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:19.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.146+0000 7fe1237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:19.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.146+0000 7fe128f41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:19.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.146+0000 7fe128f41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:19.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.146+0000 7fe128f41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:19.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.146+0000 7fe123fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:19.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.146+0000 7fe123fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:19.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.146+0000 7fe123fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:19.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 10 monclient(hunting): handle_auth_done global_id 4500 payload 824 2024-01-20T11:17:19.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 1 -- 172.21.15.110:0/3214256989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1240a96f0 msgr2=0x7fe124139e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:19.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1240a96f0 0x7fe124139e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 1 -- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1240a9ad0 msgr2=0x7fe124136b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:19.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1240a9ad0 0x7fe124136b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 10 monclient: _finish_hunting 0 2024-01-20T11:17:19.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 1 monclient: found mon.noname-a 2024-01-20T11:17:19.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:19.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:19.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe128f41640 1 -- 172.21.15.110:0/3214256989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c009a70 con 0x7fe124142af0 2024-01-20T11:17:19.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.149+0000 7fe1237fe640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1240a9ad0 0x7fe124136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:19.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.151+0000 7fe128f41640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe124142af0 0x7fe1241370d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe11c02c4c0 tx=0x7fe11c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:19.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.151+0000 7fe120ff9640 10 client.?.objecter ms_handle_connect 0x7fe124142af0 2024-01-20T11:17:19.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.151+0000 7fe120ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:19.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe120ff9640 1 -- 172.21.15.110:0/3214256989 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c03e610 con 0x7fe124142af0 2024-01-20T11:17:19.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe120ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:19.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe120ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:19.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe120ff9640 10 monclient: dump: 2024-01-20T11:17:19.155 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:19.155 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:19.155 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:19.155 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe120ff9640 10 monclient: _finish_auth 0 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe120ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe120ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe12a9a4640 5 monclient: authenticate success, global_id 4500 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe12a9a4640 10 monclient: _renew_subs 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe12a9a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.154+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12413a530 con 0x7fe124142af0 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.155+0000 7fe12a9a4640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:19.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.155+0000 7fe12a9a4640 10 monclient: _renew_subs 2024-01-20T11:17:19.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.155+0000 7fe12a9a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:19.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.155+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1241394b0 con 0x7fe124142af0 2024-01-20T11:17:19.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.155+0000 7fe12a9a4640 1 librados: init done 2024-01-20T11:17:19.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.156+0000 7fe12a9a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:19.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.156+0000 7fe12a9a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.156+0000 7fe12a9a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.156+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12414dd20 con 0x7fe124142af0 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.157+0000 7fe120ff9640 1 -- 172.21.15.110:0/3214256989 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c03e770 con 0x7fe124142af0 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.157+0000 7fe120ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.157+0000 7fe120ff9640 1 -- 172.21.15.110:0/3214256989 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c03eae0 con 0x7fe124142af0 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.157+0000 7fe120ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.157+0000 7fe120ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.157+0000 7fe120ff9640 10 monclient: dump: 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:19.158 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:19.159 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:19.159 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:19.159 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:19.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.157+0000 7fe121ffb640 4 set_mon_vals no callback set 2024-01-20T11:17:19.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.402+0000 7fe120ff9640 1 -- 172.21.15.110:0/3214256989 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe11c03b050 con 0x7fe124142af0 2024-01-20T11:17:19.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.402+0000 7fe120ff9640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe11006c860 0x7fe11006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:19.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.403+0000 7fe123fff640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe11006c860 0x7fe11006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:19.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.403+0000 7fe123fff640 10 monclient: get_auth_request con 0x7fe11006c860 auth_method 0 2024-01-20T11:17:19.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.403+0000 7fe120ff9640 10 client.4500.objecter ms_handle_connect 0x7fe11006c860 2024-01-20T11:17:19.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.403+0000 7fe123fff640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe11006c860 0x7fe11006ed20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe118001780 tx=0x7fe118009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:19.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.417+0000 7fe120ff9640 1 -- 172.21.15.110:0/3214256989 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe11c07ab20 con 0x7fe124142af0 2024-01-20T11:17:19.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.417+0000 7fe120ff9640 10 client.4500.objecter ms_dispatch 0x7fe124000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:19.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.417+0000 7fe120ff9640 3 client.4500.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:19.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.417+0000 7fe120ff9640 3 client.4500.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:19.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.417+0000 7fe120ff9640 20 client.4500.objecter dump_active .. 0 homeless 2024-01-20T11:17:19.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.417+0000 7fe120ff9640 1 -- 172.21.15.110:0/3214256989 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe11c076e60 con 0x7fe124142af0 2024-01-20T11:17:19.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.417+0000 7fe120ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:19.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.417+0000 7fe120ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:19.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.619+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1240b4400 con 0x7fe11006c860 2024-01-20T11:17:19.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.622+0000 7fe120ff9640 1 -- 172.21.15.110:0/3214256989 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64372 (secure 0 0 0) 0x7fe118001420 con 0x7fe11006c860 2024-01-20T11:17:19.622 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:19.623 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 10 librados: watch_flush enter 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 10 librados: watch_flush exit 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 20 client.4500.objecter shutdown clearing up homeless session... 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 10 client.4500.objecter successfully canceled tick 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe11006c860 msgr2=0x7fe11006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe11006c860 0x7fe11006ed20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe118001780 tx=0x7fe118009040 comp rx=0 tx=0).stop 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 10 monclient: shutdown 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe124142af0 msgr2=0x7fe1241370d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe124142af0 0x7fe1241370d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe11c02c4c0 tx=0x7fe11c004a40 comp rx=0 tx=0).stop 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 shutdown_connections 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe11006c860 0x7fe11006ed20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe124142af0 0x7fe1241370d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1240a9ad0 0x7fe124136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 --2- 172.21.15.110:0/3214256989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1240a96f0 0x7fe124139e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:19.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 >> 172.21.15.110:0/3214256989 conn(0x7fe1240b7bc0 msgr2=0x7fe12413f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:19.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 shutdown_connections 2024-01-20T11:17:19.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 -- 172.21.15.110:0/3214256989 wait complete. 2024-01-20T11:17:19.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:19.624+0000 7fe12a9a4640 1 librados: shutdown 2024-01-20T11:17:19.634 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-01-20T11:17:18.735381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001848"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1929999999999996},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.6449999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.9960000000000004},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.061999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":15.018000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.739000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.563000000000001},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":14.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":17.559000000000001},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":13.294}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.603999999999999},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":18.77}]}]},{"osd":7,"up_from":21,"seq":90194313245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.989000000000001},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.0519999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.5350000000000001},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":15.191000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":17.376999999999999},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.6470000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.42},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":19.763000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.927},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":16.513000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.113},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":18.716000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313246,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.0899999999999999},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.1260000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.855},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.189}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":7.9779999999999998},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.493}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.143000000000001},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.426}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":8.7029999999999994},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":12.089}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":15.231999999999999},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.913}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444060,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.9050000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.6429999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.446999999999999},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.7690000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":17.277999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.34}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":16.474},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.085000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.694000000000001},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.550000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.186999999999999},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":13.983000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444061,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.3159999999999998},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":6.6210000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.744},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.298}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":12.073},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.1679999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":13.319000000000001},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.411999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.667},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":13.972}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":17.141999999999999},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.576000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444061,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":7.758},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.1539999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.3650000000000002},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.393000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":15.411},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.318}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.907999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":19.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":17.108000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":14.704000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":18.379999999999999},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":12.622}]}]},{"osd":6,"up_from":15,"seq":64424509469,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":10.471},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.718}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.0429999999999993},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.394}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.843},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3819999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.036},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.666}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.786999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.333},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.885999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:22.637 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:22.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.742+0000 7f2b937e9640 10 monclient: get_monmap_and_config 2024-01-20T11:17:22.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.742+0000 7f2b937e9640 10 monclient: build_initial_monmap 2024-01-20T11:17:22.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.742+0000 7f2b937e9640 10 monclient: monmap: 2024-01-20T11:17:22.742 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:22.742 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:22.742 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:22.742624+0000 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:22.742624+0000 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.742+0000 7f2b937e9640 1 Processor -- start 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.742+0000 7f2b937e9640 1 -- start start 2024-01-20T11:17:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.742+0000 7f2b937e9640 10 monclient: init 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2b8c0a96f0 0x7f2b8c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 10 monclient(hunting): picked mon.noname-a con 0x7f2b8c0a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c13fa10 0x7f2b8c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 10 monclient(hunting): picked mon.noname-b con 0x7f2b8c13fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b8c0a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8c0b25c0 con 0x7f2b8c0a9ad0 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8c0aaea0 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0ab0a0 con 0x7f2b8c13fa10 2024-01-20T11:17:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b937e9640 10 monclient(hunting): authenticate will time out at 1486.093609s 2024-01-20T11:17:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.743+0000 7f2b90d84640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c13fa10 0x7f2b8c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.744+0000 7f2b91d86640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2b8c0a9ad0 0x7f2b8c142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:53996/0 (socket says 172.21.15.110:53996) 2024-01-20T11:17:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.744+0000 7f2b91d86640 1 -- 172.21.15.110:0/2546943730 learned_addr learned my addr 172.21.15.110:0/2546943730 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.744+0000 7f2b90d84640 10 monclient(hunting): get_auth_request con 0x7f2b8c13fa10 auth_method 0 2024-01-20T11:17:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.745+0000 7f2b90d84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.745+0000 7f2b90d84640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.745+0000 7f2b90d84640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.746+0000 7f2b90d84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.746+0000 7f2b90d84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.746+0000 7f2b90d84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.746+0000 7f2b83fff640 1 -- 172.21.15.110:0/2546943730 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197255699 0 0) 0x7f2b8c0b25c0 con 0x7f2b8c0a9ad0 2024-01-20T11:17:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.746+0000 7f2b83fff640 10 monclient(hunting): _negotiate 2024-01-20T11:17:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.746+0000 7f2b83fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.746+0000 7f2b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.746+0000 7f2b83fff640 10 monclient(hunting): my global_id is 4561 2024-01-20T11:17:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.746+0000 7f2b83fff640 1 -- 172.21.15.110:0/2546943730 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b78003520 con 0x7f2b8c0a9ad0 2024-01-20T11:17:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.748+0000 7f2b90d84640 10 monclient(hunting): handle_auth_done global_id 4400 payload 274 2024-01-20T11:17:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.748+0000 7f2b90d84640 1 -- 172.21.15.110:0/2546943730 >> v2:172.21.15.110:3300/0 conn(0x7f2b8c0a96f0 msgr2=0x7f2b8c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.748+0000 7f2b90d84640 1 --2- 172.21.15.110:0/2546943730 >> v2:172.21.15.110:3300/0 conn(0x7f2b8c0a96f0 0x7f2b8c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.748+0000 7f2b90d84640 1 -- 172.21.15.110:0/2546943730 >> v1:172.21.15.134:6789/0 conn(0x7f2b8c0a9ad0 legacy=0x7f2b8c142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.748+0000 7f2b90d84640 10 monclient: _finish_hunting 0 2024-01-20T11:17:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.748+0000 7f2b90d84640 1 monclient: found mon.noname-b 2024-01-20T11:17:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.748+0000 7f2b90d84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.748+0000 7f2b90d84640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.748+0000 7f2b90d84640 1 -- 172.21.15.110:0/2546943730 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c0b3fb0 con 0x7f2b8c13fa10 2024-01-20T11:17:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.750+0000 7f2b90d84640 1 --2- 172.21.15.110:0/2546943730 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c13fa10 0x7f2b8c1423e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0097a0 tx=0x7f2b7c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d984e937e3c681cc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.752+0000 7f2b83fff640 1 -- 172.21.15.110:0/2546943730 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c02e070 con 0x7f2b8c13fa10 2024-01-20T11:17:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.752+0000 7f2b83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.752+0000 7f2b83fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.752+0000 7f2b83fff640 10 monclient: dump: 2024-01-20T11:17:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.752+0000 7f2b83fff640 10 monclient: _finish_auth 0 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.752+0000 7f2b83fff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.752+0000 7f2b83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.752+0000 7f2b937e9640 5 monclient: authenticate success, global_id 4400 2024-01-20T11:17:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.752+0000 7f2b937e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.754+0000 7f2b83fff640 1 -- 172.21.15.110:0/2546943730 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c0315a0 con 0x7f2b8c13fa10 2024-01-20T11:17:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.754+0000 7f2b83fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.754+0000 7f2b83fff640 1 -- 172.21.15.110:0/2546943730 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c031910 con 0x7f2b8c13fa10 2024-01-20T11:17:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.754+0000 7f2b937e9640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.754+0000 7f2b937e9640 4 set_mon_vals no callback set 2024-01-20T11:17:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.754+0000 7f2b83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.754+0000 7f2b83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.754+0000 7f2b83fff640 10 monclient: dump: 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:22.755 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 10 monclient: shutdown 2024-01-20T11:17:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 1 -- 172.21.15.110:0/2546943730 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c13fa10 msgr2=0x7f2b8c1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 1 --2- 172.21.15.110:0/2546943730 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c13fa10 0x7f2b8c1423e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0097a0 tx=0x7f2b7c02b840 comp rx=0 tx=0).stop 2024-01-20T11:17:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 1 -- 172.21.15.110:0/2546943730 shutdown_connections 2024-01-20T11:17:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 1 --2- 172.21.15.110:0/2546943730 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c13fa10 0x7f2b8c1423e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 1 --2- 172.21.15.110:0/2546943730 >> v2:172.21.15.110:3300/0 conn(0x7f2b8c0a96f0 0x7f2b8c13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 1 -- 172.21.15.110:0/2546943730 >> 172.21.15.110:0/2546943730 conn(0x7f2b8c0b7bc0 msgr2=0x7f2b8c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 1 -- 172.21.15.110:0/2546943730 shutdown_connections 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.755+0000 7f2b937e9640 1 -- 172.21.15.110:0/2546943730 wait complete. 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.756+0000 7f2b937e9640 10 monclient: build_initial_monmap 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.756+0000 7f2b937e9640 10 monclient: monmap: 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.756+0000 7f2b937e9640 1 librados: starting msgr at 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.756+0000 7f2b937e9640 1 librados: starting objecter 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.756+0000 7f2b937e9640 1 Processor -- start 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.756+0000 7f2b937e9640 1 -- start start 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.756+0000 7f2b937e9640 1 librados: setting wanted keys 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.756+0000 7f2b937e9640 1 librados: calling monclient init 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.756+0000 7f2b937e9640 10 monclient: init 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c0a96f0 0x7f2b8c134a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient(hunting): picked mon.noname-b con 0x7f2b8c0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c0a9ad0 0x7f2b8c135740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient(hunting): picked mon.noname-c con 0x7f2b8c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c135c80 0x7f2b8c1d73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient(hunting): picked mon.noname-a con 0x7f2b8c135c80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c1340b0 con 0x7f2b8c135c80 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c1d6ee0 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c1403c0 con 0x7f2b8c0a9ad0 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b937e9640 10 monclient(hunting): authenticate will time out at 1486.107658s 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.757+0000 7f2b91585640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c0a96f0 0x7f2b8c134a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.758+0000 7f2b91585640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c0a96f0 0x7f2b8c134a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:53110/0 (socket says 172.21.15.110:53110) 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.758+0000 7f2b91585640 1 -- 172.21.15.110:0/3119120676 learned_addr learned my addr 172.21.15.110:0/3119120676 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.758+0000 7f2b91585640 10 monclient(hunting): get_auth_request con 0x7f2b8c0a96f0 auth_method 0 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.758+0000 7f2b91585640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.758+0000 7f2b91585640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.758+0000 7f2b91585640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:22.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.758+0000 7f2b91585640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:22.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.758+0000 7f2b91585640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:22.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.758+0000 7f2b91585640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.759+0000 7f2b91d86640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c135c80 0x7f2b8c1d73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b90d84640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c0a9ad0 0x7f2b8c135740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b91d86640 10 monclient(hunting): get_auth_request con 0x7f2b8c135c80 auth_method 0 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b91d86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b91d86640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b91d86640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b90d84640 10 monclient(hunting): get_auth_request con 0x7f2b8c0a9ad0 auth_method 0 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b90d84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b90d84640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b90d84640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b91d86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b91d86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.760+0000 7f2b91d86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 10 monclient(hunting): handle_auth_done global_id 4564 payload 824 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 1 -- 172.21.15.110:0/3119120676 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c0a9ad0 msgr2=0x7f2b8c135740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c0a9ad0 0x7f2b8c135740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 1 -- 172.21.15.110:0/3119120676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c135c80 msgr2=0x7f2b8c1d73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c135c80 0x7f2b8c1d73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 10 monclient: _finish_hunting 0 2024-01-20T11:17:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 1 monclient: found mon.noname-b 2024-01-20T11:17:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.761+0000 7f2b91585640 1 -- 172.21.15.110:0/3119120676 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c009400 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.763+0000 7f2b91585640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c0a96f0 0x7f2b8c134a30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800b4a0 tx=0x7f2b8800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.763+0000 7f2b81ffb640 10 client.?.objecter ms_handle_connect 0x7f2b8c0a96f0 2024-01-20T11:17:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.763+0000 7f2b81ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.766+0000 7f2b81ffb640 1 -- 172.21.15.110:0/3119120676 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88010610 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.766+0000 7f2b81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.766+0000 7f2b81ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.766+0000 7f2b81ffb640 10 monclient: dump: 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b81ffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b81ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b937e9640 5 monclient: authenticate success, global_id 4564 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b81ffb640 1 -- 172.21.15.110:0/3119120676 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b88010770 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b81ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b937e9640 10 monclient: _renew_subs 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b81ffb640 1 -- 172.21.15.110:0/3119120676 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b88010b20 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b937e9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c1d7900 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b837fe640 4 set_mon_vals no callback set 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b81ffb640 10 monclient: dump: 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b937e9640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:22.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b937e9640 10 monclient: _renew_subs 2024-01-20T11:17:22.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b937e9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:22.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8c1da2b0 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.767+0000 7f2b937e9640 1 librados: init done 2024-01-20T11:17:22.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.768+0000 7f2b937e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:22.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.768+0000 7f2b937e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:22.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.768+0000 7f2b937e9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:22.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.768+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c1d70a0 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.786+0000 7f2b81ffb640 1 -- 172.21.15.110:0/3119120676 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2b88010c80 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.786+0000 7f2b81ffb640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b7006c8b0 0x7f2b7006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:22.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.786+0000 7f2b90d84640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b7006c8b0 0x7f2b7006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:22.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.786+0000 7f2b90d84640 10 monclient: get_auth_request con 0x7f2b7006c8b0 auth_method 0 2024-01-20T11:17:22.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.787+0000 7f2b81ffb640 10 client.4564.objecter ms_handle_connect 0x7f2b7006c8b0 2024-01-20T11:17:22.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.787+0000 7f2b90d84640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b7006c8b0 0x7f2b7006ed70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c036000 tx=0x7f2b7c008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:22.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.788+0000 7f2b81ffb640 1 -- 172.21.15.110:0/3119120676 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2b88058210 con 0x7f2b8c0a96f0 2024-01-20T11:17:22.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.788+0000 7f2b81ffb640 10 client.4564.objecter ms_dispatch 0x7f2b8c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:22.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.788+0000 7f2b81ffb640 3 client.4564.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:22.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.788+0000 7f2b81ffb640 3 client.4564.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:22.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:22.788+0000 7f2b81ffb640 20 client.4564.objecter dump_active .. 0 homeless 2024-01-20T11:17:23.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.028+0000 7f2b81ffb640 1 -- 172.21.15.110:0/3119120676 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2b88054550 con 0x7f2b8c0a96f0 2024-01-20T11:17:23.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.028+0000 7f2b81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:23.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.028+0000 7f2b81ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:23.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.200+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b8c136090 con 0x7f2b7006c8b0 2024-01-20T11:17:23.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.203+0000 7f2b81ffb640 1 -- 172.21.15.110:0/3119120676 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64331 (secure 0 0 0) 0x7f2b7c0044a0 con 0x7f2b7006c8b0 2024-01-20T11:17:23.203 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:23.204 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 10 librados: watch_flush enter 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 10 librados: watch_flush exit 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 20 client.4564.objecter shutdown clearing up homeless session... 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 10 client.4564.objecter successfully canceled tick 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b7006c8b0 msgr2=0x7f2b7006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b7006c8b0 0x7f2b7006ed70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c036000 tx=0x7f2b7c008030 comp rx=0 tx=0).stop 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 10 monclient: shutdown 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c0a96f0 msgr2=0x7f2b8c134a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c0a96f0 0x7f2b8c134a30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800b4a0 tx=0x7f2b8800b960 comp rx=0 tx=0).stop 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 shutdown_connections 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b7006c8b0 0x7f2b7006ed70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c135c80 0x7f2b8c1d73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8c0a9ad0 0x7f2b8c135740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 --2- 172.21.15.110:0/3119120676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c0a96f0 0x7f2b8c134a30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 >> 172.21.15.110:0/3119120676 conn(0x7f2b8c0b7bc0 msgr2=0x7f2b8c1372a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:23.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 shutdown_connections 2024-01-20T11:17:23.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 -- 172.21.15.110:0/3119120676 wait complete. 2024-01-20T11:17:23.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.205+0000 7f2b937e9640 1 librados: shutdown 2024-01-20T11:17:23.216 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-01-20T11:17:22.736074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001909"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1929999999999996},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.6449999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.9960000000000004},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.061999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":15.018000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.739000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.563000000000001},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":14.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":17.559000000000001},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":13.294}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.603999999999999},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":18.77}]}]},{"osd":7,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.6029999999999998},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4949999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.25},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.6099999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.23},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":12.256}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.090999999999999},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.863}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.760999999999999},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":17.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.984},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":18.516999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313247,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.399},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.7910000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.1449999999999996},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.4210000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":15.436999999999999},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.507999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.18},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.144}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":14.81},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":10.906000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":13.525},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":10.081}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444060,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.9050000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.6429999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.446999999999999},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.7690000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":17.277999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.34}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":16.474},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.085000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.694000000000001},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.550000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.186999999999999},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":13.983000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444061,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.3159999999999998},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":6.6210000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.744},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.298}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":12.073},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.1679999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":13.319000000000001},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.411999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.667},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":13.972}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":17.141999999999999},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.576000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444061,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":7.758},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.1539999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.3650000000000002},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.393000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":15.411},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.318}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.907999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":19.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":17.108000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":14.704000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":18.379999999999999},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":12.622}]}]},{"osd":6,"up_from":15,"seq":64424509470,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":10.471},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.718}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.0429999999999993},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.394}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.843},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3819999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.036},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.666}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.786999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.333},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.885999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:23.217 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:23.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.320+0000 7f72b0ff5640 10 monclient: get_monmap_and_config 2024-01-20T11:17:23.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.320+0000 7f72b0ff5640 10 monclient: build_initial_monmap 2024-01-20T11:17:23.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.320+0000 7f72b0ff5640 10 monclient: monmap: 2024-01-20T11:17:23.320 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:23.320841+0000 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:23.320841+0000 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.320+0000 7f72b0ff5640 1 Processor -- start 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 1 -- start start 2024-01-20T11:17:23.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient: init 2024-01-20T11:17:23.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:23.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-01-20T11:17:23.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72ac0a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:23.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac0a9ad0 0x7f72ac1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient(hunting): picked mon.noname-b con 0x7f72ac0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:23.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f72ac142920 0x7f72ac14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient(hunting): picked mon.noname-a con 0x7f72ac142920 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:23.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0b25c0 con 0x7f72ac0a96f0 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ac0aaea0 con 0x7f72ac142920 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac13fbe0 con 0x7f72ac0a9ad0 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.321+0000 7f72b0ff5640 10 monclient(hunting): authenticate will time out at 1486.671773s 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.322+0000 7f72aad9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f72ac142920 0x7f72ac14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.322+0000 7f72aa59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f72ac0a96f0 0x7f72ac13f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:57684/0 (socket says 172.21.15.110:57684) 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.322+0000 7f72aa59c640 1 -- 172.21.15.110:0/941961158 learned_addr learned my addr 172.21.15.110:0/941961158 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.322+0000 7f72aad9d640 10 monclient(hunting): get_auth_request con 0x7f72ac142920 auth_method 0 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.322+0000 7f72aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.322+0000 7f72aad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.322+0000 7f72aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:23.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.323+0000 7f72aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:23.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.323+0000 7f72aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:23.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.323+0000 7f72aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:23.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.327+0000 7f72aad9d640 10 monclient(hunting): handle_auth_done global_id 4506 payload 274 2024-01-20T11:17:23.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.327+0000 7f72aad9d640 1 -- 172.21.15.110:0/941961158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac0a9ad0 msgr2=0x7f72ac1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:23.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.327+0000 7f72aad9d640 1 --2- 172.21.15.110:0/941961158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac0a9ad0 0x7f72ac1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.327+0000 7f72aad9d640 1 -- 172.21.15.110:0/941961158 >> v1:172.21.15.110:6789/0 conn(0x7f72ac0a96f0 legacy=0x7f72ac13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.327+0000 7f72aad9d640 10 monclient: _finish_hunting 0 2024-01-20T11:17:23.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.327+0000 7f72aad9d640 1 monclient: found mon.noname-a 2024-01-20T11:17:23.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.327+0000 7f72aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:23.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.327+0000 7f72aad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:23.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.327+0000 7f72aad9d640 1 -- 172.21.15.110:0/941961158 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac0ab0a0 con 0x7f72ac142920 2024-01-20T11:17:23.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.331+0000 7f72aad9d640 1 --2- 172.21.15.110:0/941961158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72ac142920 0x7f72ac14c850 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009d90 tx=0x7f72a402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ad9bc6a6e9568e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:23.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 1 -- 172.21.15.110:0/941961158 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a402cbc0 con 0x7f72ac142920 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 10 monclient: dump: 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:23.336 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 10 monclient: _finish_auth 0 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 1 -- 172.21.15.110:0/941961158 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a402cd20 con 0x7f72ac142920 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72b0ff5640 5 monclient: authenticate success, global_id 4506 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72b0ff5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.335+0000 7f72a9d9b640 1 -- 172.21.15.110:0/941961158 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4034660 con 0x7f72ac142920 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 4 set_mon_vals no callback set 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:23.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72a9d9b640 10 monclient: dump: 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:23.338 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 10 monclient: shutdown 2024-01-20T11:17:23.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 1 -- 172.21.15.110:0/941961158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72ac142920 msgr2=0x7f72ac14c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 1 --2- 172.21.15.110:0/941961158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72ac142920 0x7f72ac14c850 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009d90 tx=0x7f72a402be10 comp rx=0 tx=0).stop 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 1 -- 172.21.15.110:0/941961158 shutdown_connections 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 1 --2- 172.21.15.110:0/941961158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72ac142920 0x7f72ac14c850 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 1 --2- 172.21.15.110:0/941961158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac0a9ad0 0x7f72ac1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 1 -- 172.21.15.110:0/941961158 >> 172.21.15.110:0/941961158 conn(0x7f72ac0b7bc0 msgr2=0x7f72ac0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 1 -- 172.21.15.110:0/941961158 shutdown_connections 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.336+0000 7f72b0ff5640 1 -- 172.21.15.110:0/941961158 wait complete. 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.337+0000 7f72b0ff5640 10 monclient: build_initial_monmap 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.337+0000 7f72b0ff5640 10 monclient: monmap: 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:23.342 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.337+0000 7f72b0ff5640 1 librados: starting msgr at 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.337+0000 7f72b0ff5640 1 librados: starting objecter 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.337+0000 7f72b0ff5640 1 Processor -- start 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 1 -- start start 2024-01-20T11:17:23.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 1 librados: setting wanted keys 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 1 librados: calling monclient init 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient: init 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72ac0a96f0 0x7f72ac1dabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient(hunting): picked mon.noname-b con 0x7f72ac0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72ac0a9ad0 0x7f72ac0af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient(hunting): picked mon.noname-a con 0x7f72ac0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac142920 0x7f72ac0afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient(hunting): picked mon.noname-c con 0x7f72ac142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac1db700 con 0x7f72ac0a9ad0 2024-01-20T11:17:23.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac11fb20 con 0x7f72ac0a96f0 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac0b0410 con 0x7f72ac142920 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.338+0000 7f72b0ff5640 10 monclient(hunting): authenticate will time out at 1486.688807s 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac142920 0x7f72ac0afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aa59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72ac0a96f0 0x7f72ac1dabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72a3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72ac0a9ad0 0x7f72ac0af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac142920 0x7f72ac0afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46790/0 (socket says 172.21.15.110:46790) 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aad9d640 1 -- 172.21.15.110:0/1090427759 learned_addr learned my addr 172.21.15.110:0/1090427759 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aad9d640 10 monclient(hunting): get_auth_request con 0x7f72ac142920 auth_method 0 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:23.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72a3fff640 10 monclient(hunting): get_auth_request con 0x7f72ac0a9ad0 auth_method 0 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72a3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aa59c640 10 monclient(hunting): get_auth_request con 0x7f72ac0a96f0 auth_method 0 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aa59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.339+0000 7f72aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.340+0000 7f72aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.340+0000 7f72aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.340+0000 7f72aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.340+0000 7f72a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.340+0000 7f72a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.340+0000 7f72a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:23.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.340+0000 7f72aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:23.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.340+0000 7f72aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:23.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.340+0000 7f72aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:23.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 10 monclient(hunting): handle_auth_done global_id 4567 payload 824 2024-01-20T11:17:23.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 1 -- 172.21.15.110:0/1090427759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac142920 msgr2=0x7f72ac0afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac142920 0x7f72ac0afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 1 -- 172.21.15.110:0/1090427759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72ac0a9ad0 msgr2=0x7f72ac0af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72ac0a9ad0 0x7f72ac0af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 10 monclient: _finish_hunting 0 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 1 monclient: found mon.noname-b 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.342+0000 7f72aa59c640 1 -- 172.21.15.110:0/1090427759 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4009a70 con 0x7f72ac0a96f0 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.344+0000 7f72aa59c640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72ac0a96f0 0x7f72ac1dabf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f729c000f60 tx=0x7f729c00c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.344+0000 7f72a2ffd640 10 client.?.objecter ms_handle_connect 0x7f72ac0a96f0 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.344+0000 7f72a2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 1 -- 172.21.15.110:0/1090427759 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c005ce0 con 0x7f72ac0a96f0 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:23.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 10 monclient: dump: 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:23.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72b0ff5640 5 monclient: authenticate success, global_id 4567 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 1 -- 172.21.15.110:0/1090427759 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729c005e40 con 0x7f72ac0a96f0 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72b0ff5640 10 monclient: _renew_subs 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72b0ff5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac1db130 con 0x7f72ac0a96f0 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 1 -- 172.21.15.110:0/1090427759 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c007310 con 0x7f72ac0a96f0 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a959a640 4 set_mon_vals no callback set 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72a2ffd640 10 monclient: dump: 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.351 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72b0ff5640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72b0ff5640 10 monclient: _renew_subs 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72b0ff5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ac1393e0 con 0x7f72ac0a96f0 2024-01-20T11:17:23.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.348+0000 7f72b0ff5640 1 librados: init done 2024-01-20T11:17:23.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.349+0000 7f72b0ff5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:23.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.349+0000 7f72b0ff5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:23.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.349+0000 7f72b0ff5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:23.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.349+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ac1db870 con 0x7f72ac0a96f0 2024-01-20T11:17:23.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.367+0000 7f72a2ffd640 1 -- 172.21.15.110:0/1090427759 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f729c007470 con 0x7f72ac0a96f0 2024-01-20T11:17:23.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.367+0000 7f72a2ffd640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f728c06c860 0x7f728c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.368+0000 7f72a3fff640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f728c06c860 0x7f728c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.368+0000 7f72a3fff640 10 monclient: get_auth_request con 0x7f728c06c860 auth_method 0 2024-01-20T11:17:23.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.368+0000 7f72a2ffd640 10 client.4567.objecter ms_handle_connect 0x7f728c06c860 2024-01-20T11:17:23.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.368+0000 7f72a3fff640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f728c06c860 0x7f728c06ed20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0b8460 tx=0x7f7294009110 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:23.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.370+0000 7f72a2ffd640 1 -- 172.21.15.110:0/1090427759 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f729c007720 con 0x7f72ac0a96f0 2024-01-20T11:17:23.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.370+0000 7f72a2ffd640 10 client.4567.objecter ms_dispatch 0x7f72ac000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:23.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.370+0000 7f72a2ffd640 3 client.4567.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:23.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.370+0000 7f72a2ffd640 3 client.4567.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:23.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.370+0000 7f72a2ffd640 20 client.4567.objecter dump_active .. 0 homeless 2024-01-20T11:17:23.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.602+0000 7f72a2ffd640 1 -- 172.21.15.110:0/1090427759 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f729c053e40 con 0x7f72ac0a96f0 2024-01-20T11:17:23.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.602+0000 7f72a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:23.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.602+0000 7f72a2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:23.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.773+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72ac0aa510 con 0x7f728c06c860 2024-01-20T11:17:23.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.778+0000 7f72a2ffd640 1 -- 172.21.15.110:0/1090427759 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64331 (secure 0 0 0) 0x7f72ac0aa510 con 0x7f728c06c860 2024-01-20T11:17:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:23.780 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:23.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 10 librados: watch_flush enter 2024-01-20T11:17:23.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 10 librados: watch_flush exit 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 20 client.4567.objecter shutdown clearing up homeless session... 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 10 client.4567.objecter successfully canceled tick 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f728c06c860 msgr2=0x7f728c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f728c06c860 0x7f728c06ed20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0b8460 tx=0x7f7294009110 comp rx=0 tx=0).stop 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 10 monclient: shutdown 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72ac0a96f0 msgr2=0x7f72ac1dabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72ac0a96f0 0x7f72ac1dabf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f729c000f60 tx=0x7f729c00c740 comp rx=0 tx=0).stop 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 shutdown_connections 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f728c06c860 0x7f728c06ed20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72ac142920 0x7f72ac0afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72ac0a9ad0 0x7f72ac0af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 --2- 172.21.15.110:0/1090427759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72ac0a96f0 0x7f72ac1dabf0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 >> 172.21.15.110:0/1090427759 conn(0x7f72ac0b7bc0 msgr2=0x7f72ac13b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:23.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 shutdown_connections 2024-01-20T11:17:23.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 -- 172.21.15.110:0/1090427759 wait complete. 2024-01-20T11:17:23.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.780+0000 7f72b0ff5640 1 librados: shutdown 2024-01-20T11:17:23.790 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-01-20T11:17:22.736074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001909"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1929999999999996},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.6449999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.9960000000000004},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.061999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":15.018000000000001},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.739000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.563000000000001},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":14.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":17.559000000000001},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":13.294}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.603999999999999},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":18.77}]}]},{"osd":7,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.6029999999999998},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4949999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.25},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.6099999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.23},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":12.256}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.090999999999999},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.863}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.760999999999999},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":17.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.984},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":18.516999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313247,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.399},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.7910000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.1449999999999996},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.4210000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":15.436999999999999},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.507999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.18},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.144}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":14.81},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":10.906000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":13.525},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":10.081}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444060,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.9050000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.6429999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.446999999999999},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.7690000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":17.277999999999999},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.34}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":16.474},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.085000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.694000000000001},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.550000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.186999999999999},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":13.983000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444061,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.3159999999999998},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":6.6210000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.744},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.298}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":27.030999999999999,"5min":27.030999999999999,"15min":27.030999999999999},"min":{"1min":7.9240000000000004,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":259.483,"5min":259.483,"15min":259.483},"last":12.073},{"interface":"front","average":{"1min":21.640000000000001,"5min":21.640000000000001,"15min":21.640000000000001},"min":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":243.24199999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.1679999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":13.319000000000001},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":16.411999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.667},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":13.972}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":32.277000000000001,"5min":32.277000000000001,"15min":32.277000000000001},"min":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":344.98099999999999,"5min":344.98099999999999,"15min":344.98099999999999},"last":17.141999999999999},{"interface":"front","average":{"1min":32.536000000000001,"5min":32.536000000000001,"15min":32.536000000000001},"min":{"1min":12.576000000000001,"5min":12.576000000000001,"15min":12.576000000000001},"max":{"1min":313.45999999999998,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.576000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444061,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":7.758},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.1539999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.3650000000000002},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.393000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":15.411},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.318}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.907999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":19.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":17.108000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":14.704000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":18.379999999999999},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":12.622}]}]},{"osd":6,"up_from":15,"seq":64424509470,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":10.471},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.718}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.0429999999999993},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.394}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.843},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3819999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.036},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.666}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.786999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.333},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.885999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:23.792 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:23.792 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:23.792 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:23.793 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.002387+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.002387+0000', 'last_peered': '2024-01-20T11:16:53.002387+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.002387+0000', 'last_undegraded': '2024-01-20T11:16:53.002387+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:23.793 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:23.793 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.858298+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.858298+0000', 'last_peered': '2024-01-20T11:16:52.858298+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.858298+0000', 'last_undegraded': '2024-01-20T11:16:52.858298+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:23.793 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:23.793 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:23.793 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:23.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.892+0000 7f2258dd0640 10 monclient: get_monmap_and_config 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.892+0000 7f2258dd0640 10 monclient: build_initial_monmap 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.892+0000 7f2258dd0640 10 monclient: monmap: 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:23.892994+0000 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:23.892994+0000 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 1 Processor -- start 2024-01-20T11:17:23.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 1 -- start start 2024-01-20T11:17:23.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient: init 2024-01-20T11:17:23.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22540a96f0 0x7f225413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient(hunting): picked mon.noname-b con 0x7f22540a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f225413fa10 0x7f22541423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient(hunting): picked mon.noname-c con 0x7f225413fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22540a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22540b25c0 con 0x7f22540a9ad0 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22540aaea0 con 0x7f225413fa10 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22540ab0a0 con 0x7f22540a96f0 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:23.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.893+0000 7f2258dd0640 10 monclient(hunting): authenticate will time out at 1487.244057s 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.894+0000 7f225259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22540a96f0 0x7f225413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.894+0000 7f224bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f225413fa10 0x7f22541423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.894+0000 7f225259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22540a96f0 0x7f225413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46806/0 (socket says 172.21.15.110:46806) 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.894+0000 7f225259c640 1 -- 172.21.15.110:0/2135055687 learned_addr learned my addr 172.21.15.110:0/2135055687 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.894+0000 7f225259c640 10 monclient(hunting): get_auth_request con 0x7f22540a96f0 auth_method 0 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.894+0000 7f225259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.894+0000 7f225259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.894+0000 7f225259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f224bfff640 10 monclient(hunting): get_auth_request con 0x7f225413fa10 auth_method 0 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f224bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f224bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f224bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f225259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f225259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f225259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:23.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:23.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:23.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.895+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:23.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.897+0000 7f225259c640 10 monclient(hunting): handle_auth_done global_id 4409 payload 274 2024-01-20T11:17:23.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.897+0000 7f225259c640 1 -- 172.21.15.110:0/2135055687 >> v2:172.21.15.134:3300/0 conn(0x7f225413fa10 msgr2=0x7f22541423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:23.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.897+0000 7f225259c640 1 --2- 172.21.15.110:0/2135055687 >> v2:172.21.15.134:3300/0 conn(0x7f225413fa10 0x7f22541423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.897+0000 7f225259c640 1 -- 172.21.15.110:0/2135055687 >> v1:172.21.15.134:6789/0 conn(0x7f22540a9ad0 legacy=0x7f2254142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:23.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.897+0000 7f225259c640 10 monclient: _finish_hunting 0 2024-01-20T11:17:23.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.897+0000 7f225259c640 1 monclient: found mon.noname-b 2024-01-20T11:17:23.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.897+0000 7f225259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:23.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.897+0000 7f225259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:23.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.897+0000 7f225259c640 1 -- 172.21.15.110:0/2135055687 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540b3fb0 con 0x7f22540a96f0 2024-01-20T11:17:23.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.899+0000 7f225259c640 1 --2- 172.21.15.110:0/2135055687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22540a96f0 0x7f225413f4d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2244009870 tx=0x7f224402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d14d1b82f561640c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:23.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 1 -- 172.21.15.110:0/2135055687 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22440317c0 con 0x7f22540a96f0 2024-01-20T11:17:23.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:23.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:23.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 10 monclient: dump: 2024-01-20T11:17:23.902 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:23.902 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.902 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.902 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.902 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 10 monclient: _finish_auth 0 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 1 -- 172.21.15.110:0/2135055687 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2244031920 con 0x7f22540a96f0 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2258dd0640 5 monclient: authenticate success, global_id 4409 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2258dd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2258dd0640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2258dd0640 4 set_mon_vals no callback set 2024-01-20T11:17:23.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 1 -- 172.21.15.110:0/2135055687 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244031cb0 con 0x7f22540a96f0 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.902+0000 7f2251d9b640 10 monclient: dump: 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:23.904 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 10 monclient: shutdown 2024-01-20T11:17:23.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:23.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 1 -- 172.21.15.110:0/2135055687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22540a96f0 msgr2=0x7f225413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 1 --2- 172.21.15.110:0/2135055687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22540a96f0 0x7f225413f4d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2244009870 tx=0x7f224402b860 comp rx=0 tx=0).stop 2024-01-20T11:17:23.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 1 -- 172.21.15.110:0/2135055687 shutdown_connections 2024-01-20T11:17:23.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 1 --2- 172.21.15.110:0/2135055687 >> v2:172.21.15.134:3300/0 conn(0x7f225413fa10 0x7f22541423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 1 --2- 172.21.15.110:0/2135055687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22540a96f0 0x7f225413f4d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 1 -- 172.21.15.110:0/2135055687 >> 172.21.15.110:0/2135055687 conn(0x7f22540b7bc0 msgr2=0x7f22540b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 1 -- 172.21.15.110:0/2135055687 shutdown_connections 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.903+0000 7f2258dd0640 1 -- 172.21.15.110:0/2135055687 wait complete. 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.904+0000 7f2258dd0640 10 monclient: build_initial_monmap 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.904+0000 7f2258dd0640 10 monclient: monmap: 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:23.909 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.904+0000 7f2258dd0640 1 librados: starting msgr at 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.904+0000 7f2258dd0640 1 librados: starting objecter 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.904+0000 7f2258dd0640 1 Processor -- start 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.904+0000 7f2258dd0640 1 -- start start 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.904+0000 7f2258dd0640 1 librados: setting wanted keys 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.904+0000 7f2258dd0640 1 librados: calling monclient init 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.904+0000 7f2258dd0640 10 monclient: init 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22540a96f0 0x7f225413ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient(hunting): picked mon.noname-a con 0x7f22540a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22540a9ad0 0x7f2254133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient(hunting): picked mon.noname-b con 0x7f22540a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:23.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2254137ff0 0x7f22541341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient(hunting): picked mon.noname-c con 0x7f2254137ff0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2254134720 con 0x7f22540a96f0 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22541452e0 con 0x7f22540a9ad0 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2254146000 con 0x7f2254137ff0 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2258dd0640 10 monclient(hunting): authenticate will time out at 1487.255386s 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f225259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22540a96f0 0x7f225413ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f2252d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2254137ff0 0x7f22541341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.905+0000 7f224bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22540a9ad0 0x7f2254133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f224bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22540a9ad0 0x7f2254133ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:53152/0 (socket says 172.21.15.110:53152) 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f224bfff640 1 -- 172.21.15.110:0/1602755282 learned_addr learned my addr 172.21.15.110:0/1602755282 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:23.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f224bfff640 10 monclient(hunting): get_auth_request con 0x7f22540a9ad0 auth_method 0 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f224bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f224bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f224bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f2252d9d640 10 monclient(hunting): get_auth_request con 0x7f2254137ff0 auth_method 0 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f2252d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f2252d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f2252d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f225259c640 10 monclient(hunting): get_auth_request con 0x7f22540a96f0 auth_method 0 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f225259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f225259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f225259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:23.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f224bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:23.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f225259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:23.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f225259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:23.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f225259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:23.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f2252d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:23.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f2252d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:23.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.906+0000 7f2252d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:23.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 10 monclient(hunting): handle_auth_done global_id 4573 payload 824 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 1 -- 172.21.15.110:0/1602755282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2254137ff0 msgr2=0x7f22541341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2254137ff0 0x7f22541341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 1 -- 172.21.15.110:0/1602755282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22540a96f0 msgr2=0x7f225413ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22540a96f0 0x7f225413ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 1 monclient: found mon.noname-b 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.909+0000 7f224bfff640 1 -- 172.21.15.110:0/1602755282 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22440094d0 con 0x7f22540a9ad0 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.911+0000 7f224bfff640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22540a9ad0 0x7f2254133ca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f223c00e840 tx=0x7f223c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.911+0000 7f224affd640 10 client.?.objecter ms_handle_connect 0x7f22540a9ad0 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.911+0000 7f224affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:23.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 1 -- 172.21.15.110:0/1602755282 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c00c730 con 0x7f22540a9ad0 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 10 monclient: dump: 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:23.916 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 10 monclient: _finish_auth 0 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 1 -- 172.21.15.110:0/1602755282 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223c00c890 con 0x7f22540a9ad0 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f2258dd0640 5 monclient: authenticate success, global_id 4573 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f2258dd0640 10 monclient: _renew_subs 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f2258dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2254145a70 con 0x7f22540a9ad0 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 1 -- 172.21.15.110:0/1602755282 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c00cc40 con 0x7f22540a9ad0 2024-01-20T11:17:23.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f224affd640 10 monclient: dump: 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f225159a640 4 set_mon_vals no callback set 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f2258dd0640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f2258dd0640 10 monclient: _renew_subs 2024-01-20T11:17:23.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f2258dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:23.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22541d7250 con 0x7f22540a9ad0 2024-01-20T11:17:23.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.914+0000 7f2258dd0640 1 librados: init done 2024-01-20T11:17:23.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.915+0000 7f2258dd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:23.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.915+0000 7f2258dd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:23.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.915+0000 7f2258dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:23.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:23.915+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22541461c0 con 0x7f22540a9ad0 2024-01-20T11:17:24.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.160+0000 7f224affd640 1 -- 172.21.15.110:0/1602755282 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f223c013470 con 0x7f22540a9ad0 2024-01-20T11:17:24.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.160+0000 7f224affd640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f223406c890 0x7f223406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:24.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.160+0000 7f225259c640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f223406c890 0x7f223406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:24.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.160+0000 7f225259c640 10 monclient: get_auth_request con 0x7f223406c890 auth_method 0 2024-01-20T11:17:24.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.161+0000 7f225259c640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f223406c890 0x7f223406ed50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2244036000 tx=0x7f224403d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:24.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.161+0000 7f224affd640 10 client.4573.objecter ms_handle_connect 0x7f223406c890 2024-01-20T11:17:24.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.162+0000 7f224affd640 1 -- 172.21.15.110:0/1602755282 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f223c0589d0 con 0x7f22540a9ad0 2024-01-20T11:17:24.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.162+0000 7f224affd640 10 client.4573.objecter ms_dispatch 0x7f2254000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:24.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.162+0000 7f224affd640 3 client.4573.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:24.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.162+0000 7f224affd640 3 client.4573.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:24.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.163+0000 7f224affd640 20 client.4573.objecter dump_active .. 0 homeless 2024-01-20T11:17:24.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.174+0000 7f224affd640 1 -- 172.21.15.110:0/1602755282 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f223c054d10 con 0x7f22540a9ad0 2024-01-20T11:17:24.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.174+0000 7f224affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:24.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.174+0000 7f224affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:24.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.434+0000 7f2258dd0640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:24.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.434+0000 7f2258dd0640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:24.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.434+0000 7f2258dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:24.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.434+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2254145df0 con 0x7f22540a9ad0 2024-01-20T11:17:24.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.440+0000 7f224affd640 1 -- 172.21.15.110:0/1602755282 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7f223c021510 con 0x7f22540a9ad0 2024-01-20T11:17:24.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.440+0000 7f224affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:24.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.440+0000 7f224affd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:24.440 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:24.440 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":178,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 10 librados: watch_flush enter 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 10 librados: watch_flush exit 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 20 client.4573.objecter shutdown clearing up homeless session... 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 10 client.4573.objecter successfully canceled tick 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f223406c890 msgr2=0x7f223406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f223406c890 0x7f223406ed50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2244036000 tx=0x7f224403d040 comp rx=0 tx=0).stop 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 10 monclient: shutdown 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22540a9ad0 msgr2=0x7f2254133ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:24.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22540a9ad0 0x7f2254133ca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f223c00e840 tx=0x7f223c00ed00 comp rx=0 tx=0).stop 2024-01-20T11:17:24.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 shutdown_connections 2024-01-20T11:17:24.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f223406c890 0x7f223406ed50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:24.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2254137ff0 0x7f22541341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:24.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22540a9ad0 0x7f2254133ca0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:24.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 --2- 172.21.15.110:0/1602755282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22540a96f0 0x7f225413ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:24.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 >> 172.21.15.110:0/1602755282 conn(0x7f22540b7bc0 msgr2=0x7f22540b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:24.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 shutdown_connections 2024-01-20T11:17:24.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.442+0000 7f2258dd0640 1 -- 172.21.15.110:0/1602755282 wait complete. 2024-01-20T11:17:24.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.443+0000 7f2258dd0640 1 librados: shutdown 2024-01-20T11:17:24.454 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:24.455 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:24.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.552+0000 7f07ac076640 10 monclient: get_monmap_and_config 2024-01-20T11:17:24.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.552+0000 7f07ac076640 10 monclient: build_initial_monmap 2024-01-20T11:17:24.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.552+0000 7f07ac076640 10 monclient: monmap: 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:24.553364+0000 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:24.553364+0000 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:24.553 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.553+0000 7f07ac076640 1 Processor -- start 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.553+0000 7f07ac076640 1 -- start start 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.553+0000 7f07ac076640 10 monclient: init 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07a4145760 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f07a40a9d80 0x7f07a4145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 10 monclient(hunting): picked mon.noname-a con 0x7f07a40a9d80 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07a413f8b0 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a4146250 con 0x7f07a413f8b0 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b20c0 con 0x7f07a4145760 2024-01-20T11:17:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a40ab050 con 0x7f07a40a9d80 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07ac076640 10 monclient(hunting): authenticate will time out at 1487.904439s 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07a9e12640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f07a4145760 0x7f07a413f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:54000/0 (socket says 172.21.15.110:54000) 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07a9e12640 1 -- 172.21.15.110:0/3924497165 learned_addr learned my addr 172.21.15.110:0/3924497165 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.554+0000 7f07a9611640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f07a40a9d80 0x7f07a4145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.555+0000 7f07a9611640 10 monclient(hunting): get_auth_request con 0x7f07a40a9d80 auth_method 0 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.555+0000 7f07a9611640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.555+0000 7f07a9611640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:24.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.555+0000 7f07a9611640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:24.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.555+0000 7f07a9611640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:24.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.555+0000 7f07a9611640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:24.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.555+0000 7f07a9611640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.556+0000 7f07a8e10640 1 -- 172.21.15.110:0/3924497165 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032365291 0 0) 0x7f07a40b20c0 con 0x7f07a4145760 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.556+0000 7f07a8e10640 10 monclient(hunting): _negotiate 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.556+0000 7f07a8e10640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.556+0000 7f07a8e10640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.556+0000 7f07a8e10640 10 monclient(hunting): my global_id is 4576 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.556+0000 7f07a8e10640 1 -- 172.21.15.110:0/3924497165 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0790003520 con 0x7f07a4145760 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.557+0000 7f07a9611640 10 monclient(hunting): handle_auth_done global_id 4518 payload 274 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.557+0000 7f07a9611640 1 -- 172.21.15.110:0/3924497165 >> v1:172.21.15.134:6789/0 conn(0x7f07a4145760 legacy=0x7f07a413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.557+0000 7f07a9611640 1 -- 172.21.15.110:0/3924497165 >> v1:172.21.15.110:6789/0 conn(0x7f07a413f8b0 legacy=0x7f07a40b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.557+0000 7f07a9611640 10 monclient: _finish_hunting 0 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.557+0000 7f07a9611640 1 monclient: found mon.noname-a 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.557+0000 7f07a9611640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.557+0000 7f07a9611640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:24.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.557+0000 7f07a9611640 1 -- 172.21.15.110:0/3924497165 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40b2d80 con 0x7f07a40a9d80 2024-01-20T11:17:24.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.560+0000 7f07a9611640 1 --2- 172.21.15.110:0/3924497165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a40a9d80 0x7f07a4145b40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f07940097d0 tx=0x7f079402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c2cfa17093767 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:24.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 1 -- 172.21.15.110:0/3924497165 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794030610 con 0x7f07a40a9d80 2024-01-20T11:17:24.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:24.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:24.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 10 monclient: dump: 2024-01-20T11:17:24.564 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:24.564 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 10 monclient: _finish_auth 0 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 10 monclient: _check_auth_tickets 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 1 -- 172.21.15.110:0/3924497165 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794030770 con 0x7f07a40a9d80 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 1 -- 172.21.15.110:0/3924497165 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794030b00 con 0x7f07a40a9d80 2024-01-20T11:17:24.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07a8e10640 10 monclient: dump: 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07ac076640 5 monclient: authenticate success, global_id 4518 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07ac076640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:24.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.564+0000 7f07ac076640 4 set_mon_vals no callback set 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.565+0000 7f07ac076640 10 monclient: shutdown 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.565+0000 7f07ac076640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.565+0000 7f07ac076640 1 -- 172.21.15.110:0/3924497165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a40a9d80 msgr2=0x7f07a4145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.565+0000 7f07ac076640 1 --2- 172.21.15.110:0/3924497165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a40a9d80 0x7f07a4145b40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f07940097d0 tx=0x7f079402b960 comp rx=0 tx=0).stop 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.565+0000 7f07ac076640 1 -- 172.21.15.110:0/3924497165 shutdown_connections 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.565+0000 7f07ac076640 1 --2- 172.21.15.110:0/3924497165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a40a9d80 0x7f07a4145b40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.565+0000 7f07ac076640 1 -- 172.21.15.110:0/3924497165 >> 172.21.15.110:0/3924497165 conn(0x7f07a40663b0 msgr2=0x7f07a40667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.565+0000 7f07ac076640 1 -- 172.21.15.110:0/3924497165 shutdown_connections 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.565+0000 7f07ac076640 1 -- 172.21.15.110:0/3924497165 wait complete. 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.566+0000 7f07ac076640 10 monclient: build_initial_monmap 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.566+0000 7f07ac076640 10 monclient: monmap: 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:24.570 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.566+0000 7f07ac076640 1 librados: starting msgr at 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.566+0000 7f07ac076640 1 librados: starting objecter 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.566+0000 7f07ac076640 1 Processor -- start 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.566+0000 7f07ac076640 1 -- start start 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.566+0000 7f07ac076640 1 librados: setting wanted keys 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.566+0000 7f07ac076640 1 librados: calling monclient init 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.566+0000 7f07ac076640 10 monclient: init 2024-01-20T11:17:24.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07a40a9d80 0x7f07a4137040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient(hunting): picked mon.noname-b con 0x7f07a40a9d80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a413f8b0 0x7f07a413fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient(hunting): picked mon.noname-a con 0x7f07a413f8b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07a4145760 0x7f07a41402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient(hunting): picked mon.noname-c con 0x7f07a4145760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4138430 con 0x7f07a413f8b0 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a41407f0 con 0x7f07a40a9d80 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a41424e0 con 0x7f07a4145760 2024-01-20T11:17:24.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07ac076640 10 monclient(hunting): authenticate will time out at 1487.917451s 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07a9611640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a413f8b0 0x7f07a413fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07a9e12640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07a40a9d80 0x7f07a4137040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.567+0000 7f07aa613640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07a4145760 0x7f07a41402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9611640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a413f8b0 0x7f07a413fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56538/0 (socket says 172.21.15.110:56538) 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9611640 1 -- 172.21.15.110:0/1173607361 learned_addr learned my addr 172.21.15.110:0/1173607361 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9611640 10 monclient(hunting): get_auth_request con 0x7f07a413f8b0 auth_method 0 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9611640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9611640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9611640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9e12640 10 monclient(hunting): get_auth_request con 0x7f07a40a9d80 auth_method 0 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9e12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9e12640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9e12640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:24.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07aa613640 10 monclient(hunting): get_auth_request con 0x7f07a4145760 auth_method 0 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07aa613640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07aa613640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07aa613640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9611640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9611640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9611640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9e12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9e12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07a9e12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07aa613640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07aa613640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:24.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.568+0000 7f07aa613640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:24.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 10 monclient(hunting): handle_auth_done global_id 4579 payload 824 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 1 -- 172.21.15.110:0/1173607361 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07a4145760 msgr2=0x7f07a41402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07a4145760 0x7f07a41402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 1 -- 172.21.15.110:0/1173607361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a413f8b0 msgr2=0x7f07a413fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a413f8b0 0x7f07a413fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 10 monclient: _finish_hunting 0 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 1 monclient: found mon.noname-b 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.571+0000 7f07a9e12640 1 -- 172.21.15.110:0/1173607361 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0794009400 con 0x7f07a40a9d80 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.573+0000 7f07a9e12640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07a40a9d80 0x7f07a4137040 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f079c00cbc0 tx=0x7f079c00a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.573+0000 7f079a7fc640 10 client.?.objecter ms_handle_connect 0x7f07a40a9d80 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.573+0000 7f079a7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:24.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f079a7fc640 1 -- 172.21.15.110:0/1173607361 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c00aca0 con 0x7f07a40a9d80 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f079a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f079a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f079a7fc640 10 monclient: dump: 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f079a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f079a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:17:24.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f079a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f07ac076640 5 monclient: authenticate success, global_id 4579 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f07ac076640 10 monclient: _renew_subs 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f07ac076640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a4138670 con 0x7f07a40a9d80 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f07ac076640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f07ac076640 10 monclient: _renew_subs 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f07ac076640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a4137580 con 0x7f07a40a9d80 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.575+0000 7f07ac076640 1 librados: init done 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f07ac076640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f07ac076640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f07ac076640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a4075490 con 0x7f07a40a9d80 2024-01-20T11:17:24.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f079a7fc640 1 -- 172.21.15.110:0/1173607361 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c00ae00 con 0x7f07a40a9d80 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f079a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f079a7fc640 1 -- 172.21.15.110:0/1173607361 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c01c640 con 0x7f07a40a9d80 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f079a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f079a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f079a7fc640 10 monclient: dump: 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:24.580 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:24.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.576+0000 7f079bfff640 4 set_mon_vals no callback set 2024-01-20T11:17:24.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.820+0000 7f079a7fc640 1 -- 172.21.15.110:0/1173607361 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f079c012400 con 0x7f07a40a9d80 2024-01-20T11:17:24.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.820+0000 7f079a7fc640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f078806c860 0x7f078806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:24.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.821+0000 7f07a9611640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f078806c860 0x7f078806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:24.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.821+0000 7f07a9611640 10 monclient: get_auth_request con 0x7f078806c860 auth_method 0 2024-01-20T11:17:24.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.821+0000 7f07a9611640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f078806c860 0x7f078806ed20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07a41dbbe0 tx=0x7f079403b040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:24.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.821+0000 7f079a7fc640 10 client.4579.objecter ms_handle_connect 0x7f078806c860 2024-01-20T11:17:24.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.823+0000 7f079a7fc640 1 -- 172.21.15.110:0/1173607361 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f079c0589e0 con 0x7f07a40a9d80 2024-01-20T11:17:24.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.823+0000 7f079a7fc640 10 client.4579.objecter ms_dispatch 0x7f07a4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:24.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.823+0000 7f079a7fc640 3 client.4579.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:24.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.823+0000 7f079a7fc640 3 client.4579.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:24.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.823+0000 7f079a7fc640 20 client.4579.objecter dump_active .. 0 homeless 2024-01-20T11:17:24.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.833+0000 7f079a7fc640 1 -- 172.21.15.110:0/1173607361 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f079c054d20 con 0x7f07a40a9d80 2024-01-20T11:17:24.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.833+0000 7f079a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:24.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:24.833+0000 7f079a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:25.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.000+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07a40a9950 con 0x7f078806c860 2024-01-20T11:17:25.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.002+0000 7f079a7fc640 1 -- 172.21.15.110:0/1173607361 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64296 (secure 0 0 0) 0x7f07a40a9950 con 0x7f078806c860 2024-01-20T11:17:25.003 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:25.004 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.004+0000 7f07ac076640 10 librados: watch_flush enter 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.004+0000 7f07ac076640 10 librados: watch_flush exit 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.004+0000 7f07ac076640 20 client.4579.objecter shutdown clearing up homeless session... 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.004+0000 7f07ac076640 10 client.4579.objecter successfully canceled tick 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.004+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f078806c860 msgr2=0x7f078806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f078806c860 0x7f078806ed20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07a41dbbe0 tx=0x7f079403b040 comp rx=0 tx=0).stop 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 10 monclient: shutdown 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07a40a9d80 msgr2=0x7f07a4137040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07a40a9d80 0x7f07a4137040 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f079c00cbc0 tx=0x7f079c00a060 comp rx=0 tx=0).stop 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 shutdown_connections 2024-01-20T11:17:25.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f078806c860 0x7f078806ed20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:25.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07a4145760 0x7f07a41402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:25.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07a413f8b0 0x7f07a413fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:25.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 --2- 172.21.15.110:0/1173607361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07a40a9d80 0x7f07a4137040 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:25.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 >> 172.21.15.110:0/1173607361 conn(0x7f07a40663b0 msgr2=0x7f07a40b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:25.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 shutdown_connections 2024-01-20T11:17:25.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 -- 172.21.15.110:0/1173607361 wait complete. 2024-01-20T11:17:25.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:25.005+0000 7f07ac076640 1 librados: shutdown 2024-01-20T11:17:25.015 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-01-20T11:17:24.736390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001944"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.4280000000000008},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":13.289999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.0310000000000006},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.9969999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.058999999999999},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":15.156000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.7240000000000002},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":10.242000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.944000000000001},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":11.683}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":15.877000000000001},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":12.696}]}]},{"osd":7,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.384,"15min":10.384},"min":{"1min":4.5570000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.6029999999999998},{"interface":"front","average":{"1min":14.058999999999999,"5min":14.058999999999999,"15min":14.058999999999999},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":76.718999999999994,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4949999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":10.651999999999999,"15min":10.651999999999999},"min":{"1min":5.1440000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":18.859000000000002,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.25},{"interface":"front","average":{"1min":13.494999999999999,"5min":13.494999999999999,"15min":13.494999999999999},"min":{"1min":7.4610000000000003,"5min":7.4610000000000003,"15min":7.4610000000000003},"max":{"1min":43.774000000000001,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.6099999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":17.495999999999999,"5min":17.495999999999999,"15min":17.495999999999999},"min":{"1min":10.166,"5min":10.166,"15min":10.166},"max":{"1min":49.210000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.23},{"interface":"front","average":{"1min":17.161000000000001,"5min":17.161000000000001,"15min":17.161000000000001},"min":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":8.6219999999999999},"max":{"1min":78.548000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":12.256}]},{"osd":3,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":15.467000000000001,"5min":15.467000000000001,"15min":15.467000000000001},"min":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":55.771000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.090999999999999},{"interface":"front","average":{"1min":17.434999999999999,"5min":17.434999999999999,"15min":17.434999999999999},"min":{"1min":10.83,"5min":10.83,"15min":10.83},"max":{"1min":49.829999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.863}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.760999999999999},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":17.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:16:22 2024","interfaces":[{"interface":"back","average":{"1min":19.263999999999999,"5min":19.263999999999999,"15min":19.263999999999999},"min":{"1min":10.353,"5min":10.353,"15min":10.353},"max":{"1min":80.102000000000004,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.984},{"interface":"front","average":{"1min":19.509,"5min":19.509,"15min":19.509},"min":{"1min":13.6,"5min":13.6,"15min":13.6},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":18.516999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313247,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.399},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.7910000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.1449999999999996},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.4210000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":15.436999999999999},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.507999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.18},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.144}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":14.81},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":10.906000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":13.525},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":10.081}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444061,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.4440000000000008},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.782}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.7709999999999999},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.586}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":16.800999999999998},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.435}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.66},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.795999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":14.696999999999999},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.744999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.942},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":19.43}]}]},{"osd":1,"up_from":17,"seq":73014444062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":9.4280000000000008},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":7.3220000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.1180000000000003},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.202}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":13.423},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.2260000000000009}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":12.117000000000001},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":10.784000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":16.533999999999999},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":12.843999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.574999999999999},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.169}]}]},{"osd":0,"up_from":17,"seq":73014444062,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":9.5980000000000008},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":6.8029999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.8109999999999999},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.507999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":17.588999999999999},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":14.058999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":16.631},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":17.035},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":19.553000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":18.779},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":14.901999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509470,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":10.471},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.718}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.0429999999999993},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.394}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.843},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3819999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.286000000000001,"5min":16.286000000000001,"15min":16.286000000000001},"min":{"1min":7.1870000000000003,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":81.373999999999995,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.036},{"interface":"front","average":{"1min":16.678999999999998,"5min":16.678999999999998,"15min":16.678999999999998},"min":{"1min":6.6900000000000004,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":84.524000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.666}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.786999999999999},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":16.754000000000001,"5min":16.754000000000001,"15min":16.754000000000001},"min":{"1min":11.314,"5min":11.314,"15min":11.314},"max":{"1min":25.303999999999998,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.333},{"interface":"front","average":{"1min":16.222000000000001,"5min":16.222000000000001,"15min":16.222000000000001},"min":{"1min":10.805999999999999,"5min":10.805999999999999,"15min":10.805999999999999},"max":{"1min":22.838999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.885999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:28.018 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:28.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.121+0000 7f27fc7cf640 10 monclient: get_monmap_and_config 2024-01-20T11:17:28.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.121+0000 7f27fc7cf640 10 monclient: build_initial_monmap 2024-01-20T11:17:28.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.121+0000 7f27fc7cf640 10 monclient: monmap: 2024-01-20T11:17:28.122 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:28.122 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.122 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:28.122331+0000 2024-01-20T11:17:28.122 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:28.122331+0000 2024-01-20T11:17:28.122 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:28.122 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.123 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:28.123 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:28.123 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:28.123 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:28.123 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:28.123 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.122+0000 7f27fc7cf640 1 Processor -- start 2024-01-20T11:17:28.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.122+0000 7f27fc7cf640 1 -- start start 2024-01-20T11:17:28.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.122+0000 7f27fc7cf640 10 monclient: init 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f27f4071c60 0x7f27f4142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f27f4071c60 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27f40ac540 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27f40b0930 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f40ad610 con 0x7f27f40ac540 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f41426c0 con 0x7f27f40b0930 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:28.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f4072040 con 0x7f27f4071c60 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fc7cf640 10 monclient(hunting): authenticate will time out at 1491.473285s 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fa56b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f27f4071c60 0x7f27f4142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fad6c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f27f40b0930 0x7f27f4149510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:54014/0 (socket says 172.21.15.110:54014) 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.123+0000 7f27fad6c640 1 -- 172.21.15.110:0/428195657 learned_addr learned my addr 172.21.15.110:0/428195657 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.124+0000 7f27fa56b640 10 monclient(hunting): get_auth_request con 0x7f27f4071c60 auth_method 0 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.124+0000 7f27fa56b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.124+0000 7f27fa56b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.124+0000 7f27fa56b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.124+0000 7f27fa56b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.124+0000 7f27fa56b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:28.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.124+0000 7f27fa56b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:28.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.126+0000 7f27f9569640 1 -- 172.21.15.110:0/428195657 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184189520 0 0) 0x7f27f41426c0 con 0x7f27f40b0930 2024-01-20T11:17:28.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.126+0000 7f27f9569640 10 monclient(hunting): _negotiate 2024-01-20T11:17:28.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.126+0000 7f27f9569640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:28.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.126+0000 7f27f9569640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.126+0000 7f27f9569640 10 monclient(hunting): my global_id is 4582 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.126+0000 7f27f9569640 1 -- 172.21.15.110:0/428195657 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e0003520 con 0x7f27f40b0930 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.126+0000 7f27fa56b640 10 monclient(hunting): handle_auth_done global_id 4524 payload 274 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.127+0000 7f27fa56b640 1 -- 172.21.15.110:0/428195657 >> v1:172.21.15.134:6789/0 conn(0x7f27f40b0930 legacy=0x7f27f4149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.127+0000 7f27fa56b640 1 -- 172.21.15.110:0/428195657 >> v1:172.21.15.110:6789/0 conn(0x7f27f40ac540 legacy=0x7f27f40ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.127+0000 7f27fa56b640 10 monclient: _finish_hunting 0 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.127+0000 7f27fa56b640 1 monclient: found mon.noname-a 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.127+0000 7f27fa56b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.127+0000 7f27fa56b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:28.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.127+0000 7f27fa56b640 1 -- 172.21.15.110:0/428195657 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f40721b0 con 0x7f27f4071c60 2024-01-20T11:17:28.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.131+0000 7f27fa56b640 1 --2- 172.21.15.110:0/428195657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f4071c60 0x7f27f4142180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f27f00097a0 tx=0x7f27f002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7de516a49f0869d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:28.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 1 -- 172.21.15.110:0/428195657 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0030610 con 0x7f27f4071c60 2024-01-20T11:17:28.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:28.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:28.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 10 monclient: dump: 2024-01-20T11:17:28.134 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 10 monclient: _finish_auth 0 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 10 monclient: _check_auth_tickets 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 1 -- 172.21.15.110:0/428195657 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f0030770 con 0x7f27f4071c60 2024-01-20T11:17:28.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 1 -- 172.21.15.110:0/428195657 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0030b00 con 0x7f27f4071c60 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27f9569640 10 monclient: dump: 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27fc7cf640 5 monclient: authenticate success, global_id 4524 2024-01-20T11:17:28.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27fc7cf640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:28.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.134+0000 7f27fc7cf640 4 set_mon_vals no callback set 2024-01-20T11:17:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.135+0000 7f27fc7cf640 10 monclient: shutdown 2024-01-20T11:17:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.135+0000 7f27fc7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.135+0000 7f27fc7cf640 1 -- 172.21.15.110:0/428195657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f4071c60 msgr2=0x7f27f4142180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.135+0000 7f27fc7cf640 1 --2- 172.21.15.110:0/428195657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f4071c60 0x7f27f4142180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f27f00097a0 tx=0x7f27f002b960 comp rx=0 tx=0).stop 2024-01-20T11:17:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.135+0000 7f27fc7cf640 1 -- 172.21.15.110:0/428195657 shutdown_connections 2024-01-20T11:17:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.135+0000 7f27fc7cf640 1 --2- 172.21.15.110:0/428195657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f4071c60 0x7f27f4142180 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.135+0000 7f27fc7cf640 1 -- 172.21.15.110:0/428195657 >> 172.21.15.110:0/428195657 conn(0x7f27f40b3e60 msgr2=0x7f27f40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.135+0000 7f27fc7cf640 1 -- 172.21.15.110:0/428195657 shutdown_connections 2024-01-20T11:17:28.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.135+0000 7f27fc7cf640 1 -- 172.21.15.110:0/428195657 wait complete. 2024-01-20T11:17:28.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.136+0000 7f27fc7cf640 10 monclient: build_initial_monmap 2024-01-20T11:17:28.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.136+0000 7f27fc7cf640 10 monclient: monmap: 2024-01-20T11:17:28.143 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:28.143 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:28.143 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:28.143 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:28.143 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.136+0000 7f27fc7cf640 1 librados: starting msgr at 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.136+0000 7f27fc7cf640 1 librados: starting objecter 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.136+0000 7f27fc7cf640 1 Processor -- start 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.136+0000 7f27fc7cf640 1 -- start start 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.136+0000 7f27fc7cf640 1 librados: setting wanted keys 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.136+0000 7f27fc7cf640 1 librados: calling monclient init 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.136+0000 7f27fc7cf640 10 monclient: init 2024-01-20T11:17:28.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4071c60 0x7f27f41363a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f27f4071c60 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27f40ac540 0x7f27f41d87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient(hunting): picked mon.noname-c con 0x7f27f40ac540 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f40b0930 0x7f27f41d8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f27f40b0930 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f413ca00 con 0x7f27f40b0930 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f413cca0 con 0x7f27f4071c60 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f41d8300 con 0x7f27f40ac540 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:28.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fc7cf640 10 monclient(hunting): authenticate will time out at 1491.487534s 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fad6c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f40b0930 0x7f27f41d8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27f9d6a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27f40ac540 0x7f27f41d87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.137+0000 7f27fa56b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4071c60 0x7f27f41363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fa56b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4071c60 0x7f27f41363a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:53178/0 (socket says 172.21.15.110:53178) 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fad6c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f40b0930 0x7f27f41d8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56556/0 (socket says 172.21.15.110:56556) 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fa56b640 1 -- 172.21.15.110:0/1747935115 learned_addr learned my addr 172.21.15.110:0/1747935115 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fa56b640 10 monclient(hunting): get_auth_request con 0x7f27f4071c60 auth_method 0 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fa56b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fa56b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fa56b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fad6c640 10 monclient(hunting): get_auth_request con 0x7f27f40b0930 auth_method 0 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fad6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fad6c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fad6c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27f9d6a640 10 monclient(hunting): get_auth_request con 0x7f27f40ac540 auth_method 0 2024-01-20T11:17:28.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27f9d6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27f9d6a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27f9d6a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fad6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fad6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fad6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fa56b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fa56b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27fa56b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27f9d6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27f9d6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.138+0000 7f27f9d6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:28.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 10 monclient(hunting): handle_auth_done global_id 4585 payload 824 2024-01-20T11:17:28.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 1 -- 172.21.15.110:0/1747935115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27f40ac540 msgr2=0x7f27f41d87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27f40ac540 0x7f27f41d87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 1 -- 172.21.15.110:0/1747935115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f40b0930 msgr2=0x7f27f41d8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f40b0930 0x7f27f41d8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 10 monclient: _finish_hunting 0 2024-01-20T11:17:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 1 monclient: found mon.noname-b 2024-01-20T11:17:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.141+0000 7f27fa56b640 1 -- 172.21.15.110:0/1747935115 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f0009400 con 0x7f27f4071c60 2024-01-20T11:17:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.143+0000 7f27fa56b640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4071c60 0x7f27f41363a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f27f002bf70 tx=0x7f27f0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.143+0000 7f27eaffd640 10 client.?.objecter ms_handle_connect 0x7f27f4071c60 2024-01-20T11:17:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.143+0000 7f27eaffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:28.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 1 -- 172.21.15.110:0/1747935115 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f003e040 con 0x7f27f4071c60 2024-01-20T11:17:28.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:28.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:28.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 10 monclient: dump: 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 1 -- 172.21.15.110:0/1747935115 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f0039a40 con 0x7f27f4071c60 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27fc7cf640 5 monclient: authenticate success, global_id 4585 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27fc7cf640 10 monclient: _renew_subs 2024-01-20T11:17:28.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27fc7cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f40aa120 con 0x7f27f4071c60 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 1 -- 172.21.15.110:0/1747935115 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0039df0 con 0x7f27f4071c60 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27eaffd640 10 monclient: dump: 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:28.163 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27f8d68640 4 set_mon_vals no callback set 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27fc7cf640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27fc7cf640 10 monclient: _renew_subs 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27fc7cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f413d430 con 0x7f27f4071c60 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.161+0000 7f27fc7cf640 1 librados: init done 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.162+0000 7f27fc7cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.162+0000 7f27fc7cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.162+0000 7f27fc7cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:28.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.162+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f40aa3f0 con 0x7f27f4071c60 2024-01-20T11:17:28.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.418+0000 7f27eaffd640 1 -- 172.21.15.110:0/1747935115 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f27f004d3b0 con 0x7f27f4071c60 2024-01-20T11:17:28.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.418+0000 7f27eaffd640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f27d806c8b0 0x7f27d806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.419+0000 7f27f9d6a640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f27d806c8b0 0x7f27d806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:28.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.419+0000 7f27f9d6a640 10 monclient: get_auth_request con 0x7f27d806c8b0 auth_method 0 2024-01-20T11:17:28.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.419+0000 7f27eaffd640 10 client.4585.objecter ms_handle_connect 0x7f27d806c8b0 2024-01-20T11:17:28.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.419+0000 7f27f9d6a640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f27d806c8b0 0x7f27d806ed70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f27f413e700 tx=0x7f27e4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:28.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.421+0000 7f27eaffd640 1 -- 172.21.15.110:0/1747935115 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f27f007a910 con 0x7f27f4071c60 2024-01-20T11:17:28.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.421+0000 7f27eaffd640 10 client.4585.objecter ms_dispatch 0x7f27f4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:28.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.421+0000 7f27eaffd640 3 client.4585.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:28.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.421+0000 7f27eaffd640 3 client.4585.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:28.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.421+0000 7f27eaffd640 20 client.4585.objecter dump_active .. 0 homeless 2024-01-20T11:17:28.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.436+0000 7f27eaffd640 1 -- 172.21.15.110:0/1747935115 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f27f0076c50 con 0x7f27f4071c60 2024-01-20T11:17:28.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.436+0000 7f27eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:28.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.436+0000 7f27eaffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.615+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27f414ac30 con 0x7f27d806c8b0 2024-01-20T11:17:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.618+0000 7f27eaffd640 1 -- 172.21.15.110:0/1747935115 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64289 (secure 0 0 0) 0x7f27e4001420 con 0x7f27d806c8b0 2024-01-20T11:17:28.618 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:28.619 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.619+0000 7f27fc7cf640 10 librados: watch_flush enter 2024-01-20T11:17:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 10 librados: watch_flush exit 2024-01-20T11:17:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 20 client.4585.objecter shutdown clearing up homeless session... 2024-01-20T11:17:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 10 client.4585.objecter successfully canceled tick 2024-01-20T11:17:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f27d806c8b0 msgr2=0x7f27d806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f27d806c8b0 0x7f27d806ed70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f27f413e700 tx=0x7f27e4009040 comp rx=0 tx=0).stop 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 10 monclient: shutdown 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4071c60 msgr2=0x7f27f41363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4071c60 0x7f27f41363a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f27f002bf70 tx=0x7f27f0004730 comp rx=0 tx=0).stop 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 shutdown_connections 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f27d806c8b0 0x7f27d806ed70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f40b0930 0x7f27f41d8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27f40ac540 0x7f27f41d87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 --2- 172.21.15.110:0/1747935115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4071c60 0x7f27f41363a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 >> 172.21.15.110:0/1747935115 conn(0x7f27f40b3e60 msgr2=0x7f27f40aa660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 shutdown_connections 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 -- 172.21.15.110:0/1747935115 wait complete. 2024-01-20T11:17:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.620+0000 7f27fc7cf640 1 librados: shutdown 2024-01-20T11:17:28.630 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-01-20T11:17:26.736771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":15.359999999999999,"5min":15.359999999999999,"15min":15.359999999999999},"min":{"1min":4.3220000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":100.05800000000001,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.4280000000000008},{"interface":"front","average":{"1min":16.765999999999998,"5min":16.765999999999998,"15min":16.765999999999998},"min":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":91.789000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":13.289999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":18.356999999999999},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":98.700000000000003,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.0310000000000006},{"interface":"front","average":{"1min":17.056999999999999,"5min":17.056999999999999,"15min":17.056999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":95.603999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.9969999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":19.321999999999999,"5min":19.321999999999999,"15min":19.321999999999999},"min":{"1min":7.3819999999999997,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":131.09700000000001,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.058999999999999},{"interface":"front","average":{"1min":18.225999999999999,"5min":18.225999999999999,"15min":18.225999999999999},"min":{"1min":6.1989999999999998,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":105.43899999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":15.156000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":21.908999999999999,"5min":21.908999999999999,"15min":21.908999999999999},"min":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"max":{"1min":94.911000000000001,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.7240000000000002},{"interface":"front","average":{"1min":23.600000000000001,"5min":23.600000000000001,"15min":23.600000000000001},"min":{"1min":8.2799999999999994,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":100.633,"5min":100.633,"15min":100.633},"last":10.242000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"min":{"1min":8.1129999999999995,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":131.70599999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.944000000000001},{"interface":"front","average":{"1min":21.763999999999999,"5min":21.763999999999999,"15min":21.763999999999999},"min":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":10.159000000000001},"max":{"1min":107.962,"5min":107.962,"15min":107.962},"last":11.683}]},{"osd":7,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":20.282,"5min":20.282,"15min":20.282},"min":{"1min":8.8369999999999997,"5min":8.8369999999999997,"15min":8.8369999999999997},"max":{"1min":96.911000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":15.877000000000001},{"interface":"front","average":{"1min":23.795999999999999,"5min":23.795999999999999,"15min":23.795999999999999},"min":{"1min":9.6739999999999995,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":129.828,"5min":129.828,"15min":129.828},"last":12.696}]}]},{"osd":7,"up_from":21,"seq":90194313247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.1760000000000002},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4760000000000009}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.7080000000000002},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.225}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.228},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.045}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":17.696000000000002},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.348000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.789000000000001},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":18.335000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.667999999999999},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":19.251000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313248,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4249999999999998},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.7130000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7409999999999997},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.099}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":8.4540000000000006},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.489000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.478999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.1259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":10.401999999999999},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":15.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":16.216000000000001},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.728999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444061,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.4440000000000008},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.782}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.7709999999999999},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.586}]},{"osd":3,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":17.765999999999998,"5min":17.765999999999998,"15min":17.765999999999998},"min":{"1min":10.888999999999999,"5min":10.888999999999999,"15min":10.888999999999999},"max":{"1min":49.237000000000002,"5min":49.237000000000002,"15min":49.237000000000002},"last":16.800999999999998},{"interface":"front","average":{"1min":15.041,"5min":15.041,"15min":15.041},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":52.905999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.435}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.66},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.795999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":14.696999999999999},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.744999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:21 2024","interfaces":[{"interface":"back","average":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"min":{"1min":8.9120000000000008,"5min":8.9120000000000008,"15min":8.9120000000000008},"max":{"1min":24.266999999999999,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.942},{"interface":"front","average":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"min":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":28.148,"5min":28.148,"15min":28.148},"last":19.43}]}]},{"osd":1,"up_from":17,"seq":73014444062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":9.4280000000000008},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":7.3220000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.1180000000000003},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.202}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":13.423},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.2260000000000009}]},{"osd":4,"last update":"Sat Jan 20 11:16:24 2024","interfaces":[{"interface":"back","average":{"1min":14.294,"5min":14.294,"15min":14.294},"min":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"max":{"1min":21.984000000000002,"5min":21.984000000000002,"15min":21.984000000000002},"last":12.117000000000001},{"interface":"front","average":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":24.167999999999999,"5min":24.167999999999999,"15min":24.167999999999999},"last":10.784000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":16.533999999999999},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":12.843999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.574999999999999},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.169}]}]},{"osd":0,"up_from":17,"seq":73014444062,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":9.5980000000000008},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":6.8029999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.8109999999999999},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.507999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":17.588999999999999},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":14.058999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":16.631},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":17.035},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":19.553000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":18.779},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":14.901999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509471,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":11.795999999999999},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.569}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.565},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":7.2370000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.183},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.2620000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.966000000000001},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.888999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.222},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.087}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.058},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":17.782}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:28.632 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.731+0000 7f4cb9ccf640 10 monclient: get_monmap_and_config 2024-01-20T11:17:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.731+0000 7f4cb9ccf640 10 monclient: build_initial_monmap 2024-01-20T11:17:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.731+0000 7f4cb9ccf640 10 monclient: monmap: 2024-01-20T11:17:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:28.731714+0000 2024-01-20T11:17:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:28.731714+0000 2024-01-20T11:17:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:28.738 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.731+0000 7f4cb9ccf640 1 Processor -- start 2024-01-20T11:17:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.731+0000 7f4cb9ccf640 1 -- start start 2024-01-20T11:17:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.731+0000 7f4cb9ccf640 10 monclient: init 2024-01-20T11:17:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-01-20T11:17:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cb40a96f0 0x7f4cb413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb40a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4cb413fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4cb40a9ad0 0x7f4cb4142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb40a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb40b25c0 con 0x7f4cb413fa10 2024-01-20T11:17:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:28.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb40aaea0 con 0x7f4cb40a96f0 2024-01-20T11:17:28.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:28.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb40ab0a0 con 0x7f4cb40a9ad0 2024-01-20T11:17:28.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:28.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.732+0000 7f4cb9ccf640 10 monclient(hunting): authenticate will time out at 1492.082659s 2024-01-20T11:17:28.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb37fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cb40a96f0 0x7f4cb413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:28.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb3fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4cb40a9ad0 0x7f4cb4142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:28.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb37fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cb40a96f0 0x7f4cb413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56568/0 (socket says 172.21.15.110:56568) 2024-01-20T11:17:28.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb37fe640 1 -- 172.21.15.110:0/4201321234 learned_addr learned my addr 172.21.15.110:0/4201321234 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:28.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb37fe640 10 monclient(hunting): get_auth_request con 0x7f4cb40a96f0 auth_method 0 2024-01-20T11:17:28.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:28.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:28.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:28.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb3fff640 10 monclient(hunting): get_auth_request con 0x7f4cb40a9ad0 auth_method 0 2024-01-20T11:17:28.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:28.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:28.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:28.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:28.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.733+0000 7f4cb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:28.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.734+0000 7f4cb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:28.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.734+0000 7f4cb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:28.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.734+0000 7f4cb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:28.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.734+0000 7f4cb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:28.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.735+0000 7f4cb37fe640 10 monclient(hunting): handle_auth_done global_id 4533 payload 274 2024-01-20T11:17:28.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.735+0000 7f4cb37fe640 1 -- 172.21.15.110:0/4201321234 >> v2:172.21.15.134:3300/0 conn(0x7f4cb40a9ad0 msgr2=0x7f4cb4142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:28.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.735+0000 7f4cb37fe640 1 --2- 172.21.15.110:0/4201321234 >> v2:172.21.15.134:3300/0 conn(0x7f4cb40a9ad0 0x7f4cb4142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.735+0000 7f4cb37fe640 1 -- 172.21.15.110:0/4201321234 >> v1:172.21.15.134:6789/0 conn(0x7f4cb413fa10 legacy=0x7f4cb41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.735+0000 7f4cb37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:28.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.735+0000 7f4cb37fe640 1 monclient: found mon.noname-a 2024-01-20T11:17:28.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.735+0000 7f4cb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:28.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.735+0000 7f4cb37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:28.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.735+0000 7f4cb37fe640 1 -- 172.21.15.110:0/4201321234 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb40b3fb0 con 0x7f4cb40a96f0 2024-01-20T11:17:28.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.737+0000 7f4cb37fe640 1 --2- 172.21.15.110:0/4201321234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cb40a96f0 0x7f4cb413f4d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4074d70 tx=0x7f4ca402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa84ce6d403e08d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:28.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 1 -- 172.21.15.110:0/4201321234 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4031870 con 0x7f4cb40a96f0 2024-01-20T11:17:28.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:28.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:28.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 10 monclient: dump: 2024-01-20T11:17:28.760 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:28.760 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.761 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.761 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.761 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:28.761 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.761 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:28.761 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:28.762 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:28.762 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:28.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:28.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:28.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 1 -- 172.21.15.110:0/4201321234 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca40319d0 con 0x7f4cb40a96f0 2024-01-20T11:17:28.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:28.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 1 -- 172.21.15.110:0/4201321234 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4031d60 con 0x7f4cb40a96f0 2024-01-20T11:17:28.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:28.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:28.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb2ffd640 10 monclient: dump: 2024-01-20T11:17:28.763 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:28.764 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.764 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.764 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.764 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:28.764 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.764 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:28.765 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:28.765 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:28.765 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb9ccf640 5 monclient: authenticate success, global_id 4533 2024-01-20T11:17:28.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb9ccf640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:28.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.741+0000 7f4cb9ccf640 4 set_mon_vals no callback set 2024-01-20T11:17:28.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 10 monclient: shutdown 2024-01-20T11:17:28.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:28.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/4201321234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cb40a96f0 msgr2=0x7f4cb413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 1 --2- 172.21.15.110:0/4201321234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cb40a96f0 0x7f4cb413f4d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4074d70 tx=0x7f4ca402b910 comp rx=0 tx=0).stop 2024-01-20T11:17:28.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/4201321234 shutdown_connections 2024-01-20T11:17:28.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 1 --2- 172.21.15.110:0/4201321234 >> v2:172.21.15.134:3300/0 conn(0x7f4cb40a9ad0 0x7f4cb4142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 1 --2- 172.21.15.110:0/4201321234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cb40a96f0 0x7f4cb413f4d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/4201321234 >> 172.21.15.110:0/4201321234 conn(0x7f4cb40b7bc0 msgr2=0x7f4cb40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:28.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/4201321234 shutdown_connections 2024-01-20T11:17:28.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.742+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/4201321234 wait complete. 2024-01-20T11:17:28.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.743+0000 7f4cb9ccf640 10 monclient: build_initial_monmap 2024-01-20T11:17:28.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.743+0000 7f4cb9ccf640 10 monclient: monmap: 2024-01-20T11:17:28.768 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:28.768 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:28.768 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:28.768 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:28.768 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:28.768 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.769 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:28.769 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:28.769 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:28.769 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.743+0000 7f4cb9ccf640 1 librados: starting msgr at 2024-01-20T11:17:28.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.743+0000 7f4cb9ccf640 1 librados: starting objecter 2024-01-20T11:17:28.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.743+0000 7f4cb9ccf640 1 Processor -- start 2024-01-20T11:17:28.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.743+0000 7f4cb9ccf640 1 -- start start 2024-01-20T11:17:28.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.743+0000 7f4cb9ccf640 1 librados: setting wanted keys 2024-01-20T11:17:28.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.743+0000 7f4cb9ccf640 1 librados: calling monclient init 2024-01-20T11:17:28.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.743+0000 7f4cb9ccf640 10 monclient: init 2024-01-20T11:17:28.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:28.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cb40a96f0 0x7f4cb413acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb40a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cb40a9ad0 0x7f4cb4138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb40a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cb413b2f0 0x7f4cb4138740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb413b2f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb40abf80 con 0x7f4cb413b2f0 2024-01-20T11:17:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4138c80 con 0x7f4cb40a9ad0 2024-01-20T11:17:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb413b860 con 0x7f4cb40a96f0 2024-01-20T11:17:28.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:28.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb9ccf640 10 monclient(hunting): authenticate will time out at 1492.094428s 2024-01-20T11:17:28.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4cb37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cb40a96f0 0x7f4cb413acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:28.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.744+0000 7f4caaffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cb40a9ad0 0x7f4cb4138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:28.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4caaffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cb40a9ad0 0x7f4cb4138200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:53192/0 (socket says 172.21.15.110:53192) 2024-01-20T11:17:28.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4caaffd640 1 -- 172.21.15.110:0/2850408206 learned_addr learned my addr 172.21.15.110:0/2850408206 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:28.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4caaffd640 10 monclient(hunting): get_auth_request con 0x7f4cb40a9ad0 auth_method 0 2024-01-20T11:17:28.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4caaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:28.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4caaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:28.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:28.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4cb37fe640 10 monclient(hunting): get_auth_request con 0x7f4cb40a96f0 auth_method 0 2024-01-20T11:17:28.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4cb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:28.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4cb37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:28.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4cb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:28.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4caaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:28.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4caaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:28.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4caaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:28.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4cb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:28.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4cb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:28.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.745+0000 7f4cb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:28.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 10 monclient(hunting): handle_auth_done global_id 4594 payload 824 2024-01-20T11:17:28.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 1 -- 172.21.15.110:0/2850408206 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cb40a96f0 msgr2=0x7f4cb413acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cb40a96f0 0x7f4cb413acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 1 -- 172.21.15.110:0/2850408206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cb413b2f0 msgr2=0x7f4cb4138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:28.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cb413b2f0 0x7f4cb4138740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:28.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:28.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 1 monclient: found mon.noname-b 2024-01-20T11:17:28.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:28.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:28.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.748+0000 7f4caaffd640 1 -- 172.21.15.110:0/2850408206 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca40094d0 con 0x7f4cb40a9ad0 2024-01-20T11:17:28.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.750+0000 7f4caaffd640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cb40a9ad0 0x7f4cb4138200 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00c3c0 tx=0x7f4c9c00c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:28.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.750+0000 7f4cb0ff9640 10 client.?.objecter ms_handle_connect 0x7f4cb40a9ad0 2024-01-20T11:17:28.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.750+0000 7f4cb0ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:28.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 1 -- 172.21.15.110:0/2850408206 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0052d0 con 0x7f4cb40a9ad0 2024-01-20T11:17:28.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:28.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:28.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 10 monclient: dump: 2024-01-20T11:17:28.779 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:28.779 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.779 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.779 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.779 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:28.779 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.780 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:28.780 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:28.780 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:28.780 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 10 monclient: _finish_auth 0 2024-01-20T11:17:28.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:17:28.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:28.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 1 -- 172.21.15.110:0/2850408206 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c005430 con 0x7f4cb40a9ad0 2024-01-20T11:17:28.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:28.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 1 -- 172.21.15.110:0/2850408206 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c007260 con 0x7f4cb40a9ad0 2024-01-20T11:17:28.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb9ccf640 5 monclient: authenticate success, global_id 4594 2024-01-20T11:17:28.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:28.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:28.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb0ff9640 10 monclient: dump: 2024-01-20T11:17:28.782 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:28.782 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:28.782 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.782 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:28.783 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:28.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb9ccf640 10 monclient: _renew_subs 2024-01-20T11:17:28.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb27fc640 4 set_mon_vals no callback set 2024-01-20T11:17:28.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb9ccf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:28.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb40ac160 con 0x7f4cb40a9ad0 2024-01-20T11:17:28.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb9ccf640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:28.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb9ccf640 10 monclient: _renew_subs 2024-01-20T11:17:28.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb9ccf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:28.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb40ac720 con 0x7f4cb40a9ad0 2024-01-20T11:17:28.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.755+0000 7f4cb9ccf640 1 librados: init done 2024-01-20T11:17:28.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.756+0000 7f4cb9ccf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:28.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.756+0000 7f4cb9ccf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:28.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.756+0000 7f4cb9ccf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:28.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:28.756+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb413b9f0 con 0x7f4cb40a9ad0 2024-01-20T11:17:29.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.004+0000 7f4cb0ff9640 1 -- 172.21.15.110:0/2850408206 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4c9c0073c0 con 0x7f4cb40a9ad0 2024-01-20T11:17:29.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.004+0000 7f4cb0ff9640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9406c8b0 0x7f4c9406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.005+0000 7f4cb37fe640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9406c8b0 0x7f4c9406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.005+0000 7f4cb37fe640 10 monclient: get_auth_request con 0x7f4c9406c8b0 auth_method 0 2024-01-20T11:17:29.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.005+0000 7f4cb0ff9640 10 client.4594.objecter ms_handle_connect 0x7f4c9406c8b0 2024-01-20T11:17:29.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.005+0000 7f4cb37fe640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9406c8b0 0x7f4c9406ed70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4036000 tx=0x7f4ca403c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:29.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.007+0000 7f4cb0ff9640 1 -- 172.21.15.110:0/2850408206 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4c9c007670 con 0x7f4cb40a9ad0 2024-01-20T11:17:29.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.007+0000 7f4cb0ff9640 10 client.4594.objecter ms_dispatch 0x7f4cb4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:29.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.007+0000 7f4cb0ff9640 3 client.4594.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:29.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.007+0000 7f4cb0ff9640 3 client.4594.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:29.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.007+0000 7f4cb0ff9640 20 client.4594.objecter dump_active .. 0 homeless 2024-01-20T11:17:29.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.017+0000 7f4cb0ff9640 1 -- 172.21.15.110:0/2850408206 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4c9c053c50 con 0x7f4cb40a9ad0 2024-01-20T11:17:29.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.017+0000 7f4cb0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:29.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.017+0000 7f4cb0ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:29.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.183+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4cb4075490 con 0x7f4c9406c8b0 2024-01-20T11:17:29.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.186+0000 7f4cb0ff9640 1 -- 172.21.15.110:0/2850408206 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64124 (secure 0 0 0) 0x7f4cb4075490 con 0x7f4c9406c8b0 2024-01-20T11:17:29.186 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:29.187 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:29.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 10 librados: watch_flush enter 2024-01-20T11:17:29.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 10 librados: watch_flush exit 2024-01-20T11:17:29.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 20 client.4594.objecter shutdown clearing up homeless session... 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 10 client.4594.objecter successfully canceled tick 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9406c8b0 msgr2=0x7f4c9406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9406c8b0 0x7f4c9406ed70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4036000 tx=0x7f4ca403c040 comp rx=0 tx=0).stop 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 10 monclient: shutdown 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cb40a9ad0 msgr2=0x7f4cb4138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cb40a9ad0 0x7f4cb4138200 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00c3c0 tx=0x7f4c9c00c7c0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 shutdown_connections 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9406c8b0 0x7f4c9406ed70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cb413b2f0 0x7f4cb4138740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cb40a9ad0 0x7f4cb4138200 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 --2- 172.21.15.110:0/2850408206 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cb40a96f0 0x7f4cb413acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 >> 172.21.15.110:0/2850408206 conn(0x7f4cb40b7bc0 msgr2=0x7f4cb4137310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:29.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 shutdown_connections 2024-01-20T11:17:29.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 -- 172.21.15.110:0/2850408206 wait complete. 2024-01-20T11:17:29.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.188+0000 7f4cb9ccf640 1 librados: shutdown 2024-01-20T11:17:29.198 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-01-20T11:17:28.737107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002026"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.9640000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.0899999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.137},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.3089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.087},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.8889999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.5660000000000007},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":16.152000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.194000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":10.539}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.903},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.277000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.1760000000000002},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4760000000000009}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.7080000000000002},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.225}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.228},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.045}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":17.696000000000002},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.348000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.789000000000001},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":18.335000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.667999999999999},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":19.251000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313248,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4249999999999998},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.7130000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7409999999999997},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.099}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":8.4540000000000006},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.489000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.478999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.1259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":10.401999999999999},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":15.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":16.216000000000001},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.728999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.8250000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.0499999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.7629999999999999},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.4550000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.954000000000001},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.459}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.571},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.688000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.265000000000001},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.1289999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.6739999999999995},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":16.34}]}]},{"osd":1,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":10.922000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":5.7359999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.6879999999999997},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.194000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":15.125999999999999},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.3099999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":10.188000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.0589999999999993},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":11.563000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.5079999999999991},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.814}]}]},{"osd":0,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":4.5510000000000002},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":6.7800000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.3970000000000002},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.0739999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":9.4269999999999996},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.606999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.555999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7710000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.151999999999999},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":14.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.092000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.074999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509471,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":11.795999999999999},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.569}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.565},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":7.2370000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.183},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.2620000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.966000000000001},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.888999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.222},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.087}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.058},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":17.782}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:29.200 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:29.200 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:29.200 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:29.200 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.002387+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.002387+0000', 'last_peered': '2024-01-20T11:16:53.002387+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.002387+0000', 'last_undegraded': '2024-01-20T11:16:53.002387+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:29.200 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:29.200 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.858298+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.858298+0000', 'last_peered': '2024-01-20T11:16:52.858298+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.858298+0000', 'last_undegraded': '2024-01-20T11:16:52.858298+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:29.201 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:29.201 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:29.201 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:29.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.298+0000 7f945d3a9640 10 monclient: get_monmap_and_config 2024-01-20T11:17:29.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.298+0000 7f945d3a9640 10 monclient: build_initial_monmap 2024-01-20T11:17:29.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.298+0000 7f945d3a9640 10 monclient: monmap: 2024-01-20T11:17:29.298 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:29.298 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:29.298 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:29.298990+0000 2024-01-20T11:17:29.298 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:29.298990+0000 2024-01-20T11:17:29.299 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:29.299 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.299 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:29.299 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:29.299 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:29.299 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:29.299 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:29.299 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.298+0000 7f945d3a9640 1 Processor -- start 2024-01-20T11:17:29.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 1 -- start start 2024-01-20T11:17:29.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient: init 2024-01-20T11:17:29.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:29.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-01-20T11:17:29.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f94580b4490 0x7f94580b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient(hunting): picked mon.noname-c con 0x7f94580b4490 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 0x7f94580af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient(hunting): picked mon.noname-b con 0x7f94580b0fb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f94580af740 0x7f945813efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient(hunting): picked mon.noname-a con 0x7f94580af740 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94580b1390 con 0x7f94580af740 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94580aec70 con 0x7f94580b4490 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945813fbc0 con 0x7f94580b0fb0 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:29.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.299+0000 7f945d3a9640 10 monclient(hunting): authenticate will time out at 1492.649874s 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f9456ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f94580b4490 0x7f94580b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94577fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f94580af740 0x7f945813efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94567fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 0x7f94580af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f9456ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f94580b4490 0x7f94580b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:53198/0 (socket says 172.21.15.110:53198) 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f9456ffd640 1 -- 172.21.15.110:0/2881424802 learned_addr learned my addr 172.21.15.110:0/2881424802 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f9456ffd640 10 monclient(hunting): get_auth_request con 0x7f94580b4490 auth_method 0 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f9456ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f9456ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f9456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94567fc640 10 monclient(hunting): get_auth_request con 0x7f94580b0fb0 auth_method 0 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94567fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94567fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:29.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94577fe640 10 monclient(hunting): get_auth_request con 0x7f94580af740 auth_method 0 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94577fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.300+0000 7f94577fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.301+0000 7f9456ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.301+0000 7f9456ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.301+0000 7f9456ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.301+0000 7f94567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.301+0000 7f94567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.301+0000 7f94567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.301+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.301+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.301+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 10 monclient(hunting): handle_auth_done global_id 4597 payload 274 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 1 -- 172.21.15.110:0/2881424802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 msgr2=0x7f94580af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:29.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 1 --2- 172.21.15.110:0/2881424802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 0x7f94580af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 1 -- 172.21.15.110:0/2881424802 >> v2:172.21.15.110:3300/0 conn(0x7f94580af740 msgr2=0x7f945813efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:29.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 1 --2- 172.21.15.110:0/2881424802 >> v2:172.21.15.110:3300/0 conn(0x7f94580af740 0x7f945813efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:29.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 1 monclient: found mon.noname-c 2024-01-20T11:17:29.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:29.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:29.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.302+0000 7f9456ffd640 1 -- 172.21.15.110:0/2881424802 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580ae900 con 0x7f94580b4490 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.304+0000 7f9456ffd640 1 --2- 172.21.15.110:0/2881424802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 0x7f94580b4870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f94580b4ef0 tx=0x7f945002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec72b7340cec55b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 1 -- 172.21.15.110:0/2881424802 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94500317e0 con 0x7f94580b4490 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 10 monclient: dump: 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.308 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 1 -- 172.21.15.110:0/2881424802 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9450031940 con 0x7f94580b4490 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 1 -- 172.21.15.110:0/2881424802 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450031cd0 con 0x7f94580b4490 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 5 monclient: authenticate success, global_id 4597 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 4 set_mon_vals no callback set 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:29.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f9455ffb640 10 monclient: dump: 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:29.310 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 10 monclient: shutdown 2024-01-20T11:17:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 1 -- 172.21.15.110:0/2881424802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 msgr2=0x7f94580b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 1 --2- 172.21.15.110:0/2881424802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 0x7f94580b4870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f94580b4ef0 tx=0x7f945002b880 comp rx=0 tx=0).stop 2024-01-20T11:17:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 1 -- 172.21.15.110:0/2881424802 shutdown_connections 2024-01-20T11:17:29.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 1 --2- 172.21.15.110:0/2881424802 >> v2:172.21.15.110:3300/0 conn(0x7f94580af740 0x7f945813efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 1 --2- 172.21.15.110:0/2881424802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 0x7f94580af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 1 --2- 172.21.15.110:0/2881424802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 0x7f94580b4870 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 1 -- 172.21.15.110:0/2881424802 >> 172.21.15.110:0/2881424802 conn(0x7f945806c150 msgr2=0x7f945806c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 1 -- 172.21.15.110:0/2881424802 shutdown_connections 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.308+0000 7f945d3a9640 1 -- 172.21.15.110:0/2881424802 wait complete. 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.309+0000 7f945d3a9640 10 monclient: build_initial_monmap 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.309+0000 7f945d3a9640 10 monclient: monmap: 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:29.315 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.309+0000 7f945d3a9640 1 librados: starting msgr at 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.309+0000 7f945d3a9640 1 librados: starting objecter 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 Processor -- start 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 -- start start 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 librados: setting wanted keys 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 librados: calling monclient init 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient: init 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94580af740 0x7f94580a9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient(hunting): picked mon.noname-a con 0x7f94580af740 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 0x7f94580aa780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient(hunting): picked mon.noname-c con 0x7f94580b0fb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:29.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 0x7f94580ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient(hunting): picked mon.noname-b con 0x7f94580b4490 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580aacc0 con 0x7f94580af740 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580aae30 con 0x7f94580b4490 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94580ae4d0 con 0x7f94580b0fb0 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.310+0000 7f945d3a9640 10 monclient(hunting): authenticate will time out at 1492.660831s 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f9456ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94580af740 0x7f94580a9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 0x7f94580ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94567fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 0x7f94580aa780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 0x7f94580ad7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:53202/0 (socket says 172.21.15.110:53202) 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 1 -- 172.21.15.110:0/4239444065 learned_addr learned my addr 172.21.15.110:0/4239444065 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 10 monclient(hunting): get_auth_request con 0x7f94580b4490 auth_method 0 2024-01-20T11:17:29.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94567fc640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 0x7f94580aa780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46858/0 (socket says 172.21.15.110:46858) 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94567fc640 10 monclient(hunting): get_auth_request con 0x7f94580b0fb0 auth_method 0 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94567fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94567fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f9456ffd640 10 monclient(hunting): get_auth_request con 0x7f94580af740 auth_method 0 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f9456ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f9456ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f9456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.311+0000 7f94577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.312+0000 7f94567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:29.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.312+0000 7f94567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.312+0000 7f94567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.312+0000 7f9456ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.312+0000 7f9456ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.312+0000 7f9456ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 10 monclient(hunting): handle_auth_done global_id 4600 payload 824 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 1 -- 172.21.15.110:0/4239444065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 msgr2=0x7f94580aa780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 0x7f94580aa780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 1 -- 172.21.15.110:0/4239444065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94580af740 msgr2=0x7f94580a9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94580af740 0x7f94580a9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 1 monclient: found mon.noname-b 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:29.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:29.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.314+0000 7f94577fe640 1 -- 172.21.15.110:0/4239444065 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94500094d0 con 0x7f94580b4490 2024-01-20T11:17:29.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.316+0000 7f94577fe640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 0x7f94580ad7e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f944800efc0 tx=0x7f944800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:29.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.316+0000 7f943f7fe640 10 client.?.objecter ms_handle_connect 0x7f94580b4490 2024-01-20T11:17:29.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.316+0000 7f943f7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:29.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 1 -- 172.21.15.110:0/4239444065 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448014480 con 0x7f94580b4490 2024-01-20T11:17:29.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 10 monclient: dump: 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:29.322 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 1 -- 172.21.15.110:0/4239444065 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944800f040 con 0x7f94580b4490 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f945d3a9640 5 monclient: authenticate success, global_id 4600 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f945d3a9640 10 monclient: _renew_subs 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f945d3a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94580add20 con 0x7f94580b4490 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 1 -- 172.21.15.110:0/4239444065 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448014820 con 0x7f94580b4490 2024-01-20T11:17:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f943f7fe640 10 monclient: dump: 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:29.324 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f945d3a9640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f945d3a9640 10 monclient: _renew_subs 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f945d3a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94580ae230 con 0x7f94580b4490 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f945d3a9640 1 librados: init done 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.320+0000 7f9454ff9640 4 set_mon_vals no callback set 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.321+0000 7f945d3a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.321+0000 7f945d3a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.321+0000 7f945d3a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:29.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.321+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94580ac2c0 con 0x7f94580b4490 2024-01-20T11:17:29.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.340+0000 7f943f7fe640 1 -- 172.21.15.110:0/4239444065 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9448014980 con 0x7f94580b4490 2024-01-20T11:17:29.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.340+0000 7f943f7fe640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f944406c8b0 0x7f944406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.341+0000 7f9456ffd640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f944406c8b0 0x7f944406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.341+0000 7f9456ffd640 10 monclient: get_auth_request con 0x7f944406c8b0 auth_method 0 2024-01-20T11:17:29.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.341+0000 7f9456ffd640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f944406c8b0 0x7f944406ed70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9450036000 tx=0x7f945003d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:29.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.341+0000 7f943f7fe640 10 client.4600.objecter ms_handle_connect 0x7f944406c8b0 2024-01-20T11:17:29.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.577+0000 7f943f7fe640 1 -- 172.21.15.110:0/4239444065 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9448059300 con 0x7f94580b4490 2024-01-20T11:17:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.577+0000 7f943f7fe640 10 client.4600.objecter ms_dispatch 0x7f9458000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.577+0000 7f943f7fe640 3 client.4600.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.577+0000 7f943f7fe640 3 client.4600.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:29.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.577+0000 7f943f7fe640 20 client.4600.objecter dump_active .. 0 homeless 2024-01-20T11:17:29.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.588+0000 7f943f7fe640 1 -- 172.21.15.110:0/4239444065 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9448055640 con 0x7f94580b4490 2024-01-20T11:17:29.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.588+0000 7f943f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:29.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.588+0000 7f943f7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.841+0000 7f945d3a9640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.841+0000 7f945d3a9640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.841+0000 7f945d3a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.841+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f94580ac2c0 con 0x7f94580b4490 2024-01-20T11:17:29.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.847+0000 7f943f7fe640 1 -- 172.21.15.110:0/4239444065 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7f94580ac2c0 con 0x7f94580b4490 2024-01-20T11:17:29.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.847+0000 7f943f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:29.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.847+0000 7f943f7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:29.848 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:29.848 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":184,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:29.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 10 librados: watch_flush enter 2024-01-20T11:17:29.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 10 librados: watch_flush exit 2024-01-20T11:17:29.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 20 client.4600.objecter shutdown clearing up homeless session... 2024-01-20T11:17:29.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 10 client.4600.objecter successfully canceled tick 2024-01-20T11:17:29.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f944406c8b0 msgr2=0x7f944406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:29.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f944406c8b0 0x7f944406ed70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9450036000 tx=0x7f945003d040 comp rx=0 tx=0).stop 2024-01-20T11:17:29.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 10 monclient: shutdown 2024-01-20T11:17:29.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:29.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 msgr2=0x7f94580ad7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:29.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 0x7f94580ad7e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f944800efc0 tx=0x7f944800c490 comp rx=0 tx=0).stop 2024-01-20T11:17:29.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 shutdown_connections 2024-01-20T11:17:29.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f944406c8b0 0x7f944406ed70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94580b4490 0x7f94580ad7e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94580b0fb0 0x7f94580aa780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 --2- 172.21.15.110:0/4239444065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94580af740 0x7f94580a9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 >> 172.21.15.110:0/4239444065 conn(0x7f945806c150 msgr2=0x7f945813dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:29.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 shutdown_connections 2024-01-20T11:17:29.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 -- 172.21.15.110:0/4239444065 wait complete. 2024-01-20T11:17:29.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.849+0000 7f945d3a9640 1 librados: shutdown 2024-01-20T11:17:29.860 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:29.860 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:29.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.961+0000 7f9bcd947640 10 monclient: get_monmap_and_config 2024-01-20T11:17:29.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.961+0000 7f9bcd947640 10 monclient: build_initial_monmap 2024-01-20T11:17:29.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.961+0000 7f9bcd947640 10 monclient: monmap: 2024-01-20T11:17:29.961 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:29.961764+0000 2024-01-20T11:17:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:29.961764+0000 2024-01-20T11:17:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:29.963 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:29.963 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:29.963 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:29.963 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.961+0000 7f9bcd947640 1 Processor -- start 2024-01-20T11:17:29.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.961+0000 7f9bcd947640 1 -- start start 2024-01-20T11:17:29.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.961+0000 7f9bcd947640 10 monclient: init 2024-01-20T11:17:29.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:29.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-01-20T11:17:29.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9bc80a96f0 0x7f9bc813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 10 monclient(hunting): picked mon.noname-c con 0x7f9bc80a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9bc813fa10 0x7f9bc81423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 10 monclient(hunting): picked mon.noname-a con 0x7f9bc813fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bc80a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc80b25c0 con 0x7f9bc80a9ad0 2024-01-20T11:17:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc80aaea0 con 0x7f9bc813fa10 2024-01-20T11:17:29.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc80ab0a0 con 0x7f9bc80a96f0 2024-01-20T11:17:29.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:29.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bcd947640 10 monclient(hunting): authenticate will time out at 1493.312621s 2024-01-20T11:17:29.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.962+0000 7f9bc67fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9bc813fa10 0x7f9bc81423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc6ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9bc80a96f0 0x7f9bc813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:29.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc67fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9bc813fa10 0x7f9bc81423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43734/0 (socket says 172.21.15.110:43734) 2024-01-20T11:17:29.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc67fc640 1 -- 172.21.15.110:0/2352019930 learned_addr learned my addr 172.21.15.110:0/2352019930 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:29.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc67fc640 10 monclient(hunting): get_auth_request con 0x7f9bc813fa10 auth_method 0 2024-01-20T11:17:29.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:29.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc67fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:29.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:29.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc6ffd640 10 monclient(hunting): get_auth_request con 0x7f9bc80a96f0 auth_method 0 2024-01-20T11:17:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.963+0000 7f9bc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.964+0000 7f9bc67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.964+0000 7f9bc67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.964+0000 7f9bc67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.965+0000 7f9bc6ffd640 10 monclient(hunting): handle_auth_done global_id 4603 payload 274 2024-01-20T11:17:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.965+0000 7f9bc6ffd640 1 -- 172.21.15.110:0/2352019930 >> v2:172.21.15.110:3300/0 conn(0x7f9bc813fa10 msgr2=0x7f9bc81423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.965+0000 7f9bc6ffd640 1 --2- 172.21.15.110:0/2352019930 >> v2:172.21.15.110:3300/0 conn(0x7f9bc813fa10 0x7f9bc81423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.965+0000 7f9bc6ffd640 1 -- 172.21.15.110:0/2352019930 >> v1:172.21.15.110:6789/0 conn(0x7f9bc80a9ad0 legacy=0x7f9bc8142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:29.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.965+0000 7f9bc6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:29.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.965+0000 7f9bc6ffd640 1 monclient: found mon.noname-c 2024-01-20T11:17:29.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.965+0000 7f9bc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:29.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.965+0000 7f9bc6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:29.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.965+0000 7f9bc6ffd640 1 -- 172.21.15.110:0/2352019930 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc80b3fb0 con 0x7f9bc80a96f0 2024-01-20T11:17:29.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.967+0000 7f9bc6ffd640 1 --2- 172.21.15.110:0/2352019930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 0x7f9bc813f4d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80b5720 tx=0x7f9bb802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eab6c72160d584e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:29.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 1 -- 172.21.15.110:0/2352019930 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb802cbf0 con 0x7f9bc80a96f0 2024-01-20T11:17:29.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:29.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:29.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 10 monclient: dump: 2024-01-20T11:17:29.988 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:29.988 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:29.990 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 1 -- 172.21.15.110:0/2352019930 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb802cd50 con 0x7f9bc80a96f0 2024-01-20T11:17:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 5 monclient: authenticate success, global_id 4603 2024-01-20T11:17:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 1 -- 172.21.15.110:0/2352019930 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8034660 con 0x7f9bc80a96f0 2024-01-20T11:17:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 4 set_mon_vals no callback set 2024-01-20T11:17:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bc5ffb640 10 monclient: dump: 2024-01-20T11:17:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:29.993 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 10 monclient: shutdown 2024-01-20T11:17:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 1 -- 172.21.15.110:0/2352019930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 msgr2=0x7f9bc813f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 1 --2- 172.21.15.110:0/2352019930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 0x7f9bc813f4d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80b5720 tx=0x7f9bb802be40 comp rx=0 tx=0).stop 2024-01-20T11:17:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 1 -- 172.21.15.110:0/2352019930 shutdown_connections 2024-01-20T11:17:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 1 --2- 172.21.15.110:0/2352019930 >> v2:172.21.15.110:3300/0 conn(0x7f9bc813fa10 0x7f9bc81423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 1 --2- 172.21.15.110:0/2352019930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 0x7f9bc813f4d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.971+0000 7f9bcd947640 1 -- 172.21.15.110:0/2352019930 >> 172.21.15.110:0/2352019930 conn(0x7f9bc80b7bc0 msgr2=0x7f9bc80b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.972+0000 7f9bcd947640 1 -- 172.21.15.110:0/2352019930 shutdown_connections 2024-01-20T11:17:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.972+0000 7f9bcd947640 1 -- 172.21.15.110:0/2352019930 wait complete. 2024-01-20T11:17:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.972+0000 7f9bcd947640 10 monclient: build_initial_monmap 2024-01-20T11:17:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.972+0000 7f9bcd947640 10 monclient: monmap: 2024-01-20T11:17:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:29.996 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:29.996 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:29.996 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:29.996 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:29.996 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:29.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.972+0000 7f9bcd947640 1 librados: starting msgr at 2024-01-20T11:17:29.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.972+0000 7f9bcd947640 1 librados: starting objecter 2024-01-20T11:17:29.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.972+0000 7f9bcd947640 1 Processor -- start 2024-01-20T11:17:29.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 1 -- start start 2024-01-20T11:17:29.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 1 librados: setting wanted keys 2024-01-20T11:17:29.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 1 librados: calling monclient init 2024-01-20T11:17:29.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient: init 2024-01-20T11:17:29.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:29.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:17:29.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 0x7f9bc8133340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient(hunting): picked mon.noname-b con 0x7f9bc80a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:29.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc80a9ad0 0x7f9bc81d81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient(hunting): picked mon.noname-a con 0x7f9bc80a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:29.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc81d8720 0x7f9bc81db020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient(hunting): picked mon.noname-c con 0x7f9bc81d8720 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8133aa0 con 0x7f9bc80a9ad0 2024-01-20T11:17:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81d7cc0 con 0x7f9bc80a96f0 2024-01-20T11:17:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8134640 con 0x7f9bc81d8720 2024-01-20T11:17:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.973+0000 7f9bcd947640 10 monclient(hunting): authenticate will time out at 1493.323940s 2024-01-20T11:17:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc81d8720 0x7f9bc81db020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc6ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 0x7f9bc8133340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc67fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc80a9ad0 0x7f9bc81d81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc6ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 0x7f9bc8133340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46326/0 (socket says 172.21.15.110:46326) 2024-01-20T11:17:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc6ffd640 1 -- 172.21.15.110:0/1605231273 learned_addr learned my addr 172.21.15.110:0/1605231273 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc6ffd640 10 monclient(hunting): get_auth_request con 0x7f9bc80a96f0 auth_method 0 2024-01-20T11:17:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc77fe640 10 monclient(hunting): get_auth_request con 0x7f9bc81d8720 auth_method 0 2024-01-20T11:17:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.974+0000 7f9bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:30.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc67fc640 10 monclient(hunting): get_auth_request con 0x7f9bc80a9ad0 auth_method 0 2024-01-20T11:17:30.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:30.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc67fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:30.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:30.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:30.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:30.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.975+0000 7f9bc67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 10 monclient(hunting): handle_auth_done global_id 4606 payload 824 2024-01-20T11:17:30.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 1 -- 172.21.15.110:0/1605231273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc81d8720 msgr2=0x7f9bc81db020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:30.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc81d8720 0x7f9bc81db020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:30.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 1 -- 172.21.15.110:0/1605231273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc80a9ad0 msgr2=0x7f9bc81d81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:30.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc80a9ad0 0x7f9bc81d81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:30.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:30.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 1 monclient: found mon.noname-b 2024-01-20T11:17:30.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:30.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:30.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.977+0000 7f9bc6ffd640 1 -- 172.21.15.110:0/1605231273 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb8009a70 con 0x7f9bc80a96f0 2024-01-20T11:17:30.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.979+0000 7f9bc6ffd640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 0x7f9bc8133340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009e80 tx=0x7f9bb8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:30.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.979+0000 7f9baffff640 10 client.?.objecter ms_handle_connect 0x7f9bc80a96f0 2024-01-20T11:17:30.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.979+0000 7f9baffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:30.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 1 -- 172.21.15.110:0/1605231273 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb803e610 con 0x7f9bc80a96f0 2024-01-20T11:17:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 10 monclient: dump: 2024-01-20T11:17:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:30.007 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:30.007 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:30.007 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:30.007 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:30.007 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:30.007 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:30.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 10 monclient: _finish_auth 0 2024-01-20T11:17:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 1 -- 172.21.15.110:0/1605231273 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb803e770 con 0x7f9bc80a96f0 2024-01-20T11:17:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bcd947640 5 monclient: authenticate success, global_id 4606 2024-01-20T11:17:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bcd947640 10 monclient: _renew_subs 2024-01-20T11:17:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bcd947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc8132530 con 0x7f9bc80a96f0 2024-01-20T11:17:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 1 -- 172.21.15.110:0/1605231273 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb803eb20 con 0x7f9bc80a96f0 2024-01-20T11:17:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9baffff640 10 monclient: dump: 2024-01-20T11:17:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:30.010 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:30.010 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:30.010 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:30.010 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:30.010 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:30.010 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:30.010 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:30.011 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bc4ff9640 4 set_mon_vals no callback set 2024-01-20T11:17:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bcd947640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bcd947640 10 monclient: _renew_subs 2024-01-20T11:17:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bcd947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc813edc0 con 0x7f9bc80a96f0 2024-01-20T11:17:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.983+0000 7f9bcd947640 1 librados: init done 2024-01-20T11:17:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.984+0000 7f9bcd947640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.984+0000 7f9bcd947640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.984+0000 7f9bcd947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:29.984+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc81d7b20 con 0x7f9bc80a96f0 2024-01-20T11:17:30.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.002+0000 7f9baffff640 1 -- 172.21.15.110:0/1605231273 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9bb803b050 con 0x7f9bc80a96f0 2024-01-20T11:17:30.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.003+0000 7f9baffff640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bb406c8b0 0x7f9bb406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:30.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.003+0000 7f9bc67fc640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bb406c8b0 0x7f9bb406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:30.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.003+0000 7f9bc67fc640 10 monclient: get_auth_request con 0x7f9bb406c8b0 auth_method 0 2024-01-20T11:17:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.004+0000 7f9baffff640 10 client.4606.objecter ms_handle_connect 0x7f9bb406c8b0 2024-01-20T11:17:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.004+0000 7f9bc67fc640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bb406c8b0 0x7f9bb406ed70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80b73b0 tx=0x7f9bbc009240 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.006+0000 7f9baffff640 1 -- 172.21.15.110:0/1605231273 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9bb807aca0 con 0x7f9bc80a96f0 2024-01-20T11:17:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.006+0000 7f9baffff640 10 client.4606.objecter ms_dispatch 0x7f9bc8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.006+0000 7f9baffff640 3 client.4606.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.006+0000 7f9baffff640 3 client.4606.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.006+0000 7f9baffff640 20 client.4606.objecter dump_active .. 0 homeless 2024-01-20T11:17:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.248+0000 7f9baffff640 1 -- 172.21.15.110:0/1605231273 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9bb8076fe0 con 0x7f9bc80a96f0 2024-01-20T11:17:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.248+0000 7f9baffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:30.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.248+0000 7f9baffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:30.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.420+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bc81430f0 con 0x7f9bb406c8b0 2024-01-20T11:17:30.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.425+0000 7f9baffff640 1 -- 172.21.15.110:0/1605231273 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64124 (secure 0 0 0) 0x7f9bc81430f0 con 0x7f9bb406c8b0 2024-01-20T11:17:30.425 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:30.426 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 10 librados: watch_flush enter 2024-01-20T11:17:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 10 librados: watch_flush exit 2024-01-20T11:17:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 20 client.4606.objecter shutdown clearing up homeless session... 2024-01-20T11:17:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 10 client.4606.objecter successfully canceled tick 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bb406c8b0 msgr2=0x7f9bb406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bb406c8b0 0x7f9bb406ed70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80b73b0 tx=0x7f9bbc009240 comp rx=0 tx=0).stop 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 10 monclient: shutdown 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 msgr2=0x7f9bc8133340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 0x7f9bc8133340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009e80 tx=0x7f9bb8004a40 comp rx=0 tx=0).stop 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 shutdown_connections 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bb406c8b0 0x7f9bb406ed70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc81d8720 0x7f9bc81db020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc80a9ad0 0x7f9bc81d81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 --2- 172.21.15.110:0/1605231273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bc80a96f0 0x7f9bc8133340 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 >> 172.21.15.110:0/1605231273 conn(0x7f9bc80b7bc0 msgr2=0x7f9bc80b68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 shutdown_connections 2024-01-20T11:17:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 -- 172.21.15.110:0/1605231273 wait complete. 2024-01-20T11:17:30.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:30.427+0000 7f9bcd947640 1 librados: shutdown 2024-01-20T11:17:30.438 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-01-20T11:17:28.737107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002026"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.858298+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.858298+0000","last_peered":"2024-01-20T11:16:52.858298+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.858298+0000","last_undegraded":"2024-01-20T11:16:52.858298+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.9640000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.0899999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.137},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.3089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.087},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.8889999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.5660000000000007},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":16.152000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.194000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":10.539}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.903},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.277000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.1760000000000002},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4760000000000009}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.7080000000000002},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.225}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.228},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.045}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":17.696000000000002},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.348000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"min":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":91.667000000000002,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.789000000000001},{"interface":"front","average":{"1min":18.317,"5min":18.317,"15min":18.317},"min":{"1min":8.6799999999999997,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":92.613,"5min":92.613,"15min":92.613},"last":18.335000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.667999999999999},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":19.251000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313248,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.425999999999998,"5min":17.425999999999998,"15min":17.425999999999998},"min":{"1min":4.992,"5min":4.992,"15min":4.992},"max":{"1min":155.27799999999999,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4249999999999998},{"interface":"front","average":{"1min":15.284000000000001,"5min":15.284000000000001,"15min":15.284000000000001},"min":{"1min":6.165,"5min":6.165,"15min":6.165},"max":{"1min":144.16499999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.7130000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":16.623000000000001,"5min":16.623000000000001,"15min":16.623000000000001},"min":{"1min":6.6879999999999997,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":149.55000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7409999999999997},{"interface":"front","average":{"1min":20.513000000000002,"5min":20.513000000000002,"15min":20.513000000000002},"min":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":155.91200000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.099}]},{"osd":2,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"min":{"1min":7.3949999999999996,"5min":7.3949999999999996,"15min":7.3949999999999996},"max":{"1min":145.553,"5min":145.553,"15min":145.553},"last":8.4540000000000006},{"interface":"front","average":{"1min":19.026,"5min":19.026,"15min":19.026},"min":{"1min":8.9109999999999996,"5min":8.9109999999999996,"15min":8.9109999999999996},"max":{"1min":147.66300000000001,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.489000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":27.265999999999998,"5min":27.265999999999998,"15min":27.265999999999998},"min":{"1min":9.125,"5min":9.125,"15min":9.125},"max":{"1min":284.00700000000001,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.478999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":28.222000000000001,"15min":28.222000000000001},"min":{"1min":9.1590000000000007,"5min":9.1590000000000007,"15min":9.1590000000000007},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.1259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":21.652000000000001,"5min":21.652000000000001,"15min":21.652000000000001},"min":{"1min":8.5250000000000004,"5min":8.5250000000000004,"15min":8.5250000000000004},"max":{"1min":146.744,"5min":146.744,"15min":146.744},"last":10.401999999999999},{"interface":"front","average":{"1min":22.077999999999999,"5min":22.077999999999999,"15min":22.077999999999999},"min":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":150.542,"5min":150.542,"15min":150.542},"last":15.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":24.161000000000001,"5min":24.161000000000001,"15min":24.161000000000001},"min":{"1min":11.300000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":159.179,"5min":159.179,"15min":159.179},"last":16.216000000000001},{"interface":"front","average":{"1min":24.43,"5min":24.43,"15min":24.43},"min":{"1min":11.234999999999999,"5min":11.234999999999999,"15min":11.234999999999999},"max":{"1min":157.35599999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.728999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.8250000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.0499999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.7629999999999999},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.4550000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.954000000000001},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.459}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.571},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.688000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.265000000000001},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.1289999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.6739999999999995},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":16.34}]}]},{"osd":1,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":10.922000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":5.7359999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.6879999999999997},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.194000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":15.125999999999999},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.3099999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":10.188000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.0589999999999993},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":11.563000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.5079999999999991},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.814}]}]},{"osd":0,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":4.5510000000000002},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":6.7800000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.3970000000000002},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.0739999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":9.4269999999999996},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.606999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.555999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7710000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.151999999999999},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":14.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.092000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.074999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509471,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":11.795999999999999},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.569}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.565},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":7.2370000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.183},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.2620000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.966000000000001},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.888999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:26 2024","interfaces":[{"interface":"back","average":{"1min":17.469000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"min":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":70.254999999999995,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.222},{"interface":"front","average":{"1min":17.512,"5min":17.512,"15min":17.512},"min":{"1min":8.016,"5min":8.016,"15min":8.016},"max":{"1min":72.670000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.087}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.058},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":17.782}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:33.440 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.541+0000 7f8223bdc640 10 monclient: get_monmap_and_config 2024-01-20T11:17:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.541+0000 7f8223bdc640 10 monclient: build_initial_monmap 2024-01-20T11:17:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.541+0000 7f8223bdc640 10 monclient: monmap: 2024-01-20T11:17:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:33.541 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:33.542020+0000 2024-01-20T11:17:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:33.542020+0000 2024-01-20T11:17:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:33.542 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:33.542 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:33.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.541+0000 7f8223bdc640 1 Processor -- start 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 1 -- start start 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient: init 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient(hunting): picked mon.noname-b con 0x7f821c071c60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f821c0ac540 0x7f821c0ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient(hunting): picked mon.noname-c con 0x7f821c0ac540 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f821c0b0930 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f821c0ad610 con 0x7f821c0b0930 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:33.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821c072040 con 0x7f821c0ac540 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821c1426c0 con 0x7f821c071c60 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.542+0000 7f8223bdc640 10 monclient(hunting): authenticate will time out at 1496.893020s 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.543+0000 7f8221978640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.543+0000 7f8221177640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f821c0ac540 0x7f821c0ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.543+0000 7f8221978640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c142180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46894/0 (socket says 172.21.15.110:46894) 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.543+0000 7f8221978640 1 -- 172.21.15.110:0/1119802978 learned_addr learned my addr 172.21.15.110:0/1119802978 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.543+0000 7f8221978640 10 monclient(hunting): get_auth_request con 0x7f821c071c60 auth_method 0 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.543+0000 7f8221978640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.543+0000 7f8221978640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.543+0000 7f8221978640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:33.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.544+0000 7f8221177640 10 monclient(hunting): get_auth_request con 0x7f821c0ac540 auth_method 0 2024-01-20T11:17:33.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.544+0000 7f8221177640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:33.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.544+0000 7f8221177640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:33.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.544+0000 7f8221177640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:33.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.544+0000 7f8221978640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:33.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.544+0000 7f8221978640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:33.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.544+0000 7f8221978640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.545+0000 7f8221177640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.545+0000 7f8221177640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.545+0000 7f8221177640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.546+0000 7f8221978640 10 monclient(hunting): handle_auth_done global_id 4433 payload 274 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.546+0000 7f8221978640 1 -- 172.21.15.110:0/1119802978 >> v2:172.21.15.134:3300/0 conn(0x7f821c0ac540 msgr2=0x7f821c0ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.546+0000 7f8221978640 1 --2- 172.21.15.110:0/1119802978 >> v2:172.21.15.134:3300/0 conn(0x7f821c0ac540 0x7f821c0ab4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.546+0000 7f8221978640 1 -- 172.21.15.110:0/1119802978 >> v1:172.21.15.134:6789/0 conn(0x7f821c0b0930 legacy=0x7f821c0af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.546+0000 7f8221978640 10 monclient: _finish_hunting 0 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.546+0000 7f8221978640 1 monclient: found mon.noname-b 2024-01-20T11:17:33.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.546+0000 7f8221978640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.546+0000 7f8221978640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.546+0000 7f8221978640 1 -- 172.21.15.110:0/1119802978 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c0aba00 con 0x7f821c071c60 2024-01-20T11:17:33.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.548+0000 7f8221978640 1 --2- 172.21.15.110:0/1119802978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c142180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8218009870 tx=0x7f821802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f3029f44964f1e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 1 -- 172.21.15.110:0/1119802978 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8218030840 con 0x7f821c071c60 2024-01-20T11:17:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 10 monclient: dump: 2024-01-20T11:17:33.552 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 10 monclient: _finish_auth 0 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 10 monclient: _check_auth_tickets 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 1 -- 172.21.15.110:0/1119802978 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82180309a0 con 0x7f821c071c60 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:33.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 1 -- 172.21.15.110:0/1119802978 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8218030d30 con 0x7f821c071c60 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8223bdc640 5 monclient: authenticate success, global_id 4433 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8223bdc640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8223bdc640 4 set_mon_vals no callback set 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.551+0000 7f8220976640 10 monclient: dump: 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:33.554 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 10 monclient: shutdown 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 1 -- 172.21.15.110:0/1119802978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 msgr2=0x7f821c142180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 1 --2- 172.21.15.110:0/1119802978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c142180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8218009870 tx=0x7f821802b980 comp rx=0 tx=0).stop 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 1 -- 172.21.15.110:0/1119802978 shutdown_connections 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 1 --2- 172.21.15.110:0/1119802978 >> v2:172.21.15.134:3300/0 conn(0x7f821c0ac540 0x7f821c0ab4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 1 --2- 172.21.15.110:0/1119802978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c142180 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 1 -- 172.21.15.110:0/1119802978 >> 172.21.15.110:0/1119802978 conn(0x7f821c0b3e60 msgr2=0x7f821c0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 1 -- 172.21.15.110:0/1119802978 shutdown_connections 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.552+0000 7f8223bdc640 1 -- 172.21.15.110:0/1119802978 wait complete. 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.553+0000 7f8223bdc640 10 monclient: build_initial_monmap 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.553+0000 7f8223bdc640 10 monclient: monmap: 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:33.555 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:33.556 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:33.556 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.553+0000 7f8223bdc640 1 librados: starting msgr at 2024-01-20T11:17:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.553+0000 7f8223bdc640 1 librados: starting objecter 2024-01-20T11:17:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.553+0000 7f8223bdc640 1 Processor -- start 2024-01-20T11:17:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.553+0000 7f8223bdc640 1 -- start start 2024-01-20T11:17:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.553+0000 7f8223bdc640 1 librados: setting wanted keys 2024-01-20T11:17:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.553+0000 7f8223bdc640 1 librados: calling monclient init 2024-01-20T11:17:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.553+0000 7f8223bdc640 10 monclient: init 2024-01-20T11:17:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:33.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient(hunting): picked mon.noname-c con 0x7f821c071c60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f821c0ac540 0x7f821c131800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient(hunting): picked mon.noname-a con 0x7f821c0ac540 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c0b0930 0x7f821c06ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient(hunting): picked mon.noname-b con 0x7f821c0b0930 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c131d40 con 0x7f821c0ac540 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c06e600 con 0x7f821c0b0930 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821c06fd00 con 0x7f821c071c60 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:33.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8223bdc640 10 monclient(hunting): authenticate will time out at 1496.904309s 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8221978640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8221177640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f821c0ac540 0x7f821c131800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8222179640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c0b0930 0x7f821c06ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8221978640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46902/0 (socket says 172.21.15.110:46902) 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8221978640 1 -- 172.21.15.110:0/351711694 learned_addr learned my addr 172.21.15.110:0/351711694 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8221978640 10 monclient(hunting): get_auth_request con 0x7f821c071c60 auth_method 0 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8221978640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8221978640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.554+0000 7f8221978640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221177640 10 monclient(hunting): get_auth_request con 0x7f821c0ac540 auth_method 0 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221177640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221177640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221177640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8222179640 10 monclient(hunting): get_auth_request con 0x7f821c0b0930 auth_method 0 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8222179640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:33.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8222179640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8222179640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221978640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221978640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221978640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8222179640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8222179640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8222179640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221177640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221177640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.555+0000 7f8221177640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 10 monclient(hunting): handle_auth_done global_id 4615 payload 824 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 1 -- 172.21.15.110:0/351711694 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 msgr2=0x7f821c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c06ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 1 -- 172.21.15.110:0/351711694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f821c0ac540 msgr2=0x7f821c131800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:33.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f821c0ac540 0x7f821c131800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 10 monclient: _finish_hunting 0 2024-01-20T11:17:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 1 monclient: found mon.noname-b 2024-01-20T11:17:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:33.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.558+0000 7f8222179640 1 -- 172.21.15.110:0/351711694 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82180094d0 con 0x7f821c0b0930 2024-01-20T11:17:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.560+0000 7f8222179640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c0b0930 0x7f821c06ea90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f821400c730 tx=0x7f821400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.560+0000 7f82127fc640 10 client.?.objecter ms_handle_connect 0x7f821c0b0930 2024-01-20T11:17:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.560+0000 7f82127fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:33.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 1 -- 172.21.15.110:0/351711694 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214003b70 con 0x7f821c0b0930 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 10 monclient: dump: 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:33.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 10 monclient: _finish_auth 0 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 10 monclient: _check_auth_tickets 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 1 -- 172.21.15.110:0/351711694 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8214003cd0 con 0x7f821c0b0930 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 1 -- 172.21.15.110:0/351711694 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214005310 con 0x7f821c0b0930 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f8223bdc640 5 monclient: authenticate success, global_id 4615 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f8223bdc640 10 monclient: _renew_subs 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f8223bdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821c06ffe0 con 0x7f821c0b0930 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f8213fff640 4 set_mon_vals no callback set 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f82127fc640 10 monclient: dump: 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:33.566 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f8223bdc640 10 client.4615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.563+0000 7f8223bdc640 10 monclient: _renew_subs 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.564+0000 7f8223bdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.564+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821c070cf0 con 0x7f821c0b0930 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.564+0000 7f8223bdc640 1 librados: init done 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.564+0000 7f8223bdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.564+0000 7f8223bdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:33.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.564+0000 7f8223bdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:33.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.564+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821c06e790 con 0x7f821c0b0930 2024-01-20T11:17:33.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.582+0000 7f82127fc640 1 -- 172.21.15.110:0/351711694 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8214005680 con 0x7f821c0b0930 2024-01-20T11:17:33.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.583+0000 7f82127fc640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f820006c8b0 0x7f820006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:33.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.583+0000 7f8221978640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f820006c8b0 0x7f820006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:33.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.583+0000 7f8221978640 10 monclient: get_auth_request con 0x7f820006c8b0 auth_method 0 2024-01-20T11:17:33.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.583+0000 7f82127fc640 10 client.4615.objecter ms_handle_connect 0x7f820006c8b0 2024-01-20T11:17:33.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.583+0000 7f8221978640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f820006c8b0 0x7f820006ed70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f821802b9c0 tx=0x7f8218008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:33.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.586+0000 7f82127fc640 1 -- 172.21.15.110:0/351711694 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8214058ab0 con 0x7f821c0b0930 2024-01-20T11:17:33.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.586+0000 7f82127fc640 10 client.4615.objecter ms_dispatch 0x7f821c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:33.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.586+0000 7f82127fc640 3 client.4615.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:33.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.586+0000 7f82127fc640 3 client.4615.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:33.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.586+0000 7f82127fc640 20 client.4615.objecter dump_active .. 0 homeless 2024-01-20T11:17:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.820+0000 7f82127fc640 1 -- 172.21.15.110:0/351711694 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8214054df0 con 0x7f821c0b0930 2024-01-20T11:17:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.820+0000 7f82127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.820+0000 7f82127fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:33.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.987+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f821c0701d0 con 0x7f820006c8b0 2024-01-20T11:17:33.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.989+0000 7f82127fc640 1 -- 172.21.15.110:0/351711694 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64127 (secure 0 0 0) 0x7f821c0701d0 con 0x7f820006c8b0 2024-01-20T11:17:33.990 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:33.990 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:33.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.991+0000 7f8223bdc640 10 librados: watch_flush enter 2024-01-20T11:17:33.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.991+0000 7f8223bdc640 10 librados: watch_flush exit 2024-01-20T11:17:33.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.991+0000 7f8223bdc640 20 client.4615.objecter shutdown clearing up homeless session... 2024-01-20T11:17:33.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.991+0000 7f8223bdc640 10 client.4615.objecter successfully canceled tick 2024-01-20T11:17:33.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f820006c8b0 msgr2=0x7f820006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f820006c8b0 0x7f820006ed70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f821802b9c0 tx=0x7f8218008030 comp rx=0 tx=0).stop 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 10 monclient: shutdown 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c0b0930 msgr2=0x7f821c06ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c0b0930 0x7f821c06ea90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f821400c730 tx=0x7f821400cbf0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 shutdown_connections 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f820006c8b0 0x7f820006ed70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f821c0b0930 0x7f821c06ea90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f821c0ac540 0x7f821c131800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 --2- 172.21.15.110:0/351711694 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f821c071c60 0x7f821c06ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 >> 172.21.15.110:0/351711694 conn(0x7f821c0b3e60 msgr2=0x7f821c1361b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 shutdown_connections 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 -- 172.21.15.110:0/351711694 wait complete. 2024-01-20T11:17:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:33.992+0000 7f8223bdc640 1 librados: shutdown 2024-01-20T11:17:34.002 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-01-20T11:17:32.737859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002146"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.9640000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.0899999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.137},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.3089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.087},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.8889999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.5660000000000007},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":16.152000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.194000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":10.539}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.903},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.277000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313248,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.0169999999999995},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.3360000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.75},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":14.817}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.512},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":11.491}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":18.474},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.255000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.113},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":13.35}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":18.879999999999999},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.837}]}]},{"osd":3,"up_from":21,"seq":90194313249,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":13.712999999999999},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.0090000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.3360000000000003},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.393000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":5.1719999999999997},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.163}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.754},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.5380000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":7.617},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":13.327}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.816000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.023999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.8250000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.0499999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.7629999999999999},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.4550000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.954000000000001},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.459}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.571},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.688000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.265000000000001},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.1289999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.6739999999999995},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":16.34}]}]},{"osd":1,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":10.922000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":5.7359999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.6879999999999997},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.194000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":15.125999999999999},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.3099999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":10.188000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.0589999999999993},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":11.563000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.5079999999999991},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.814}]}]},{"osd":0,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":4.5510000000000002},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":6.7800000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.3970000000000002},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.0739999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":9.4269999999999996},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.606999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.555999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7710000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.151999999999999},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":14.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.092000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.074999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":7.9240000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.6020000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":15.57},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.337}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.3119999999999994},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.795999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.749000000000001},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.667}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.965},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.2949999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.382999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.346}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:34.004 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:34.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.103+0000 7ff808755640 10 monclient: get_monmap_and_config 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.103+0000 7ff808755640 10 monclient: build_initial_monmap 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.103+0000 7ff808755640 10 monclient: monmap: 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:34.103638+0000 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:34.103638+0000 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:34.103 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.103+0000 7ff808755640 1 Processor -- start 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.103+0000 7ff808755640 1 -- start start 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.103+0000 7ff808755640 10 monclient: init 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff8000b4d10 0x7ff8000b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient(hunting): picked mon.noname-a con 0x7ff8000b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8000b17c0 0x7ff8000b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient(hunting): picked mon.noname-b con 0x7ff8000b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff800145980 0x7ff800145e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient(hunting): picked mon.noname-c con 0x7ff800145980 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8000b50f0 con 0x7ff8000b4d10 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8000b2120 con 0x7ff800145980 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000aade0 con 0x7ff8000b17c0 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.104+0000 7ff808755640 10 monclient(hunting): authenticate will time out at 1497.454644s 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff8064f1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff8000b4d10 0x7ff8000b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff805cf0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8000b17c0 0x7ff8000b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff806cf2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff800145980 0x7ff800145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff8064f1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff8000b4d10 0x7ff8000b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43770/0 (socket says 172.21.15.110:43770) 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff805cf0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8000b17c0 0x7ff8000b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46908/0 (socket says 172.21.15.110:46908) 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff8064f1640 1 -- 172.21.15.110:0/2957361776 learned_addr learned my addr 172.21.15.110:0/2957361776 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff8064f1640 10 monclient(hunting): get_auth_request con 0x7ff8000b4d10 auth_method 0 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff8064f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff8064f1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff8064f1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff805cf0640 10 monclient(hunting): get_auth_request con 0x7ff8000b17c0 auth_method 0 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff805cf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff805cf0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff805cf0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff806cf2640 10 monclient(hunting): get_auth_request con 0x7ff800145980 auth_method 0 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff806cf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff806cf2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff806cf2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff8064f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.105+0000 7ff8064f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.106+0000 7ff8064f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.106+0000 7ff805cf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.106+0000 7ff805cf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.106+0000 7ff805cf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.106+0000 7ff806cf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.106+0000 7ff806cf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.106+0000 7ff806cf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 10 monclient(hunting): handle_auth_done global_id 4618 payload 274 2024-01-20T11:17:34.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 1 -- 172.21.15.110:0/2957361776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8000b17c0 msgr2=0x7ff8000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:34.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 1 --2- 172.21.15.110:0/2957361776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8000b17c0 0x7ff8000b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 1 -- 172.21.15.110:0/2957361776 >> v2:172.21.15.110:3300/0 conn(0x7ff8000b4d10 msgr2=0x7ff8000b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:34.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 1 --2- 172.21.15.110:0/2957361776 >> v2:172.21.15.110:3300/0 conn(0x7ff8000b4d10 0x7ff8000b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 10 monclient: _finish_hunting 0 2024-01-20T11:17:34.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 1 monclient: found mon.noname-c 2024-01-20T11:17:34.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:34.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:34.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff806cf2640 1 -- 172.21.15.110:0/2957361776 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800145460 con 0x7ff800145980 2024-01-20T11:17:34.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff805cf0640 1 --2- 172.21.15.110:0/2957361776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8000b17c0 0x7ff8000b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:34.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.107+0000 7ff8064f1640 1 --2- 172.21.15.110:0/2957361776 >> v2:172.21.15.110:3300/0 conn(0x7ff8000b4d10 0x7ff8000b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:34.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.109+0000 7ff806cf2640 1 --2- 172.21.15.110:0/2957361776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff800145980 0x7ff800145e10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff8000b53c0 tx=0x7ff7fc02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6083b3d54b566c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:34.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 1 -- 172.21.15.110:0/2957361776 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc02f050 con 0x7ff800145980 2024-01-20T11:17:34.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:34.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:34.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 10 monclient: dump: 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 10 monclient: _finish_auth 0 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 10 monclient: _check_auth_tickets 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 1 -- 172.21.15.110:0/2957361776 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc031700 con 0x7ff800145980 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:34.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 1 -- 172.21.15.110:0/2957361776 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc031a90 con 0x7ff800145980 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff808755640 5 monclient: authenticate success, global_id 4618 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff808755640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff808755640 4 set_mon_vals no callback set 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.112+0000 7ff8054ef640 10 monclient: dump: 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:34.114 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 10 monclient: shutdown 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 1 -- 172.21.15.110:0/2957361776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff800145980 msgr2=0x7ff800145e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 1 --2- 172.21.15.110:0/2957361776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff800145980 0x7ff800145e10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff8000b53c0 tx=0x7ff7fc02b7a0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 1 -- 172.21.15.110:0/2957361776 shutdown_connections 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 1 --2- 172.21.15.110:0/2957361776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff800145980 0x7ff800145e10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 1 --2- 172.21.15.110:0/2957361776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8000b17c0 0x7ff8000b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 1 --2- 172.21.15.110:0/2957361776 >> v2:172.21.15.110:3300/0 conn(0x7ff8000b4d10 0x7ff8000b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 1 -- 172.21.15.110:0/2957361776 >> 172.21.15.110:0/2957361776 conn(0x7ff800066590 msgr2=0x7ff8000669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 1 -- 172.21.15.110:0/2957361776 shutdown_connections 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.113+0000 7ff808755640 1 -- 172.21.15.110:0/2957361776 wait complete. 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.114+0000 7ff808755640 10 monclient: build_initial_monmap 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.114+0000 7ff808755640 10 monclient: monmap: 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:34.118 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.114+0000 7ff808755640 1 librados: starting msgr at 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.114+0000 7ff808755640 1 librados: starting objecter 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 Processor -- start 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 -- start start 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 librados: setting wanted keys 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 librados: calling monclient init 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient: init 2024-01-20T11:17:34.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8000b17c0 0x7ff8001446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient(hunting): picked mon.noname-b con 0x7ff8000b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8000b4d10 0x7ff8000acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient(hunting): picked mon.noname-a con 0x7ff8000b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff800145980 0x7ff8000adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient(hunting): picked mon.noname-c con 0x7ff800145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000ad310 con 0x7ff8000b4d10 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000aee70 con 0x7ff8000b17c0 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000af150 con 0x7ff800145980 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:34.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.115+0000 7ff808755640 10 monclient(hunting): authenticate will time out at 1497.466103s 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.116+0000 7ff8064f1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8000b17c0 0x7ff8001446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.116+0000 7ff805cf0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8000b4d10 0x7ff8000acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.116+0000 7ff806cf2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff800145980 0x7ff8000adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.116+0000 7ff8064f1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8000b17c0 0x7ff8001446c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46356/0 (socket says 172.21.15.110:46356) 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.116+0000 7ff8064f1640 1 -- 172.21.15.110:0/215165063 learned_addr learned my addr 172.21.15.110:0/215165063 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.116+0000 7ff8064f1640 10 monclient(hunting): get_auth_request con 0x7ff8000b17c0 auth_method 0 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.116+0000 7ff8064f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.116+0000 7ff8064f1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.116+0000 7ff8064f1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff806cf2640 10 monclient(hunting): get_auth_request con 0x7ff800145980 auth_method 0 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff806cf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff806cf2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff806cf2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff805cf0640 10 monclient(hunting): get_auth_request con 0x7ff8000b4d10 auth_method 0 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff805cf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff805cf0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff805cf0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff8064f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff8064f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff8064f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff806cf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff806cf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff806cf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff805cf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff805cf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.117+0000 7ff805cf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 10 monclient(hunting): handle_auth_done global_id 4621 payload 824 2024-01-20T11:17:34.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 1 -- 172.21.15.110:0/215165063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff800145980 msgr2=0x7ff8000adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:34.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff800145980 0x7ff8000adab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 1 -- 172.21.15.110:0/215165063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8000b4d10 msgr2=0x7ff8000acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:34.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8000b4d10 0x7ff8000acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 10 monclient: _finish_hunting 0 2024-01-20T11:17:34.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 1 monclient: found mon.noname-b 2024-01-20T11:17:34.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:34.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:34.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.119+0000 7ff8064f1640 1 -- 172.21.15.110:0/215165063 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f0009400 con 0x7ff8000b17c0 2024-01-20T11:17:34.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.121+0000 7ff8064f1640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8000b17c0 0x7ff8001446c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800efc0 tx=0x7ff7f800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:34.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.121+0000 7ff7f6ffd640 10 client.?.objecter ms_handle_connect 0x7ff8000b17c0 2024-01-20T11:17:34.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.121+0000 7ff7f6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:34.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.125+0000 7ff7f6ffd640 1 -- 172.21.15.110:0/215165063 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8014480 con 0x7ff8000b17c0 2024-01-20T11:17:34.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:34.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:34.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 10 monclient: dump: 2024-01-20T11:17:34.127 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:34.128 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.128 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.128 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.128 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:34.128 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.128 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:34.128 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:34.128 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:34.128 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 1 -- 172.21.15.110:0/215165063 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f800f040 con 0x7ff8000b17c0 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff808755640 5 monclient: authenticate success, global_id 4621 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff808755640 10 monclient: _renew_subs 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff808755640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f8009b30 con 0x7ff8000b17c0 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:34.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 1 -- 172.21.15.110:0/215165063 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80147e0 con 0x7ff8000b17c0 2024-01-20T11:17:34.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:34.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:34.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff7f6ffd640 10 monclient: dump: 2024-01-20T11:17:34.130 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:34.130 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.130 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.130 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.130 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:34.130 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff804cee640 4 set_mon_vals no callback set 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff808755640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff808755640 10 monclient: _renew_subs 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff808755640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc009400 con 0x7ff8000b17c0 2024-01-20T11:17:34.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.126+0000 7ff808755640 1 librados: init done 2024-01-20T11:17:34.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.127+0000 7ff808755640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:34.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.127+0000 7ff808755640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:34.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.127+0000 7ff808755640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:34.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.127+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8000ad4e0 con 0x7ff8000b17c0 2024-01-20T11:17:34.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.370+0000 7ff7f6ffd640 1 -- 172.21.15.110:0/215165063 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff7f8014940 con 0x7ff8000b17c0 2024-01-20T11:17:34.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.371+0000 7ff7f6ffd640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff7e406c8e0 0x7ff7e406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.371+0000 7ff805cf0640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff7e406c8e0 0x7ff7e406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.371+0000 7ff805cf0640 10 monclient: get_auth_request con 0x7ff7e406c8e0 auth_method 0 2024-01-20T11:17:34.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.372+0000 7ff805cf0640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff7e406c8e0 0x7ff7e406eda0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff800144420 tx=0x7ff7f0009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:34.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.372+0000 7ff7f6ffd640 10 client.4621.objecter ms_handle_connect 0x7ff7e406c8e0 2024-01-20T11:17:34.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.374+0000 7ff7f6ffd640 1 -- 172.21.15.110:0/215165063 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff7f8059430 con 0x7ff8000b17c0 2024-01-20T11:17:34.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.374+0000 7ff7f6ffd640 10 client.4621.objecter ms_dispatch 0x7ff800000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:34.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.374+0000 7ff7f6ffd640 3 client.4621.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:34.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.374+0000 7ff7f6ffd640 3 client.4621.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:34.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.374+0000 7ff7f6ffd640 20 client.4621.objecter dump_active .. 0 homeless 2024-01-20T11:17:34.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.385+0000 7ff7f6ffd640 1 -- 172.21.15.110:0/215165063 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff7f8055770 con 0x7ff8000b17c0 2024-01-20T11:17:34.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.385+0000 7ff7f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:34.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.385+0000 7ff7f6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.588+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff8000b2660 con 0x7ff7e406c8e0 2024-01-20T11:17:34.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.590+0000 7ff7f6ffd640 1 -- 172.21.15.110:0/215165063 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64127 (secure 0 0 0) 0x7ff7f0011af0 con 0x7ff7e406c8e0 2024-01-20T11:17:34.591 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:34.591 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:34.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 10 librados: watch_flush enter 2024-01-20T11:17:34.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 10 librados: watch_flush exit 2024-01-20T11:17:34.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 20 client.4621.objecter shutdown clearing up homeless session... 2024-01-20T11:17:34.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 10 client.4621.objecter successfully canceled tick 2024-01-20T11:17:34.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff7e406c8e0 msgr2=0x7ff7e406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:34.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff7e406c8e0 0x7ff7e406eda0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff800144420 tx=0x7ff7f0009190 comp rx=0 tx=0).stop 2024-01-20T11:17:34.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 10 monclient: shutdown 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8000b17c0 msgr2=0x7ff8001446c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8000b17c0 0x7ff8001446c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800efc0 tx=0x7ff7f800c490 comp rx=0 tx=0).stop 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 shutdown_connections 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff7e406c8e0 0x7ff7e406eda0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff800145980 0x7ff8000adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8000b4d10 0x7ff8000acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 --2- 172.21.15.110:0/215165063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8000b17c0 0x7ff8001446c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 >> 172.21.15.110:0/215165063 conn(0x7ff800066590 msgr2=0x7ff8000b83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 shutdown_connections 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 -- 172.21.15.110:0/215165063 wait complete. 2024-01-20T11:17:34.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.592+0000 7ff808755640 1 librados: shutdown 2024-01-20T11:17:34.603 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-01-20T11:17:32.737859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002146"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.9640000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.0899999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.137},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.3089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.087},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.8889999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.5660000000000007},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":16.152000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.194000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":10.539}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.903},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.277000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313248,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.0169999999999995},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.3360000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.75},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":14.817}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.512},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":11.491}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":18.474},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.255000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.113},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":13.35}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":18.879999999999999},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.837}]}]},{"osd":3,"up_from":21,"seq":90194313249,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":13.712999999999999},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.0090000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.3360000000000003},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.393000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":5.1719999999999997},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.163}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.754},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.5380000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":7.617},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":13.327}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.816000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.023999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.8250000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.0499999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.7629999999999999},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.4550000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.954000000000001},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.459}]},{"osd":4,"last update":"Sat Jan 20 11:16:27 2024","interfaces":[{"interface":"back","average":{"1min":18.023,"5min":18.023,"15min":18.023},"min":{"1min":11.689,"5min":11.689,"15min":11.689},"max":{"1min":51.000999999999998,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.571},{"interface":"front","average":{"1min":14.852,"5min":14.852,"15min":14.852},"min":{"1min":9.407,"5min":9.407,"15min":9.407},"max":{"1min":48.299999999999997,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.688000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.265000000000001},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.1289999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.6739999999999995},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":16.34}]}]},{"osd":1,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":10.922000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":5.7359999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.6879999999999997},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.194000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":15.125999999999999},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.3099999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":10.188000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.41}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.0589999999999993},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":11.563000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.5079999999999991},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.814}]}]},{"osd":0,"up_from":17,"seq":73014444063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":4.5510000000000002},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":6.7800000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.3970000000000002},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.0739999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":29.998999999999999,"5min":29.998999999999999,"15min":29.998999999999999},"min":{"1min":8.8379999999999992,"5min":8.8379999999999992,"15min":8.8379999999999992},"max":{"1min":358.738,"5min":358.738,"15min":358.738},"last":9.4269999999999996},{"interface":"front","average":{"1min":13.255000000000001,"5min":13.255000000000001,"15min":13.255000000000001},"min":{"1min":8.9930000000000003,"5min":8.9930000000000003,"15min":8.9930000000000003},"max":{"1min":21.004999999999999,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.606999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":32.514000000000003,"5min":32.514000000000003,"15min":32.514000000000003},"min":{"1min":10.763,"5min":10.763,"15min":10.763},"max":{"1min":372.02300000000002,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.555999999999999},{"interface":"front","average":{"1min":29.655999999999999,"5min":29.655999999999999,"15min":29.655999999999999},"min":{"1min":5.96,"5min":5.96,"15min":5.96},"max":{"1min":372.70800000000003,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7710000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.151999999999999},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":14.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:16:25 2024","interfaces":[{"interface":"back","average":{"1min":17.532,"5min":17.532,"15min":17.532},"min":{"1min":12.504,"5min":12.504,"15min":12.504},"max":{"1min":27.385999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.092000000000001},{"interface":"front","average":{"1min":12.464,"5min":12.464,"15min":12.464},"min":{"1min":8.3279999999999994,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":21.529,"5min":21.529,"15min":21.529},"last":10.074999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":7.9240000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.6020000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":15.57},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.337}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.3119999999999994},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.795999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.749000000000001},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.667}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.965},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.2949999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.382999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.346}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:34.604 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:34.604 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:34.604 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:34.604 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.002387+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.002387+0000', 'last_peered': '2024-01-20T11:16:53.002387+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.002387+0000', 'last_undegraded': '2024-01-20T11:16:53.002387+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:34.604 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:34.604 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:34.605 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:34.605 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:34.605 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.706+0000 7fa0225a4640 10 monclient: get_monmap_and_config 2024-01-20T11:17:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.706+0000 7fa0225a4640 10 monclient: build_initial_monmap 2024-01-20T11:17:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.706+0000 7fa0225a4640 10 monclient: monmap: 2024-01-20T11:17:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:34.706727+0000 2024-01-20T11:17:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:34.706727+0000 2024-01-20T11:17:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:34.707 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.706+0000 7fa0225a4640 1 Processor -- start 2024-01-20T11:17:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.706+0000 7fa0225a4640 1 -- start start 2024-01-20T11:17:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.706+0000 7fa0225a4640 10 monclient: init 2024-01-20T11:17:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-01-20T11:17:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa01c0a96f0 0x7fa01c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 10 monclient(hunting): picked mon.noname-c con 0x7fa01c0a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa01c13fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c0a9ad0 0x7fa01c142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 10 monclient(hunting): picked mon.noname-b con 0x7fa01c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01c0b25c0 con 0x7fa01c13fa10 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c0aaea0 con 0x7fa01c0a96f0 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0ab0a0 con 0x7fa01c0a9ad0 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.707+0000 7fa0225a4640 10 monclient(hunting): authenticate will time out at 1498.057785s 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.708+0000 7fa020b41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c0a9ad0 0x7fa01c142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.708+0000 7fa01bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa01c0a96f0 0x7fa01c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.708+0000 7fa020b41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c0a9ad0 0x7fa01c142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46924/0 (socket says 172.21.15.110:46924) 2024-01-20T11:17:34.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.708+0000 7fa020b41640 1 -- 172.21.15.110:0/4219936930 learned_addr learned my addr 172.21.15.110:0/4219936930 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:34.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.708+0000 7fa020b41640 10 monclient(hunting): get_auth_request con 0x7fa01c0a9ad0 auth_method 0 2024-01-20T11:17:34.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.708+0000 7fa020b41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.708+0000 7fa020b41640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.708+0000 7fa020b41640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa020b41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa020b41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa020b41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa01bfff640 10 monclient(hunting): get_auth_request con 0x7fa01c0a96f0 auth_method 0 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.709+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.710+0000 7fa020b41640 10 monclient(hunting): handle_auth_done global_id 4445 payload 274 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.710+0000 7fa020b41640 1 -- 172.21.15.110:0/4219936930 >> v2:172.21.15.134:3300/0 conn(0x7fa01c0a96f0 msgr2=0x7fa01c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.710+0000 7fa020b41640 1 --2- 172.21.15.110:0/4219936930 >> v2:172.21.15.134:3300/0 conn(0x7fa01c0a96f0 0x7fa01c13f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.710+0000 7fa020b41640 1 -- 172.21.15.110:0/4219936930 >> v1:172.21.15.134:6789/0 conn(0x7fa01c13fa10 legacy=0x7fa01c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.710+0000 7fa020b41640 10 monclient: _finish_hunting 0 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.710+0000 7fa020b41640 1 monclient: found mon.noname-b 2024-01-20T11:17:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.710+0000 7fa020b41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.710+0000 7fa020b41640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.710+0000 7fa020b41640 1 -- 172.21.15.110:0/4219936930 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c0b3fb0 con 0x7fa01c0a9ad0 2024-01-20T11:17:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.712+0000 7fa020b41640 1 --2- 172.21.15.110:0/4219936930 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c0a9ad0 0x7fa01c142af0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa014009e50 tx=0x7fa01402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8df47a41a33d3896 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 1 -- 172.21.15.110:0/4219936930 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01402ca70 con 0x7fa01c0a9ad0 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 10 monclient: dump: 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 10 monclient: _finish_auth 0 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 1 -- 172.21.15.110:0/4219936930 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01402cbd0 con 0x7fa01c0a9ad0 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 5 monclient: authenticate success, global_id 4445 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 1 -- 172.21.15.110:0/4219936930 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0140343f0 con 0x7fa01c0a9ad0 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 4 set_mon_vals no callback set 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa01affd640 10 monclient: dump: 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:34.718 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 10 monclient: shutdown 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 1 -- 172.21.15.110:0/4219936930 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c0a9ad0 msgr2=0x7fa01c142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 1 --2- 172.21.15.110:0/4219936930 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c0a9ad0 0x7fa01c142af0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa014009e50 tx=0x7fa01402bed0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 1 -- 172.21.15.110:0/4219936930 shutdown_connections 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 1 --2- 172.21.15.110:0/4219936930 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c0a9ad0 0x7fa01c142af0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 1 --2- 172.21.15.110:0/4219936930 >> v2:172.21.15.134:3300/0 conn(0x7fa01c0a96f0 0x7fa01c13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 1 -- 172.21.15.110:0/4219936930 >> 172.21.15.110:0/4219936930 conn(0x7fa01c0b7bc0 msgr2=0x7fa01c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 1 -- 172.21.15.110:0/4219936930 shutdown_connections 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.716+0000 7fa0225a4640 1 -- 172.21.15.110:0/4219936930 wait complete. 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.717+0000 7fa0225a4640 10 monclient: build_initial_monmap 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.717+0000 7fa0225a4640 10 monclient: monmap: 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:34.721 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.717+0000 7fa0225a4640 1 librados: starting msgr at 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.717+0000 7fa0225a4640 1 librados: starting objecter 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.717+0000 7fa0225a4640 1 Processor -- start 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 1 -- start start 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 1 librados: setting wanted keys 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 1 librados: calling monclient init 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient: init 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa01c0a96f0 0x7fa01c132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient(hunting): picked mon.noname-b con 0x7fa01c0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa01c0a9ad0 0x7fa01c1320b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient(hunting): picked mon.noname-a con 0x7fa01c0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c13fe00 0x7fa01c140250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient(hunting): picked mon.noname-c con 0x7fa01c13fe00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c132f60 con 0x7fa01c0a9ad0 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c1325f0 con 0x7fa01c0a96f0 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c1336b0 con 0x7fa01c13fe00 2024-01-20T11:17:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.718+0000 7fa0225a4640 10 monclient(hunting): authenticate will time out at 1498.068995s 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa020b41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c13fe00 0x7fa01c140250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa01c0a96f0 0x7fa01c132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa01c0a9ad0 0x7fa01c1320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa01c0a96f0 0x7fa01c132a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46374/0 (socket says 172.21.15.110:46374) 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01bfff640 1 -- 172.21.15.110:0/1316296012 learned_addr learned my addr 172.21.15.110:0/1316296012 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01bfff640 10 monclient(hunting): get_auth_request con 0x7fa01c0a96f0 auth_method 0 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01b7fe640 10 monclient(hunting): get_auth_request con 0x7fa01c0a9ad0 auth_method 0 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa020b41640 10 monclient(hunting): get_auth_request con 0x7fa01c13fe00 auth_method 0 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa020b41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa020b41640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.719+0000 7fa020b41640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.720+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.720+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.720+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.720+0000 7fa01b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.720+0000 7fa01b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.720+0000 7fa01b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.720+0000 7fa020b41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.720+0000 7fa020b41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.720+0000 7fa020b41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 10 monclient(hunting): handle_auth_done global_id 4630 payload 824 2024-01-20T11:17:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 1 -- 172.21.15.110:0/1316296012 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c13fe00 msgr2=0x7fa01c140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c13fe00 0x7fa01c140250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 1 -- 172.21.15.110:0/1316296012 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa01c0a9ad0 msgr2=0x7fa01c1320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa01c0a9ad0 0x7fa01c1320b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 1 monclient: found mon.noname-b 2024-01-20T11:17:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:34.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.723+0000 7fa01bfff640 1 -- 172.21.15.110:0/1316296012 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa014009b30 con 0x7fa01c0a96f0 2024-01-20T11:17:34.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.725+0000 7fa01bfff640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa01c0a96f0 0x7fa01c132a20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00e9b0 tx=0x7fa00c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:34.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.726+0000 7fa018ff9640 10 client.?.objecter ms_handle_connect 0x7fa01c0a96f0 2024-01-20T11:17:34.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.726+0000 7fa018ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 1 -- 172.21.15.110:0/1316296012 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c009670 con 0x7fa01c0a96f0 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 10 monclient: dump: 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 10 monclient: _finish_auth 0 2024-01-20T11:17:34.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 1 -- 172.21.15.110:0/1316296012 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c00c9a0 con 0x7fa01c0a96f0 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 1 -- 172.21.15.110:0/1316296012 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c00cd50 con 0x7fa01c0a96f0 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa018ff9640 10 monclient: dump: 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:34.732 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa0225a4640 5 monclient: authenticate success, global_id 4630 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa0225a4640 10 monclient: _renew_subs 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa0225a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c1343f0 con 0x7fa01c0a96f0 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa01a7fc640 4 set_mon_vals no callback set 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa0225a4640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa0225a4640 10 monclient: _renew_subs 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa0225a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c140790 con 0x7fa01c0a96f0 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.730+0000 7fa0225a4640 1 librados: init done 2024-01-20T11:17:34.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.731+0000 7fa0225a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:34.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.731+0000 7fa0225a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:34.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.731+0000 7fa0225a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:34.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.731+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c1409b0 con 0x7fa01c0a96f0 2024-01-20T11:17:34.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.760+0000 7fa018ff9640 1 -- 172.21.15.110:0/1316296012 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa00c013470 con 0x7fa01c0a96f0 2024-01-20T11:17:34.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.761+0000 7fa018ff9640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00806c860 0x7fa00806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:34.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.761+0000 7fa01b7fe640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00806c860 0x7fa00806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:34.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.761+0000 7fa01b7fe640 10 monclient: get_auth_request con 0x7fa00806c860 auth_method 0 2024-01-20T11:17:34.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.761+0000 7fa01b7fe640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00806c860 0x7fa00806ed20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0b7450 tx=0x7fa010009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:34.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.761+0000 7fa018ff9640 10 client.4630.objecter ms_handle_connect 0x7fa00806c860 2024-01-20T11:17:34.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.993+0000 7fa018ff9640 1 -- 172.21.15.110:0/1316296012 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa00c058c60 con 0x7fa01c0a96f0 2024-01-20T11:17:34.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.993+0000 7fa018ff9640 10 client.4630.objecter ms_dispatch 0x7fa01c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:34.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.993+0000 7fa018ff9640 3 client.4630.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:34.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.993+0000 7fa018ff9640 3 client.4630.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:34.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:34.994+0000 7fa018ff9640 20 client.4630.objecter dump_active .. 0 homeless 2024-01-20T11:17:35.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.012+0000 7fa018ff9640 1 -- 172.21.15.110:0/1316296012 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa00c054fa0 con 0x7fa01c0a96f0 2024-01-20T11:17:35.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.012+0000 7fa018ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:35.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.012+0000 7fa018ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:35.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.252+0000 7fa0225a4640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:35.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.252+0000 7fa0225a4640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:35.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.252+0000 7fa0225a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:35.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.252+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa01c075490 con 0x7fa01c0a96f0 2024-01-20T11:17:35.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.259+0000 7fa018ff9640 1 -- 172.21.15.110:0/1316296012 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7fa00c021510 con 0x7fa01c0a96f0 2024-01-20T11:17:35.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.259+0000 7fa018ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:35.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.259+0000 7fa018ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:35.259 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:35.259 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":189,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:35.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 10 librados: watch_flush enter 2024-01-20T11:17:35.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 10 librados: watch_flush exit 2024-01-20T11:17:35.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 20 client.4630.objecter shutdown clearing up homeless session... 2024-01-20T11:17:35.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 10 client.4630.objecter successfully canceled tick 2024-01-20T11:17:35.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00806c860 msgr2=0x7fa00806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:35.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00806c860 0x7fa00806ed20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0b7450 tx=0x7fa010009040 comp rx=0 tx=0).stop 2024-01-20T11:17:35.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 10 monclient: shutdown 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa01c0a96f0 msgr2=0x7fa01c132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa01c0a96f0 0x7fa01c132a20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00e9b0 tx=0x7fa00c00ee70 comp rx=0 tx=0).stop 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 shutdown_connections 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00806c860 0x7fa00806ed20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01c13fe00 0x7fa01c140250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa01c0a9ad0 0x7fa01c1320b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 --2- 172.21.15.110:0/1316296012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa01c0a96f0 0x7fa01c132a20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 >> 172.21.15.110:0/1316296012 conn(0x7fa01c0b7bc0 msgr2=0x7fa01c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 shutdown_connections 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 -- 172.21.15.110:0/1316296012 wait complete. 2024-01-20T11:17:35.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.261+0000 7fa0225a4640 1 librados: shutdown 2024-01-20T11:17:35.272 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:35.272 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:35.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.370+0000 7f8bbf947640 10 monclient: get_monmap_and_config 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.370+0000 7f8bbf947640 10 monclient: build_initial_monmap 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.371+0000 7f8bbf947640 10 monclient: monmap: 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:35.371597+0000 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:35.371597+0000 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:35.371 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:35.372 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:35.372 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.371+0000 7f8bbf947640 1 Processor -- start 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.371+0000 7f8bbf947640 1 -- start start 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.371+0000 7f8bbf947640 10 monclient: init 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bb806bb00 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8bb80af790 0x7f8bb80ae410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 10 monclient(hunting): picked mon.noname-a con 0x7f8bb80af790 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8bb80ae950 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb806bee0 con 0x7f8bb806bb00 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb80ada30 con 0x7f8bb80ae950 2024-01-20T11:17:35.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb80add10 con 0x7f8bb80af790 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.372+0000 7f8bbf947640 10 monclient(hunting): authenticate will time out at 1498.722815s 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbcee2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8bb80af790 0x7f8bb80ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbdee4640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8bb80ae950 0x7f8bb80aedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:36702/0 (socket says 172.21.15.110:36702) 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbdee4640 1 -- 172.21.15.110:0/1424181433 learned_addr learned my addr 172.21.15.110:0/1424181433 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbcee2640 10 monclient(hunting): get_auth_request con 0x7f8bb80af790 auth_method 0 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbcee2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbcee2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbcee2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbcee2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbcee2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:35.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.373+0000 7f8bbcee2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:35.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bb7fff640 1 -- 172.21.15.110:0/1424181433 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170818718 0 0) 0x7f8bb80ada30 con 0x7f8bb80ae950 2024-01-20T11:17:35.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bb7fff640 10 monclient(hunting): _negotiate 2024-01-20T11:17:35.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bb7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:35.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:35.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bb7fff640 10 monclient(hunting): my global_id is 4633 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bb7fff640 1 -- 172.21.15.110:0/1424181433 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba0003520 con 0x7f8bb80ae950 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bbcee2640 10 monclient(hunting): handle_auth_done global_id 4563 payload 274 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bbcee2640 1 -- 172.21.15.110:0/1424181433 >> v1:172.21.15.134:6789/0 conn(0x7f8bb80ae950 legacy=0x7f8bb80aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bbcee2640 1 -- 172.21.15.110:0/1424181433 >> v1:172.21.15.110:6789/0 conn(0x7f8bb806bb00 legacy=0x7f8bb80b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bbcee2640 10 monclient: _finish_hunting 0 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bbcee2640 1 monclient: found mon.noname-a 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bbcee2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bbcee2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.375+0000 7f8bbcee2640 1 -- 172.21.15.110:0/1424181433 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb80adff0 con 0x7f8bb80af790 2024-01-20T11:17:35.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.377+0000 7f8bbcee2640 1 --2- 172.21.15.110:0/1424181433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80af790 0x7f8bb80ae410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009790 tx=0x7f8bac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a09ce6b4b776c68 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 1 -- 172.21.15.110:0/1424181433 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac02e070 con 0x7f8bb80af790 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 10 monclient: dump: 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:35.382 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 10 monclient: _finish_auth 0 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 1 -- 172.21.15.110:0/1424181433 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac0315a0 con 0x7f8bb80af790 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 5 monclient: authenticate success, global_id 4563 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 4 set_mon_vals no callback set 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 1 -- 172.21.15.110:0/1424181433 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac031930 con 0x7f8bb80af790 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bb7fff640 10 monclient: dump: 2024-01-20T11:17:35.383 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:35.384 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:35.384 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:35.384 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:35.384 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:35.384 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:35.384 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:35.384 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:35.384 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:35.384 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:35.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 10 monclient: shutdown 2024-01-20T11:17:35.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:35.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 1 -- 172.21.15.110:0/1424181433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80af790 msgr2=0x7f8bb80ae410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:35.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 1 --2- 172.21.15.110:0/1424181433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80af790 0x7f8bb80ae410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009790 tx=0x7f8bac02b840 comp rx=0 tx=0).stop 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 1 -- 172.21.15.110:0/1424181433 shutdown_connections 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 1 --2- 172.21.15.110:0/1424181433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80af790 0x7f8bb80ae410 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 1 -- 172.21.15.110:0/1424181433 >> 172.21.15.110:0/1424181433 conn(0x7f8bb8066d60 msgr2=0x7f8bb80b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 1 -- 172.21.15.110:0/1424181433 shutdown_connections 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.382+0000 7f8bbf947640 1 -- 172.21.15.110:0/1424181433 wait complete. 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.383+0000 7f8bbf947640 10 monclient: build_initial_monmap 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.383+0000 7f8bbf947640 10 monclient: monmap: 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:35.389 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.383+0000 7f8bbf947640 1 librados: starting msgr at 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.383+0000 7f8bbf947640 1 librados: starting objecter 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 Processor -- start 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 -- start start 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 librados: setting wanted keys 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 librados: calling monclient init 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient: init 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bb806bb00 0x7f8bb8070380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:35.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient(hunting): picked mon.noname-b con 0x7f8bb806bb00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80ae950 0x7f8bb81da2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient(hunting): picked mon.noname-a con 0x7f8bb80ae950 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8bb80af790 0x7f8bb81da830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient(hunting): picked mon.noname-c con 0x7f8bb80af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8120010 con 0x7f8bb80ae950 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb81daf60 con 0x7f8bb806bb00 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb813c850 con 0x7f8bb80af790 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.384+0000 7f8bbf947640 10 monclient(hunting): authenticate will time out at 1498.734832s 2024-01-20T11:17:35.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80ae950 0x7f8bb81da2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbd6e3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bb806bb00 0x7f8bb8070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbdee4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8bb80af790 0x7f8bb81da830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80ae950 0x7f8bb81da2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43810/0 (socket says 172.21.15.110:43810) 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 1 -- 172.21.15.110:0/26880482 learned_addr learned my addr 172.21.15.110:0/26880482 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 10 monclient(hunting): get_auth_request con 0x7f8bb80ae950 auth_method 0 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbdee4640 10 monclient(hunting): get_auth_request con 0x7f8bb80af790 auth_method 0 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbdee4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbdee4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbdee4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbd6e3640 10 monclient(hunting): get_auth_request con 0x7f8bb806bb00 auth_method 0 2024-01-20T11:17:35.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbd6e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbd6e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbd6e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.385+0000 7f8bbcee2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.386+0000 7f8bbdee4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.386+0000 7f8bbdee4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.386+0000 7f8bbdee4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.386+0000 7f8bbd6e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.386+0000 7f8bbd6e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.386+0000 7f8bbd6e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 10 monclient(hunting): handle_auth_done global_id 4566 payload 824 2024-01-20T11:17:35.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 1 -- 172.21.15.110:0/26880482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8bb80af790 msgr2=0x7f8bb81da830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:35.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8bb80af790 0x7f8bb81da830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 1 -- 172.21.15.110:0/26880482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bb806bb00 msgr2=0x7f8bb8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:35.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bb806bb00 0x7f8bb8070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 10 monclient: _finish_hunting 0 2024-01-20T11:17:35.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 1 monclient: found mon.noname-a 2024-01-20T11:17:35.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:35.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:35.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.388+0000 7f8bbcee2640 1 -- 172.21.15.110:0/26880482 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac0093a0 con 0x7f8bb80ae950 2024-01-20T11:17:35.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.393+0000 7f8bbcee2640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80ae950 0x7f8bb81da2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0098b0 tx=0x7f8bac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:35.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.393+0000 7f8bb5ffb640 10 client.?.objecter ms_handle_connect 0x7f8bb80ae950 2024-01-20T11:17:35.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.393+0000 7f8bb5ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 1 -- 172.21.15.110:0/26880482 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac031ee0 con 0x7f8bb80ae950 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 10 monclient: dump: 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 1 -- 172.21.15.110:0/26880482 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac032640 con 0x7f8bb80ae950 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bbf947640 5 monclient: authenticate success, global_id 4566 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bbf947640 10 monclient: _renew_subs 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bbf947640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb813cad0 con 0x7f8bb80ae950 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 1 -- 172.21.15.110:0/26880482 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac032a10 con 0x7f8bb80ae950 2024-01-20T11:17:35.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb5ffb640 10 monclient: dump: 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bbf947640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bbf947640 10 monclient: _renew_subs 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bbf947640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:35.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb813cf80 con 0x7f8bb80ae950 2024-01-20T11:17:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bb6ffd640 4 set_mon_vals no callback set 2024-01-20T11:17:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.401+0000 7f8bbf947640 1 librados: init done 2024-01-20T11:17:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.402+0000 7f8bbf947640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.402+0000 7f8bbf947640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.402+0000 7f8bbf947640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:35.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.402+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb813d230 con 0x7f8bb80ae950 2024-01-20T11:17:35.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.648+0000 7f8bb5ffb640 1 -- 172.21.15.110:0/26880482 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8bac032b70 con 0x7f8bb80ae950 2024-01-20T11:17:35.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.648+0000 7f8bb5ffb640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ba406c8b0 0x7f8ba406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:35.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.648+0000 7f8bbd6e3640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ba406c8b0 0x7f8ba406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:35.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.648+0000 7f8bbd6e3640 10 monclient: get_auth_request con 0x7f8ba406c8b0 auth_method 0 2024-01-20T11:17:35.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.649+0000 7f8bb5ffb640 10 client.4566.objecter ms_handle_connect 0x7f8ba406c8b0 2024-01-20T11:17:35.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.649+0000 7f8bbd6e3640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ba406c8b0 0x7f8ba406ed70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0001ef0 tx=0x7f8bb0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:35.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.662+0000 7f8bb5ffb640 1 -- 172.21.15.110:0/26880482 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8bac032e20 con 0x7f8bb80ae950 2024-01-20T11:17:35.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.662+0000 7f8bb5ffb640 10 client.4566.objecter ms_dispatch 0x7f8bb8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:35.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.662+0000 7f8bb5ffb640 3 client.4566.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:35.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.662+0000 7f8bb5ffb640 3 client.4566.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:35.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.662+0000 7f8bb5ffb640 20 client.4566.objecter dump_active .. 0 homeless 2024-01-20T11:17:35.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.662+0000 7f8bb5ffb640 1 -- 172.21.15.110:0/26880482 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8bac0761d0 con 0x7f8bb80ae950 2024-01-20T11:17:35.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.662+0000 7f8bb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:35.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.662+0000 7f8bb5ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.841+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8bb8075490 con 0x7f8ba406c8b0 2024-01-20T11:17:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.844+0000 7f8bb5ffb640 1 -- 172.21.15.110:0/26880482 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64187 (secure 0 0 0) 0x7f8bb8075490 con 0x7f8ba406c8b0 2024-01-20T11:17:35.844 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:35.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.845+0000 7f8bbf947640 10 librados: watch_flush enter 2024-01-20T11:17:35.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.845+0000 7f8bbf947640 10 librados: watch_flush exit 2024-01-20T11:17:35.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.845+0000 7f8bbf947640 20 client.4566.objecter shutdown clearing up homeless session... 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.845+0000 7f8bbf947640 10 client.4566.objecter successfully canceled tick 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ba406c8b0 msgr2=0x7f8ba406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ba406c8b0 0x7f8ba406ed70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0001ef0 tx=0x7f8bb0009040 comp rx=0 tx=0).stop 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 10 monclient: shutdown 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80ae950 msgr2=0x7f8bb81da2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80ae950 0x7f8bb81da2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0098b0 tx=0x7f8bac0046c0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 shutdown_connections 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8ba406c8b0 0x7f8ba406ed70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8bb80af790 0x7f8bb81da830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80ae950 0x7f8bb81da2f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 --2- 172.21.15.110:0/26880482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bb806bb00 0x7f8bb8070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 >> 172.21.15.110:0/26880482 conn(0x7f8bb8066d60 msgr2=0x7f8bb8144dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 shutdown_connections 2024-01-20T11:17:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 -- 172.21.15.110:0/26880482 wait complete. 2024-01-20T11:17:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:35.846+0000 7f8bbf947640 1 librados: shutdown 2024-01-20T11:17:35.856 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-01-20T11:17:34.738179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002105"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.743872+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.743872+0000","last_peered":"2024-01-20T11:16:52.743872+0000","last_clean":"2024-01-20T11:16:52.743872+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.743872+0000","last_undegraded":"2024-01-20T11:16:52.743872+0000","last_fullsized":"2024-01-20T11:16:52.743872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.871684+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.871684+0000","last_peered":"2024-01-20T11:16:52.871684+0000","last_clean":"2024-01-20T11:16:52.871684+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.871684+0000","last_undegraded":"2024-01-20T11:16:52.871684+0000","last_fullsized":"2024-01-20T11:16:52.871684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.839432+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.839432+0000","last_peered":"2024-01-20T11:16:52.839432+0000","last_clean":"2024-01-20T11:16:52.839432+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.839432+0000","last_undegraded":"2024-01-20T11:16:52.839432+0000","last_fullsized":"2024-01-20T11:16:52.839432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.002387+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.002387+0000","last_peered":"2024-01-20T11:16:53.002387+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.002387+0000","last_undegraded":"2024-01-20T11:16:53.002387+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.847279+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.847279+0000","last_peered":"2024-01-20T11:16:52.847279+0000","last_clean":"2024-01-20T11:16:52.847279+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.847279+0000","last_undegraded":"2024-01-20T11:16:52.847279+0000","last_fullsized":"2024-01-20T11:16:52.847279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280546,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.6549999999999994},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.7229999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.368},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.7430000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.318},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":8.9619999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":12.140000000000001},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":10.805}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.244999999999999},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":10.254}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.2360000000000007},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.91}]}]},{"osd":7,"up_from":21,"seq":90194313248,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.0169999999999995},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.3360000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.75},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":14.817}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.512},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":11.491}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":18.474},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.255000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.113},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":13.35}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":18.879999999999999},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.837}]}]},{"osd":3,"up_from":21,"seq":90194313249,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":13.712999999999999},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.0090000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.3360000000000003},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.393000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":5.1719999999999997},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.163}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.754},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.5380000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":7.617},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":13.327}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.816000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.023999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444063,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.2940000000000005},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.4379999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.1360000000000001},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.885999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.281000000000001},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":5.6769999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.175000000000001},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":7.9059999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.534000000000001},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.646000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.085000000000001},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":14.974}]}]},{"osd":1,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.4089999999999998},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":8.6869999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.869},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.5920000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":15.069000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.731}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":7.8200000000000003},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.712}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.8360000000000003},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":10.279}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.326000000000001},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.983000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444064,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":6.4000000000000004},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":9.2620000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.0500000000000007},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.548}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":10.065},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.958}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.805},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":16.239000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.691000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.034000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.356},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":15.417}]}]},{"osd":6,"up_from":15,"seq":64424509472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":7.9240000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.6020000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":15.57},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.337}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.3119999999999994},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.795999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.749000000000001},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.667}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.965},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.2949999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.382999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.346}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:38.313 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:29.338 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:38.317 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:29.341 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:38.331 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:29.355 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:38.363 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:03:29.369 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:38.365 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:29.389 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:38.365 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:03:29.372 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:38.367 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:29.392 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:38.379 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:03:29.385 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:38.382 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:29.406 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:38.859 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:38.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.957+0000 7fe831861640 10 monclient: get_monmap_and_config 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.957+0000 7fe831861640 10 monclient: build_initial_monmap 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.957+0000 7fe831861640 10 monclient: monmap: 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:38.958015+0000 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:38.958015+0000 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:38.958 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:38.959 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:38.959 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:38.959 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:38.959 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:38.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.957+0000 7fe831861640 1 Processor -- start 2024-01-20T11:17:38.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 1 -- start start 2024-01-20T11:17:38.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient: init 2024-01-20T11:17:38.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe82c0b4d10 0x7fe82c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient(hunting): picked mon.noname-a con 0x7fe82c0b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 0x7fe82c0b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient(hunting): picked mon.noname-b con 0x7fe82c0b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe82c145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82c0b2120 con 0x7fe82c145980 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82c0b50f0 con 0x7fe82c0b4d10 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0aade0 con 0x7fe82c0b17c0 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:38.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.958+0000 7fe831861640 10 monclient(hunting): authenticate will time out at 1502.308781s 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe82affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe82c0b4d10 0x7fe82c0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe8227fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 0x7fe82c0b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe82affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe82c0b4d10 0x7fe82c0b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43816/0 (socket says 172.21.15.110:43816) 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe82affd640 1 -- 172.21.15.110:0/1020989192 learned_addr learned my addr 172.21.15.110:0/1020989192 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe82affd640 10 monclient(hunting): get_auth_request con 0x7fe82c0b4d10 auth_method 0 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe82affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe82affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe82affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe8227fc640 10 monclient(hunting): get_auth_request con 0x7fe82c0b17c0 auth_method 0 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe8227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe8227fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:38.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.959+0000 7fe8227fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.960+0000 7fe82affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.960+0000 7fe82affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.960+0000 7fe82affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.960+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.960+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.960+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.961+0000 7fe82affd640 10 monclient(hunting): handle_auth_done global_id 4569 payload 274 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.961+0000 7fe82affd640 1 -- 172.21.15.110:0/1020989192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 msgr2=0x7fe82c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.961+0000 7fe82affd640 1 --2- 172.21.15.110:0/1020989192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 0x7fe82c0b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.961+0000 7fe82affd640 1 -- 172.21.15.110:0/1020989192 >> v1:172.21.15.134:6789/0 conn(0x7fe82c145980 legacy=0x7fe82c145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:38.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.961+0000 7fe82affd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:38.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.961+0000 7fe82affd640 1 monclient: found mon.noname-a 2024-01-20T11:17:38.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.961+0000 7fe82affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:38.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.961+0000 7fe82affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:38.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.961+0000 7fe82affd640 1 -- 172.21.15.110:0/1020989192 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c145460 con 0x7fe82c0b4d10 2024-01-20T11:17:38.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.963+0000 7fe82affd640 1 --2- 172.21.15.110:0/1020989192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c0b4d10 0x7fe82c0b3da0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009870 tx=0x7fe81c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d554eb7021d541 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:38.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 1 -- 172.21.15.110:0/1020989192 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c031640 con 0x7fe82c0b4d10 2024-01-20T11:17:38.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:38.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:38.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 10 monclient: dump: 2024-01-20T11:17:38.967 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:38.967 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:38.967 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 1 -- 172.21.15.110:0/1020989192 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c0317a0 con 0x7fe82c0b4d10 2024-01-20T11:17:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 1 -- 172.21.15.110:0/1020989192 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c031b30 con 0x7fe82c0b4d10 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe831861640 5 monclient: authenticate success, global_id 4569 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe831861640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe831861640 4 set_mon_vals no callback set 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe82a7fc640 10 monclient: dump: 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:38.969 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:38.970 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:38.970 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:38.970 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:38.970 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:38.970 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:38.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe831861640 10 monclient: shutdown 2024-01-20T11:17:38.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe831861640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:38.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.967+0000 7fe831861640 1 -- 172.21.15.110:0/1020989192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c0b4d10 msgr2=0x7fe82c0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:38.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.968+0000 7fe831861640 1 --2- 172.21.15.110:0/1020989192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c0b4d10 0x7fe82c0b3da0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009870 tx=0x7fe81c02b8f0 comp rx=0 tx=0).stop 2024-01-20T11:17:38.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.968+0000 7fe831861640 1 -- 172.21.15.110:0/1020989192 shutdown_connections 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.968+0000 7fe831861640 1 --2- 172.21.15.110:0/1020989192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 0x7fe82c0b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.968+0000 7fe831861640 1 --2- 172.21.15.110:0/1020989192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c0b4d10 0x7fe82c0b3da0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.968+0000 7fe831861640 1 -- 172.21.15.110:0/1020989192 >> 172.21.15.110:0/1020989192 conn(0x7fe82c066590 msgr2=0x7fe82c0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.968+0000 7fe831861640 1 -- 172.21.15.110:0/1020989192 shutdown_connections 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.968+0000 7fe831861640 1 -- 172.21.15.110:0/1020989192 wait complete. 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.968+0000 7fe831861640 10 monclient: build_initial_monmap 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.968+0000 7fe831861640 10 monclient: monmap: 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 librados: starting msgr at 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 librados: starting objecter 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 Processor -- start 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 -- start start 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 librados: setting wanted keys 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 librados: calling monclient init 2024-01-20T11:17:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient: init 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 0x7fe82c1467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient(hunting): picked mon.noname-c con 0x7fe82c0b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe82c0b4d10 0x7fe82c140280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient(hunting): picked mon.noname-b con 0x7fe82c0b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c145980 0x7fe82c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient(hunting): picked mon.noname-a con 0x7fe82c145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c140d00 con 0x7fe82c145980 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:38.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c147fa0 con 0x7fe82c0b4d10 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c148280 con 0x7fe82c0b17c0 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.969+0000 7fe831861640 10 monclient(hunting): authenticate will time out at 1502.320108s 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe8227fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe82c0b4d10 0x7fe82c140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe82b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c145980 0x7fe82c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe82b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c145980 0x7fe82c1407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43824/0 (socket says 172.21.15.110:43824) 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe8227fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe82c0b4d10 0x7fe82c140280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46396/0 (socket says 172.21.15.110:46396) 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe82b7fe640 1 -- 172.21.15.110:0/3962482527 learned_addr learned my addr 172.21.15.110:0/3962482527 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe82b7fe640 10 monclient(hunting): get_auth_request con 0x7fe82c145980 auth_method 0 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe82b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:38.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe82b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe82b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe8227fc640 10 monclient(hunting): get_auth_request con 0x7fe82c0b4d10 auth_method 0 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.970+0000 7fe8227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe8227fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe8227fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe82affd640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 0x7fe82c1467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe8227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe82b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe82b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe82b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe82affd640 10 monclient(hunting): get_auth_request con 0x7fe82c0b17c0 auth_method 0 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe82affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe82affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.971+0000 7fe82affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.972+0000 7fe82affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.972+0000 7fe82affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.972+0000 7fe82affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 10 monclient(hunting): handle_auth_done global_id 4642 payload 824 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 1 -- 172.21.15.110:0/3962482527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 msgr2=0x7fe82c1467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 0x7fe82c1467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 1 -- 172.21.15.110:0/3962482527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c145980 msgr2=0x7fe82c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:38.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c145980 0x7fe82c1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:38.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 10 monclient: _finish_hunting 0 2024-01-20T11:17:38.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 1 monclient: found mon.noname-b 2024-01-20T11:17:38.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:38.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:38.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.974+0000 7fe8227fc640 1 -- 172.21.15.110:0/3962482527 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c0094d0 con 0x7fe82c0b4d10 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.976+0000 7fe8227fc640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe82c0b4d10 0x7fe82c140280 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe81400e850 tx=0x7fe81400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.976+0000 7fe828ff9640 10 client.?.objecter ms_handle_connect 0x7fe82c0b4d10 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.976+0000 7fe828ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 1 -- 172.21.15.110:0/3962482527 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8140095b0 con 0x7fe82c0b4d10 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 10 monclient: dump: 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:38.983 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 10 monclient: _finish_auth 0 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 1 -- 172.21.15.110:0/3962482527 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81400c940 con 0x7fe82c0b4d10 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe831861640 5 monclient: authenticate success, global_id 4642 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe831861640 10 monclient: _renew_subs 2024-01-20T11:17:38.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe831861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82c148560 con 0x7fe82c0b4d10 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 1 -- 172.21.15.110:0/3962482527 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81400ccd0 con 0x7fe82c0b4d10 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe828ff9640 10 monclient: dump: 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe829ffb640 4 set_mon_vals no callback set 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe831861640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe831861640 10 monclient: _renew_subs 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe831861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82c146d00 con 0x7fe82c0b4d10 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.980+0000 7fe831861640 1 librados: init done 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.981+0000 7fe831861640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.981+0000 7fe831861640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.981+0000 7fe831861640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.981+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82c075490 con 0x7fe82c0b4d10 2024-01-20T11:17:38.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.999+0000 7fe828ff9640 1 -- 172.21.15.110:0/3962482527 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe814013470 con 0x7fe82c0b4d10 2024-01-20T11:17:39.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:38.999+0000 7fe828ff9640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe80c06c860 0x7fe80c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:39.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.000+0000 7fe82affd640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe80c06c860 0x7fe80c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:39.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.000+0000 7fe82affd640 10 monclient: get_auth_request con 0x7fe80c06c860 auth_method 0 2024-01-20T11:17:39.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.000+0000 7fe82affd640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe80c06c860 0x7fe80c06ed20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe81c036000 tx=0x7fe81c03c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:39.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.000+0000 7fe828ff9640 10 client.4642.objecter ms_handle_connect 0x7fe80c06c860 2024-01-20T11:17:39.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.233+0000 7fe828ff9640 1 -- 172.21.15.110:0/3962482527 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe81400ce30 con 0x7fe82c0b4d10 2024-01-20T11:17:39.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.233+0000 7fe828ff9640 10 client.4642.objecter ms_dispatch 0x7fe82c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:39.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.233+0000 7fe828ff9640 3 client.4642.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:39.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.233+0000 7fe828ff9640 3 client.4642.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:39.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.233+0000 7fe828ff9640 20 client.4642.objecter dump_active .. 0 homeless 2024-01-20T11:17:39.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.250+0000 7fe828ff9640 1 -- 172.21.15.110:0/3962482527 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe814054ea0 con 0x7fe82c0b4d10 2024-01-20T11:17:39.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.250+0000 7fe828ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:39.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.250+0000 7fe828ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:39.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.425+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe82c142b20 con 0x7fe80c06c860 2024-01-20T11:17:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.428+0000 7fe828ff9640 1 -- 172.21.15.110:0/3962482527 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64144 (secure 0 0 0) 0x7fe82c142b20 con 0x7fe80c06c860 2024-01-20T11:17:39.428 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 10 librados: watch_flush enter 2024-01-20T11:17:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 10 librados: watch_flush exit 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 20 client.4642.objecter shutdown clearing up homeless session... 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 10 client.4642.objecter successfully canceled tick 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe80c06c860 msgr2=0x7fe80c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe80c06c860 0x7fe80c06ed20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe81c036000 tx=0x7fe81c03c040 comp rx=0 tx=0).stop 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 10 monclient: shutdown 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe82c0b4d10 msgr2=0x7fe82c140280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe82c0b4d10 0x7fe82c140280 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe81400e850 tx=0x7fe81400ed10 comp rx=0 tx=0).stop 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 shutdown_connections 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe80c06c860 0x7fe80c06ed20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe82c145980 0x7fe82c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe82c0b4d10 0x7fe82c140280 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 --2- 172.21.15.110:0/3962482527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe82c0b17c0 0x7fe82c1467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 >> 172.21.15.110:0/3962482527 conn(0x7fe82c066590 msgr2=0x7fe82c0b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 shutdown_connections 2024-01-20T11:17:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 -- 172.21.15.110:0/3962482527 wait complete. 2024-01-20T11:17:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.430+0000 7fe831861640 1 librados: shutdown 2024-01-20T11:17:39.440 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-01-20T11:17:38.738813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002042"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.8310000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.117}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.874000000000001},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.7199999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.1509999999999998},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.654}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.273999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":12.509}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.6},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":13.930999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.9100000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":13.186}]}]},{"osd":7,"up_from":21,"seq":90194313249,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":13.448},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.758}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.1280000000000001},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":16.030999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":16.879000000000001},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.3650000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.051},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.634}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.050000000000001},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.959}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.304},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.788}]}]},{"osd":3,"up_from":21,"seq":90194313250,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.8789999999999996},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.1430000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.4369999999999994},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.176}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":13.055999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.721}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":17.469999999999999},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.736000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":15.585000000000001},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":14.852}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":18.268999999999998},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.327999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.9179999999999993},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.327}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.8259999999999996},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":15.897}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.314},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.996}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":15},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.194000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.612},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.1829999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.872},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":16.75}]}]},{"osd":1,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.5800000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":7.8799999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.557},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.6500000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":13.255000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.4649999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":15.241},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.31}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.849},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":16.006}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.957000000000001},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.780999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":4.5599999999999996},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":9.9629999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9029999999999996},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":6.1849999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":11.686},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.388}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.394},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.282999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.917999999999999},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.577}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.241},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":16.128}]}]},{"osd":6,"up_from":15,"seq":64424509473,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.3090000000000002},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.4179999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.561999999999999},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":6.7039999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.124000000000001},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.1}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.260999999999999},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.281000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":10.69},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.6379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.962999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8.5950000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:39.442 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:39.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.552+0000 7fda0f7dc640 10 monclient: get_monmap_and_config 2024-01-20T11:17:39.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.552+0000 7fda0f7dc640 10 monclient: build_initial_monmap 2024-01-20T11:17:39.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.552+0000 7fda0f7dc640 10 monclient: monmap: 2024-01-20T11:17:39.581 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:39.581 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:39.581 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:39.553454+0000 2024-01-20T11:17:39.581 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:39.553454+0000 2024-01-20T11:17:39.581 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:39.581 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:39.582 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:39.582 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:39.582 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:39.582 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:39.582 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:39.582 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.553+0000 7fda0f7dc640 1 Processor -- start 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.553+0000 7fda0f7dc640 1 -- start start 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.553+0000 7fda0f7dc640 10 monclient: init 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda080b4d10 0x7fda080b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient(hunting): picked mon.noname-b con 0x7fda080b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fda080b17c0 0x7fda080b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient(hunting): picked mon.noname-a con 0x7fda080b17c0 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fda08145980 0x7fda08145e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient(hunting): picked mon.noname-c con 0x7fda08145980 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:39.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda080b50f0 con 0x7fda080b17c0 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda080b2120 con 0x7fda08145980 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda080aade0 con 0x7fda080b4d10 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.554+0000 7fda0f7dc640 10 monclient(hunting): authenticate will time out at 1502.904916s 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0d578640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda080b4d10 0x7fda080b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0dd79640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fda08145980 0x7fda08145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0d578640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda080b4d10 0x7fda080b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:53790/0 (socket says 172.21.15.110:53790) 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0d578640 1 -- 172.21.15.110:0/1066812164 learned_addr learned my addr 172.21.15.110:0/1066812164 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0d578640 10 monclient(hunting): get_auth_request con 0x7fda080b4d10 auth_method 0 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0d578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:39.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0d578640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0d578640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0dd79640 10 monclient(hunting): get_auth_request con 0x7fda08145980 auth_method 0 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0dd79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0dd79640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.555+0000 7fda0dd79640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.556+0000 7fda0dd79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.556+0000 7fda0dd79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.556+0000 7fda0dd79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.556+0000 7fda0d578640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.556+0000 7fda0d578640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.556+0000 7fda0d578640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0cd77640 1 --2- 172.21.15.110:0/1066812164 >> v2:172.21.15.110:3300/0 conn(0x7fda080b17c0 0x7fda080b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 10 monclient(hunting): handle_auth_done global_id 4645 payload 274 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 1 -- 172.21.15.110:0/1066812164 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda080b4d10 msgr2=0x7fda080b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:39.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 1 --2- 172.21.15.110:0/1066812164 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda080b4d10 0x7fda080b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 1 -- 172.21.15.110:0/1066812164 >> v2:172.21.15.110:3300/0 conn(0x7fda080b17c0 msgr2=0x7fda080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 1 --2- 172.21.15.110:0/1066812164 >> v2:172.21.15.110:3300/0 conn(0x7fda080b17c0 0x7fda080b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 10 monclient: _finish_hunting 0 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 1 monclient: found mon.noname-c 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.557+0000 7fda0dd79640 1 -- 172.21.15.110:0/1066812164 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda08145460 con 0x7fda08145980 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.560+0000 7fda0dd79640 1 --2- 172.21.15.110:0/1066812164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda08145980 0x7fda08145e10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda080b53c0 tx=0x7fda0002bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d24d15febc8d40eb server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.563+0000 7fd9fffff640 1 -- 172.21.15.110:0/1066812164 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0002cb40 con 0x7fda08145980 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 10 monclient: dump: 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:39.593 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 10 monclient: _finish_auth 0 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:39.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 1 -- 172.21.15.110:0/1066812164 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0002cca0 con 0x7fda08145980 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 1 -- 172.21.15.110:0/1066812164 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00034660 con 0x7fda08145980 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 5 monclient: authenticate success, global_id 4645 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 4 set_mon_vals no callback set 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fd9fffff640 10 monclient: dump: 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:39.595 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 10 monclient: shutdown 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 1 -- 172.21.15.110:0/1066812164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda08145980 msgr2=0x7fda08145e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/1066812164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda08145980 0x7fda08145e10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda080b53c0 tx=0x7fda0002bd90 comp rx=0 tx=0).stop 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 1 -- 172.21.15.110:0/1066812164 shutdown_connections 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/1066812164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda08145980 0x7fda08145e10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/1066812164 >> v2:172.21.15.110:3300/0 conn(0x7fda080b17c0 0x7fda080b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/1066812164 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda080b4d10 0x7fda080b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.564+0000 7fda0f7dc640 1 -- 172.21.15.110:0/1066812164 >> 172.21.15.110:0/1066812164 conn(0x7fda08066590 msgr2=0x7fda080669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:39.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.565+0000 7fda0f7dc640 1 -- 172.21.15.110:0/1066812164 shutdown_connections 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.565+0000 7fda0f7dc640 1 -- 172.21.15.110:0/1066812164 wait complete. 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.565+0000 7fda0f7dc640 10 monclient: build_initial_monmap 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.565+0000 7fda0f7dc640 10 monclient: monmap: 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.565+0000 7fda0f7dc640 1 librados: starting msgr at 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.565+0000 7fda0f7dc640 1 librados: starting objecter 2024-01-20T11:17:39.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 Processor -- start 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 -- start start 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 librados: setting wanted keys 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 librados: calling monclient init 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient: init 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda080b17c0 0x7fda080ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient(hunting): picked mon.noname-a con 0x7fda080b17c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda080b4d10 0x7fda080ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient(hunting): picked mon.noname-b con 0x7fda080b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda08145980 0x7fda0813ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient(hunting): picked mon.noname-c con 0x7fda08145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:39.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda080adbb0 con 0x7fda080b17c0 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda080ade90 con 0x7fda080b4d10 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda080acd10 con 0x7fda08145980 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.566+0000 7fda0f7dc640 10 monclient(hunting): authenticate will time out at 1502.916849s 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda080b17c0 0x7fda080ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0dd79640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda08145980 0x7fda0813ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0cd77640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda080b4d10 0x7fda080ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda080b17c0 0x7fda080ac110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:35354/0 (socket says 172.21.15.110:35354) 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 1 -- 172.21.15.110:0/2984121779 learned_addr learned my addr 172.21.15.110:0/2984121779 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 10 monclient(hunting): get_auth_request con 0x7fda080b17c0 auth_method 0 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:39.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0dd79640 10 monclient(hunting): get_auth_request con 0x7fda08145980 auth_method 0 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0dd79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0dd79640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0dd79640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0cd77640 10 monclient(hunting): get_auth_request con 0x7fda080b4d10 auth_method 0 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0cd77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0cd77640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0cd77640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.567+0000 7fda0d578640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.568+0000 7fda0dd79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.568+0000 7fda0dd79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:39.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.568+0000 7fda0dd79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.568+0000 7fda0cd77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.568+0000 7fda0cd77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.568+0000 7fda0cd77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 10 monclient(hunting): handle_auth_done global_id 4463 payload 824 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 1 -- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda080b4d10 msgr2=0x7fda080ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda080b4d10 0x7fda080ac650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 1 -- 172.21.15.110:0/2984121779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda080b17c0 msgr2=0x7fda080ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda080b17c0 0x7fda080ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 10 monclient: _finish_hunting 0 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 1 monclient: found mon.noname-c 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.570+0000 7fda0dd79640 1 -- 172.21.15.110:0/2984121779 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda00009a70 con 0x7fda08145980 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.571+0000 7fda0d578640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda080b17c0 0x7fda080ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:39.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.571+0000 7fda0cd77640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda080b4d10 0x7fda080ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.573+0000 7fda0dd79640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda08145980 0x7fda0813ca10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda00033000 tx=0x7fda00004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.573+0000 7fd9fdffb640 10 client.?.objecter ms_handle_connect 0x7fda08145980 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.573+0000 7fd9fdffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.576+0000 7fd9fdffb640 1 -- 172.21.15.110:0/2984121779 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00040650 con 0x7fda08145980 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.576+0000 7fd9fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.576+0000 7fd9fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.576+0000 7fd9fdffb640 10 monclient: dump: 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:39.602 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9fdffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9fdffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9fdffb640 1 -- 172.21.15.110:0/2984121779 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda000407b0 con 0x7fda08145980 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9fdffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9fdffb640 1 -- 172.21.15.110:0/2984121779 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00040b60 con 0x7fda08145980 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fda0f7dc640 5 monclient: authenticate success, global_id 4463 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fda0f7dc640 10 monclient: _renew_subs 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fda0f7dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0813cf50 con 0x7fda08145980 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9ff7fe640 4 set_mon_vals no callback set 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:39.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fd9fdffb640 10 monclient: dump: 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fda0f7dc640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fda0f7dc640 10 monclient: _renew_subs 2024-01-20T11:17:39.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fda0f7dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:39.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0813d3d0 con 0x7fda08145980 2024-01-20T11:17:39.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.577+0000 7fda0f7dc640 1 librados: init done 2024-01-20T11:17:39.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.578+0000 7fda0f7dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:39.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.578+0000 7fda0f7dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:39.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.578+0000 7fda0f7dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:39.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.578+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda080acf30 con 0x7fda08145980 2024-01-20T11:17:39.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.597+0000 7fd9fdffb640 1 -- 172.21.15.110:0/2984121779 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fda00040cc0 con 0x7fda08145980 2024-01-20T11:17:39.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.597+0000 7fd9fdffb640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9ec06c8b0 0x7fd9ec06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:39.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.597+0000 7fda0d578640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9ec06c8b0 0x7fd9ec06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:39.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.597+0000 7fda0d578640 10 monclient: get_auth_request con 0x7fd9ec06c8b0 auth_method 0 2024-01-20T11:17:39.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.598+0000 7fd9fdffb640 10 client.4463.objecter ms_handle_connect 0x7fd9ec06c8b0 2024-01-20T11:17:39.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.598+0000 7fda0d578640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9ec06c8b0 0x7fd9ec06ed70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fda040094a0 tx=0x7fda04009240 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:39.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.600+0000 7fd9fdffb640 1 -- 172.21.15.110:0/2984121779 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fda0007b2e0 con 0x7fda08145980 2024-01-20T11:17:39.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.600+0000 7fd9fdffb640 10 client.4463.objecter ms_dispatch 0x7fda08000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:39.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.600+0000 7fd9fdffb640 3 client.4463.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:39.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.600+0000 7fd9fdffb640 3 client.4463.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:39.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.600+0000 7fd9fdffb640 20 client.4463.objecter dump_active .. 0 homeless 2024-01-20T11:17:39.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.833+0000 7fd9fdffb640 1 -- 172.21.15.110:0/2984121779 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fda00077620 con 0x7fda08145980 2024-01-20T11:17:39.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.833+0000 7fd9fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:39.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:39.833+0000 7fd9fdffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:40.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.009+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fda080b2660 con 0x7fd9ec06c8b0 2024-01-20T11:17:40.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.012+0000 7fd9fdffb640 1 -- 172.21.15.110:0/2984121779 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64144 (secure 0 0 0) 0x7fda04004030 con 0x7fd9ec06c8b0 2024-01-20T11:17:40.013 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:40.013 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 10 librados: watch_flush enter 2024-01-20T11:17:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 10 librados: watch_flush exit 2024-01-20T11:17:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 20 client.4463.objecter shutdown clearing up homeless session... 2024-01-20T11:17:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 10 client.4463.objecter successfully canceled tick 2024-01-20T11:17:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9ec06c8b0 msgr2=0x7fd9ec06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9ec06c8b0 0x7fd9ec06ed70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fda040094a0 tx=0x7fda04009240 comp rx=0 tx=0).stop 2024-01-20T11:17:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 10 monclient: shutdown 2024-01-20T11:17:40.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda08145980 msgr2=0x7fda0813ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda08145980 0x7fda0813ca10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda00033000 tx=0x7fda00004a40 comp rx=0 tx=0).stop 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 shutdown_connections 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9ec06c8b0 0x7fd9ec06ed70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda08145980 0x7fda0813ca10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda080b4d10 0x7fda080ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 --2- 172.21.15.110:0/2984121779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda080b17c0 0x7fda080ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 >> 172.21.15.110:0/2984121779 conn(0x7fda08066590 msgr2=0x7fda080b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 shutdown_connections 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 -- 172.21.15.110:0/2984121779 wait complete. 2024-01-20T11:17:40.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.014+0000 7fda0f7dc640 1 librados: shutdown 2024-01-20T11:17:40.026 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-01-20T11:17:38.738813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002042"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.8310000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.117}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.874000000000001},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.7199999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.1509999999999998},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.654}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.273999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":12.509}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.6},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":13.930999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.9100000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":13.186}]}]},{"osd":7,"up_from":21,"seq":90194313249,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":13.448},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.758}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.1280000000000001},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":16.030999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":16.879000000000001},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.3650000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.051},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.634}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.050000000000001},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.959}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.304},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.788}]}]},{"osd":3,"up_from":21,"seq":90194313250,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.8789999999999996},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.1430000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.4369999999999994},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.176}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":13.055999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.721}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":17.469999999999999},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.736000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":15.585000000000001},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":14.852}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":18.268999999999998},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.327999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.9179999999999993},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.327}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.8259999999999996},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":15.897}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.314},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.996}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":15},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.194000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.612},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.1829999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.872},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":16.75}]}]},{"osd":1,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.5800000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":7.8799999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.557},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.6500000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":13.255000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.4649999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":15.241},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.31}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.849},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":16.006}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.957000000000001},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.780999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":4.5599999999999996},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":9.9629999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9029999999999996},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":6.1849999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":11.686},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.388}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.394},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.282999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.917999999999999},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.577}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.241},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":16.128}]}]},{"osd":6,"up_from":15,"seq":64424509473,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.3090000000000002},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.4179999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.561999999999999},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":6.7039999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.124000000000001},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.1}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.260999999999999},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.281000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":10.69},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.6379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.962999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8.5950000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:40.028 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:40.028 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:40.028 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:40.028 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:40.029 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:40.029 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:40.029 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:40.029 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:40.029 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.126+0000 7f93c8d3a640 10 monclient: get_monmap_and_config 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.126+0000 7f93c8d3a640 10 monclient: build_initial_monmap 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.126+0000 7f93c8d3a640 10 monclient: monmap: 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:40.127535+0000 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:40.127535+0000 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:40.127 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.127+0000 7f93c8d3a640 1 Processor -- start 2024-01-20T11:17:40.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.127+0000 7f93c8d3a640 1 -- start start 2024-01-20T11:17:40.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.127+0000 7f93c8d3a640 10 monclient: init 2024-01-20T11:17:40.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:40.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-01-20T11:17:40.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c406bb00 0x7f93c40b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 10 monclient(hunting): picked mon.noname-b con 0x7f93c406bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:40.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f93c40af790 0x7f93c4145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 10 monclient(hunting): picked mon.noname-c con 0x7f93c40af790 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93c40b0900 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c406c220 con 0x7f93c40b0900 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c406bee0 con 0x7f93c40af790 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c40ada30 con 0x7f93c406bb00 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c8d3a640 10 monclient(hunting): authenticate will time out at 1503.478462s 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c2d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f93c40b0900 0x7f93c4145cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51976/0 (socket says 172.21.15.110:51976) 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c2d9d640 1 -- 172.21.15.110:0/4223427343 learned_addr learned my addr 172.21.15.110:0/4223427343 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.128+0000 7f93c259c640 1 --2- 172.21.15.110:0/4223427343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c406bb00 0x7f93c40b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c1d9b640 1 --2- 172.21.15.110:0/4223427343 >> v2:172.21.15.134:3300/0 conn(0x7f93c40af790 0x7f93c4145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c259c640 10 monclient(hunting): get_auth_request con 0x7f93c406bb00 auth_method 0 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c1d9b640 10 monclient(hunting): get_auth_request con 0x7f93c40af790 auth_method 0 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.129+0000 7f93c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c159a640 1 -- 172.21.15.110:0/4223427343 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086283579 0 0) 0x7f93c406c220 con 0x7f93c40b0900 2024-01-20T11:17:40.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c159a640 10 monclient(hunting): _negotiate 2024-01-20T11:17:40.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c159a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c159a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c159a640 10 monclient(hunting): my global_id is 4578 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c159a640 1 -- 172.21.15.110:0/4223427343 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93ac003520 con 0x7f93c40b0900 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c259c640 10 monclient(hunting): handle_auth_done global_id 4466 payload 274 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c259c640 1 -- 172.21.15.110:0/4223427343 >> v2:172.21.15.134:3300/0 conn(0x7f93c40af790 msgr2=0x7f93c4145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c259c640 1 --2- 172.21.15.110:0/4223427343 >> v2:172.21.15.134:3300/0 conn(0x7f93c40af790 0x7f93c4145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c259c640 1 -- 172.21.15.110:0/4223427343 >> v1:172.21.15.110:6789/0 conn(0x7f93c40b0900 legacy=0x7f93c4145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c259c640 10 monclient: _finish_hunting 0 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c259c640 1 monclient: found mon.noname-b 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.131+0000 7f93c259c640 1 -- 172.21.15.110:0/4223427343 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c40add10 con 0x7f93c406bb00 2024-01-20T11:17:40.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.133+0000 7f93c259c640 1 --2- 172.21.15.110:0/4223427343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c406bb00 0x7f93c40b03c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0097a0 tx=0x7f93bc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65e965afe481e402 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:40.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 1 -- 172.21.15.110:0/4223427343 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc02e070 con 0x7f93c406bb00 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 10 monclient: dump: 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 10 monclient: _finish_auth 0 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 10 monclient: _check_auth_tickets 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 1 -- 172.21.15.110:0/4223427343 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc0315a0 con 0x7f93c406bb00 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 5 monclient: authenticate success, global_id 4466 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 1 -- 172.21.15.110:0/4223427343 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc031930 con 0x7f93c406bb00 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 4 set_mon_vals no callback set 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c159a640 10 monclient: dump: 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.138 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.139 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:40.139 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.139 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:40.139 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:40.139 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:40.139 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 10 monclient: shutdown 2024-01-20T11:17:40.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:40.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 1 -- 172.21.15.110:0/4223427343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c406bb00 msgr2=0x7f93c40b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 1 --2- 172.21.15.110:0/4223427343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c406bb00 0x7f93c40b03c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0097a0 tx=0x7f93bc02b840 comp rx=0 tx=0).stop 2024-01-20T11:17:40.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 1 -- 172.21.15.110:0/4223427343 shutdown_connections 2024-01-20T11:17:40.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 1 --2- 172.21.15.110:0/4223427343 >> v2:172.21.15.134:3300/0 conn(0x7f93c40af790 0x7f93c4145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 1 --2- 172.21.15.110:0/4223427343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c406bb00 0x7f93c40b03c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 1 -- 172.21.15.110:0/4223427343 >> 172.21.15.110:0/4223427343 conn(0x7f93c4066d60 msgr2=0x7f93c40b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 1 -- 172.21.15.110:0/4223427343 shutdown_connections 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.137+0000 7f93c8d3a640 1 -- 172.21.15.110:0/4223427343 wait complete. 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.138+0000 7f93c8d3a640 10 monclient: build_initial_monmap 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.138+0000 7f93c8d3a640 10 monclient: monmap: 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:40.143 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.138+0000 7f93c8d3a640 1 librados: starting msgr at 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.138+0000 7f93c8d3a640 1 librados: starting objecter 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 Processor -- start 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 -- start start 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 librados: setting wanted keys 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 librados: calling monclient init 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient: init 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93c406bb00 0x7f93c40b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient(hunting): picked mon.noname-a con 0x7f93c406bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c40af790 0x7f93c411d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient(hunting): picked mon.noname-c con 0x7f93c40af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40b0900 0x7f93c413bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient(hunting): picked mon.noname-b con 0x7f93c40b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c411de00 con 0x7f93c406bb00 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c411dae0 con 0x7f93c40b0900 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c4071300 con 0x7f93c40af790 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.139+0000 7f93c8d3a640 10 monclient(hunting): authenticate will time out at 1503.489860s 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c1d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c40af790 0x7f93c411d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93c406bb00 0x7f93c40b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c2d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40b0900 0x7f93c413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93c406bb00 0x7f93c40b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:35364/0 (socket says 172.21.15.110:35364) 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c259c640 1 -- 172.21.15.110:0/2689523172 learned_addr learned my addr 172.21.15.110:0/2689523172 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c259c640 10 monclient(hunting): get_auth_request con 0x7f93c406bb00 auth_method 0 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c1d9b640 10 monclient(hunting): get_auth_request con 0x7f93c40af790 auth_method 0 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c2d9d640 10 monclient(hunting): get_auth_request con 0x7f93c40b0900 auth_method 0 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.140+0000 7f93c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.141+0000 7f93c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.141+0000 7f93c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.141+0000 7f93c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.141+0000 7f93c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.141+0000 7f93c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.141+0000 7f93c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.141+0000 7f93c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.141+0000 7f93c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.141+0000 7f93c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 10 monclient(hunting): handle_auth_done global_id 4654 payload 824 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 1 -- 172.21.15.110:0/2689523172 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c40af790 msgr2=0x7f93c411d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c40af790 0x7f93c411d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 1 -- 172.21.15.110:0/2689523172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93c406bb00 msgr2=0x7f93c40b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93c406bb00 0x7f93c40b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 1 monclient: found mon.noname-b 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.143+0000 7f93c2d9d640 1 -- 172.21.15.110:0/2689523172 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc009400 con 0x7f93c40b0900 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.145+0000 7f93c2d9d640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40b0900 0x7f93c413bd10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f93b400c340 tx=0x7f93b4001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.145+0000 7f93b2ffd640 10 client.?.objecter ms_handle_connect 0x7f93c40b0900 2024-01-20T11:17:40.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.145+0000 7f93b2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:40.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 1 -- 172.21.15.110:0/2689523172 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4005ce0 con 0x7f93c40b0900 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 10 monclient: dump: 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:40.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 1 -- 172.21.15.110:0/2689523172 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b4005e40 con 0x7f93c40b0900 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c8d3a640 5 monclient: authenticate success, global_id 4654 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c8d3a640 10 monclient: _renew_subs 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c8d3a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c413c250 con 0x7f93c40b0900 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 1 -- 172.21.15.110:0/2689523172 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b40072d0 con 0x7f93c40b0900 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93b2ffd640 10 monclient: dump: 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.151 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c0d99640 4 set_mon_vals no callback set 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c8d3a640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c8d3a640 10 monclient: _renew_subs 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c8d3a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c413c760 con 0x7f93c40b0900 2024-01-20T11:17:40.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.149+0000 7f93c8d3a640 1 librados: init done 2024-01-20T11:17:40.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.150+0000 7f93c8d3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:40.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.150+0000 7f93c8d3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:40.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.150+0000 7f93c8d3a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:40.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.150+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c4071580 con 0x7f93c40b0900 2024-01-20T11:17:40.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.391+0000 7f93b2ffd640 1 -- 172.21.15.110:0/2689523172 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f93b4007430 con 0x7f93c40b0900 2024-01-20T11:17:40.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.391+0000 7f93b2ffd640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93a406c8b0 0x7f93a406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.392+0000 7f93c259c640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93a406c8b0 0x7f93a406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.392+0000 7f93c259c640 10 monclient: get_auth_request con 0x7f93a406c8b0 auth_method 0 2024-01-20T11:17:40.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.392+0000 7f93c259c640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93a406c8b0 0x7f93a406ed70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f93bc000c00 tx=0x7f93bc03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:40.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.392+0000 7f93b2ffd640 10 client.4654.objecter ms_handle_connect 0x7f93a406c8b0 2024-01-20T11:17:40.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.395+0000 7f93b2ffd640 1 -- 172.21.15.110:0/2689523172 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f93b4057bd0 con 0x7f93c40b0900 2024-01-20T11:17:40.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.395+0000 7f93b2ffd640 10 client.4654.objecter ms_dispatch 0x7f93c4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:40.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.395+0000 7f93b2ffd640 3 client.4654.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:40.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.395+0000 7f93b2ffd640 3 client.4654.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:40.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.395+0000 7f93b2ffd640 20 client.4654.objecter dump_active .. 0 homeless 2024-01-20T11:17:40.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.406+0000 7f93b2ffd640 1 -- 172.21.15.110:0/2689523172 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f93b4053f10 con 0x7f93c40b0900 2024-01-20T11:17:40.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.407+0000 7f93b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:40.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.407+0000 7f93b2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:40.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.652+0000 7f93c8d3a640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:40.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.652+0000 7f93c8d3a640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:40.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.652+0000 7f93c8d3a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:40.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.652+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f93c411dc20 con 0x7f93c40b0900 2024-01-20T11:17:40.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.658+0000 7f93b2ffd640 1 -- 172.21.15.110:0/2689523172 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7f93b4021510 con 0x7f93c40b0900 2024-01-20T11:17:40.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.658+0000 7f93b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:40.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.658+0000 7f93b2ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:40.658 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:40.658 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":195,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:40.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 10 librados: watch_flush enter 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 10 librados: watch_flush exit 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 20 client.4654.objecter shutdown clearing up homeless session... 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 10 client.4654.objecter successfully canceled tick 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93a406c8b0 msgr2=0x7f93a406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93a406c8b0 0x7f93a406ed70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f93bc000c00 tx=0x7f93bc03d040 comp rx=0 tx=0).stop 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 10 monclient: shutdown 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40b0900 msgr2=0x7f93c413bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40b0900 0x7f93c413bd10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f93b400c340 tx=0x7f93b4001e70 comp rx=0 tx=0).stop 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 shutdown_connections 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93a406c8b0 0x7f93a406ed70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40b0900 0x7f93c413bd10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f93c40af790 0x7f93c411d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 --2- 172.21.15.110:0/2689523172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93c406bb00 0x7f93c40b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 >> 172.21.15.110:0/2689523172 conn(0x7f93c4066d60 msgr2=0x7f93c4144ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:40.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 shutdown_connections 2024-01-20T11:17:40.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 -- 172.21.15.110:0/2689523172 wait complete. 2024-01-20T11:17:40.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.660+0000 7f93c8d3a640 1 librados: shutdown 2024-01-20T11:17:40.671 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:40.671 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:40.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.773+0000 7f08a5e90640 10 monclient: get_monmap_and_config 2024-01-20T11:17:40.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.773+0000 7f08a5e90640 10 monclient: build_initial_monmap 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.773+0000 7f08a5e90640 10 monclient: monmap: 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:40.774441+0000 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:40.774441+0000 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:40.774 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.774+0000 7f08a5e90640 1 Processor -- start 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.774+0000 7f08a5e90640 1 -- start start 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.774+0000 7f08a5e90640 10 monclient: init 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08a00a96f0 0x7f08a013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 10 monclient(hunting): picked mon.noname-a con 0x7f08a00a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08a013fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f08a00a9ad0 0x7f08a0142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 10 monclient(hunting): picked mon.noname-c con 0x7f08a00a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a00b25c0 con 0x7f08a013fa10 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a00aaea0 con 0x7f08a00a96f0 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a00ab0a0 con 0x7f08a00a9ad0 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f08a5e90640 10 monclient(hunting): authenticate will time out at 1504.125428s 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f089f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08a00a96f0 0x7f08a013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.775+0000 7f089ffff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f08a00a9ad0 0x7f08a0142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08a00a96f0 0x7f08a013f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:35380/0 (socket says 172.21.15.110:35380) 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089f7fe640 1 -- 172.21.15.110:0/3727041884 learned_addr learned my addr 172.21.15.110:0/3727041884 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089f7fe640 10 monclient(hunting): get_auth_request con 0x7f08a00a96f0 auth_method 0 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089ffff640 10 monclient(hunting): get_auth_request con 0x7f08a00a9ad0 auth_method 0 2024-01-20T11:17:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.776+0000 7f089ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.778+0000 7f089f7fe640 10 monclient(hunting): handle_auth_done global_id 4584 payload 274 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.778+0000 7f089f7fe640 1 -- 172.21.15.110:0/3727041884 >> v2:172.21.15.134:3300/0 conn(0x7f08a00a9ad0 msgr2=0x7f08a0142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.778+0000 7f089f7fe640 1 --2- 172.21.15.110:0/3727041884 >> v2:172.21.15.134:3300/0 conn(0x7f08a00a9ad0 0x7f08a0142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.778+0000 7f089f7fe640 1 -- 172.21.15.110:0/3727041884 >> v1:172.21.15.110:6789/0 conn(0x7f08a013fa10 legacy=0x7f08a01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.778+0000 7f089f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.778+0000 7f089f7fe640 1 monclient: found mon.noname-a 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.778+0000 7f089f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.778+0000 7f089f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.778+0000 7f089f7fe640 1 -- 172.21.15.110:0/3727041884 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a00b3fb0 con 0x7f08a00a96f0 2024-01-20T11:17:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.780+0000 7f089f7fe640 1 --2- 172.21.15.110:0/3727041884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f08a013f4d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08a0074d70 tx=0x7f089002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec66d104f2b1bf12 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:40.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 1 -- 172.21.15.110:0/3727041884 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08900317e0 con 0x7f08a00a96f0 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 10 monclient: dump: 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 1 -- 172.21.15.110:0/3727041884 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0890031940 con 0x7f08a00a96f0 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 1 -- 172.21.15.110:0/3727041884 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890031cd0 con 0x7f08a00a96f0 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f089e7fc640 10 monclient: dump: 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:40.787 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f08a5e90640 5 monclient: authenticate success, global_id 4584 2024-01-20T11:17:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f08a5e90640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.784+0000 7f08a5e90640 4 set_mon_vals no callback set 2024-01-20T11:17:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 10 monclient: shutdown 2024-01-20T11:17:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 1 -- 172.21.15.110:0/3727041884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 msgr2=0x7f08a013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 1 --2- 172.21.15.110:0/3727041884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f08a013f4d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08a0074d70 tx=0x7f089002b880 comp rx=0 tx=0).stop 2024-01-20T11:17:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 1 -- 172.21.15.110:0/3727041884 shutdown_connections 2024-01-20T11:17:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 1 --2- 172.21.15.110:0/3727041884 >> v2:172.21.15.134:3300/0 conn(0x7f08a00a9ad0 0x7f08a0142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 1 --2- 172.21.15.110:0/3727041884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f08a013f4d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 1 -- 172.21.15.110:0/3727041884 >> 172.21.15.110:0/3727041884 conn(0x7f08a00b7bc0 msgr2=0x7f08a00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 1 -- 172.21.15.110:0/3727041884 shutdown_connections 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.785+0000 7f08a5e90640 1 -- 172.21.15.110:0/3727041884 wait complete. 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.786+0000 7f08a5e90640 10 monclient: build_initial_monmap 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.786+0000 7f08a5e90640 10 monclient: monmap: 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.786+0000 7f08a5e90640 1 librados: starting msgr at 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.786+0000 7f08a5e90640 1 librados: starting objecter 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.786+0000 7f08a5e90640 1 Processor -- start 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.786+0000 7f08a5e90640 1 -- start start 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.786+0000 7f08a5e90640 1 librados: setting wanted keys 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.786+0000 7f08a5e90640 1 librados: calling monclient init 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.786+0000 7f08a5e90640 10 monclient: init 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f08a01d7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient(hunting): picked mon.noname-a con 0x7f08a00a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 0x7f08a0133c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient(hunting): picked mon.noname-c con 0x7f08a00a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0137f80 0x7f08a0134170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient(hunting): picked mon.noname-b con 0x7f08a0137f80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a00ac070 con 0x7f08a00a96f0 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a01d7180 con 0x7f08a0137f80 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a013a390 con 0x7f08a00a9ad0 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f08a5e90640 10 monclient(hunting): authenticate will time out at 1504.137560s 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f089f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f08a01d7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.787+0000 7f089ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0137f80 0x7f08a0134170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f08a01d7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:35386/0 (socket says 172.21.15.110:35386) 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089f7fe640 1 -- 172.21.15.110:0/3564446506 learned_addr learned my addr 172.21.15.110:0/3564446506 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089f7fe640 10 monclient(hunting): get_auth_request con 0x7f08a00a96f0 auth_method 0 2024-01-20T11:17:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089effd640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 0x7f08a0133c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089effd640 10 monclient(hunting): get_auth_request con 0x7f08a00a9ad0 auth_method 0 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089ffff640 10 monclient(hunting): get_auth_request con 0x7f08a0137f80 auth_method 0 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.788+0000 7f089effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 10 monclient(hunting): handle_auth_done global_id 4660 payload 824 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 1 -- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 msgr2=0x7f08a0133c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 0x7f08a0133c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 1 -- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 msgr2=0x7f08a01d7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f08a01d7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 1 monclient: found mon.noname-b 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.791+0000 7f089ffff640 1 -- 172.21.15.110:0/3564446506 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08900094d0 con 0x7f08a0137f80 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.793+0000 7f089ffff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0137f80 0x7f08a0134170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f089800efc0 tx=0x7f089800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:40.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.793+0000 7f0887fff640 10 client.?.objecter ms_handle_connect 0x7f08a0137f80 2024-01-20T11:17:40.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.793+0000 7f0887fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f089ffff640 1 -- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0137f80 msgr2=0x7f08a0134170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f089ffff640 1 -- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0137f80 msgr2=0x7f08a0134170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f089ffff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0137f80 0x7f08a0134170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f089800efc0 tx=0x7f089800c490 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f089ffff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0137f80 0x7f08a0134170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f089800efc0 tx=0x7f089800c490 comp rx=0 tx=0).stop 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0137f80 msgr2=0x7f08a0134170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0137f80 0x7f08a0134170 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f088c006e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient(hunting): picked mon.noname-a con 0x7f08a00a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f088c007870 0x7f088c009d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient(hunting): picked mon.noname-b con 0x7f088c007870 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 0x7f088c00c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient(hunting): picked mon.noname-c con 0x7f08a00a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c00d2d0 con 0x7f08a00a96f0 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c00d410 con 0x7f088c007870 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c00d5d0 con 0x7f08a00a9ad0 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f0887fff640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f089ffff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 0x7f088c00c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.797+0000 7f089f7fe640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f088c006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089f7fe640 10 monclient(hunting): get_auth_request con 0x7f08a00a96f0 auth_method 0 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089ffff640 10 monclient(hunting): get_auth_request con 0x7f08a00a9ad0 auth_method 0 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:40.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.798+0000 7f089ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:17:40.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 10 monclient(hunting): handle_auth_done global_id 4660 payload 293 2024-01-20T11:17:40.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 1 -- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f088c007870 msgr2=0x7f088c009d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f088c007870 0x7f088c009d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 1 -- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 msgr2=0x7f088c006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f088c006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 1 monclient: found mon.noname-c 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089ffff640 1 -- 172.21.15.110:0/3564446506 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c00d850 con 0x7f08a00a9ad0 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.800+0000 7f089f7fe640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f088c006e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.802+0000 7f0887fff640 10 client.?.objecter ms_handle_connect 0x7f08a00a9ad0 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.802+0000 7f0887fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:40.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.802+0000 7f089ffff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 0x7f088c00c8c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0898016000 tx=0x7f089800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:40.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.805+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0898023070 con 0x7f08a00a9ad0 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.805+0000 7f0887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.805+0000 7f0887fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.805+0000 7f0887fff640 10 monclient: dump: 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.805+0000 7f0887fff640 10 monclient: _finish_auth 0 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.805+0000 7f0887fff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f0887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089800f040 con 0x7f08a00a9ad0 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 5 monclient: authenticate success, global_id 4660 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f0887fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0898009bd0 con 0x7f08a00a9ad0 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 10 monclient: _renew_subs 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a01d7660 con 0x7f08a00a9ad0 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f0887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f0887fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f0887fff640 10 monclient: dump: 2024-01-20T11:17:40.808 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f089dffb640 4 set_mon_vals no callback set 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 10 monclient: _renew_subs 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:40.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a01d7b60 con 0x7f08a00a9ad0 2024-01-20T11:17:40.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 1 librados: init done 2024-01-20T11:17:40.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:40.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:40.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:40.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:40.806+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a01d6fe0 con 0x7f08a00a9ad0 2024-01-20T11:17:41.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.048+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f0898003990 con 0x7f08a00a9ad0 2024-01-20T11:17:41.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.048+0000 7f0887fff640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f08a0137f80 0x7f088c076d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:41.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.048+0000 7f089f7fe640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f08a0137f80 0x7f088c076d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:41.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.048+0000 7f089f7fe640 10 monclient: get_auth_request con 0x7f08a0137f80 auth_method 0 2024-01-20T11:17:41.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.049+0000 7f089f7fe640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f08a0137f80 0x7f088c076d80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0890034040 tx=0x7f0890008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:41.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.049+0000 7f0887fff640 10 client.4660.objecter ms_handle_connect 0x7f08a0137f80 2024-01-20T11:17:41.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.060+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0898059f50 con 0x7f08a00a9ad0 2024-01-20T11:17:41.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.060+0000 7f0887fff640 10 client.4660.objecter ms_dispatch 0x7f08a0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:41.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.060+0000 7f0887fff640 3 client.4660.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:41.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.060+0000 7f0887fff640 3 client.4660.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:41.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.060+0000 7f0887fff640 20 client.4660.objecter dump_active .. 0 homeless 2024-01-20T11:17:41.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.060+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0898056290 con 0x7f08a00a9ad0 2024-01-20T11:17:41.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.060+0000 7f0887fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:41.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.060+0000 7f0887fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:41.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.230+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08a01d6fe0 con 0x7f08a0137f80 2024-01-20T11:17:41.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.233+0000 7f0887fff640 1 -- 172.21.15.110:0/3564446506 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64144 (secure 0 0 0) 0x7f08a01d6fe0 con 0x7f08a0137f80 2024-01-20T11:17:41.233 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:41.234 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 10 librados: watch_flush enter 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 10 librados: watch_flush exit 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 20 client.4660.objecter shutdown clearing up homeless session... 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 10 client.4660.objecter successfully canceled tick 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f08a0137f80 msgr2=0x7f088c076d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f08a0137f80 0x7f088c076d80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0890034040 tx=0x7f0890008030 comp rx=0 tx=0).stop 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 10 monclient: shutdown 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 msgr2=0x7f088c00c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 0x7f088c00c8c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0898016000 tx=0x7f089800b040 comp rx=0 tx=0).stop 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 shutdown_connections 2024-01-20T11:17:41.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f08a0137f80 0x7f088c076d80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:41.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a00a9ad0 0x7f088c00c8c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:41.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f088c007870 0x7f088c009d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:41.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 --2- 172.21.15.110:0/3564446506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a00a96f0 0x7f088c006e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:41.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 >> 172.21.15.110:0/3564446506 conn(0x7f08a00b7bc0 msgr2=0x7f08a013c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:41.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 shutdown_connections 2024-01-20T11:17:41.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 -- 172.21.15.110:0/3564446506 wait complete. 2024-01-20T11:17:41.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:41.235+0000 7f08a5e90640 1 librados: shutdown 2024-01-20T11:17:41.245 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-01-20T11:17:40.739085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001978"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.876479+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.876479+0000","last_peered":"2024-01-20T11:16:52.876479+0000","last_clean":"2024-01-20T11:16:52.876479+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.876479+0000","last_undegraded":"2024-01-20T11:16:52.876479+0000","last_fullsized":"2024-01-20T11:16:52.876479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.8310000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.117}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.874000000000001},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.7199999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.1509999999999998},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.654}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.273999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":12.509}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.6},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":13.930999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.9100000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":13.186}]}]},{"osd":7,"up_from":21,"seq":90194313249,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":13.448},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.758}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.1280000000000001},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":16.030999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":16.879000000000001},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.3650000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.051},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.634}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.050000000000001},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.959}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.304},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.788}]}]},{"osd":3,"up_from":21,"seq":90194313250,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.8789999999999996},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.1430000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.4369999999999994},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.176}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":13.055999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.721}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":17.469999999999999},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.736000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":15.585000000000001},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":14.852}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":18.268999999999998},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.327999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.9179999999999993},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":8.327}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.8259999999999996},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":15.897}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.314},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.996}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":15},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.194000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.612},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.1829999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.872},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":16.75}]}]},{"osd":1,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.5800000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":7.8799999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.557},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.6500000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":13.255000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.4649999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":15.241},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.31}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.849},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":16.006}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.957000000000001},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.780999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":4.5599999999999996},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":9.9629999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9029999999999996},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":6.1849999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":11.686},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.388}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.394},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.282999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.917999999999999},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.577}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.241},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":16.128}]}]},{"osd":6,"up_from":15,"seq":64424509473,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.3090000000000002},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.4179999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.561999999999999},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":6.7039999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.124000000000001},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.1}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.260999999999999},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.281000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":10.69},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.6379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.962999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8.5950000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:44.247 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:44.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.354+0000 7f52589a4640 10 monclient: get_monmap_and_config 2024-01-20T11:17:44.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.354+0000 7f52589a4640 10 monclient: build_initial_monmap 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.354+0000 7f52589a4640 10 monclient: monmap: 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:44.355425+0000 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:44.355425+0000 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:44.355 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.355+0000 7f52589a4640 1 Processor -- start 2024-01-20T11:17:44.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.355+0000 7f52589a4640 1 -- start start 2024-01-20T11:17:44.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.355+0000 7f52589a4640 10 monclient: init 2024-01-20T11:17:44.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:44.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-01-20T11:17:44.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52540a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:44.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52540a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:44.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5254142af0 0x7f525414d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 10 monclient(hunting): picked mon.noname-b con 0x7f5254142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52540b25c0 con 0x7f52540a96f0 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5254142f50 con 0x7f52540a9ad0 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52540aaea0 con 0x7f5254142af0 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f52589a4640 10 monclient(hunting): authenticate will time out at 1507.706444s 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f525259c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f52540a96f0 0x7f525413f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51984/0 (socket says 172.21.15.110:51984) 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f525259c640 1 -- 172.21.15.110:0/1194884413 learned_addr learned my addr 172.21.15.110:0/1194884413 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.356+0000 7f5252d9d640 1 --2- 172.21.15.110:0/1194884413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5254142af0 0x7f525414d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.357+0000 7f5252d9d640 10 monclient(hunting): get_auth_request con 0x7f5254142af0 auth_method 0 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.357+0000 7f5252d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.357+0000 7f5252d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.357+0000 7f5252d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.357+0000 7f5252d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:44.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.357+0000 7f5252d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:44.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.357+0000 7f5252d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:44.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.358+0000 7f5251d9b640 1 -- 172.21.15.110:0/1194884413 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831185371 0 0) 0x7f52540b25c0 con 0x7f52540a96f0 2024-01-20T11:17:44.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.358+0000 7f5251d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:17:44.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.358+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.358+0000 7f5251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.358+0000 7f5251d9b640 10 monclient(hunting): my global_id is 4590 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.358+0000 7f5251d9b640 1 -- 172.21.15.110:0/1194884413 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5240003520 con 0x7f52540a96f0 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5251d9b640 1 -- 172.21.15.110:0/1194884413 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423369954 0 0) 0x7f523c001110 con 0x7f52540a9ad0 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5251d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5251d9b640 10 monclient(hunting): my global_id is 4663 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5251d9b640 1 -- 172.21.15.110:0/1194884413 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52540b25c0 con 0x7f52540a9ad0 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5252d9d640 10 monclient(hunting): handle_auth_done global_id 4475 payload 274 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5252d9d640 1 -- 172.21.15.110:0/1194884413 >> v1:172.21.15.134:6789/0 conn(0x7f52540a9ad0 legacy=0x7f52541423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5252d9d640 1 -- 172.21.15.110:0/1194884413 >> v1:172.21.15.110:6789/0 conn(0x7f52540a96f0 legacy=0x7f525413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5252d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5252d9d640 1 monclient: found mon.noname-b 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5252d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:44.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5252d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:44.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.359+0000 7f5252d9d640 1 -- 172.21.15.110:0/1194884413 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525413fbe0 con 0x7f5254142af0 2024-01-20T11:17:44.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.361+0000 7f5252d9d640 1 --2- 172.21.15.110:0/1194884413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5254142af0 0x7f525414d860 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f524c009dc0 tx=0x7f524c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84775a48ddb585e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 1 -- 172.21.15.110:0/1194884413 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c02cc40 con 0x7f5254142af0 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 10 monclient: dump: 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:44.365 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 10 monclient: _finish_auth 0 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 1 -- 172.21.15.110:0/1194884413 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524c02cda0 con 0x7f5254142af0 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 1 -- 172.21.15.110:0/1194884413 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c034660 con 0x7f5254142af0 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 5 monclient: authenticate success, global_id 4475 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 4 set_mon_vals no callback set 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f5251d9b640 10 monclient: dump: 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:44.367 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 10 monclient: shutdown 2024-01-20T11:17:44.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:44.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 1 -- 172.21.15.110:0/1194884413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5254142af0 msgr2=0x7f525414d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 1 --2- 172.21.15.110:0/1194884413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5254142af0 0x7f525414d860 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f524c009dc0 tx=0x7f524c02be90 comp rx=0 tx=0).stop 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 1 -- 172.21.15.110:0/1194884413 shutdown_connections 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 1 --2- 172.21.15.110:0/1194884413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5254142af0 0x7f525414d860 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 1 -- 172.21.15.110:0/1194884413 >> 172.21.15.110:0/1194884413 conn(0x7f52540b7bc0 msgr2=0x7f52540b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 1 -- 172.21.15.110:0/1194884413 shutdown_connections 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.365+0000 7f52589a4640 1 -- 172.21.15.110:0/1194884413 wait complete. 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.366+0000 7f52589a4640 10 monclient: build_initial_monmap 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.366+0000 7f52589a4640 10 monclient: monmap: 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:44.371 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 librados: starting msgr at 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 librados: starting objecter 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 Processor -- start 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 -- start start 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 librados: setting wanted keys 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 librados: calling monclient init 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient: init 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52540a96f0 0x7f52540ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient(hunting): picked mon.noname-c con 0x7f52540a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52540a9ad0 0x7f52541d76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient(hunting): picked mon.noname-a con 0x7f52540a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5254142af0 0x7f5254135d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient(hunting): picked mon.noname-b con 0x7f5254142af0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52541d7290 con 0x7f52540a9ad0 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52540ad7b0 con 0x7f5254142af0 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52540abd00 con 0x7f52540a96f0 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.367+0000 7f52589a4640 10 monclient(hunting): authenticate will time out at 1507.718158s 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f525259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52540a96f0 0x7f52540ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f5252d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5254142af0 0x7f5254135d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f524bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52540a9ad0 0x7f52541d76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f524bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52540a9ad0 0x7f52541d76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:35406/0 (socket says 172.21.15.110:35406) 2024-01-20T11:17:44.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f5252d9d640 10 monclient(hunting): get_auth_request con 0x7f5254142af0 auth_method 0 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f524bfff640 1 -- 172.21.15.110:0/2250174090 learned_addr learned my addr 172.21.15.110:0/2250174090 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f5252d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f525259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52540a96f0 0x7f52540ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:53846/0 (socket says 172.21.15.110:53846) 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f5252d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f5252d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f524bfff640 10 monclient(hunting): get_auth_request con 0x7f52540a9ad0 auth_method 0 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f524bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f524bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f524bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f525259c640 10 monclient(hunting): get_auth_request con 0x7f52540a96f0 auth_method 0 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f525259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f525259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.368+0000 7f525259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.369+0000 7f5252d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:44.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.369+0000 7f5252d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:44.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.369+0000 7f5252d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:44.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.369+0000 7f525259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:44.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.369+0000 7f525259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:44.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.369+0000 7f525259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:44.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.369+0000 7f524bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:44.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.369+0000 7f524bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:44.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.369+0000 7f524bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:44.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 10 monclient(hunting): handle_auth_done global_id 4666 payload 824 2024-01-20T11:17:44.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 1 -- 172.21.15.110:0/2250174090 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52540a96f0 msgr2=0x7f52540ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52540a96f0 0x7f52540ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 1 -- 172.21.15.110:0/2250174090 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52540a9ad0 msgr2=0x7f52541d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52540a9ad0 0x7f52541d76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:17:44.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 1 monclient: found mon.noname-b 2024-01-20T11:17:44.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:44.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:44.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.371+0000 7f5252d9d640 1 -- 172.21.15.110:0/2250174090 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524c009a70 con 0x7f5254142af0 2024-01-20T11:17:44.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.373+0000 7f5252d9d640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5254142af0 0x7f5254135d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f524c033000 tx=0x7f524c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:44.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.373+0000 7f524affd640 10 client.?.objecter ms_handle_connect 0x7f5254142af0 2024-01-20T11:17:44.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.373+0000 7f524affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:44.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 1 -- 172.21.15.110:0/2250174090 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c03e610 con 0x7f5254142af0 2024-01-20T11:17:44.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:44.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:44.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 10 monclient: dump: 2024-01-20T11:17:44.378 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 10 monclient: _finish_auth 0 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 1 -- 172.21.15.110:0/2250174090 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524c03e770 con 0x7f5254142af0 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 1 -- 172.21.15.110:0/2250174090 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c03eb00 con 0x7f5254142af0 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f52589a4640 5 monclient: authenticate success, global_id 4666 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f52589a4640 10 monclient: _renew_subs 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f52589a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52541383a0 con 0x7f5254142af0 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f525159a640 4 set_mon_vals no callback set 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f524affd640 10 monclient: dump: 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.380 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f52589a4640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f52589a4640 10 monclient: _renew_subs 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f52589a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5254138690 con 0x7f5254142af0 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.377+0000 7f52589a4640 1 librados: init done 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.378+0000 7f52589a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.378+0000 7f52589a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.378+0000 7f52589a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:44.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.378+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52541d7400 con 0x7f5254142af0 2024-01-20T11:17:44.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.395+0000 7f524affd640 1 -- 172.21.15.110:0/2250174090 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f524c03b050 con 0x7f5254142af0 2024-01-20T11:17:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.396+0000 7f524affd640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523406c890 0x7f523406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.396+0000 7f525259c640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523406c890 0x7f523406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.396+0000 7f525259c640 10 monclient: get_auth_request con 0x7f523406c890 auth_method 0 2024-01-20T11:17:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.396+0000 7f524affd640 10 client.4666.objecter ms_handle_connect 0x7f523406c890 2024-01-20T11:17:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.396+0000 7f525259c640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523406c890 0x7f523406ed50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f52440010e0 tx=0x7f5244009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:44.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.397+0000 7f524affd640 1 -- 172.21.15.110:0/2250174090 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f524c07ab00 con 0x7f5254142af0 2024-01-20T11:17:44.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.397+0000 7f524affd640 10 client.4666.objecter ms_dispatch 0x7f5254000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:44.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.397+0000 7f524affd640 3 client.4666.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:44.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.397+0000 7f524affd640 3 client.4666.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:44.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.397+0000 7f524affd640 20 client.4666.objecter dump_active .. 0 homeless 2024-01-20T11:17:44.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.635+0000 7f524affd640 1 -- 172.21.15.110:0/2250174090 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f524c076e40 con 0x7f5254142af0 2024-01-20T11:17:44.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.635+0000 7f524affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:44.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.635+0000 7f524affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:44.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.804+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52540b4400 con 0x7f523406c890 2024-01-20T11:17:44.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.807+0000 7f524affd640 1 -- 172.21.15.110:0/2250174090 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64219 (secure 0 0 0) 0x7f52440014f0 con 0x7f523406c890 2024-01-20T11:17:44.808 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:44.808 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 10 librados: watch_flush enter 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 10 librados: watch_flush exit 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 20 client.4666.objecter shutdown clearing up homeless session... 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 10 client.4666.objecter successfully canceled tick 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523406c890 msgr2=0x7f523406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523406c890 0x7f523406ed50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f52440010e0 tx=0x7f5244009040 comp rx=0 tx=0).stop 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 10 monclient: shutdown 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5254142af0 msgr2=0x7f5254135d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5254142af0 0x7f5254135d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f524c033000 tx=0x7f524c004a40 comp rx=0 tx=0).stop 2024-01-20T11:17:44.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 shutdown_connections 2024-01-20T11:17:44.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523406c890 0x7f523406ed50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5254142af0 0x7f5254135d50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52540a9ad0 0x7f52541d76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 --2- 172.21.15.110:0/2250174090 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52540a96f0 0x7f52540ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 >> 172.21.15.110:0/2250174090 conn(0x7f52540b7bc0 msgr2=0x7f52540b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:44.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 shutdown_connections 2024-01-20T11:17:44.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 -- 172.21.15.110:0/2250174090 wait complete. 2024-01-20T11:17:44.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.809+0000 7f52589a4640 1 librados: shutdown 2024-01-20T11:17:44.819 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-01-20T11:17:44.739717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001857"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.0419999999999998},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.2140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.725},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.005000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.512},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.037000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.282},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":14.281000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.797000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":11.897}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.0990000000000002},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":15.050000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.758},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":12.315}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.0039999999999996},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":7.3319999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.8390000000000004},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":18.869}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.9},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.4939999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.2270000000000003},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":12.750999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":19.93},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.462}]}]},{"osd":3,"up_from":21,"seq":90194313251,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":12.731},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.6829999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.5579999999999998},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.365}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":14.673999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.589}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.433},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.569000000000001},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":18.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":17.512},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.323}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":10.670999999999999},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":7.1020000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.2840000000000007},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.647}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.587999999999999},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.3840000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.564},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":17.300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":17.917000000000002},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":15.593999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":18.609000000000002},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":14.669}]}]},{"osd":1,"up_from":17,"seq":73014444066,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.2290000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":9.4969999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":14.565},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.141}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":10.124000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":5.9930000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":12.037000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":10.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.362},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":15.352}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.9190000000000005},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.744}]}]},{"osd":0,"up_from":17,"seq":73014444066,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":11.503},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.2770000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.5899999999999999},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.244999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":8.1319999999999997},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":5.6870000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":8.7319999999999993},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.460000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.474},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":10.836}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":9.2509999999999994},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":14.321999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":4.2380000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.7230000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":16.265999999999998},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":10.558999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.675000000000001},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":17.050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.340999999999999},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.749000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.755000000000001},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":17.965}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.7590000000000003},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:44.821 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:44.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.920+0000 7f4ace7c9640 10 monclient: get_monmap_and_config 2024-01-20T11:17:44.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.920+0000 7f4ace7c9640 10 monclient: build_initial_monmap 2024-01-20T11:17:44.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.920+0000 7f4ace7c9640 10 monclient: monmap: 2024-01-20T11:17:44.920 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:44.920 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.920 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:44.921127+0000 2024-01-20T11:17:44.920 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:44.921127+0000 2024-01-20T11:17:44.921 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:44.921 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.921 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:44.921 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:44.921 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:44.921 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:44.921 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:44.921 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 1 Processor -- start 2024-01-20T11:17:44.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 1 -- start start 2024-01-20T11:17:44.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient: init 2024-01-20T11:17:44.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:44.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-01-20T11:17:44.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ac806f920 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:44.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4ac806fd00 0x7f4ac8143950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac806fd00 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:44.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac8143e90 0x7f4ac814a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac8143e90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac80715b0 con 0x7f4ac806f920 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ac81442e0 con 0x7f4ac806fd00 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac8071890 con 0x7f4ac8143e90 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.921+0000 7f4ace7c9640 10 monclient(hunting): authenticate will time out at 1508.272088s 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.922+0000 7f4accd66640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac8143e90 0x7f4ac814a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.922+0000 7f4ac77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4ac806fd00 0x7f4ac8143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.922+0000 7f4accd66640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac8143e90 0x7f4ac814a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:53856/0 (socket says 172.21.15.110:53856) 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.922+0000 7f4accd66640 1 -- 172.21.15.110:0/4203394549 learned_addr learned my addr 172.21.15.110:0/4203394549 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.922+0000 7f4accd66640 10 monclient(hunting): get_auth_request con 0x7f4ac8143e90 auth_method 0 2024-01-20T11:17:44.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.922+0000 7f4accd66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:44.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.922+0000 7f4accd66640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.922+0000 7f4accd66640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.923+0000 7f4ac77fe640 10 monclient(hunting): get_auth_request con 0x7f4ac806fd00 auth_method 0 2024-01-20T11:17:44.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.923+0000 7f4ac77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:44.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.923+0000 7f4ac77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.923+0000 7f4ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.923+0000 7f4accd66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:44.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.923+0000 7f4accd66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:44.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.923+0000 7f4accd66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:44.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.924+0000 7f4ac77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:44.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.924+0000 7f4ac77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:44.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.924+0000 7f4ac77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.924+0000 7f4accd66640 10 monclient(hunting): handle_auth_done global_id 4481 payload 274 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.925+0000 7f4accd66640 1 -- 172.21.15.110:0/4203394549 >> v2:172.21.15.110:3300/0 conn(0x7f4ac806fd00 msgr2=0x7f4ac8143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.925+0000 7f4accd66640 1 --2- 172.21.15.110:0/4203394549 >> v2:172.21.15.110:3300/0 conn(0x7f4ac806fd00 0x7f4ac8143950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.925+0000 7f4accd66640 1 -- 172.21.15.110:0/4203394549 >> v1:172.21.15.110:6789/0 conn(0x7f4ac806f920 legacy=0x7f4ac813d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.925+0000 7f4accd66640 10 monclient: _finish_hunting 0 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.925+0000 7f4accd66640 1 monclient: found mon.noname-b 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.925+0000 7f4accd66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.925+0000 7f4accd66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.925+0000 7f4accd66640 1 -- 172.21.15.110:0/4203394549 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac8071b70 con 0x7f4ac8143e90 2024-01-20T11:17:44.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.926+0000 7f4accd66640 1 --2- 172.21.15.110:0/4203394549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac8143e90 0x7f4ac814a720 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0009e50 tx=0x7f4ac002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0738486fe3815cc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:44.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 1 -- 172.21.15.110:0/4203394549 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac002ca70 con 0x7f4ac8143e90 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 10 monclient: dump: 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:44.935 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 1 -- 172.21.15.110:0/4203394549 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac002cbd0 con 0x7f4ac8143e90 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 1 -- 172.21.15.110:0/4203394549 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00343f0 con 0x7f4ac8143e90 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 5 monclient: authenticate success, global_id 4481 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 4 set_mon_vals no callback set 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ac6ffd640 10 monclient: dump: 2024-01-20T11:17:44.936 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 10 monclient: shutdown 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 1 -- 172.21.15.110:0/4203394549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac8143e90 msgr2=0x7f4ac814a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 1 --2- 172.21.15.110:0/4203394549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac8143e90 0x7f4ac814a720 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0009e50 tx=0x7f4ac002bed0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 1 -- 172.21.15.110:0/4203394549 shutdown_connections 2024-01-20T11:17:44.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 1 --2- 172.21.15.110:0/4203394549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac8143e90 0x7f4ac814a720 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 1 --2- 172.21.15.110:0/4203394549 >> v2:172.21.15.110:3300/0 conn(0x7f4ac806fd00 0x7f4ac8143950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 1 -- 172.21.15.110:0/4203394549 >> 172.21.15.110:0/4203394549 conn(0x7f4ac8066d60 msgr2=0x7f4ac80b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.930+0000 7f4ace7c9640 1 -- 172.21.15.110:0/4203394549 shutdown_connections 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.931+0000 7f4ace7c9640 1 -- 172.21.15.110:0/4203394549 wait complete. 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.931+0000 7f4ace7c9640 10 monclient: build_initial_monmap 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.931+0000 7f4ace7c9640 10 monclient: monmap: 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:44.939 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.932+0000 7f4ace7c9640 1 librados: starting msgr at 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.932+0000 7f4ace7c9640 1 librados: starting objecter 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.932+0000 7f4ace7c9640 1 Processor -- start 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.932+0000 7f4ace7c9640 1 -- start start 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.932+0000 7f4ace7c9640 1 librados: setting wanted keys 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.932+0000 7f4ace7c9640 1 librados: calling monclient init 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.932+0000 7f4ace7c9640 10 monclient: init 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac806f920 0x7f4ac81da6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac806f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:44.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ac806fd00 0x7f4ac80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac806fd00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac8143e90 0x7f4ac806cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac8143e90 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac806b800 con 0x7f4ac8143e90 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac81db680 con 0x7f4ac806fd00 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac806c280 con 0x7f4ac806f920 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ace7c9640 10 monclient(hunting): authenticate will time out at 1508.283282s 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac806f920 0x7f4ac81da6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4accd66640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac8143e90 0x7f4ac806cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ac806fd00 0x7f4ac80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac806f920 0x7f4ac81da6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:53860/0 (socket says 172.21.15.110:53860) 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac7fff640 1 -- 172.21.15.110:0/522215242 learned_addr learned my addr 172.21.15.110:0/522215242 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac7fff640 10 monclient(hunting): get_auth_request con 0x7f4ac806f920 auth_method 0 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac77fe640 10 monclient(hunting): get_auth_request con 0x7f4ac806fd00 auth_method 0 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.933+0000 7f4ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4accd66640 10 monclient(hunting): get_auth_request con 0x7f4ac8143e90 auth_method 0 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4accd66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4accd66640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4accd66640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4ac77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:44.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4ac77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:44.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4ac77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:44.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4ac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:44.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4ac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:44.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4ac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:44.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4accd66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:44.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4accd66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:44.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.934+0000 7f4accd66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 10 monclient(hunting): handle_auth_done global_id 4669 payload 824 2024-01-20T11:17:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 1 -- 172.21.15.110:0/522215242 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac806f920 msgr2=0x7f4ac81da6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac806f920 0x7f4ac81da6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 1 -- 172.21.15.110:0/522215242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac8143e90 msgr2=0x7f4ac806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac8143e90 0x7f4ac806cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 1 monclient: found mon.noname-b 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.937+0000 7f4ac77fe640 1 -- 172.21.15.110:0/522215242 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac0009b30 con 0x7f4ac806fd00 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.939+0000 7f4ac77fe640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ac806fd00 0x7f4ac80aeba0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00e870 tx=0x7f4abc00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.939+0000 7f4ac4ff9640 10 client.?.objecter ms_handle_connect 0x7f4ac806fd00 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.939+0000 7f4ac4ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 1 -- 172.21.15.110:0/522215242 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc0095a0 con 0x7f4ac806fd00 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 10 monclient: dump: 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.946 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.947 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:44.947 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.947 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:44.947 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:44.947 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:44.947 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 10 monclient: _finish_auth 0 2024-01-20T11:17:44.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:17:44.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 1 -- 172.21.15.110:0/522215242 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4abc00c940 con 0x7f4ac806fd00 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 1 -- 172.21.15.110:0/522215242 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc00ccb0 con 0x7f4ac806fd00 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac4ff9640 10 monclient: dump: 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:44.948 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ace7c9640 5 monclient: authenticate success, global_id 4669 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ace7c9640 10 monclient: _renew_subs 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ace7c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac80ae150 con 0x7f4ac806fd00 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ac67fc640 4 set_mon_vals no callback set 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ace7c9640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ace7c9640 10 monclient: _renew_subs 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ace7c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac80af6b0 con 0x7f4ac806fd00 2024-01-20T11:17:44.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.942+0000 7f4ace7c9640 1 librados: init done 2024-01-20T11:17:44.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.943+0000 7f4ace7c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:44.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.943+0000 7f4ace7c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:44.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.943+0000 7f4ace7c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:44.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.943+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac8075490 con 0x7f4ac806fd00 2024-01-20T11:17:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.962+0000 7f4ac4ff9640 1 -- 172.21.15.110:0/522215242 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4abc005990 con 0x7f4ac806fd00 2024-01-20T11:17:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.962+0000 7f4ac4ff9640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ab406c8b0 0x7f4ab406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.962+0000 7f4ac7fff640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ab406c8b0 0x7f4ab406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.962+0000 7f4ac7fff640 10 monclient: get_auth_request con 0x7f4ab406c8b0 auth_method 0 2024-01-20T11:17:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.963+0000 7f4ac7fff640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ab406c8b0 0x7f4ab406ed70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4ab80017a0 tx=0x7f4ab8009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.963+0000 7f4ac4ff9640 10 client.4669.objecter ms_handle_connect 0x7f4ab406c8b0 2024-01-20T11:17:44.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.964+0000 7f4ac4ff9640 1 -- 172.21.15.110:0/522215242 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4abc058b70 con 0x7f4ac806fd00 2024-01-20T11:17:44.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.964+0000 7f4ac4ff9640 10 client.4669.objecter ms_dispatch 0x7f4ac8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:44.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.964+0000 7f4ac4ff9640 3 client.4669.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:44.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.964+0000 7f4ac4ff9640 3 client.4669.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:44.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:44.964+0000 7f4ac4ff9640 20 client.4669.objecter dump_active .. 0 homeless 2024-01-20T11:17:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.203+0000 7f4ac4ff9640 1 -- 172.21.15.110:0/522215242 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4abc054eb0 con 0x7f4ac806fd00 2024-01-20T11:17:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.203+0000 7f4ac4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.203+0000 7f4ac4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:45.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.380+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ac80b03c0 con 0x7f4ab406c8b0 2024-01-20T11:17:45.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.383+0000 7f4ac4ff9640 1 -- 172.21.15.110:0/522215242 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64219 (secure 0 0 0) 0x7f4ab8001440 con 0x7f4ab406c8b0 2024-01-20T11:17:45.383 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:45.384 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 10 librados: watch_flush enter 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 10 librados: watch_flush exit 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 20 client.4669.objecter shutdown clearing up homeless session... 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 10 client.4669.objecter successfully canceled tick 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ab406c8b0 msgr2=0x7f4ab406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ab406c8b0 0x7f4ab406ed70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4ab80017a0 tx=0x7f4ab8009040 comp rx=0 tx=0).stop 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 10 monclient: shutdown 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ac806fd00 msgr2=0x7f4ac80aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ac806fd00 0x7f4ac80aeba0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00e870 tx=0x7f4abc00ed30 comp rx=0 tx=0).stop 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 shutdown_connections 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ab406c8b0 0x7f4ab406ed70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac8143e90 0x7f4ac806cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ac806fd00 0x7f4ac80aeba0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 --2- 172.21.15.110:0/522215242 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ac806f920 0x7f4ac81da6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 >> 172.21.15.110:0/522215242 conn(0x7f4ac8066d60 msgr2=0x7f4ac813adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 shutdown_connections 2024-01-20T11:17:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 -- 172.21.15.110:0/522215242 wait complete. 2024-01-20T11:17:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.385+0000 7f4ace7c9640 1 librados: shutdown 2024-01-20T11:17:45.396 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-01-20T11:17:44.739717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001857"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.0419999999999998},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.2140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.725},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.005000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.512},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.037000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.282},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":14.281000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.797000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":11.897}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.0990000000000002},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":15.050000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.758},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":12.315}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.0039999999999996},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":7.3319999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.8390000000000004},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":18.869}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.9},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.4939999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.2270000000000003},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":12.750999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":19.93},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.462}]}]},{"osd":3,"up_from":21,"seq":90194313251,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":12.731},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.6829999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.5579999999999998},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.365}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":14.673999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.589}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.433},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.569000000000001},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":18.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":17.512},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.323}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":10.670999999999999},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":7.1020000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.2840000000000007},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.647}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.587999999999999},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.3840000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.564},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":17.300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":17.917000000000002},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":15.593999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":18.609000000000002},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":14.669}]}]},{"osd":1,"up_from":17,"seq":73014444066,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.2290000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":9.4969999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":14.565},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.141}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":10.124000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":5.9930000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":12.037000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":10.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.362},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":15.352}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.9190000000000005},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.744}]}]},{"osd":0,"up_from":17,"seq":73014444066,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":11.503},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.2770000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.5899999999999999},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.244999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":8.1319999999999997},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":5.6870000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":8.7319999999999993},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.460000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.474},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":10.836}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":9.2509999999999994},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":14.321999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":4.2380000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.7230000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":16.265999999999998},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":10.558999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.675000000000001},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":17.050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.340999999999999},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.749000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.755000000000001},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":17.965}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.7590000000000003},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:45.397 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:45.397 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:45.398 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:45.398 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:45.398 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:45.398 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:45.398 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:45.398 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:45.398 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.498+0000 7f2fb24ec640 10 monclient: get_monmap_and_config 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.498+0000 7f2fb24ec640 10 monclient: build_initial_monmap 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.498+0000 7f2fb24ec640 10 monclient: monmap: 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:45.498680+0000 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:45.498680+0000 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:45.499 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.498+0000 7f2fb24ec640 1 Processor -- start 2024-01-20T11:17:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.498+0000 7f2fb24ec640 1 -- start start 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.498+0000 7f2fb24ec640 10 monclient: init 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 0x7f2fac13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 10 monclient(hunting): picked mon.noname-b con 0x7f2fac0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2fac13fa10 0x7f2fac1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 10 monclient(hunting): picked mon.noname-c con 0x7f2fac13fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2fac0a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac0b25c0 con 0x7f2fac0a9ad0 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac0aaea0 con 0x7f2fac13fa10 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0ab0a0 con 0x7f2fac0a96f0 2024-01-20T11:17:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fb24ec640 10 monclient(hunting): authenticate will time out at 1508.849629s 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fab7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2fac13fa10 0x7f2fac1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.499+0000 7f2fabfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 0x7f2fac13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fab7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2fac13fa10 0x7f2fac1423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:57820/0 (socket says 172.21.15.110:57820) 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fab7fe640 1 -- 172.21.15.110:0/1062547663 learned_addr learned my addr 172.21.15.110:0/1062547663 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fab7fe640 10 monclient(hunting): get_auth_request con 0x7f2fac13fa10 auth_method 0 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fab7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fabfff640 10 monclient(hunting): get_auth_request con 0x7f2fac0a96f0 auth_method 0 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fabfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fabfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.500+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_done global_id 4672 payload 274 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fab7fe640 1 -- 172.21.15.110:0/1062547663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 msgr2=0x7f2fac13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fab7fe640 1 --2- 172.21.15.110:0/1062547663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 0x7f2fac13f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fab7fe640 1 -- 172.21.15.110:0/1062547663 >> v1:172.21.15.134:6789/0 conn(0x7f2fac0a9ad0 legacy=0x7f2fac142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fab7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fab7fe640 1 monclient: found mon.noname-c 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fab7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fab7fe640 1 -- 172.21.15.110:0/1062547663 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac0b3fb0 con 0x7f2fac13fa10 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.502+0000 7f2fabfff640 1 --2- 172.21.15.110:0/1062547663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 0x7f2fac13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.504+0000 7f2fab7fe640 1 --2- 172.21.15.110:0/1062547663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac13fa10 0x7f2fac1423e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0009870 tx=0x7f2fa002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d81572636fc448c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 1 -- 172.21.15.110:0/1062547663 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00317c0 con 0x7f2fac13fa10 2024-01-20T11:17:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 10 monclient: dump: 2024-01-20T11:17:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 1 -- 172.21.15.110:0/1062547663 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa0031920 con 0x7f2fac13fa10 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 1 -- 172.21.15.110:0/1062547663 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0031cb0 con 0x7f2fac13fa10 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2fb24ec640 5 monclient: authenticate success, global_id 4672 2024-01-20T11:17:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2fb24ec640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2fb24ec640 4 set_mon_vals no callback set 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.508+0000 7f2faaffd640 10 monclient: dump: 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:45.510 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:45.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 10 monclient: shutdown 2024-01-20T11:17:45.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:45.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 1 -- 172.21.15.110:0/1062547663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac13fa10 msgr2=0x7f2fac1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:45.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 1 --2- 172.21.15.110:0/1062547663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac13fa10 0x7f2fac1423e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0009870 tx=0x7f2fa002b860 comp rx=0 tx=0).stop 2024-01-20T11:17:45.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 1 -- 172.21.15.110:0/1062547663 shutdown_connections 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 1 --2- 172.21.15.110:0/1062547663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac13fa10 0x7f2fac1423e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 1 --2- 172.21.15.110:0/1062547663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 0x7f2fac13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 1 -- 172.21.15.110:0/1062547663 >> 172.21.15.110:0/1062547663 conn(0x7f2fac0b7bc0 msgr2=0x7f2fac0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 1 -- 172.21.15.110:0/1062547663 shutdown_connections 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 1 -- 172.21.15.110:0/1062547663 wait complete. 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 10 monclient: build_initial_monmap 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.509+0000 7f2fb24ec640 10 monclient: monmap: 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 librados: starting msgr at 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 librados: starting objecter 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 Processor -- start 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 -- start start 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 librados: setting wanted keys 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 librados: calling monclient init 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient: init 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:17:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 0x7f2fac1386a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient(hunting): picked mon.noname-c con 0x7f2fac0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0a9ad0 0x7f2fac13a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient(hunting): picked mon.noname-a con 0x7f2fac0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac138e80 0x7f2fac13ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient(hunting): picked mon.noname-b con 0x7f2fac138e80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac137fa0 con 0x7f2fac0a9ad0 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac139ec0 con 0x7f2fac138e80 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac13a1a0 con 0x7f2fac0a96f0 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.510+0000 7f2fb24ec640 10 monclient(hunting): authenticate will time out at 1508.861084s 2024-01-20T11:17:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fabfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 0x7f2fac1386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fb0a89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac138e80 0x7f2fac13ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fab7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0a9ad0 0x7f2fac13a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fb0a89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac138e80 0x7f2fac13ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:57824/0 (socket says 172.21.15.110:57824) 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fb0a89640 1 -- 172.21.15.110:0/2012269578 learned_addr learned my addr 172.21.15.110:0/2012269578 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fb0a89640 10 monclient(hunting): get_auth_request con 0x7f2fac138e80 auth_method 0 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fb0a89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fb0a89640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fb0a89640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fabfff640 10 monclient(hunting): get_auth_request con 0x7f2fac0a96f0 auth_method 0 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fabfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fabfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fab7fe640 10 monclient(hunting): get_auth_request con 0x7f2fac0a9ad0 auth_method 0 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fab7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.511+0000 7f2fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.512+0000 7f2fb0a89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.512+0000 7f2fb0a89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.512+0000 7f2fb0a89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.512+0000 7f2fabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.512+0000 7f2fabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.512+0000 7f2fabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.512+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.512+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.512+0000 7f2fab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 10 monclient(hunting): handle_auth_done global_id 4675 payload 824 2024-01-20T11:17:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 1 -- 172.21.15.110:0/2012269578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 msgr2=0x7f2fac1386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 0x7f2fac1386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 1 -- 172.21.15.110:0/2012269578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0a9ad0 msgr2=0x7f2fac13a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0a9ad0 0x7f2fac13a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 10 monclient: _finish_hunting 0 2024-01-20T11:17:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 1 monclient: found mon.noname-b 2024-01-20T11:17:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.514+0000 7f2fb0a89640 1 -- 172.21.15.110:0/2012269578 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c009400 con 0x7f2fac138e80 2024-01-20T11:17:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.516+0000 7f2fb0a89640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac138e80 0x7f2fac13ae70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400c780 tx=0x7f2fa400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.516+0000 7f2fa8ff9640 10 client.?.objecter ms_handle_connect 0x7f2fac138e80 2024-01-20T11:17:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.516+0000 7f2fa8ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 1 -- 172.21.15.110:0/2012269578 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4003d80 con 0x7f2fac138e80 2024-01-20T11:17:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 10 monclient: dump: 2024-01-20T11:17:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 10 monclient: _finish_auth 0 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 1 -- 172.21.15.110:0/2012269578 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa400ce90 con 0x7f2fac138e80 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fb24ec640 5 monclient: authenticate success, global_id 4675 2024-01-20T11:17:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fb24ec640 10 monclient: _renew_subs 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fb24ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa00094d0 con 0x7f2fac138e80 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 1 -- 172.21.15.110:0/2012269578 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40052b0 con 0x7f2fac138e80 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fa8ff9640 10 monclient: dump: 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2faa7fc640 4 set_mon_vals no callback set 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fb24ec640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fb24ec640 10 monclient: _renew_subs 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fb24ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac146190 con 0x7f2fac138e80 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.520+0000 7f2fb24ec640 1 librados: init done 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.521+0000 7f2fb24ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.521+0000 7f2fb24ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.521+0000 7f2fb24ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.521+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac1463e0 con 0x7f2fac138e80 2024-01-20T11:17:45.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.538+0000 7f2fa8ff9640 1 -- 172.21.15.110:0/2012269578 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2fa4005410 con 0x7f2fac138e80 2024-01-20T11:17:45.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.539+0000 7f2fa8ff9640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f9806c890 0x7f2f9806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:45.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.539+0000 7f2fabfff640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f9806c890 0x7f2f9806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:45.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.539+0000 7f2fabfff640 10 monclient: get_auth_request con 0x7f2f9806c890 auth_method 0 2024-01-20T11:17:45.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.539+0000 7f2fa8ff9640 10 client.4675.objecter ms_handle_connect 0x7f2f9806c890 2024-01-20T11:17:45.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.539+0000 7f2fabfff640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f9806c890 0x7f2f9806ed50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009cd0 tx=0x7f2f9c009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.540+0000 7f2fa8ff9640 1 -- 172.21.15.110:0/2012269578 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2fa4057e80 con 0x7f2fac138e80 2024-01-20T11:17:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.540+0000 7f2fa8ff9640 10 client.4675.objecter ms_dispatch 0x7f2fac000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.540+0000 7f2fa8ff9640 3 client.4675.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.540+0000 7f2fa8ff9640 3 client.4675.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.541+0000 7f2fa8ff9640 20 client.4675.objecter dump_active .. 0 homeless 2024-01-20T11:17:45.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.777+0000 7f2fa8ff9640 1 -- 172.21.15.110:0/2012269578 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2fa40541c0 con 0x7f2fac138e80 2024-01-20T11:17:45.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.777+0000 7f2fa8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:45.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:45.777+0000 7f2fa8ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:46.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.010+0000 7f2fb24ec640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:46.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.010+0000 7f2fb24ec640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:46.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.010+0000 7f2fb24ec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:46.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.010+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2fac138110 con 0x7f2fac138e80 2024-01-20T11:17:46.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.018+0000 7f2fa8ff9640 1 -- 172.21.15.110:0/2012269578 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7f2fa4017510 con 0x7f2fac138e80 2024-01-20T11:17:46.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.018+0000 7f2fa8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:46.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.018+0000 7f2fa8ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:46.018 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:46.018 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":200,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 10 librados: watch_flush enter 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 10 librados: watch_flush exit 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 20 client.4675.objecter shutdown clearing up homeless session... 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 10 client.4675.objecter successfully canceled tick 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f9806c890 msgr2=0x7f2f9806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f9806c890 0x7f2f9806ed50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009cd0 tx=0x7f2f9c009190 comp rx=0 tx=0).stop 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 10 monclient: shutdown 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac138e80 msgr2=0x7f2fac13ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac138e80 0x7f2fac13ae70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400c780 tx=0x7f2fa400cc40 comp rx=0 tx=0).stop 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 shutdown_connections 2024-01-20T11:17:46.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f9806c890 0x7f2f9806ed50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2fac138e80 0x7f2fac13ae70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fac0a9ad0 0x7f2fac13a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 --2- 172.21.15.110:0/2012269578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fac0a96f0 0x7f2fac1386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 >> 172.21.15.110:0/2012269578 conn(0x7f2fac0b7bc0 msgr2=0x7f2fac0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:46.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 shutdown_connections 2024-01-20T11:17:46.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 -- 172.21.15.110:0/2012269578 wait complete. 2024-01-20T11:17:46.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.020+0000 7f2fb24ec640 1 librados: shutdown 2024-01-20T11:17:46.030 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:46.031 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:46.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.130+0000 7fe87e1a4640 10 monclient: get_monmap_and_config 2024-01-20T11:17:46.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.130+0000 7fe87e1a4640 10 monclient: build_initial_monmap 2024-01-20T11:17:46.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.130+0000 7fe87e1a4640 10 monclient: monmap: 2024-01-20T11:17:46.130 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:46.130 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:46.130 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:46.131119+0000 2024-01-20T11:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:46.131119+0000 2024-01-20T11:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:46.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 1 Processor -- start 2024-01-20T11:17:46.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 1 -- start start 2024-01-20T11:17:46.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient: init 2024-01-20T11:17:46.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:46.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-01-20T11:17:46.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 0x7fe8780b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient(hunting): picked mon.noname-b con 0x7fe87806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe8780af790 0x7fe878145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient(hunting): picked mon.noname-c con 0x7fe8780af790 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8780b0900 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87806c220 con 0x7fe8780b0900 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe87806bee0 con 0x7fe8780af790 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8780ada30 con 0x7fe87806bb00 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.131+0000 7fe87e1a4640 10 monclient(hunting): authenticate will time out at 1509.482043s 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe876ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe8780af790 0x7fe878145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe877fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe8780b0900 0x7fe878145cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:52004/0 (socket says 172.21.15.110:52004) 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe877fff640 1 -- 172.21.15.110:0/4094137279 learned_addr learned my addr 172.21.15.110:0/4094137279 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:46.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe8777fe640 1 --2- 172.21.15.110:0/4094137279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 0x7fe8780b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe876ffd640 10 monclient(hunting): get_auth_request con 0x7fe8780af790 auth_method 0 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe876ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe876ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe876ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe8777fe640 10 monclient(hunting): get_auth_request con 0x7fe87806bb00 auth_method 0 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe8777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe8777fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.132+0000 7fe8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.133+0000 7fe876ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.133+0000 7fe876ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.133+0000 7fe876ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.133+0000 7fe8777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.133+0000 7fe8777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:46.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.133+0000 7fe8777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:46.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.134+0000 7fe876ffd640 10 monclient(hunting): handle_auth_done global_id 4678 payload 274 2024-01-20T11:17:46.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.134+0000 7fe876ffd640 1 -- 172.21.15.110:0/4094137279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 msgr2=0x7fe8780b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:46.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.134+0000 7fe876ffd640 1 --2- 172.21.15.110:0/4094137279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 0x7fe8780b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.134+0000 7fe876ffd640 1 -- 172.21.15.110:0/4094137279 >> v1:172.21.15.110:6789/0 conn(0x7fe8780b0900 legacy=0x7fe878145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:46.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.134+0000 7fe876ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:46.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.134+0000 7fe876ffd640 1 monclient: found mon.noname-c 2024-01-20T11:17:46.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.134+0000 7fe876ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:46.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.134+0000 7fe876ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:46.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.134+0000 7fe876ffd640 1 -- 172.21.15.110:0/4094137279 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8780add10 con 0x7fe8780af790 2024-01-20T11:17:46.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.136+0000 7fe876ffd640 1 --2- 172.21.15.110:0/4094137279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780af790 0x7fe878145770 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009790 tx=0x7fe86c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe146891e36cc6f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:46.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 1 -- 172.21.15.110:0/4094137279 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c031770 con 0x7fe8780af790 2024-01-20T11:17:46.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:46.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:46.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 10 monclient: dump: 2024-01-20T11:17:46.140 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 10 monclient: _finish_auth 0 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 10 monclient: _check_auth_tickets 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 1 -- 172.21.15.110:0/4094137279 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86c0318d0 con 0x7fe8780af790 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 1 -- 172.21.15.110:0/4094137279 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c031c60 con 0x7fe8780af790 2024-01-20T11:17:46.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe8767fc640 10 monclient: dump: 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe87e1a4640 5 monclient: authenticate success, global_id 4678 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe87e1a4640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:46.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.140+0000 7fe87e1a4640 4 set_mon_vals no callback set 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 10 monclient: shutdown 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 1 -- 172.21.15.110:0/4094137279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780af790 msgr2=0x7fe878145770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 1 --2- 172.21.15.110:0/4094137279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780af790 0x7fe878145770 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009790 tx=0x7fe86c02b810 comp rx=0 tx=0).stop 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 1 -- 172.21.15.110:0/4094137279 shutdown_connections 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 1 --2- 172.21.15.110:0/4094137279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780af790 0x7fe878145770 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 1 --2- 172.21.15.110:0/4094137279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 0x7fe8780b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 1 -- 172.21.15.110:0/4094137279 >> 172.21.15.110:0/4094137279 conn(0x7fe878066d60 msgr2=0x7fe8780b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 1 -- 172.21.15.110:0/4094137279 shutdown_connections 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.141+0000 7fe87e1a4640 1 -- 172.21.15.110:0/4094137279 wait complete. 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.142+0000 7fe87e1a4640 10 monclient: build_initial_monmap 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.142+0000 7fe87e1a4640 10 monclient: monmap: 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:46.146 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.142+0000 7fe87e1a4640 1 librados: starting msgr at 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.142+0000 7fe87e1a4640 1 librados: starting objecter 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.142+0000 7fe87e1a4640 1 Processor -- start 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.142+0000 7fe87e1a4640 1 -- start start 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.142+0000 7fe87e1a4640 1 librados: setting wanted keys 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.142+0000 7fe87e1a4640 1 librados: calling monclient init 2024-01-20T11:17:46.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.142+0000 7fe87e1a4640 10 monclient: init 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 0x7fe878141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient(hunting): picked mon.noname-c con 0x7fe87806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8780af790 0x7fe8781da6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient(hunting): picked mon.noname-a con 0x7fe8780af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780b0900 0x7fe8781dac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient(hunting): picked mon.noname-b con 0x7fe8780b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8781db160 con 0x7fe8780af790 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87811dbe0 con 0x7fe8780b0900 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:46.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87811efc0 con 0x7fe87806bb00 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe87e1a4640 10 monclient(hunting): authenticate will time out at 1509.493364s 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe876ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8780af790 0x7fe8781da6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe8777fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 0x7fe878141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe877fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780b0900 0x7fe8781dac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.143+0000 7fe8777fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 0x7fe878141740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:53884/0 (socket says 172.21.15.110:53884) 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe8777fe640 1 -- 172.21.15.110:0/699647269 learned_addr learned my addr 172.21.15.110:0/699647269 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe8777fe640 10 monclient(hunting): get_auth_request con 0x7fe87806bb00 auth_method 0 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe8777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe8777fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe8777fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe876ffd640 10 monclient(hunting): get_auth_request con 0x7fe8780af790 auth_method 0 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe876ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe876ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe876ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe877fff640 10 monclient(hunting): get_auth_request con 0x7fe8780b0900 auth_method 0 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe877fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe877fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe8777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe8777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe8777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe876ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe876ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe876ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe877fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe877fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:46.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.144+0000 7fe877fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 10 monclient(hunting): handle_auth_done global_id 4611 payload 824 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 1 -- 172.21.15.110:0/699647269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 msgr2=0x7fe878141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 0x7fe878141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 1 -- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780b0900 msgr2=0x7fe8781dac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780b0900 0x7fe8781dac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 1 monclient: found mon.noname-a 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe876ffd640 1 -- 172.21.15.110:0/699647269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c0093a0 con 0x7fe8780af790 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.147+0000 7fe877fff640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780b0900 0x7fe8781dac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:46.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.149+0000 7fe876ffd640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8780af790 0x7fe8781da6e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe8780b6890 tx=0x7fe86c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.149+0000 7fe85ffff640 10 client.?.objecter ms_handle_connect 0x7fe8780af790 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.149+0000 7fe85ffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe85ffff640 1 -- 172.21.15.110:0/699647269 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c005da0 con 0x7fe8780af790 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe85ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe85ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe85ffff640 10 monclient: dump: 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:46.153 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe85ffff640 10 monclient: _finish_auth 0 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe85ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe85ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe87e1a4640 5 monclient: authenticate success, global_id 4611 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe87e1a4640 10 monclient: _renew_subs 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe87e1a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87811e070 con 0x7fe8780af790 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe87e1a4640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe87e1a4640 10 monclient: _renew_subs 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe87e1a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87811d780 con 0x7fe8780af790 2024-01-20T11:17:46.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.151+0000 7fe87e1a4640 1 librados: init done 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.152+0000 7fe87e1a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.152+0000 7fe87e1a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.152+0000 7fe87e1a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.152+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87811d9c0 con 0x7fe8780af790 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.154+0000 7fe85ffff640 1 -- 172.21.15.110:0/699647269 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86c006030 con 0x7fe8780af790 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.154+0000 7fe85ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.154+0000 7fe85ffff640 1 -- 172.21.15.110:0/699647269 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0063a0 con 0x7fe8780af790 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.154+0000 7fe85ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.154+0000 7fe85ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:17:46.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.154+0000 7fe85ffff640 10 monclient: dump: 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:46.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.154+0000 7fe875ffb640 4 set_mon_vals no callback set 2024-01-20T11:17:46.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.389+0000 7fe85ffff640 1 -- 172.21.15.110:0/699647269 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe86c006500 con 0x7fe8780af790 2024-01-20T11:17:46.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.390+0000 7fe85ffff640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe86406c860 0x7fe86406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:46.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.390+0000 7fe8777fe640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe86406c860 0x7fe86406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:46.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.390+0000 7fe8777fe640 10 monclient: get_auth_request con 0x7fe86406c860 auth_method 0 2024-01-20T11:17:46.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.391+0000 7fe8777fe640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe86406c860 0x7fe86406ed20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe868009c00 tx=0x7fe868009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:46.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.391+0000 7fe85ffff640 10 client.4611.objecter ms_handle_connect 0x7fe86406c860 2024-01-20T11:17:46.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.404+0000 7fe85ffff640 1 -- 172.21.15.110:0/699647269 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe86c07a0a0 con 0x7fe8780af790 2024-01-20T11:17:46.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.404+0000 7fe85ffff640 10 client.4611.objecter ms_dispatch 0x7fe878000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:46.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.404+0000 7fe85ffff640 3 client.4611.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:46.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.404+0000 7fe85ffff640 3 client.4611.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:46.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.404+0000 7fe85ffff640 20 client.4611.objecter dump_active .. 0 homeless 2024-01-20T11:17:46.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.404+0000 7fe85ffff640 1 -- 172.21.15.110:0/699647269 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe86c0763e0 con 0x7fe8780af790 2024-01-20T11:17:46.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.404+0000 7fe85ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:46.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.404+0000 7fe85ffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:46.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.584+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe878075490 con 0x7fe86406c860 2024-01-20T11:17:46.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.587+0000 7fe85ffff640 1 -- 172.21.15.110:0/699647269 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64219 (secure 0 0 0) 0x7fe878075490 con 0x7fe86406c860 2024-01-20T11:17:46.587 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:46.588 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:46.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 10 librados: watch_flush enter 2024-01-20T11:17:46.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 10 librados: watch_flush exit 2024-01-20T11:17:46.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 20 client.4611.objecter shutdown clearing up homeless session... 2024-01-20T11:17:46.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 10 client.4611.objecter successfully canceled tick 2024-01-20T11:17:46.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe86406c860 msgr2=0x7fe86406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:46.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe86406c860 0x7fe86406ed20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe868009c00 tx=0x7fe868009190 comp rx=0 tx=0).stop 2024-01-20T11:17:46.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 10 monclient: shutdown 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8780af790 msgr2=0x7fe8781da6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8780af790 0x7fe8781da6e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe8780b6890 tx=0x7fe86c0046c0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 shutdown_connections 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe86406c860 0x7fe86406ed20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8780b0900 0x7fe8781dac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8780af790 0x7fe8781da6e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 --2- 172.21.15.110:0/699647269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe87806bb00 0x7fe878141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 >> 172.21.15.110:0/699647269 conn(0x7fe878066d60 msgr2=0x7fe878142bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 shutdown_connections 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 -- 172.21.15.110:0/699647269 wait complete. 2024-01-20T11:17:46.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:46.589+0000 7fe87e1a4640 1 librados: shutdown 2024-01-20T11:17:46.599 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-01-20T11:17:44.739717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001857"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.0419999999999998},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.2140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.725},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.005000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.512},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.037000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.282},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":14.281000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.797000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":11.897}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.0990000000000002},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":15.050000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313250,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.758},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":12.315}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.0039999999999996},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":7.3319999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.8390000000000004},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":18.869}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.9},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.4939999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.2270000000000003},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":12.750999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":19.93},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.462}]}]},{"osd":3,"up_from":21,"seq":90194313251,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":12.731},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.6829999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.5579999999999998},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.365}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":14.673999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.589}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.433},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.569000000000001},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":18.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":17.512},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.323}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":10.670999999999999},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":7.1020000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.2840000000000007},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.647}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.587999999999999},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.3840000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.564},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":17.300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":17.917000000000002},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":15.593999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":18.609000000000002},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":14.669}]}]},{"osd":1,"up_from":17,"seq":73014444066,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.2290000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":9.4969999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":14.565},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.141}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":10.124000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":5.9930000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":12.037000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":10.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.362},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":15.352}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.9190000000000005},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.744}]}]},{"osd":0,"up_from":17,"seq":73014444066,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":11.503},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.2770000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.5899999999999999},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":15.244999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":8.1319999999999997},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":5.6870000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":8.7319999999999993},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.460000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.474},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":10.836}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":9.2509999999999994},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":14.321999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":4.2380000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.7230000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":16.265999999999998},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":10.558999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.675000000000001},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":17.050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.340999999999999},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.749000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.755000000000001},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":17.965}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.7590000000000003},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:49.602 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:49.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.701+0000 7fc4e2d73640 10 monclient: get_monmap_and_config 2024-01-20T11:17:49.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.701+0000 7fc4e2d73640 10 monclient: build_initial_monmap 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.701+0000 7fc4e2d73640 10 monclient: monmap: 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:49.701898+0000 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:49.701898+0000 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 1 Processor -- start 2024-01-20T11:17:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 1 -- start start 2024-01-20T11:17:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient: init 2024-01-20T11:17:49.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:49.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-01-20T11:17:49.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc4dc0a96f0 0x7fc4dc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:49.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc0a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:49.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc13fa10 0x7fc4dc1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:49.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc13fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:49.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc4dc0a9ad0 0x7fc4dc142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc0a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4dc0aaea0 con 0x7fc4dc0a9ad0 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4dc0b25c0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0ab0a0 con 0x7fc4dc13fa10 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.702+0000 7fc4e2d73640 10 monclient(hunting): authenticate will time out at 1513.052840s 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e0b0f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc4dc0a96f0 0x7fc4dc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e1310640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc4dc0a9ad0 0x7fc4dc142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4dbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc13fa10 0x7fc4dc1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e0b0f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc4dc0a96f0 0x7fc4dc13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46248/0 (socket says 172.21.15.110:46248) 2024-01-20T11:17:49.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e0b0f640 1 -- 172.21.15.110:0/3349318465 learned_addr learned my addr 172.21.15.110:0/3349318465 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e0b0f640 10 monclient(hunting): get_auth_request con 0x7fc4dc0a96f0 auth_method 0 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e0b0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e0b0f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e0b0f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4dbfff640 10 monclient(hunting): get_auth_request con 0x7fc4dc13fa10 auth_method 0 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4dbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e1310640 10 monclient(hunting): get_auth_request con 0x7fc4dc0a9ad0 auth_method 0 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e1310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e1310640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.703+0000 7fc4e1310640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.704+0000 7fc4e0b0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.704+0000 7fc4e0b0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:49.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.704+0000 7fc4e0b0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.704+0000 7fc4dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.704+0000 7fc4dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.704+0000 7fc4dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.704+0000 7fc4e1310640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.704+0000 7fc4e1310640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.704+0000 7fc4e1310640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 10 monclient(hunting): handle_auth_done global_id 4684 payload 274 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 1 -- 172.21.15.110:0/3349318465 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc13fa10 msgr2=0x7fc4dc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 1 --2- 172.21.15.110:0/3349318465 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc13fa10 0x7fc4dc1423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 1 -- 172.21.15.110:0/3349318465 >> v2:172.21.15.110:3300/0 conn(0x7fc4dc0a9ad0 msgr2=0x7fc4dc142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 1 --2- 172.21.15.110:0/3349318465 >> v2:172.21.15.110:3300/0 conn(0x7fc4dc0a9ad0 0x7fc4dc142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 10 monclient: _finish_hunting 0 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 1 monclient: found mon.noname-c 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:49.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:49.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.705+0000 7fc4e0b0f640 1 -- 172.21.15.110:0/3349318465 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc0b3fb0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.707+0000 7fc4e0b0f640 1 --2- 172.21.15.110:0/3349318465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4dc0a96f0 0x7fc4dc13f4d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0b5720 tx=0x7fc4d002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41b42d57b14a2bc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 1 -- 172.21.15.110:0/3349318465 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00317e0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 10 monclient: dump: 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:49.711 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 10 monclient: _finish_auth 0 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 1 -- 172.21.15.110:0/3349318465 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d0031940 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 5 monclient: authenticate success, global_id 4684 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 4 set_mon_vals no callback set 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 1 -- 172.21.15.110:0/3349318465 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0031cd0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:49.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4db7fe640 10 monclient: dump: 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:49.713 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 10 monclient: shutdown 2024-01-20T11:17:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3349318465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4dc0a96f0 msgr2=0x7fc4dc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3349318465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4dc0a96f0 0x7fc4dc13f4d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0b5720 tx=0x7fc4d002b880 comp rx=0 tx=0).stop 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3349318465 shutdown_connections 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3349318465 >> v2:172.21.15.110:3300/0 conn(0x7fc4dc0a9ad0 0x7fc4dc142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3349318465 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc13fa10 0x7fc4dc1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3349318465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4dc0a96f0 0x7fc4dc13f4d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3349318465 >> 172.21.15.110:0/3349318465 conn(0x7fc4dc0b7bc0 msgr2=0x7fc4dc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3349318465 shutdown_connections 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.711+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3349318465 wait complete. 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.712+0000 7fc4e2d73640 10 monclient: build_initial_monmap 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.712+0000 7fc4e2d73640 10 monclient: monmap: 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.712+0000 7fc4e2d73640 1 librados: starting msgr at 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.712+0000 7fc4e2d73640 1 librados: starting objecter 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.712+0000 7fc4e2d73640 1 Processor -- start 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.712+0000 7fc4e2d73640 1 -- start start 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.712+0000 7fc4e2d73640 1 librados: setting wanted keys 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.712+0000 7fc4e2d73640 1 librados: calling monclient init 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.712+0000 7fc4e2d73640 10 monclient: init 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc0a96f0 0x7fc4dc141460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4dc0a9ad0 0x7fc4dc1da820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4dc0ab410 0x7fc4dc1dad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc0ab410 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0acff0 con 0x7fc4dc0a9ad0 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0ad260 con 0x7fc4dc0ab410 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0ab9c0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.713+0000 7fc4e2d73640 10 monclient(hunting): authenticate will time out at 1513.063792s 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4dbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4dc0a9ad0 0x7fc4dc1da820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e1310640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4dc0ab410 0x7fc4dc1dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc0a96f0 0x7fc4dc141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:49.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc0a96f0 0x7fc4dc141460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:59382/0 (socket says 172.21.15.110:59382) 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4dbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4dc0a9ad0 0x7fc4dc1da820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40058/0 (socket says 172.21.15.110:40058) 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 1 -- 172.21.15.110:0/3368200489 learned_addr learned my addr 172.21.15.110:0/3368200489 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e1310640 10 monclient(hunting): get_auth_request con 0x7fc4dc0ab410 auth_method 0 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e1310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e1310640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e1310640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 10 monclient(hunting): get_auth_request con 0x7fc4dc0a96f0 auth_method 0 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4dbfff640 10 monclient(hunting): get_auth_request con 0x7fc4dc0a9ad0 auth_method 0 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4dbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:49.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:49.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:49.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e0b0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:49.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e1310640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:49.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.714+0000 7fc4e1310640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:49.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.715+0000 7fc4e1310640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:49.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.715+0000 7fc4dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:49.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.715+0000 7fc4dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:49.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.715+0000 7fc4dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:49.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 10 monclient(hunting): handle_auth_done global_id 4502 payload 824 2024-01-20T11:17:49.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 1 -- 172.21.15.110:0/3368200489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4dc0ab410 msgr2=0x7fc4dc1dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:49.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4dc0ab410 0x7fc4dc1dad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:49.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 1 -- 172.21.15.110:0/3368200489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4dc0a9ad0 msgr2=0x7fc4dc1da820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:49.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4dc0a9ad0 0x7fc4dc1da820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:49.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 10 monclient: _finish_hunting 0 2024-01-20T11:17:49.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 1 monclient: found mon.noname-c 2024-01-20T11:17:49.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:49.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:17:49.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.717+0000 7fc4e0b0f640 1 -- 172.21.15.110:0/3368200489 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d00094d0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.719+0000 7fc4e0b0f640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc0a96f0 0x7fc4dc141460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0000c00 tx=0x7fc4d00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:49.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.719+0000 7fc4d97fa640 10 client.?.objecter ms_handle_connect 0x7fc4dc0a96f0 2024-01-20T11:17:49.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.719+0000 7fc4d97fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:49.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 1 -- 172.21.15.110:0/3368200489 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0032880 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:49.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:49.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 10 monclient: dump: 2024-01-20T11:17:49.724 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:49.724 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:49.724 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:49.724 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 10 monclient: _finish_auth 0 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 10 monclient: _check_auth_tickets 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 1 -- 172.21.15.110:0/3368200489 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d00329e0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 1 -- 172.21.15.110:0/3368200489 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0032d90 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:49.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4d97fa640 10 monclient: dump: 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4e2d73640 5 monclient: authenticate success, global_id 4502 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4e2d73640 10 monclient: _renew_subs 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4e2d73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc145180 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4daffd640 4 set_mon_vals no callback set 2024-01-20T11:17:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4e2d73640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4e2d73640 10 monclient: _renew_subs 2024-01-20T11:17:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4e2d73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc0b09e0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.723+0000 7fc4e2d73640 1 librados: init done 2024-01-20T11:17:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.724+0000 7fc4e2d73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.724+0000 7fc4e2d73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.724+0000 7fc4e2d73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:49.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.724+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc075490 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.742+0000 7fc4d97fa640 1 -- 172.21.15.110:0/3368200489 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc4d004e450 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.742+0000 7fc4d97fa640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4c006c8b0 0x7fc4c006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:49.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.743+0000 7fc4dbfff640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4c006c8b0 0x7fc4c006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:49.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.743+0000 7fc4dbfff640 10 monclient: get_auth_request con 0x7fc4c006c8b0 auth_method 0 2024-01-20T11:17:49.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.743+0000 7fc4dbfff640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4c006c8b0 0x7fc4c006ed70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0ac8f0 tx=0x7fc4c8009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:49.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.743+0000 7fc4d97fa640 10 client.4502.objecter ms_handle_connect 0x7fc4c006c8b0 2024-01-20T11:17:49.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.745+0000 7fc4d97fa640 1 -- 172.21.15.110:0/3368200489 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc4d007a8b0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.745+0000 7fc4d97fa640 10 client.4502.objecter ms_dispatch 0x7fc4dc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:49.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.745+0000 7fc4d97fa640 3 client.4502.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:49.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.745+0000 7fc4d97fa640 3 client.4502.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:49.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.745+0000 7fc4d97fa640 20 client.4502.objecter dump_active .. 0 homeless 2024-01-20T11:17:49.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.977+0000 7fc4d97fa640 1 -- 172.21.15.110:0/3368200489 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc4d0076bf0 con 0x7fc4dc0a96f0 2024-01-20T11:17:49.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.977+0000 7fc4d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:49.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:49.977+0000 7fc4d97fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:50.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.144+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4dc145310 con 0x7fc4c006c8b0 2024-01-20T11:17:50.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.147+0000 7fc4d97fa640 1 -- 172.21.15.110:0/3368200489 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64086 (secure 0 0 0) 0x7fc4dc145310 con 0x7fc4c006c8b0 2024-01-20T11:17:50.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:50.148 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 10 librados: watch_flush enter 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 10 librados: watch_flush exit 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 20 client.4502.objecter shutdown clearing up homeless session... 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 10 client.4502.objecter successfully canceled tick 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4c006c8b0 msgr2=0x7fc4c006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4c006c8b0 0x7fc4c006ed70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0ac8f0 tx=0x7fc4c8009190 comp rx=0 tx=0).stop 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 10 monclient: shutdown 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc0a96f0 msgr2=0x7fc4dc141460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc0a96f0 0x7fc4dc141460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0000c00 tx=0x7fc4d00046c0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 shutdown_connections 2024-01-20T11:17:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4c006c8b0 0x7fc4c006ed70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4dc0ab410 0x7fc4dc1dad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4dc0a9ad0 0x7fc4dc1da820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 --2- 172.21.15.110:0/3368200489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4dc0a96f0 0x7fc4dc141460 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 >> 172.21.15.110:0/3368200489 conn(0x7fc4dc0b7bc0 msgr2=0x7fc4dc145a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 shutdown_connections 2024-01-20T11:17:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 -- 172.21.15.110:0/3368200489 wait complete. 2024-01-20T11:17:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.149+0000 7fc4e2d73640 1 librados: shutdown 2024-01-20T11:17:50.160 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-01-20T11:17:48.740363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001870"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.367},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.298}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.738},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.497999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":16.492999999999999},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.331}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.238},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":18.768999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":17.632000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":8.8079999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.925000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":19.885000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.8290000000000006},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.7869999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.218},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.567}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":16.056999999999999},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.9120000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.343},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.510999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.243},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.347}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":9.5579999999999998},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.137}]}]},{"osd":3,"up_from":21,"seq":90194313252,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":12.731},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.6829999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.5579999999999998},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.365}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":14.673999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.589}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.433},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.569000000000001},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":18.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":17.512},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.323}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444066,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.415},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":9.4749999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":4.5890000000000004},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.1370000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.978},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.005000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.73},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.43}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.583},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.2720000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.836},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.65}]}]},{"osd":1,"up_from":17,"seq":73014444067,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":12.638999999999999},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":8.7699999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.202},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.5640000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":10.739000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.9100000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":14.276},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.725}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.802},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":16.329999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.406000000000001},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.311}]}]},{"osd":0,"up_from":17,"seq":73014444067,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":5.7679999999999998},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.0979999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.7140000000000004},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.76}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":8.2370000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":15.044}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.786},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.352}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.35},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.351000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":10.023999999999999},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":11.061}]}]},{"osd":6,"up_from":15,"seq":64424509475,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":9.577},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.9160000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.3130000000000006},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":6.8849999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.868},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.645}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.84},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.1880000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.702999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.67}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.454999999999998},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.965}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:50.161 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:50.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.260+0000 7f4c0b361640 10 monclient: get_monmap_and_config 2024-01-20T11:17:50.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.260+0000 7f4c0b361640 10 monclient: build_initial_monmap 2024-01-20T11:17:50.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.260+0000 7f4c0b361640 10 monclient: monmap: 2024-01-20T11:17:50.261 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:50.261186+0000 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:50.261186+0000 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.261+0000 7f4c0b361640 1 Processor -- start 2024-01-20T11:17:50.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.261+0000 7f4c0b361640 1 -- start start 2024-01-20T11:17:50.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.261+0000 7f4c0b361640 10 monclient: init 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.261+0000 7f4c0b361640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.261+0000 7f4c0b361640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.261+0000 7f4c0b361640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4c040af3a0 0x7f4c0413b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.261+0000 7f4c0b361640 10 monclient(hunting): picked mon.noname-c con 0x7f4c040af3a0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.261+0000 7f4c0b361640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c040ab6d0 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c0b361640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040adb30 0x7f4c0413bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c0b361640 10 monclient(hunting): picked mon.noname-b con 0x7f4c040adb30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c0b361640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c04141930 con 0x7f4c040ab6d0 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c0b361640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:50.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c0b361640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c04142920 con 0x7f4c040af3a0 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c0b361640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c0b361640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c040af780 con 0x7f4c040adb30 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c0b361640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c0b361640 10 monclient(hunting): authenticate will time out at 1513.612232s 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c098fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040adb30 0x7f4c0413bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.262+0000 7f4c090fd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4c040af3a0 0x7f4c0413b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c098fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040adb30 0x7f4c0413bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:59398/0 (socket says 172.21.15.110:59398) 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c098fe640 1 -- 172.21.15.110:0/1526304603 learned_addr learned my addr 172.21.15.110:0/1526304603 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c098fe640 10 monclient(hunting): get_auth_request con 0x7f4c040adb30 auth_method 0 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c098fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c098fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c098fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c090fd640 10 monclient(hunting): get_auth_request con 0x7f4c040af3a0 auth_method 0 2024-01-20T11:17:50.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c090fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c090fd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c090fd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c098fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c098fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c098fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c090fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c090fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.263+0000 7f4c090fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.265+0000 7f4c098fe640 10 monclient(hunting): handle_auth_done global_id 4505 payload 274 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.265+0000 7f4c098fe640 1 -- 172.21.15.110:0/1526304603 >> v2:172.21.15.134:3300/0 conn(0x7f4c040af3a0 msgr2=0x7f4c0413b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.265+0000 7f4c098fe640 1 --2- 172.21.15.110:0/1526304603 >> v2:172.21.15.134:3300/0 conn(0x7f4c040af3a0 0x7f4c0413b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.265+0000 7f4c098fe640 1 -- 172.21.15.110:0/1526304603 >> v1:172.21.15.134:6789/0 conn(0x7f4c040ab6d0 legacy=0x7f4c040ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.265+0000 7f4c098fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.265+0000 7f4c098fe640 1 monclient: found mon.noname-b 2024-01-20T11:17:50.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.265+0000 7f4c098fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:50.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.265+0000 7f4c098fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:50.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.265+0000 7f4c098fe640 1 -- 172.21.15.110:0/1526304603 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c040ab2b0 con 0x7f4c040adb30 2024-01-20T11:17:50.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.267+0000 7f4c098fe640 1 --2- 172.21.15.110:0/1526304603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040adb30 0x7f4c0413bc70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009e50 tx=0x7f4bfc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4629ad3c9bb3578e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.269+0000 7f4bfbfff640 1 -- 172.21.15.110:0/1526304603 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc02ca70 con 0x7f4c040adb30 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.269+0000 7f4bfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.269+0000 7f4bfbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.269+0000 7f4bfbfff640 10 monclient: dump: 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:50.269 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:50.270 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.269+0000 7f4bfbfff640 10 monclient: _finish_auth 0 2024-01-20T11:17:50.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.269+0000 7f4bfbfff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:50.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.269+0000 7f4bfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:50.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.269+0000 7f4c0b361640 5 monclient: authenticate success, global_id 4505 2024-01-20T11:17:50.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.269+0000 7f4c0b361640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:50.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.270+0000 7f4bfbfff640 1 -- 172.21.15.110:0/1526304603 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc02cbd0 con 0x7f4c040adb30 2024-01-20T11:17:50.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.270+0000 7f4bfbfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:50.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.270+0000 7f4bfbfff640 1 -- 172.21.15.110:0/1526304603 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0343f0 con 0x7f4c040adb30 2024-01-20T11:17:50.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.270+0000 7f4c0b361640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:50.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.270+0000 7f4c0b361640 4 set_mon_vals no callback set 2024-01-20T11:17:50.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.270+0000 7f4bfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:50.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.270+0000 7f4bfbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:50.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.270+0000 7f4bfbfff640 10 monclient: dump: 2024-01-20T11:17:50.272 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:50.272 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.272 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.272 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.272 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:50.272 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.272 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:50.273 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:50.273 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:50.273 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 10 monclient: shutdown 2024-01-20T11:17:50.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:50.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 1 -- 172.21.15.110:0/1526304603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040adb30 msgr2=0x7f4c0413bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 1 --2- 172.21.15.110:0/1526304603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040adb30 0x7f4c0413bc70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009e50 tx=0x7f4bfc02bed0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 1 -- 172.21.15.110:0/1526304603 shutdown_connections 2024-01-20T11:17:50.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 1 --2- 172.21.15.110:0/1526304603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040adb30 0x7f4c0413bc70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 1 --2- 172.21.15.110:0/1526304603 >> v2:172.21.15.134:3300/0 conn(0x7f4c040af3a0 0x7f4c0413b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 1 -- 172.21.15.110:0/1526304603 >> 172.21.15.110:0/1526304603 conn(0x7f4c040b3e60 msgr2=0x7f4c040b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:50.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 1 -- 172.21.15.110:0/1526304603 shutdown_connections 2024-01-20T11:17:50.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.271+0000 7f4c0b361640 1 -- 172.21.15.110:0/1526304603 wait complete. 2024-01-20T11:17:50.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.272+0000 7f4c0b361640 10 monclient: build_initial_monmap 2024-01-20T11:17:50.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.272+0000 7f4c0b361640 10 monclient: monmap: 2024-01-20T11:17:50.277 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.272+0000 7f4c0b361640 1 librados: starting msgr at 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.272+0000 7f4c0b361640 1 librados: starting objecter 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.272+0000 7f4c0b361640 1 Processor -- start 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.272+0000 7f4c0b361640 1 -- start start 2024-01-20T11:17:50.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.272+0000 7f4c0b361640 1 librados: setting wanted keys 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.272+0000 7f4c0b361640 1 librados: calling monclient init 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.272+0000 7f4c0b361640 10 monclient: init 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040ab6d0 0x7f4c041ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient(hunting): picked mon.noname-c con 0x7f4c040ab6d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c040adb30 0x7f4c041d8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient(hunting): picked mon.noname-b con 0x7f4c040adb30 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c040af3a0 0x7f4c041d8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient(hunting): picked mon.noname-a con 0x7f4c040af3a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0406dc60 con 0x7f4c040af3a0 2024-01-20T11:17:50.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c041c9be0 con 0x7f4c040adb30 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c041d92b0 con 0x7f4c040ab6d0 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c0b361640 10 monclient(hunting): authenticate will time out at 1513.623615s 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c090fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040ab6d0 0x7f4c041ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.273+0000 7f4c098fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c040af3a0 0x7f4c041d8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c090fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040ab6d0 0x7f4c041ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:59412/0 (socket says 172.21.15.110:59412) 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c090fd640 1 -- 172.21.15.110:0/109009088 learned_addr learned my addr 172.21.15.110:0/109009088 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c090fd640 10 monclient(hunting): get_auth_request con 0x7f4c040ab6d0 auth_method 0 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c090fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c090fd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c090fd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:50.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c088fc640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c040adb30 0x7f4c041d8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c098fe640 10 monclient(hunting): get_auth_request con 0x7f4c040af3a0 auth_method 0 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c098fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c098fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c098fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c090fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c090fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c090fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c098fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.274+0000 7f4c098fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.275+0000 7f4c098fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 1 -- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c040adb30 msgr2=0x7f4c041d8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c040adb30 0x7f4c041d8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 1 -- 172.21.15.110:0/109009088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c040af3a0 msgr2=0x7f4c041d8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c040af3a0 0x7f4c041d8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 10 monclient: _finish_hunting 0 2024-01-20T11:17:50.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 1 monclient: found mon.noname-c 2024-01-20T11:17:50.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:50.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:17:50.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.277+0000 7f4c090fd640 1 -- 172.21.15.110:0/109009088 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc009b30 con 0x7f4c040ab6d0 2024-01-20T11:17:50.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.280+0000 7f4c090fd640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040ab6d0 0x7f4c041ca950 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000e690 tx=0x7f4c0000eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:50.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.280+0000 7f4bf9ffb640 10 client.?.objecter ms_handle_connect 0x7f4c040ab6d0 2024-01-20T11:17:50.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.280+0000 7f4bf9ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:50.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 1 -- 172.21.15.110:0/109009088 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0000ca30 con 0x7f4c040ab6d0 2024-01-20T11:17:50.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:50.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:50.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 10 monclient: dump: 2024-01-20T11:17:50.284 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:50.284 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.284 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.284 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.284 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:50.284 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 10 monclient: _finish_auth 0 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 1 -- 172.21.15.110:0/109009088 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0000cb90 con 0x7f4c040ab6d0 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 1 -- 172.21.15.110:0/109009088 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c000113f0 con 0x7f4c040ab6d0 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bf9ffb640 10 monclient: dump: 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:50.285 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4c0b361640 5 monclient: authenticate success, global_id 4508 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4c0b361640 10 monclient: _renew_subs 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4c0b361640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4bfb7fe640 4 set_mon_vals no callback set 2024-01-20T11:17:50.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.283+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c041cae90 con 0x7f4c040ab6d0 2024-01-20T11:17:50.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.284+0000 7f4c0b361640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:50.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.284+0000 7f4c0b361640 10 monclient: _renew_subs 2024-01-20T11:17:50.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.284+0000 7f4c0b361640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:50.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.284+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c041d5310 con 0x7f4c040ab6d0 2024-01-20T11:17:50.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.284+0000 7f4c0b361640 1 librados: init done 2024-01-20T11:17:50.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.284+0000 7f4c0b361640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:50.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.284+0000 7f4c0b361640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:50.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.284+0000 7f4c0b361640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:50.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.284+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c041c9da0 con 0x7f4c040ab6d0 2024-01-20T11:17:50.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.303+0000 7f4bf9ffb640 1 -- 172.21.15.110:0/109009088 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4c000243d0 con 0x7f4c040ab6d0 2024-01-20T11:17:50.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.303+0000 7f4bf9ffb640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be806c8b0 0x7f4be806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.304+0000 7f4c088fc640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be806c8b0 0x7f4be806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.304+0000 7f4c088fc640 10 monclient: get_auth_request con 0x7f4be806c8b0 auth_method 0 2024-01-20T11:17:50.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.304+0000 7f4c088fc640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be806c8b0 0x7f4be806ed70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4c041372f0 tx=0x7f4bf4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:50.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.304+0000 7f4bf9ffb640 10 client.4508.objecter ms_handle_connect 0x7f4be806c8b0 2024-01-20T11:17:50.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.306+0000 7f4bf9ffb640 1 -- 172.21.15.110:0/109009088 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4c00058660 con 0x7f4c040ab6d0 2024-01-20T11:17:50.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.306+0000 7f4bf9ffb640 10 client.4508.objecter ms_dispatch 0x7f4c04000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:50.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.306+0000 7f4bf9ffb640 3 client.4508.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:50.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.306+0000 7f4bf9ffb640 3 client.4508.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:50.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.306+0000 7f4bf9ffb640 20 client.4508.objecter dump_active .. 0 homeless 2024-01-20T11:17:50.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.541+0000 7f4bf9ffb640 1 -- 172.21.15.110:0/109009088 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4c000549a0 con 0x7f4c040ab6d0 2024-01-20T11:17:50.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.541+0000 7f4bf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:50.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.541+0000 7f4bf9ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:50.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.715+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c040abf00 con 0x7f4be806c8b0 2024-01-20T11:17:50.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.718+0000 7f4bf9ffb640 1 -- 172.21.15.110:0/109009088 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64086 (secure 0 0 0) 0x7f4bf40050d0 con 0x7f4be806c8b0 2024-01-20T11:17:50.718 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:50.719 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:50.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 10 librados: watch_flush enter 2024-01-20T11:17:50.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 10 librados: watch_flush exit 2024-01-20T11:17:50.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 20 client.4508.objecter shutdown clearing up homeless session... 2024-01-20T11:17:50.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 10 client.4508.objecter successfully canceled tick 2024-01-20T11:17:50.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be806c8b0 msgr2=0x7f4be806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be806c8b0 0x7f4be806ed70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4c041372f0 tx=0x7f4bf4009040 comp rx=0 tx=0).stop 2024-01-20T11:17:50.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 10 monclient: shutdown 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040ab6d0 msgr2=0x7f4c041ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040ab6d0 0x7f4c041ca950 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000e690 tx=0x7f4c0000eb50 comp rx=0 tx=0).stop 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 shutdown_connections 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be806c8b0 0x7f4be806ed70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c040af3a0 0x7f4c041d8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c040adb30 0x7f4c041d8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 --2- 172.21.15.110:0/109009088 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c040ab6d0 0x7f4c041ca950 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 >> 172.21.15.110:0/109009088 conn(0x7f4c040b3e60 msgr2=0x7f4c04138e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 shutdown_connections 2024-01-20T11:17:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 -- 172.21.15.110:0/109009088 wait complete. 2024-01-20T11:17:50.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.720+0000 7f4c0b361640 1 librados: shutdown 2024-01-20T11:17:50.730 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-01-20T11:17:48.740363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001870"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.367},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.298}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.738},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.497999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":16.492999999999999},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.331}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.238},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":18.768999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":17.632000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":8.8079999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.925000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":19.885000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.8290000000000006},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.7869999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.218},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.567}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":16.056999999999999},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.9120000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.343},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.510999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.243},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.347}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":9.5579999999999998},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.137}]}]},{"osd":3,"up_from":21,"seq":90194313252,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":12.731},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.6829999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.5579999999999998},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.365}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":14.673999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.589}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.433},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.569000000000001},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":18.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":17.512},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.323}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444066,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.415},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":9.4749999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":4.5890000000000004},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.1370000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.978},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.005000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.73},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.43}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.583},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.2720000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.836},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.65}]}]},{"osd":1,"up_from":17,"seq":73014444067,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":12.638999999999999},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":8.7699999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.202},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.5640000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":10.739000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.9100000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":14.276},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.725}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.802},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":16.329999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.406000000000001},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.311}]}]},{"osd":0,"up_from":17,"seq":73014444067,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":5.7679999999999998},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.0979999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.7140000000000004},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.76}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":8.2370000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":15.044}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.786},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.352}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.35},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.351000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":10.023999999999999},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":11.061}]}]},{"osd":6,"up_from":15,"seq":64424509475,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":9.577},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.9160000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.3130000000000006},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":6.8849999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.868},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.645}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.84},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.1880000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.702999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.67}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.454999999999998},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.965}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:50.731 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:50.732 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:50.732 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:50.732 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:50.732 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:50.732 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:50.733 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:50.733 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:50.733 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:50.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.831+0000 7fd8cea34640 10 monclient: get_monmap_and_config 2024-01-20T11:17:50.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.831+0000 7fd8cea34640 10 monclient: build_initial_monmap 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.831+0000 7fd8cea34640 10 monclient: monmap: 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:50.832419+0000 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:50.832419+0000 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:50.832 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.832+0000 7fd8cea34640 1 Processor -- start 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.832+0000 7fd8cea34640 1 -- start start 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.832+0000 7fd8cea34640 10 monclient: init 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd8c806bb00 0x7fd8c80b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 10 monclient(hunting): picked mon.noname-c con 0x7fd8c806bb00 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8c80af790 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8c80b0900 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c806c220 con 0x7fd8c80af790 2024-01-20T11:17:50.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c8146590 con 0x7fd8c80b0900 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c806bee0 con 0x7fd8c806bb00 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8cea34640 10 monclient(hunting): authenticate will time out at 1514.183448s 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8c7fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd8c806bb00 0x7fd8c80b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8c77fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd8c80af790 0x7fd8c8145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56434/0 (socket says 172.21.15.110:56434) 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.833+0000 7fd8c77fe640 1 -- 172.21.15.110:0/218631147 learned_addr learned my addr 172.21.15.110:0/218631147 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.834+0000 7fd8c7fff640 10 monclient(hunting): get_auth_request con 0x7fd8c806bb00 auth_method 0 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.834+0000 7fd8c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.834+0000 7fd8c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.834+0000 7fd8c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:50.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.834+0000 7fd8c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:50.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.834+0000 7fd8c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:50.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.834+0000 7fd8c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:50.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c6ffd640 1 -- 172.21.15.110:0/218631147 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178294654 0 0) 0x7fd8c806c220 con 0x7fd8c80af790 2024-01-20T11:17:50.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c6ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:17:50.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:50.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:50.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c6ffd640 10 monclient(hunting): my global_id is 4623 2024-01-20T11:17:50.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c6ffd640 1 -- 172.21.15.110:0/218631147 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b0003520 con 0x7fd8c80af790 2024-01-20T11:17:50.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c7fff640 10 monclient(hunting): handle_auth_done global_id 4693 payload 274 2024-01-20T11:17:50.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c7fff640 1 -- 172.21.15.110:0/218631147 >> v1:172.21.15.134:6789/0 conn(0x7fd8c80b0900 legacy=0x7fd8c8145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:50.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c7fff640 1 -- 172.21.15.110:0/218631147 >> v1:172.21.15.110:6789/0 conn(0x7fd8c80af790 legacy=0x7fd8c8145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:50.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c7fff640 1 monclient: found mon.noname-c 2024-01-20T11:17:50.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:50.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:50.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.836+0000 7fd8c7fff640 1 -- 172.21.15.110:0/218631147 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c80ada30 con 0x7fd8c806bb00 2024-01-20T11:17:50.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.839+0000 7fd8c7fff640 1 --2- 172.21.15.110:0/218631147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c806bb00 0x7fd8c80b03c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80097a0 tx=0x7fd8b802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b76c283f5df8d0b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:50.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 1 -- 172.21.15.110:0/218631147 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b802e070 con 0x7fd8c806bb00 2024-01-20T11:17:50.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:50.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:50.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 10 monclient: dump: 2024-01-20T11:17:50.843 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:50.843 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.843 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.843 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.843 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 1 -- 172.21.15.110:0/218631147 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b80315a0 con 0x7fd8c806bb00 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 1 -- 172.21.15.110:0/218631147 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8031930 con 0x7fd8c806bb00 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8cea34640 5 monclient: authenticate success, global_id 4693 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8cea34640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:50.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8cea34640 4 set_mon_vals no callback set 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.843+0000 7fd8c6ffd640 10 monclient: dump: 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:50.845 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 10 monclient: shutdown 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 1 -- 172.21.15.110:0/218631147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c806bb00 msgr2=0x7fd8c80b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 1 --2- 172.21.15.110:0/218631147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c806bb00 0x7fd8c80b03c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80097a0 tx=0x7fd8b802b840 comp rx=0 tx=0).stop 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 1 -- 172.21.15.110:0/218631147 shutdown_connections 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 1 --2- 172.21.15.110:0/218631147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c806bb00 0x7fd8c80b03c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 1 -- 172.21.15.110:0/218631147 >> 172.21.15.110:0/218631147 conn(0x7fd8c8066d60 msgr2=0x7fd8c80b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 1 -- 172.21.15.110:0/218631147 shutdown_connections 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 1 -- 172.21.15.110:0/218631147 wait complete. 2024-01-20T11:17:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 10 monclient: build_initial_monmap 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.844+0000 7fd8cea34640 10 monclient: monmap: 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.845+0000 7fd8cea34640 1 librados: starting msgr at 2024-01-20T11:17:50.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.845+0000 7fd8cea34640 1 librados: starting objecter 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.845+0000 7fd8cea34640 1 Processor -- start 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.845+0000 7fd8cea34640 1 -- start start 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.845+0000 7fd8cea34640 1 librados: setting wanted keys 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.845+0000 7fd8cea34640 1 librados: calling monclient init 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.845+0000 7fd8cea34640 10 monclient: init 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8c806bb00 0x7fd8c80707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient(hunting): picked mon.noname-c con 0x7fd8c806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c80af790 0x7fd8c811fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient(hunting): picked mon.noname-b con 0x7fd8c80af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0900 0x7fd8c811d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:50.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient(hunting): picked mon.noname-a con 0x7fd8c80b0900 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c811f5c0 con 0x7fd8c80b0900 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c811ed10 con 0x7fd8c80af790 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c811dae0 con 0x7fd8c806bb00 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8cea34640 10 monclient(hunting): authenticate will time out at 1514.196313s 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8ccfd1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0900 0x7fd8c811d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8c7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8c806bb00 0x7fd8c80707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8c77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c80af790 0x7fd8c811fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8c77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c80af790 0x7fd8c811fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46290/0 (socket says 172.21.15.110:46290) 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8c77fe640 1 -- 172.21.15.110:0/961934129 learned_addr learned my addr 172.21.15.110:0/961934129 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:50.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8c77fe640 10 monclient(hunting): get_auth_request con 0x7fd8c80af790 auth_method 0 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.846+0000 7fd8c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8ccfd1640 10 monclient(hunting): get_auth_request con 0x7fd8c80b0900 auth_method 0 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8ccfd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8ccfd1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8ccfd1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c7fff640 10 monclient(hunting): get_auth_request con 0x7fd8c806bb00 auth_method 0 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:50.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8ccfd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8ccfd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8ccfd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.847+0000 7fd8c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 10 monclient(hunting): handle_auth_done global_id 4696 payload 824 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 1 -- 172.21.15.110:0/961934129 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8c806bb00 msgr2=0x7fd8c80707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8c806bb00 0x7fd8c80707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 1 -- 172.21.15.110:0/961934129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0900 msgr2=0x7fd8c811d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0900 0x7fd8c811d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 1 monclient: found mon.noname-b 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.850+0000 7fd8c77fe640 1 -- 172.21.15.110:0/961934129 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b8009400 con 0x7fd8c80af790 2024-01-20T11:17:50.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.852+0000 7fd8c77fe640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c80af790 0x7fd8c811fa50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c730 tx=0x7fd8bc00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:50.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.852+0000 7fd8c4ff9640 10 client.?.objecter ms_handle_connect 0x7fd8c80af790 2024-01-20T11:17:50.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.852+0000 7fd8c4ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:50.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8c4ff9640 1 -- 172.21.15.110:0/961934129 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc007270 con 0x7fd8c80af790 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8c4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8c4ff9640 10 monclient: dump: 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8c4ff9640 10 monclient: _finish_auth 0 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8c4ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8cea34640 5 monclient: authenticate success, global_id 4696 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8cea34640 10 monclient: _renew_subs 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8cea34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c813e640 con 0x7fd8c80af790 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8cea34640 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8cea34640 10 monclient: _renew_subs 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8cea34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c811f1f0 con 0x7fd8c80af790 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.854+0000 7fd8cea34640 1 librados: init done 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8cea34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8cea34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8cea34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c811dd60 con 0x7fd8c80af790 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8c4ff9640 1 -- 172.21.15.110:0/961934129 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8bc007cf0 con 0x7fd8c80af790 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8c4ff9640 1 -- 172.21.15.110:0/961934129 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc0075d0 con 0x7fd8c80af790 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8c4ff9640 10 monclient: dump: 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:50.861 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:50.861 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:50.861 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:50.861 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:50.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:50.855+0000 7fd8c67fc640 4 set_mon_vals no callback set 2024-01-20T11:17:51.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.094+0000 7fd8c4ff9640 1 -- 172.21.15.110:0/961934129 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd8bc00a640 con 0x7fd8c80af790 2024-01-20T11:17:51.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.094+0000 7fd8c4ff9640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd8b406c860 0x7fd8b406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:51.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.095+0000 7fd8c7fff640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd8b406c860 0x7fd8b406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:51.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.095+0000 7fd8c7fff640 10 monclient: get_auth_request con 0x7fd8b406c860 auth_method 0 2024-01-20T11:17:51.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.095+0000 7fd8c7fff640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd8b406c860 0x7fd8b406ed20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8036000 tx=0x7fd8b803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:51.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.095+0000 7fd8c4ff9640 10 client.4696.objecter ms_handle_connect 0x7fd8b406c860 2024-01-20T11:17:51.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.097+0000 7fd8c4ff9640 1 -- 172.21.15.110:0/961934129 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd8bc001d20 con 0x7fd8c80af790 2024-01-20T11:17:51.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.097+0000 7fd8c4ff9640 10 client.4696.objecter ms_dispatch 0x7fd8c8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:51.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.097+0000 7fd8c4ff9640 3 client.4696.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:51.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.097+0000 7fd8c4ff9640 3 client.4696.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:51.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.097+0000 7fd8c4ff9640 20 client.4696.objecter dump_active .. 0 homeless 2024-01-20T11:17:51.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.108+0000 7fd8c4ff9640 1 -- 172.21.15.110:0/961934129 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd8bc0542b0 con 0x7fd8c80af790 2024-01-20T11:17:51.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.108+0000 7fd8c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:51.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.108+0000 7fd8c4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:51.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.346+0000 7fd8cea34640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:51.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.346+0000 7fd8cea34640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:51.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.346+0000 7fd8cea34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:51.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.346+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd8c80adff0 con 0x7fd8c80af790 2024-01-20T11:17:51.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.351+0000 7fd8c4ff9640 1 -- 172.21.15.110:0/961934129 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1265 (secure 0 0 0) 0x7fd8bc021510 con 0x7fd8c80af790 2024-01-20T11:17:51.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.351+0000 7fd8c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:51.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.351+0000 7fd8c4ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:51.351 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:51.352 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":205,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 10 librados: watch_flush enter 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 10 librados: watch_flush exit 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 20 client.4696.objecter shutdown clearing up homeless session... 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 10 client.4696.objecter successfully canceled tick 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd8b406c860 msgr2=0x7fd8b406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd8b406c860 0x7fd8b406ed20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8036000 tx=0x7fd8b803d040 comp rx=0 tx=0).stop 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 10 monclient: shutdown 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c80af790 msgr2=0x7fd8c811fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c80af790 0x7fd8c811fa50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c730 tx=0x7fd8bc00cbf0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 shutdown_connections 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd8b406c860 0x7fd8b406ed20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0900 0x7fd8c811d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8c80af790 0x7fd8c811fa50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 --2- 172.21.15.110:0/961934129 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8c806bb00 0x7fd8c80707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 >> 172.21.15.110:0/961934129 conn(0x7fd8c8066d60 msgr2=0x7fd8c8144fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 shutdown_connections 2024-01-20T11:17:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 -- 172.21.15.110:0/961934129 wait complete. 2024-01-20T11:17:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.353+0000 7fd8cea34640 1 librados: shutdown 2024-01-20T11:17:51.365 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:51.365 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.466+0000 7f25a1ef7640 10 monclient: get_monmap_and_config 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.466+0000 7f25a1ef7640 10 monclient: build_initial_monmap 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.466+0000 7f25a1ef7640 10 monclient: monmap: 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:51.466883+0000 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:51.466883+0000 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:51.466 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:51.467 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:51.467 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:51.467 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:51.467 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:51.467 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 1 Processor -- start 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 1 -- start start 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient: init 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f259c06bb00 0x7f259c0b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient(hunting): picked mon.noname-a con 0x7f259c06bb00 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f259c0af790 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c0b0900 0x7f259c145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:51.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient(hunting): picked mon.noname-b con 0x7f259c0b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c06c220 con 0x7f259c0af790 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c06bee0 con 0x7f259c06bb00 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0ada30 con 0x7f259c0b0900 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.467+0000 7f25a1ef7640 10 monclient(hunting): authenticate will time out at 1514.817963s 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c0b0900 0x7f259c145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f259c06bb00 0x7f259c0b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c0b0900 0x7f259c145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:59440/0 (socket says 172.21.15.110:59440) 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259bfff640 1 -- 172.21.15.110:0/3888003248 learned_addr learned my addr 172.21.15.110:0/3888003248 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259bfff640 10 monclient(hunting): get_auth_request con 0x7f259c0b0900 auth_method 0 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:51.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259b7fe640 10 monclient(hunting): get_auth_request con 0x7f259c06bb00 auth_method 0 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.468+0000 7f259b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.469+0000 7f259bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.469+0000 7f259bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.469+0000 7f259bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.469+0000 7f259b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.469+0000 7f259b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:51.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.469+0000 7f259b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:51.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.470+0000 7f259bfff640 10 monclient(hunting): handle_auth_done global_id 4514 payload 274 2024-01-20T11:17:51.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.470+0000 7f259bfff640 1 -- 172.21.15.110:0/3888003248 >> v2:172.21.15.110:3300/0 conn(0x7f259c06bb00 msgr2=0x7f259c0b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:51.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.470+0000 7f259bfff640 1 --2- 172.21.15.110:0/3888003248 >> v2:172.21.15.110:3300/0 conn(0x7f259c06bb00 0x7f259c0b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.470+0000 7f259bfff640 1 -- 172.21.15.110:0/3888003248 >> v1:172.21.15.110:6789/0 conn(0x7f259c0af790 legacy=0x7f259c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:51.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.470+0000 7f259bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:51.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.470+0000 7f259bfff640 1 monclient: found mon.noname-b 2024-01-20T11:17:51.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.470+0000 7f259bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:51.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.470+0000 7f259bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:51.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.470+0000 7f259bfff640 1 -- 172.21.15.110:0/3888003248 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c0add10 con 0x7f259c0b0900 2024-01-20T11:17:51.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.472+0000 7f259bfff640 1 --2- 172.21.15.110:0/3888003248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c0b0900 0x7f259c145e80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2594009e50 tx=0x7f259402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3797ad8aafa914c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:51.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 1 -- 172.21.15.110:0/3888003248 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259402ca70 con 0x7f259c0b0900 2024-01-20T11:17:51.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:51.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:51.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 10 monclient: dump: 2024-01-20T11:17:51.476 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:51.476 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:51.476 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:51.476 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 1 -- 172.21.15.110:0/3888003248 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259402cbd0 con 0x7f259c0b0900 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f25a1ef7640 5 monclient: authenticate success, global_id 4514 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f25a1ef7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f25a1ef7640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:51.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f25a1ef7640 4 set_mon_vals no callback set 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 1 -- 172.21.15.110:0/3888003248 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940343f0 con 0x7f259c0b0900 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.476+0000 7f259a7fc640 10 monclient: dump: 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:51.478 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:51.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 10 monclient: shutdown 2024-01-20T11:17:51.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:51.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 1 -- 172.21.15.110:0/3888003248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c0b0900 msgr2=0x7f259c145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:51.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 1 --2- 172.21.15.110:0/3888003248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c0b0900 0x7f259c145e80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2594009e50 tx=0x7f259402bed0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 1 -- 172.21.15.110:0/3888003248 shutdown_connections 2024-01-20T11:17:51.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 1 --2- 172.21.15.110:0/3888003248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c0b0900 0x7f259c145e80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 1 --2- 172.21.15.110:0/3888003248 >> v2:172.21.15.110:3300/0 conn(0x7f259c06bb00 0x7f259c0b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 1 -- 172.21.15.110:0/3888003248 >> 172.21.15.110:0/3888003248 conn(0x7f259c066d60 msgr2=0x7f259c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:51.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 1 -- 172.21.15.110:0/3888003248 shutdown_connections 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 1 -- 172.21.15.110:0/3888003248 wait complete. 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 10 monclient: build_initial_monmap 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.477+0000 7f25a1ef7640 10 monclient: monmap: 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:51.483 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.478+0000 7f25a1ef7640 1 librados: starting msgr at 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.478+0000 7f25a1ef7640 1 librados: starting objecter 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.478+0000 7f25a1ef7640 1 Processor -- start 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.478+0000 7f25a1ef7640 1 -- start start 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.478+0000 7f25a1ef7640 1 librados: setting wanted keys 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.478+0000 7f25a1ef7640 1 librados: calling monclient init 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.478+0000 7f25a1ef7640 10 monclient: init 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c06bb00 0x7f259c1d7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient(hunting): picked mon.noname-c con 0x7f259c06bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f259c0af790 0x7f259c1d75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient(hunting): picked mon.noname-b con 0x7f259c0af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0b0900 0x7f259c11ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:51.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient(hunting): picked mon.noname-a con 0x7f259c0b0900 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c11eaa0 con 0x7f259c0b0900 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c11e6b0 con 0x7f259c0af790 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c1d7af0 con 0x7f259c06bb00 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f25a1ef7640 10 monclient(hunting): authenticate will time out at 1514.829324s 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f259b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c06bb00 0x7f259c1d7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f259affd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f259c0af790 0x7f259c1d75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f259bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0b0900 0x7f259c11ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f259b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c06bb00 0x7f259c1d7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:59442/0 (socket says 172.21.15.110:59442) 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f259b7fe640 1 -- 172.21.15.110:0/1264260426 learned_addr learned my addr 172.21.15.110:0/1264260426 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:51.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f259b7fe640 10 monclient(hunting): get_auth_request con 0x7f259c06bb00 auth_method 0 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f259b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f259b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.479+0000 7f259b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259affd640 10 monclient(hunting): get_auth_request con 0x7f259c0af790 auth_method 0 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259bfff640 10 monclient(hunting): get_auth_request con 0x7f259c0b0900 auth_method 0 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:51.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:51.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:51.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:51.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:51.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:51.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:51.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.480+0000 7f259bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:51.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 10 monclient(hunting): handle_auth_done global_id 4517 payload 824 2024-01-20T11:17:51.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 1 -- 172.21.15.110:0/1264260426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f259c0af790 msgr2=0x7f259c1d75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:51.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f259c0af790 0x7f259c1d75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 1 -- 172.21.15.110:0/1264260426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0b0900 msgr2=0x7f259c11ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:51.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0b0900 0x7f259c11ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:17:51.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 1 monclient: found mon.noname-c 2024-01-20T11:17:51.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:51.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:17:51.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.483+0000 7f259b7fe640 1 -- 172.21.15.110:0/1264260426 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2594009b30 con 0x7f259c06bb00 2024-01-20T11:17:51.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.485+0000 7f259b7fe640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c06bb00 0x7f259c1d7070 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e9e0 tx=0x7f258c00eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:51.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.485+0000 7f2583fff640 10 client.?.objecter ms_handle_connect 0x7f259c06bb00 2024-01-20T11:17:51.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.485+0000 7f2583fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:51.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 1 -- 172.21.15.110:0/1264260426 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c009670 con 0x7f259c06bb00 2024-01-20T11:17:51.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:51.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:51.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 10 monclient: dump: 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 10 monclient: _finish_auth 0 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 10 monclient: _check_auth_tickets 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 1 -- 172.21.15.110:0/1264260426 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c00c9d0 con 0x7f259c06bb00 2024-01-20T11:17:51.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f25a1ef7640 5 monclient: authenticate success, global_id 4517 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 1 -- 172.21.15.110:0/1264260426 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c00cd80 con 0x7f259c06bb00 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2583fff640 10 monclient: dump: 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:51.492 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f25a1ef7640 10 monclient: _renew_subs 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f2599ffb640 4 set_mon_vals no callback set 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f25a1ef7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c1d7d70 con 0x7f259c06bb00 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f25a1ef7640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f25a1ef7640 10 monclient: _renew_subs 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f25a1ef7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259c11f680 con 0x7f259c06bb00 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.489+0000 7f25a1ef7640 1 librados: init done 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.490+0000 7f25a1ef7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.490+0000 7f25a1ef7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.490+0000 7f25a1ef7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:17:51.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.490+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c075490 con 0x7f259c06bb00 2024-01-20T11:17:51.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.508+0000 7f2583fff640 1 -- 172.21.15.110:0/1264260426 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f258c013470 con 0x7f259c06bb00 2024-01-20T11:17:51.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.508+0000 7f2583fff640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f258806c860 0x7f258806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:51.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.509+0000 7f259affd640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f258806c860 0x7f258806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:51.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.509+0000 7f259affd640 10 monclient: get_auth_request con 0x7f258806c860 auth_method 0 2024-01-20T11:17:51.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.509+0000 7f2583fff640 10 client.4517.objecter ms_handle_connect 0x7f258806c860 2024-01-20T11:17:51.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.509+0000 7f259affd640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f258806c860 0x7f258806ed20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f259c11e3c0 tx=0x7f2590009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:51.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.511+0000 7f2583fff640 1 -- 172.21.15.110:0/1264260426 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f258c058a70 con 0x7f259c06bb00 2024-01-20T11:17:51.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.511+0000 7f2583fff640 10 client.4517.objecter ms_dispatch 0x7f259c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:51.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.511+0000 7f2583fff640 3 client.4517.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:51.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.511+0000 7f2583fff640 3 client.4517.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:51.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.511+0000 7f2583fff640 20 client.4517.objecter dump_active .. 0 homeless 2024-01-20T11:17:51.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.744+0000 7f2583fff640 1 -- 172.21.15.110:0/1264260426 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f258c054db0 con 0x7f259c06bb00 2024-01-20T11:17:51.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.744+0000 7f2583fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:51.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.744+0000 7f2583fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:51.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.926+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f259c11e870 con 0x7f258806c860 2024-01-20T11:17:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.928+0000 7f2583fff640 1 -- 172.21.15.110:0/1264260426 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64086 (secure 0 0 0) 0x7f259c11e870 con 0x7f258806c860 2024-01-20T11:17:51.929 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 10 librados: watch_flush enter 2024-01-20T11:17:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 10 librados: watch_flush exit 2024-01-20T11:17:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 20 client.4517.objecter shutdown clearing up homeless session... 2024-01-20T11:17:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 10 client.4517.objecter successfully canceled tick 2024-01-20T11:17:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f258806c860 msgr2=0x7f258806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f258806c860 0x7f258806ed20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f259c11e3c0 tx=0x7f2590009040 comp rx=0 tx=0).stop 2024-01-20T11:17:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 10 monclient: shutdown 2024-01-20T11:17:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c06bb00 msgr2=0x7f259c1d7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c06bb00 0x7f259c1d7070 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e9e0 tx=0x7f258c00eea0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 shutdown_connections 2024-01-20T11:17:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.930+0000 7f25a1ef7640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f258806c860 0x7f258806ed20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.931+0000 7f25a1ef7640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0b0900 0x7f259c11ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.931+0000 7f25a1ef7640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f259c0af790 0x7f259c1d75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.931+0000 7f25a1ef7640 1 --2- 172.21.15.110:0/1264260426 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f259c06bb00 0x7f259c1d7070 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.931+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 >> 172.21.15.110:0/1264260426 conn(0x7f259c066d60 msgr2=0x7f259c144ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.931+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 shutdown_connections 2024-01-20T11:17:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.931+0000 7f25a1ef7640 1 -- 172.21.15.110:0/1264260426 wait complete. 2024-01-20T11:17:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:51.931+0000 7f25a1ef7640 1 librados: shutdown 2024-01-20T11:17:51.941 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-01-20T11:17:50.740684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001870"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.367},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.298}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.738},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.497999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":16.492999999999999},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.331}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.238},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":18.768999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":17.632000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":8.8079999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.925000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":19.885000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.8290000000000006},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.7869999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.218},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.567}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":16.056999999999999},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.9120000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.343},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.510999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.243},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.347}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":9.5579999999999998},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.137}]}]},{"osd":3,"up_from":21,"seq":90194313252,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":12.731},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.6829999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.5579999999999998},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.365}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":14.673999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.589}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.433},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.569000000000001},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":18.326000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":17.512},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.323}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444066,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.415},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":9.4749999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":4.5890000000000004},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.1370000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.978},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.005000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.73},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.43}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.583},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.2720000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.836},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.65}]}]},{"osd":1,"up_from":17,"seq":73014444067,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":12.638999999999999},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":8.7699999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.202},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.5640000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":10.739000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.9100000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":14.276},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.725}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.802},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":16.329999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.406000000000001},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.311}]}]},{"osd":0,"up_from":17,"seq":73014444067,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":5.7679999999999998},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.0979999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.7140000000000004},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.76}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":8.2370000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":15.044}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.786},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.352}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.35},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.351000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":10.023999999999999},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":11.061}]}]},{"osd":6,"up_from":15,"seq":64424509475,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":9.577},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.9160000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.3130000000000006},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":6.8849999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.868},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.645}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.84},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.1880000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.702999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.67}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.454999999999998},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.965}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:54.943 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:55.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.043+0000 7f8870c9b640 10 monclient: get_monmap_and_config 2024-01-20T11:17:55.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.043+0000 7f8870c9b640 10 monclient: build_initial_monmap 2024-01-20T11:17:55.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.043+0000 7f8870c9b640 10 monclient: monmap: 2024-01-20T11:17:55.044 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:55.044 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.044 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:55.043944+0000 2024-01-20T11:17:55.044 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:55.043944+0000 2024-01-20T11:17:55.044 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:55.045 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.045 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:55.045 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:55.045 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:55.045 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:55.045 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:55.045 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 1 Processor -- start 2024-01-20T11:17:55.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 1 -- start start 2024-01-20T11:17:55.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient: init 2024-01-20T11:17:55.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:55.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-01-20T11:17:55.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c06f920 0x7f886c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient(hunting): picked mon.noname-b con 0x7f886c06f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:55.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f886c13d230 addr v1:172.21.15.134:6789/0 2024-01-20T11:17:55.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f886c144060 0x7f886c13d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient(hunting): picked mon.noname-a con 0x7f886c144060 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886c0715b0 con 0x7f886c13d230 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f886c071890 con 0x7f886c144060 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c071b70 con 0x7f886c06f920 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.044+0000 7f8870c9b640 10 monclient(hunting): authenticate will time out at 1518.394970s 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c06f920 0x7f886c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886ad9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f886c144060 0x7f886c13d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f8863fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f886c13d230 0x7f886c143950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:57132/0 (socket says 172.21.15.110:57132) 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f8863fff640 1 -- 172.21.15.110:0/2554407836 learned_addr learned my addr 172.21.15.110:0/2554407836 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886a59c640 10 monclient(hunting): get_auth_request con 0x7f886c06f920 auth_method 0 2024-01-20T11:17:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886ad9d640 10 monclient(hunting): get_auth_request con 0x7f886c144060 auth_method 0 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.045+0000 7f886ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.046+0000 7f886a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.046+0000 7f886a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.046+0000 7f886a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.046+0000 7f886ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.046+0000 7f886ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.046+0000 7f886ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.047+0000 7f8869d9b640 1 -- 172.21.15.110:0/2554407836 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290272975 0 0) 0x7f886c0715b0 con 0x7f886c13d230 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.047+0000 7f8869d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.047+0000 7f8869d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.047+0000 7f8869d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.047+0000 7f8869d9b640 10 monclient(hunting): my global_id is 4702 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.047+0000 7f8869d9b640 1 -- 172.21.15.110:0/2554407836 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8858003520 con 0x7f886c13d230 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.047+0000 7f886a59c640 10 monclient(hunting): handle_auth_done global_id 4520 payload 274 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.047+0000 7f886a59c640 1 -- 172.21.15.110:0/2554407836 >> v2:172.21.15.110:3300/0 conn(0x7f886c144060 msgr2=0x7f886c13d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.047+0000 7f886a59c640 1 --2- 172.21.15.110:0/2554407836 >> v2:172.21.15.110:3300/0 conn(0x7f886c144060 0x7f886c13d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.048+0000 7f886a59c640 1 -- 172.21.15.110:0/2554407836 >> v1:172.21.15.134:6789/0 conn(0x7f886c13d230 legacy=0x7f886c143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.048+0000 7f886a59c640 10 monclient: _finish_hunting 0 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.048+0000 7f886a59c640 1 monclient: found mon.noname-b 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.048+0000 7f886a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.048+0000 7f886a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.048+0000 7f886a59c640 1 -- 172.21.15.110:0/2554407836 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c143430 con 0x7f886c06f920 2024-01-20T11:17:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.048+0000 7f886ad9d640 1 --2- 172.21.15.110:0/2554407836 >> v2:172.21.15.110:3300/0 conn(0x7f886c144060 0x7f886c13d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:17:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.050+0000 7f886a59c640 1 --2- 172.21.15.110:0/2554407836 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c06f920 0x7f886c06fd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f885c0097a0 tx=0x7f885c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6764025a10579df server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:55.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 1 -- 172.21.15.110:0/2554407836 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c02e070 con 0x7f886c06f920 2024-01-20T11:17:55.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:55.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 10 monclient: dump: 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 10 monclient: _finish_auth 0 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:17:55.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 1 -- 172.21.15.110:0/2554407836 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c0315a0 con 0x7f886c06f920 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8870c9b640 5 monclient: authenticate success, global_id 4520 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8870c9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8870c9b640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8870c9b640 4 set_mon_vals no callback set 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 1 -- 172.21.15.110:0/2554407836 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c031930 con 0x7f886c06f920 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.053+0000 7f8869d9b640 10 monclient: dump: 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.055 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:55.056 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.056 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:55.056 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:55.056 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:55.056 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 10 monclient: shutdown 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 1 -- 172.21.15.110:0/2554407836 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c06f920 msgr2=0x7f886c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 1 --2- 172.21.15.110:0/2554407836 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c06f920 0x7f886c06fd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f885c0097a0 tx=0x7f885c02b840 comp rx=0 tx=0).stop 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 1 -- 172.21.15.110:0/2554407836 shutdown_connections 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 1 --2- 172.21.15.110:0/2554407836 >> v2:172.21.15.110:3300/0 conn(0x7f886c144060 0x7f886c13d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 1 --2- 172.21.15.110:0/2554407836 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c06f920 0x7f886c06fd00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 1 -- 172.21.15.110:0/2554407836 >> 172.21.15.110:0/2554407836 conn(0x7f886c066d60 msgr2=0x7f886c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 1 -- 172.21.15.110:0/2554407836 shutdown_connections 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.054+0000 7f8870c9b640 1 -- 172.21.15.110:0/2554407836 wait complete. 2024-01-20T11:17:55.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.055+0000 7f8870c9b640 10 monclient: build_initial_monmap 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.055+0000 7f8870c9b640 10 monclient: monmap: 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.055+0000 7f8870c9b640 1 librados: starting msgr at 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.055+0000 7f8870c9b640 1 librados: starting objecter 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.055+0000 7f8870c9b640 1 Processor -- start 2024-01-20T11:17:55.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.055+0000 7f8870c9b640 1 -- start start 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.055+0000 7f8870c9b640 1 librados: setting wanted keys 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.055+0000 7f8870c9b640 1 librados: calling monclient init 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.055+0000 7f8870c9b640 10 monclient: init 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f886c06f920 0x7f886c13a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient(hunting): picked mon.noname-b con 0x7f886c06f920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c13d230 0x7f886c138120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient(hunting): picked mon.noname-a con 0x7f886c13d230 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c144060 0x7f886c1dac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient(hunting): picked mon.noname-c con 0x7f886c144060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c1396f0 con 0x7f886c13d230 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c139940 con 0x7f886c06f920 2024-01-20T11:17:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c139c20 con 0x7f886c144060 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f8870c9b640 10 monclient(hunting): authenticate will time out at 1518.406559s 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f886ad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c144060 0x7f886c1dac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.056+0000 7f886a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f886c06f920 0x7f886c13a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f8863fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c13d230 0x7f886c138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f886c06f920 0x7f886c13a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46306/0 (socket says 172.21.15.110:46306) 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886a59c640 1 -- 172.21.15.110:0/3445031424 learned_addr learned my addr 172.21.15.110:0/3445031424 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886a59c640 10 monclient(hunting): get_auth_request con 0x7f886c06f920 auth_method 0 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886ad9d640 10 monclient(hunting): get_auth_request con 0x7f886c144060 auth_method 0 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:55.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f8863fff640 10 monclient(hunting): get_auth_request con 0x7f886c13d230 auth_method 0 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f8863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f8863fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f8863fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f886ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f8863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f8863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:55.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.057+0000 7f8863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 10 monclient(hunting): handle_auth_done global_id 4705 payload 824 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 1 -- 172.21.15.110:0/3445031424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c144060 msgr2=0x7f886c1dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c144060 0x7f886c1dac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 1 -- 172.21.15.110:0/3445031424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c13d230 msgr2=0x7f886c138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c13d230 0x7f886c138120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 10 monclient: _finish_hunting 0 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 1 monclient: found mon.noname-b 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:55.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.060+0000 7f886a59c640 1 -- 172.21.15.110:0/3445031424 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8864009a70 con 0x7f886c06f920 2024-01-20T11:17:55.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.062+0000 7f886a59c640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f886c06f920 0x7f886c13a390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f885c034040 tx=0x7f885c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:55.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.062+0000 7f8862ffd640 10 client.?.objecter ms_handle_connect 0x7f886c06f920 2024-01-20T11:17:55.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.063+0000 7f8862ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:55.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 1 -- 172.21.15.110:0/3445031424 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c032840 con 0x7f886c06f920 2024-01-20T11:17:55.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:55.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 10 monclient: dump: 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:55.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 1 -- 172.21.15.110:0/3445031424 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c0329c0 con 0x7f886c06f920 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 1 -- 172.21.15.110:0/3445031424 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c032d70 con 0x7f886c06f920 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8862ffd640 10 monclient: dump: 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:55.069 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8870c9b640 5 monclient: authenticate success, global_id 4705 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8870c9b640 10 monclient: _renew_subs 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f886959a640 4 set_mon_vals no callback set 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8870c9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885c009400 con 0x7f886c06f920 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8870c9b640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8870c9b640 10 monclient: _renew_subs 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8870c9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:55.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886c136890 con 0x7f886c06f920 2024-01-20T11:17:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.066+0000 7f8870c9b640 1 librados: init done 2024-01-20T11:17:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.067+0000 7f8870c9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.067+0000 7f8870c9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.067+0000 7f8870c9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.067+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886c075490 con 0x7f886c06f920 2024-01-20T11:17:55.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.086+0000 7f8862ffd640 1 -- 172.21.15.110:0/3445031424 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f885c04e430 con 0x7f886c06f920 2024-01-20T11:17:55.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.087+0000 7f8862ffd640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f884c06c8b0 0x7f884c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.087+0000 7f8863fff640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f884c06c8b0 0x7f884c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.087+0000 7f8863fff640 10 monclient: get_auth_request con 0x7f884c06c8b0 auth_method 0 2024-01-20T11:17:55.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.087+0000 7f8862ffd640 10 client.4705.objecter ms_handle_connect 0x7f884c06c8b0 2024-01-20T11:17:55.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.087+0000 7f8863fff640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f884c06c8b0 0x7f884c06ed70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f886c135930 tx=0x7f8854009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:55.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.089+0000 7f8862ffd640 1 -- 172.21.15.110:0/3445031424 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f885c031e90 con 0x7f886c06f920 2024-01-20T11:17:55.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.089+0000 7f8862ffd640 10 client.4705.objecter ms_dispatch 0x7f886c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:55.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.089+0000 7f8862ffd640 3 client.4705.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:55.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.089+0000 7f8862ffd640 3 client.4705.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:55.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.089+0000 7f8862ffd640 20 client.4705.objecter dump_active .. 0 homeless 2024-01-20T11:17:55.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.325+0000 7f8862ffd640 1 -- 172.21.15.110:0/3445031424 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f885c0766d0 con 0x7f886c06f920 2024-01-20T11:17:55.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.325+0000 7f8862ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:55.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.325+0000 7f8862ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:55.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.518+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f886c0b59a0 con 0x7f884c06c8b0 2024-01-20T11:17:55.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.521+0000 7f8862ffd640 1 -- 172.21.15.110:0/3445031424 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64200 (secure 0 0 0) 0x7f886c0b59a0 con 0x7f884c06c8b0 2024-01-20T11:17:55.522 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:55.523 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 10 librados: watch_flush enter 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 10 librados: watch_flush exit 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 20 client.4705.objecter shutdown clearing up homeless session... 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 10 client.4705.objecter successfully canceled tick 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f884c06c8b0 msgr2=0x7f884c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f884c06c8b0 0x7f884c06ed70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f886c135930 tx=0x7f8854009040 comp rx=0 tx=0).stop 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 10 monclient: shutdown 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f886c06f920 msgr2=0x7f886c13a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:55.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f886c06f920 0x7f886c13a390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f885c034040 tx=0x7f885c0046c0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 shutdown_connections 2024-01-20T11:17:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f884c06c8b0 0x7f884c06ed70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f886c144060 0x7f886c1dac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c13d230 0x7f886c138120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 --2- 172.21.15.110:0/3445031424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f886c06f920 0x7f886c13a390 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 >> 172.21.15.110:0/3445031424 conn(0x7f886c066d60 msgr2=0x7f886c073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 shutdown_connections 2024-01-20T11:17:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 -- 172.21.15.110:0/3445031424 wait complete. 2024-01-20T11:17:55.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.524+0000 7f8870c9b640 1 librados: shutdown 2024-01-20T11:17:55.536 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-01-20T11:17:54.741339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001946"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280550,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.0350000000000001},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.4699999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":15.256},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.388},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.632999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.5150000000000006},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":10.962}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":6.8070000000000004},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":13.227}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.1660000000000004},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":13.988}]}]},{"osd":7,"up_from":21,"seq":90194313252,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.4509999999999996},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.3970000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.589},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.276}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":14.996},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.9580000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.613},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.908000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.158000000000001},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":8.6110000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.127000000000001},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":17.061}]}]},{"osd":3,"up_from":21,"seq":90194313253,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.7309999999999999},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.6050000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.9369999999999994},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.151}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":7.1630000000000003},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.943}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.596},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.3339999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":7.899},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":14.702999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.488},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.997999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444067,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.7169999999999996},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":10.5}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":16.443000000000001},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.4920000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":9.8330000000000002},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.622999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.2260000000000009},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.239000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.66},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.5010000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.439},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":11.917}]}]},{"osd":1,"up_from":17,"seq":73014444068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.5759999999999996},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":9.3399999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.176},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.4260000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.492000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.4829999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":12.696999999999999},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.585000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.7089999999999996},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":13.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.741},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.179}]}]},{"osd":0,"up_from":17,"seq":73014444068,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":6.5339999999999998},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":12.762}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.7089999999999996},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.7490000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":15.332000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.4500000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.346},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.914999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":9.6530000000000005},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.965}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.391},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":11.532999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509476,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.843},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.9909999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.9379999999999997},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.606999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.391999999999999},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.5820000000000007}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.087},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.063000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.045},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":10.846}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":17.140000000000001},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.93}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:55.538 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.635+0000 7fbf962c9640 10 monclient: get_monmap_and_config 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.635+0000 7fbf962c9640 10 monclient: build_initial_monmap 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.635+0000 7fbf962c9640 10 monclient: monmap: 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:55.637492+0000 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:55.637492+0000 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:55.637 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.636+0000 7fbf962c9640 1 Processor -- start 2024-01-20T11:17:55.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.636+0000 7fbf962c9640 1 -- start start 2024-01-20T11:17:55.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.636+0000 7fbf962c9640 10 monclient: init 2024-01-20T11:17:55.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbf900a96f0 0x7fbf9013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 10 monclient(hunting): picked mon.noname-c con 0x7fbf900a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbf9013fa10 0x7fbf901423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 10 monclient(hunting): picked mon.noname-a con 0x7fbf9013fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf900a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900b25c0 con 0x7fbf900a9ad0 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf900aaea0 con 0x7fbf9013fa10 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf900ab0a0 con 0x7fbf900a96f0 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.637+0000 7fbf962c9640 10 monclient(hunting): authenticate will time out at 1518.988668s 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbf900a96f0 0x7fbf9013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fbf900a96f0 0x7fbf9013f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46314/0 (socket says 172.21.15.110:46314) 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 1 -- 172.21.15.110:0/525049823 learned_addr learned my addr 172.21.15.110:0/525049823 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 10 monclient(hunting): get_auth_request con 0x7fbf900a96f0 auth_method 0 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.638+0000 7fbf8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.640+0000 7fbf8ffff640 10 monclient(hunting): handle_auth_done global_id 4708 payload 274 2024-01-20T11:17:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.640+0000 7fbf8ffff640 1 -- 172.21.15.110:0/525049823 >> v2:172.21.15.110:3300/0 conn(0x7fbf9013fa10 msgr2=0x7fbf901423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.640+0000 7fbf8ffff640 1 --2- 172.21.15.110:0/525049823 >> v2:172.21.15.110:3300/0 conn(0x7fbf9013fa10 0x7fbf901423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.640+0000 7fbf8ffff640 1 -- 172.21.15.110:0/525049823 >> v1:172.21.15.110:6789/0 conn(0x7fbf900a9ad0 legacy=0x7fbf90142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.640+0000 7fbf8ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:17:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.640+0000 7fbf8ffff640 1 monclient: found mon.noname-c 2024-01-20T11:17:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.640+0000 7fbf8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.640+0000 7fbf8ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.640+0000 7fbf8ffff640 1 -- 172.21.15.110:0/525049823 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf900b3fb0 con 0x7fbf900a96f0 2024-01-20T11:17:55.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.642+0000 7fbf8ffff640 1 --2- 172.21.15.110:0/525049823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf900a96f0 0x7fbf9013f4d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf900b5720 tx=0x7fbf8002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d170b0bf35083b88 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:55.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 1 -- 172.21.15.110:0/525049823 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80031800 con 0x7fbf900a96f0 2024-01-20T11:17:55.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:55.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:55.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 10 monclient: dump: 2024-01-20T11:17:55.647 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:55.649 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 10 monclient: _finish_auth 0 2024-01-20T11:17:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 1 -- 172.21.15.110:0/525049823 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf80031960 con 0x7fbf900a96f0 2024-01-20T11:17:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf962c9640 5 monclient: authenticate success, global_id 4708 2024-01-20T11:17:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf962c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:55.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf962c9640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:55.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf962c9640 4 set_mon_vals no callback set 2024-01-20T11:17:55.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 1 -- 172.21.15.110:0/525049823 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80031cf0 con 0x7fbf900a96f0 2024-01-20T11:17:55.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:55.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:55.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.646+0000 7fbf8effd640 10 monclient: dump: 2024-01-20T11:17:55.650 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:55.651 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.651 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.651 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.651 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:55.651 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.651 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:55.651 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:55.651 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:55.651 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 10 monclient: shutdown 2024-01-20T11:17:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 1 -- 172.21.15.110:0/525049823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf900a96f0 msgr2=0x7fbf9013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 1 --2- 172.21.15.110:0/525049823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf900a96f0 0x7fbf9013f4d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf900b5720 tx=0x7fbf8002b8a0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 1 -- 172.21.15.110:0/525049823 shutdown_connections 2024-01-20T11:17:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 1 --2- 172.21.15.110:0/525049823 >> v2:172.21.15.110:3300/0 conn(0x7fbf9013fa10 0x7fbf901423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 1 --2- 172.21.15.110:0/525049823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf900a96f0 0x7fbf9013f4d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 1 -- 172.21.15.110:0/525049823 >> 172.21.15.110:0/525049823 conn(0x7fbf900b7bc0 msgr2=0x7fbf900b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 1 -- 172.21.15.110:0/525049823 shutdown_connections 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 1 -- 172.21.15.110:0/525049823 wait complete. 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 10 monclient: build_initial_monmap 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.647+0000 7fbf962c9640 10 monclient: monmap: 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.648+0000 7fbf962c9640 1 librados: starting msgr at 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.648+0000 7fbf962c9640 1 librados: starting objecter 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.648+0000 7fbf962c9640 1 Processor -- start 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.648+0000 7fbf962c9640 1 -- start start 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.648+0000 7fbf962c9640 1 librados: setting wanted keys 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.648+0000 7fbf962c9640 1 librados: calling monclient init 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.648+0000 7fbf962c9640 10 monclient: init 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf900a96f0 0x7fbf90134380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient(hunting): picked mon.noname-a con 0x7fbf900a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf900a9ad0 0x7fbf90137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient(hunting): picked mon.noname-c con 0x7fbf900a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:55.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf901da780 0x7fbf90134b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient(hunting): picked mon.noname-b con 0x7fbf901da780 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90137160 con 0x7fbf900a96f0 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90136e10 con 0x7fbf901da780 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90135040 con 0x7fbf900a9ad0 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf962c9640 10 monclient(hunting): authenticate will time out at 1519.000255s 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf94866640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf901da780 0x7fbf90134b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf8f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf900a9ad0 0x7fbf90137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf8ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf900a96f0 0x7fbf90134380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf94866640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf901da780 0x7fbf90134b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46328/0 (socket says 172.21.15.110:46328) 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf94866640 1 -- 172.21.15.110:0/258357208 learned_addr learned my addr 172.21.15.110:0/258357208 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf94866640 10 monclient(hunting): get_auth_request con 0x7fbf901da780 auth_method 0 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf94866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf94866640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.649+0000 7fbf94866640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8ffff640 10 monclient(hunting): get_auth_request con 0x7fbf900a96f0 auth_method 0 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8f7fe640 10 monclient(hunting): get_auth_request con 0x7fbf900a9ad0 auth_method 0 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf94866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:55.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf94866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf94866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.650+0000 7fbf8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.652+0000 7fbf94866640 10 monclient(hunting): handle_auth_done global_id 4711 payload 824 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.652+0000 7fbf94866640 1 -- 172.21.15.110:0/258357208 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf900a9ad0 msgr2=0x7fbf90137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.652+0000 7fbf94866640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf900a9ad0 0x7fbf90137680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.652+0000 7fbf94866640 1 -- 172.21.15.110:0/258357208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf900a96f0 msgr2=0x7fbf90134380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.652+0000 7fbf94866640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf900a96f0 0x7fbf90134380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:55.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.652+0000 7fbf94866640 10 monclient: _finish_hunting 0 2024-01-20T11:17:55.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.652+0000 7fbf94866640 1 monclient: found mon.noname-b 2024-01-20T11:17:55.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.653+0000 7fbf94866640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:55.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.653+0000 7fbf94866640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:55.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.653+0000 7fbf94866640 1 -- 172.21.15.110:0/258357208 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf800094d0 con 0x7fbf901da780 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.655+0000 7fbf94866640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf901da780 0x7fbf90134b00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800c780 tx=0x7fbf8800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.655+0000 7fbf8cff9640 10 client.?.objecter ms_handle_connect 0x7fbf901da780 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.655+0000 7fbf8cff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 1 -- 172.21.15.110:0/258357208 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88003d80 con 0x7fbf901da780 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 10 monclient: dump: 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:55.663 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 10 monclient: _finish_auth 0 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 10 monclient: _check_auth_tickets 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 1 -- 172.21.15.110:0/258357208 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8800ce30 con 0x7fbf901da780 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf962c9640 5 monclient: authenticate success, global_id 4711 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf962c9640 10 monclient: _renew_subs 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf962c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf901dac30 con 0x7fbf901da780 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 1 -- 172.21.15.110:0/258357208 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88005290 con 0x7fbf901da780 2024-01-20T11:17:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8cff9640 10 monclient: dump: 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf8e7fc640 4 set_mon_vals no callback set 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf962c9640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf962c9640 10 monclient: _renew_subs 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf962c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf90133260 con 0x7fbf901da780 2024-01-20T11:17:55.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.660+0000 7fbf962c9640 1 librados: init done 2024-01-20T11:17:55.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.661+0000 7fbf962c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:55.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.661+0000 7fbf962c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:55.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.661+0000 7fbf962c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:55.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.661+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf90136b40 con 0x7fbf901da780 2024-01-20T11:17:55.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.680+0000 7fbf8cff9640 1 -- 172.21.15.110:0/258357208 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fbf880053f0 con 0x7fbf901da780 2024-01-20T11:17:55.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.680+0000 7fbf8cff9640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbf7c06c890 0x7fbf7c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:55.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.680+0000 7fbf8ffff640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbf7c06c890 0x7fbf7c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:55.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.680+0000 7fbf8ffff640 10 monclient: get_auth_request con 0x7fbf7c06c890 auth_method 0 2024-01-20T11:17:55.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.681+0000 7fbf8ffff640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbf7c06c890 0x7fbf7c06ed50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf80036000 tx=0x7fbf8003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:55.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.681+0000 7fbf8cff9640 10 client.4711.objecter ms_handle_connect 0x7fbf7c06c890 2024-01-20T11:17:55.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.683+0000 7fbf8cff9640 1 -- 172.21.15.110:0/258357208 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbf88057ed0 con 0x7fbf901da780 2024-01-20T11:17:55.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.683+0000 7fbf8cff9640 10 client.4711.objecter ms_dispatch 0x7fbf90000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:55.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.683+0000 7fbf8cff9640 3 client.4711.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:55.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.683+0000 7fbf8cff9640 3 client.4711.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:55.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.683+0000 7fbf8cff9640 20 client.4711.objecter dump_active .. 0 homeless 2024-01-20T11:17:55.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.917+0000 7fbf8cff9640 1 -- 172.21.15.110:0/258357208 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fbf88054210 con 0x7fbf901da780 2024-01-20T11:17:55.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.917+0000 7fbf8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:55.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:55.917+0000 7fbf8cff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:56.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.084+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbf90135200 con 0x7fbf7c06c890 2024-01-20T11:17:56.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.087+0000 7fbf8cff9640 1 -- 172.21.15.110:0/258357208 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64200 (secure 0 0 0) 0x7fbf90135200 con 0x7fbf7c06c890 2024-01-20T11:17:56.088 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.088+0000 7fbf962c9640 10 librados: watch_flush enter 2024-01-20T11:17:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.088+0000 7fbf962c9640 10 librados: watch_flush exit 2024-01-20T11:17:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.088+0000 7fbf962c9640 20 client.4711.objecter shutdown clearing up homeless session... 2024-01-20T11:17:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.088+0000 7fbf962c9640 10 client.4711.objecter successfully canceled tick 2024-01-20T11:17:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbf7c06c890 msgr2=0x7fbf7c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbf7c06c890 0x7fbf7c06ed50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf80036000 tx=0x7fbf8003c040 comp rx=0 tx=0).stop 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 10 monclient: shutdown 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf901da780 msgr2=0x7fbf90134b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf901da780 0x7fbf90134b00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800c780 tx=0x7fbf8800cc40 comp rx=0 tx=0).stop 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 shutdown_connections 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbf7c06c890 0x7fbf7c06ed50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf901da780 0x7fbf90134b00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf900a9ad0 0x7fbf90137680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 --2- 172.21.15.110:0/258357208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf900a96f0 0x7fbf90134380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 >> 172.21.15.110:0/258357208 conn(0x7fbf900b7bc0 msgr2=0x7fbf900b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 shutdown_connections 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 -- 172.21.15.110:0/258357208 wait complete. 2024-01-20T11:17:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.089+0000 7fbf962c9640 1 librados: shutdown 2024-01-20T11:17:56.100 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-01-20T11:17:54.741339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001946"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280550,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.0350000000000001},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.4699999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":15.256},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.388},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.632999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.5150000000000006},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":10.962}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":6.8070000000000004},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":13.227}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.1660000000000004},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":13.988}]}]},{"osd":7,"up_from":21,"seq":90194313252,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.4509999999999996},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.3970000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.589},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.276}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":14.996},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.9580000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.613},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.908000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":16.158000000000001},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":8.6110000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.127000000000001},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":17.061}]}]},{"osd":3,"up_from":21,"seq":90194313253,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.7309999999999999},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.6050000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.9369999999999994},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.151}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":7.1630000000000003},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.943}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.596},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.3339999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":7.899},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":14.702999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.488},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.997999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444067,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.7169999999999996},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":10.5}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":16.443000000000001},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.4920000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":9.8330000000000002},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.622999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.2260000000000009},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.239000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.66},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.5010000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.439},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":11.917}]}]},{"osd":1,"up_from":17,"seq":73014444068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.5759999999999996},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":9.3399999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.176},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.4260000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.492000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.4829999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":12.696999999999999},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.585000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.7089999999999996},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":13.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.741},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.179}]}]},{"osd":0,"up_from":17,"seq":73014444068,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":6.5339999999999998},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":12.762}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.7089999999999996},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.7490000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":15.332000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.4500000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.346},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.914999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":9.6530000000000005},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.965}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.391},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":11.532999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509476,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.843},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.9909999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.9379999999999997},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.606999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.391999999999999},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.5820000000000007}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.087},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.063000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.045},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":10.846}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":17.140000000000001},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.93}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:56.102 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:17:56.102 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:17:56.102 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:17:56.102 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:17:56.102 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:17:56.102 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:17:56.102 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:17:56.102 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:17:56.102 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:17:56.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.199+0000 7fa0c4583640 10 monclient: get_monmap_and_config 2024-01-20T11:17:56.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.199+0000 7fa0c4583640 10 monclient: build_initial_monmap 2024-01-20T11:17:56.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.199+0000 7fa0c4583640 10 monclient: monmap: 2024-01-20T11:17:56.204 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:56.204 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.205 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:56.200704+0000 2024-01-20T11:17:56.205 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:56.200704+0000 2024-01-20T11:17:56.205 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:56.205 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.205 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:56.205 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:56.205 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:56.206 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:56.206 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:56.206 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.199+0000 7fa0c4583640 1 Processor -- start 2024-01-20T11:17:56.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.199+0000 7fa0c4583640 1 -- start start 2024-01-20T11:17:56.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.199+0000 7fa0c4583640 10 monclient: init 2024-01-20T11:17:56.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:56.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-01-20T11:17:56.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa0bc0a96f0 0x7fa0bc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 10 monclient(hunting): picked mon.noname-c con 0x7fa0bc0a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:56.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0bc13fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:56.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa0bc0a9ad0 0x7fa0bc142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 10 monclient(hunting): picked mon.noname-a con 0x7fa0bc0a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:17:56.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0bc0b25c0 con 0x7fa0bc13fa10 2024-01-20T11:17:56.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc0aaea0 con 0x7fa0bc0a9ad0 2024-01-20T11:17:56.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc0ab0a0 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:56.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.200+0000 7fa0c4583640 10 monclient(hunting): authenticate will time out at 1519.551681s 2024-01-20T11:17:56.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c2b20640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa0bc0a9ad0 0x7fa0bc142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa0bc0a96f0 0x7fa0bc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa0bc0a96f0 0x7fa0bc13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46340/0 (socket says 172.21.15.110:46340) 2024-01-20T11:17:56.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 1 -- 172.21.15.110:0/2063318307 learned_addr learned my addr 172.21.15.110:0/2063318307 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:56.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 10 monclient(hunting): get_auth_request con 0x7fa0bc0a96f0 auth_method 0 2024-01-20T11:17:56.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c2b20640 10 monclient(hunting): get_auth_request con 0x7fa0bc0a9ad0 auth_method 0 2024-01-20T11:17:56.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c2b20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c2b20640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c2b20640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.201+0000 7fa0c231f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.202+0000 7fa0c2b20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.202+0000 7fa0c2b20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.202+0000 7fa0c2b20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.203+0000 7fa0c231f640 10 monclient(hunting): handle_auth_done global_id 4714 payload 274 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.203+0000 7fa0c231f640 1 -- 172.21.15.110:0/2063318307 >> v2:172.21.15.110:3300/0 conn(0x7fa0bc0a9ad0 msgr2=0x7fa0bc142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.203+0000 7fa0c231f640 1 --2- 172.21.15.110:0/2063318307 >> v2:172.21.15.110:3300/0 conn(0x7fa0bc0a9ad0 0x7fa0bc142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.203+0000 7fa0c231f640 1 -- 172.21.15.110:0/2063318307 >> v1:172.21.15.110:6789/0 conn(0x7fa0bc13fa10 legacy=0x7fa0bc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.203+0000 7fa0c231f640 10 monclient: _finish_hunting 0 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.203+0000 7fa0c231f640 1 monclient: found mon.noname-c 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.203+0000 7fa0c231f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.203+0000 7fa0c231f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.203+0000 7fa0c231f640 1 -- 172.21.15.110:0/2063318307 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0b3fb0 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.205+0000 7fa0c231f640 1 --2- 172.21.15.110:0/2063318307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 0x7fa0bc13f4d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0b5720 tx=0x7fa0b802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83ad2fdf2f914612 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 1 -- 172.21.15.110:0/2063318307 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8031870 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 10 monclient: dump: 2024-01-20T11:17:56.215 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 10 monclient: _finish_auth 0 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 10 monclient: _check_auth_tickets 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:56.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 1 -- 172.21.15.110:0/2063318307 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b80319d0 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c4583640 5 monclient: authenticate success, global_id 4714 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c4583640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c4583640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c4583640 4 set_mon_vals no callback set 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 1 -- 172.21.15.110:0/2063318307 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8031d60 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.209+0000 7fa0c131d640 10 monclient: dump: 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.217 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 10 monclient: shutdown 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 1 -- 172.21.15.110:0/2063318307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 msgr2=0x7fa0bc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 1 --2- 172.21.15.110:0/2063318307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 0x7fa0bc13f4d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0b5720 tx=0x7fa0b802b910 comp rx=0 tx=0).stop 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 1 -- 172.21.15.110:0/2063318307 shutdown_connections 2024-01-20T11:17:56.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 1 --2- 172.21.15.110:0/2063318307 >> v2:172.21.15.110:3300/0 conn(0x7fa0bc0a9ad0 0x7fa0bc142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 1 --2- 172.21.15.110:0/2063318307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 0x7fa0bc13f4d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 1 -- 172.21.15.110:0/2063318307 >> 172.21.15.110:0/2063318307 conn(0x7fa0bc0b7bc0 msgr2=0x7fa0bc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 1 -- 172.21.15.110:0/2063318307 shutdown_connections 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.210+0000 7fa0c4583640 1 -- 172.21.15.110:0/2063318307 wait complete. 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 10 monclient: build_initial_monmap 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 10 monclient: monmap: 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:56.219 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 1 librados: starting msgr at 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 1 librados: starting objecter 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 1 Processor -- start 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 1 -- start start 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 1 librados: setting wanted keys 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 1 librados: calling monclient init 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 10 monclient: init 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.211+0000 7fa0c4583640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:17:56.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 0x7fa0bc1375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 10 monclient(hunting): picked mon.noname-b con 0x7fa0bc0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0bc0a9ad0 0x7fa0bc134e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 10 monclient(hunting): picked mon.noname-c con 0x7fa0bc0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0bc135b10 0x7fa0bc135380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 10 monclient(hunting): picked mon.noname-a con 0x7fa0bc135b10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc134680 con 0x7fa0bc135b10 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc131fb0 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc1358c0 con 0x7fa0bc0a9ad0 2024-01-20T11:17:56.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c4583640 10 monclient(hunting): authenticate will time out at 1519.563258s 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c2b20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0bc135b10 0x7fa0bc135380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c231f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 0x7fa0bc1375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c1b1e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0bc0a9ad0 0x7fa0bc134e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c231f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 0x7fa0bc1375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46346/0 (socket says 172.21.15.110:46346) 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c231f640 1 -- 172.21.15.110:0/3378358938 learned_addr learned my addr 172.21.15.110:0/3378358938 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c231f640 10 monclient(hunting): get_auth_request con 0x7fa0bc0a96f0 auth_method 0 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c231f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c231f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c231f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c2b20640 10 monclient(hunting): get_auth_request con 0x7fa0bc135b10 auth_method 0 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c2b20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c2b20640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c2b20640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c1b1e640 10 monclient(hunting): get_auth_request con 0x7fa0bc0a9ad0 auth_method 0 2024-01-20T11:17:56.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c1b1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c1b1e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.212+0000 7fa0c1b1e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.213+0000 7fa0c231f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.213+0000 7fa0c231f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.213+0000 7fa0c231f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.213+0000 7fa0c2b20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.213+0000 7fa0c2b20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.213+0000 7fa0c2b20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.213+0000 7fa0c1b1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.213+0000 7fa0c1b1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.213+0000 7fa0c1b1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 10 monclient(hunting): handle_auth_done global_id 4717 payload 824 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 1 -- 172.21.15.110:0/3378358938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0bc0a9ad0 msgr2=0x7fa0bc134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0bc0a9ad0 0x7fa0bc134e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 1 -- 172.21.15.110:0/3378358938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0bc135b10 msgr2=0x7fa0bc135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0bc135b10 0x7fa0bc135380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 10 monclient: _finish_hunting 0 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 1 monclient: found mon.noname-b 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.215+0000 7fa0c231f640 1 -- 172.21.15.110:0/3378358938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b80094d0 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.218+0000 7fa0c231f640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 0x7fa0bc1375e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8000c00 tx=0x7fa0b80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.218+0000 7fa0b2ffd640 10 client.?.objecter ms_handle_connect 0x7fa0bc0a96f0 2024-01-20T11:17:56.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.218+0000 7fa0b2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 1 -- 172.21.15.110:0/3378358938 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b803abe0 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 10 monclient: dump: 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:56.226 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 1 -- 172.21.15.110:0/3378358938 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b803ad60 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 1 -- 172.21.15.110:0/3378358938 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8044630 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0b2ffd640 10 monclient: dump: 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:56.228 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c4583640 5 monclient: authenticate success, global_id 4717 2024-01-20T11:17:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c4583640 10 monclient: _renew_subs 2024-01-20T11:17:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c4583640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c0b1c640 4 set_mon_vals no callback set 2024-01-20T11:17:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc136b70 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c4583640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c4583640 10 monclient: _renew_subs 2024-01-20T11:17:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c4583640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc137b20 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.221+0000 7fa0c4583640 1 librados: init done 2024-01-20T11:17:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.222+0000 7fa0c4583640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.222+0000 7fa0c4583640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.222+0000 7fa0c4583640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.222+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc13cd80 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.241+0000 7fa0b2ffd640 1 -- 172.21.15.110:0/3378358938 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa0b804d450 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.241+0000 7fa0b2ffd640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0a006c860 0x7fa0a006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.242+0000 7fa0c1b1e640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0a006c860 0x7fa0a006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.242+0000 7fa0c1b1e640 10 monclient: get_auth_request con 0x7fa0a006c860 auth_method 0 2024-01-20T11:17:56.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.242+0000 7fa0c1b1e640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0a006c860 0x7fa0a006ed20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0b85b0 tx=0x7fa0ac009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.242+0000 7fa0b2ffd640 10 client.4717.objecter ms_handle_connect 0x7fa0a006c860 2024-01-20T11:17:56.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.467+0000 7fa0b2ffd640 1 -- 172.21.15.110:0/3378358938 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa0b802d040 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.467+0000 7fa0b2ffd640 10 client.4717.objecter ms_dispatch 0x7fa0bc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:56.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.467+0000 7fa0b2ffd640 3 client.4717.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:56.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.467+0000 7fa0b2ffd640 3 client.4717.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:56.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.467+0000 7fa0b2ffd640 20 client.4717.objecter dump_active .. 0 homeless 2024-01-20T11:17:56.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.476+0000 7fa0b2ffd640 1 -- 172.21.15.110:0/3378358938 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa0b8076eb0 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.476+0000 7fa0b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:56.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.476+0000 7fa0b2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.722+0000 7fa0c4583640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:17:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.723+0000 7fa0c4583640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.723+0000 7fa0c4583640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.723+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa0bc134c20 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.730+0000 7fa0b2ffd640 1 -- 172.21.15.110:0/3378358938 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fa0b807a4d0 con 0x7fa0bc0a96f0 2024-01-20T11:17:56.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.730+0000 7fa0b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:17:56.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.730+0000 7fa0b2ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:17:56.732 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:56.732 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":211,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:17:56.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 10 librados: watch_flush enter 2024-01-20T11:17:56.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 10 librados: watch_flush exit 2024-01-20T11:17:56.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 20 client.4717.objecter shutdown clearing up homeless session... 2024-01-20T11:17:56.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 10 client.4717.objecter successfully canceled tick 2024-01-20T11:17:56.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0a006c860 msgr2=0x7fa0a006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0a006c860 0x7fa0a006ed20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0b85b0 tx=0x7fa0ac009040 comp rx=0 tx=0).stop 2024-01-20T11:17:56.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 10 monclient: shutdown 2024-01-20T11:17:56.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:56.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 msgr2=0x7fa0bc1375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 0x7fa0bc1375e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8000c00 tx=0x7fa0b80046c0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 shutdown_connections 2024-01-20T11:17:56.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0a006c860 0x7fa0a006ed20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0bc135b10 0x7fa0bc135380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0bc0a9ad0 0x7fa0bc134e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 --2- 172.21.15.110:0/3378358938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0bc0a96f0 0x7fa0bc1375e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 >> 172.21.15.110:0/3378358938 conn(0x7fa0bc0b7bc0 msgr2=0x7fa0bc0b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:56.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 shutdown_connections 2024-01-20T11:17:56.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 -- 172.21.15.110:0/3378358938 wait complete. 2024-01-20T11:17:56.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.732+0000 7fa0c4583640 1 librados: shutdown 2024-01-20T11:17:56.745 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:17:56.745 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.842+0000 7f1d3c305640 10 monclient: get_monmap_and_config 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.842+0000 7f1d3c305640 10 monclient: build_initial_monmap 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.842+0000 7f1d3c305640 10 monclient: monmap: 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:17:56.844570+0000 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:17:56.844570+0000 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:17:56.844 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:17:56.845 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:17:56.845 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:17:56.845 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:17:56.845 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.843+0000 7f1d3c305640 1 Processor -- start 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.843+0000 7f1d3c305640 1 -- start start 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.843+0000 7f1d3c305640 10 monclient: init 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d3406a860 addr v1:172.21.15.110:6789/0 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 0x7f1d341455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 10 monclient(hunting): picked mon.noname-b con 0x7f1d340af730 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1d34145af0 0x7f1d3414c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 10 monclient(hunting): picked mon.noname-c con 0x7f1d34145af0 addr v2:172.21.15.134:3300/0 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d340afc50 con 0x7f1d3406a860 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d3406e450 con 0x7f1d34145af0 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d340b0170 con 0x7f1d340af730 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.844+0000 7f1d3c305640 10 monclient(hunting): authenticate will time out at 1520.195620s 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a0a1640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1d3406a860 0x7f1d3413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56470/0 (socket says 172.21.15.110:56470) 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a0a1640 1 -- 172.21.15.110:0/2955341418 learned_addr learned my addr 172.21.15.110:0/2955341418 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a8a2640 1 --2- 172.21.15.110:0/2955341418 >> v2:172.21.15.134:3300/0 conn(0x7f1d34145af0 0x7f1d3414c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d398a0640 1 --2- 172.21.15.110:0/2955341418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 0x7f1d341455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a8a2640 10 monclient(hunting): get_auth_request con 0x7f1d34145af0 auth_method 0 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a8a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a8a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a8a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d398a0640 10 monclient(hunting): get_auth_request con 0x7f1d340af730 auth_method 0 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d398a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d398a0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d398a0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a8a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a8a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.845+0000 7f1d3a8a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.846+0000 7f1d398a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.846+0000 7f1d398a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.846+0000 7f1d398a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.847+0000 7f1d3a8a2640 10 monclient(hunting): handle_auth_done global_id 4720 payload 274 2024-01-20T11:17:56.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.847+0000 7f1d3a8a2640 1 -- 172.21.15.110:0/2955341418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 msgr2=0x7f1d341455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:17:56.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.847+0000 7f1d3a8a2640 1 --2- 172.21.15.110:0/2955341418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 0x7f1d341455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.847+0000 7f1d3a8a2640 1 -- 172.21.15.110:0/2955341418 >> v1:172.21.15.110:6789/0 conn(0x7f1d3406a860 legacy=0x7f1d3413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.847+0000 7f1d3a8a2640 10 monclient: _finish_hunting 0 2024-01-20T11:17:56.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.847+0000 7f1d3a8a2640 1 monclient: found mon.noname-c 2024-01-20T11:17:56.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.847+0000 7f1d3a8a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:56.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.847+0000 7f1d3a8a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.847+0000 7f1d3a8a2640 1 -- 172.21.15.110:0/2955341418 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d34145fa0 con 0x7f1d34145af0 2024-01-20T11:17:56.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.849+0000 7f1d3a8a2640 1 --2- 172.21.15.110:0/2955341418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 0x7f1d3414c940 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009d90 tx=0x7f1d2c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50fbf26d55e375d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:56.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 1 -- 172.21.15.110:0/2955341418 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c02cb30 con 0x7f1d34145af0 2024-01-20T11:17:56.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:56.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:17:56.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 10 monclient: dump: 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 10 monclient: _finish_auth 0 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 10 monclient: _check_auth_tickets 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:56.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3c305640 5 monclient: authenticate success, global_id 4720 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 1 -- 172.21.15.110:0/2955341418 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c02cc90 con 0x7f1d34145af0 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3c305640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3c305640 10 monclient: get_monmap_and_config success 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3c305640 4 set_mon_vals no callback set 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 1 -- 172.21.15.110:0/2955341418 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c034440 con 0x7f1d34145af0 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.852+0000 7f1d3909f640 10 monclient: dump: 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.856 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 10 monclient: shutdown 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 1 -- 172.21.15.110:0/2955341418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 msgr2=0x7f1d3414c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 1 --2- 172.21.15.110:0/2955341418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 0x7f1d3414c940 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009d90 tx=0x7f1d2c02bd80 comp rx=0 tx=0).stop 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 1 -- 172.21.15.110:0/2955341418 shutdown_connections 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 1 --2- 172.21.15.110:0/2955341418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 0x7f1d3414c940 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 1 --2- 172.21.15.110:0/2955341418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 0x7f1d341455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 1 -- 172.21.15.110:0/2955341418 >> 172.21.15.110:0/2955341418 conn(0x7f1d340b47d0 msgr2=0x7f1d340b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:56.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 1 -- 172.21.15.110:0/2955341418 shutdown_connections 2024-01-20T11:17:56.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.853+0000 7f1d3c305640 1 -- 172.21.15.110:0/2955341418 wait complete. 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.854+0000 7f1d3c305640 10 monclient: build_initial_monmap 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.854+0000 7f1d3c305640 10 monclient: monmap: 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.854+0000 7f1d3c305640 1 librados: starting msgr at 2024-01-20T11:17:56.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.854+0000 7f1d3c305640 1 librados: starting objecter 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.854+0000 7f1d3c305640 1 Processor -- start 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.854+0000 7f1d3c305640 1 -- start start 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.854+0000 7f1d3c305640 1 librados: setting wanted keys 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.854+0000 7f1d3c305640 1 librados: calling monclient init 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.854+0000 7f1d3c305640 10 monclient: init 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient: _reopen_session rank -1 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d3406a860 0x7f1d341431b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient(hunting): picked mon.noname-a con 0x7f1d3406a860 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 0x7f1d34143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient(hunting): picked mon.noname-c con 0x7f1d340af730 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 0x7f1d341d7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:56.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient(hunting): picked mon.noname-b con 0x7f1d34145af0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d341d75e0 con 0x7f1d3406a860 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d34145080 con 0x7f1d34145af0 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient(hunting): start opening mon connection 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d34142c20 con 0x7f1d340af730 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient(hunting): _renew_subs 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3c305640 10 monclient(hunting): authenticate will time out at 1520.206601s 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d398a0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 0x7f1d34143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.855+0000 7f1d3a8a2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 0x7f1d341d7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a0a1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d3406a860 0x7f1d341431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a8a2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 0x7f1d341d7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46362/0 (socket says 172.21.15.110:46362) 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a8a2640 1 -- 172.21.15.110:0/2265550403 learned_addr learned my addr 172.21.15.110:0/2265550403 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:17:56.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a8a2640 10 monclient(hunting): get_auth_request con 0x7f1d34145af0 auth_method 0 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a8a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a8a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a8a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d398a0640 10 monclient(hunting): get_auth_request con 0x7f1d340af730 auth_method 0 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d398a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d398a0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d398a0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a0a1640 10 monclient(hunting): get_auth_request con 0x7f1d3406a860 auth_method 0 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a0a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a0a1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a0a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a8a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a8a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d3a8a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d398a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d398a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.856+0000 7f1d398a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.857+0000 7f1d3a0a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.857+0000 7f1d3a0a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.857+0000 7f1d3a0a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 10 monclient(hunting): handle_auth_done global_id 4723 payload 824 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 1 -- 172.21.15.110:0/2265550403 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 msgr2=0x7f1d34143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 0x7f1d34143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 1 -- 172.21.15.110:0/2265550403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d3406a860 msgr2=0x7f1d341431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d3406a860 0x7f1d341431b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 10 monclient: _finish_hunting 0 2024-01-20T11:17:56.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 1 monclient: found mon.noname-b 2024-01-20T11:17:56.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:17:56.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.859+0000 7f1d3a8a2640 1 -- 172.21.15.110:0/2265550403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c009a70 con 0x7f1d34145af0 2024-01-20T11:17:56.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.861+0000 7f1d3a8a2640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 0x7f1d341d7a70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009eb0 tx=0x7f1d2c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:56.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.861+0000 7f1d2affd640 10 client.?.objecter ms_handle_connect 0x7f1d34145af0 2024-01-20T11:17:56.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.861+0000 7f1d2affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:17:56.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 1 -- 172.21.15.110:0/2265550403 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c03e410 con 0x7f1d34145af0 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 10 monclient: dump: 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:56.868 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 10 monclient: _finish_auth 0 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 10 monclient: _check_auth_tickets 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 1 -- 172.21.15.110:0/2265550403 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c03e570 con 0x7f1d34145af0 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3c305640 5 monclient: authenticate success, global_id 4723 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3c305640 10 monclient: _renew_subs 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3c305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d34140370 con 0x7f1d34145af0 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 1 -- 172.21.15.110:0/2265550403 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c03e900 con 0x7f1d34145af0 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:17:56.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d2affd640 10 monclient: dump: 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3889e640 4 set_mon_vals no callback set 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3c305640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3c305640 10 monclient: _renew_subs 2024-01-20T11:17:56.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3c305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d341d8190 con 0x7f1d34145af0 2024-01-20T11:17:56.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.865+0000 7f1d3c305640 1 librados: init done 2024-01-20T11:17:56.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.866+0000 7f1d3c305640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:56.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.866+0000 7f1d3c305640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:56.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.866+0000 7f1d3c305640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:17:56.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:56.866+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d341d73b0 con 0x7f1d34145af0 2024-01-20T11:17:57.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.107+0000 7f1d2affd640 1 -- 172.21.15.110:0/2265550403 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1d2c03b050 con 0x7f1d34145af0 2024-01-20T11:17:57.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.107+0000 7f1d2affd640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1d1806c860 0x7f1d1806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:17:57.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.107+0000 7f1d3a0a1640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1d1806c860 0x7f1d1806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:17:57.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.107+0000 7f1d3a0a1640 10 monclient: get_auth_request con 0x7f1d1806c860 auth_method 0 2024-01-20T11:17:57.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.108+0000 7f1d3a0a1640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1d1806c860 0x7f1d1806ed20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d30007fd0 tx=0x7f1d30007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:17:57.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.108+0000 7f1d2affd640 10 client.4723.objecter ms_handle_connect 0x7f1d1806c860 2024-01-20T11:17:57.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.110+0000 7f1d2affd640 1 -- 172.21.15.110:0/2265550403 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1d2c07abb0 con 0x7f1d34145af0 2024-01-20T11:17:57.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.110+0000 7f1d2affd640 10 client.4723.objecter ms_dispatch 0x7f1d34000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:17:57.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.110+0000 7f1d2affd640 3 client.4723.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:17:57.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.110+0000 7f1d2affd640 3 client.4723.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:17:57.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.110+0000 7f1d2affd640 20 client.4723.objecter dump_active .. 0 homeless 2024-01-20T11:17:57.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.120+0000 7f1d2affd640 1 -- 172.21.15.110:0/2265550403 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1d2c076ef0 con 0x7f1d34145af0 2024-01-20T11:17:57.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.120+0000 7f1d2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:17:57.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.120+0000 7f1d2affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:17:57.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.285+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1d340723e0 con 0x7f1d1806c860 2024-01-20T11:17:57.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.288+0000 7f1d2affd640 1 -- 172.21.15.110:0/2265550403 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64226 (secure 0 0 0) 0x7f1d300013c0 con 0x7f1d1806c860 2024-01-20T11:17:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:17:57.290 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 10 librados: watch_flush enter 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 10 librados: watch_flush exit 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 20 client.4723.objecter shutdown clearing up homeless session... 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 10 client.4723.objecter successfully canceled tick 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1d1806c860 msgr2=0x7f1d1806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1d1806c860 0x7f1d1806ed20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d30007fd0 tx=0x7f1d30007e80 comp rx=0 tx=0).stop 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 10 monclient: shutdown 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 msgr2=0x7f1d341d7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 0x7f1d341d7a70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009eb0 tx=0x7f1d2c004a40 comp rx=0 tx=0).stop 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 shutdown_connections 2024-01-20T11:17:57.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1d1806c860 0x7f1d1806ed20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:57.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d34145af0 0x7f1d341d7a70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:57.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1d340af730 0x7f1d34143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:57.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 --2- 172.21.15.110:0/2265550403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d3406a860 0x7f1d341431b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:17:57.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 >> 172.21.15.110:0/2265550403 conn(0x7f1d340b47d0 msgr2=0x7f1d340b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:17:57.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 shutdown_connections 2024-01-20T11:17:57.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 -- 172.21.15.110:0/2265550403 wait complete. 2024-01-20T11:17:57.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:17:57.290+0000 7f1d3c305640 1 librados: shutdown 2024-01-20T11:17:57.302 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-01-20T11:17:56.741659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001942"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280550,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.0350000000000001},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.4699999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":15.256},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.388},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.632999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.5150000000000006},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":10.962}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":6.8070000000000004},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":13.227}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.1660000000000004},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":13.988}]}]},{"osd":7,"up_from":21,"seq":90194313253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":3.9820000000000002},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7279999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":13.026},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.5800000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":6.9370000000000003},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.935}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.973000000000001},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":8.3689999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.284000000000001},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":15.173999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.568},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.132}]}]},{"osd":3,"up_from":21,"seq":90194313254,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.3520000000000003},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.625}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.1899999999999995},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.699}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":8.8420000000000005},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.819000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.972},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.132}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":13.090999999999999},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.481999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.861000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.307}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444067,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.7169999999999996},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":10.5}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":16.443000000000001},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.4920000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":9.8330000000000002},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.622999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.2260000000000009},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.239000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.66},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.5010000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.439},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":11.917}]}]},{"osd":1,"up_from":17,"seq":73014444068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.5759999999999996},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":9.3399999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.176},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.4260000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.492000000000001},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.4829999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":12.696999999999999},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.585000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.7089999999999996},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":13.324999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.741},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.179}]}]},{"osd":0,"up_from":17,"seq":73014444068,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":6.5339999999999998},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":12.762}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.7089999999999996},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.7490000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":15.332000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.4500000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.346},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.914999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":9.6530000000000005},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":13.965}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.391},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":11.532999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":6.2779999999999996},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.2450000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.262},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":8.7319999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.8630000000000004},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.609999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":11.878},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.933}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.907999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.238}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.837999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.994999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:17:58.359 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:49.383 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:58.363 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:49.387 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:58.379 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:49.404 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:58.406 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:03:49.412 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:58.411 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:03:49.418 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:58.412 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:49.437 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:58.414 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:49.438 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:58.427 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:03:49.433 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:17:58.430 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:49.454 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:00.304 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:00.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.403+0000 7f74d4747640 10 monclient: get_monmap_and_config 2024-01-20T11:18:00.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.403+0000 7f74d4747640 10 monclient: build_initial_monmap 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.403+0000 7f74d4747640 10 monclient: monmap: 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:00.405418+0000 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:00.405418+0000 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:00.405 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:00.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.404+0000 7f74d4747640 1 Processor -- start 2024-01-20T11:18:00.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.404+0000 7f74d4747640 1 -- start start 2024-01-20T11:18:00.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.404+0000 7f74d4747640 10 monclient: init 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74cc06bb00 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f74cc0af790 0x7f74cc0ae410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 10 monclient(hunting): picked mon.noname-c con 0x7f74cc0af790 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 0x7f74cc0aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 10 monclient(hunting): picked mon.noname-b con 0x7f74cc0ae950 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc0af320 con 0x7f74cc06bb00 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc06bee0 con 0x7f74cc0af790 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0ada30 con 0x7f74cc0ae950 2024-01-20T11:18:00.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d4747640 10 monclient(hunting): authenticate will time out at 1523.756352s 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d2ce4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 0x7f74cc0aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.405+0000 7f74d1ce2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f74cc0af790 0x7f74cc0ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d2ce4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 0x7f74cc0aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43964/0 (socket says 172.21.15.110:43964) 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d2ce4640 1 -- 172.21.15.110:0/877031340 learned_addr learned my addr 172.21.15.110:0/877031340 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d2ce4640 10 monclient(hunting): get_auth_request con 0x7f74cc0ae950 auth_method 0 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d2ce4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d2ce4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d2ce4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d1ce2640 10 monclient(hunting): get_auth_request con 0x7f74cc0af790 auth_method 0 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d1ce2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d1ce2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d1ce2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:00.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d1ce2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d1ce2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d1ce2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d2ce4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d2ce4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.406+0000 7f74d2ce4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:00.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.408+0000 7f74d1ce2640 10 monclient(hunting): handle_auth_done global_id 4726 payload 274 2024-01-20T11:18:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.408+0000 7f74d1ce2640 1 -- 172.21.15.110:0/877031340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 msgr2=0x7f74cc0aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.408+0000 7f74d1ce2640 1 --2- 172.21.15.110:0/877031340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 0x7f74cc0aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.408+0000 7f74d1ce2640 1 -- 172.21.15.110:0/877031340 >> v1:172.21.15.134:6789/0 conn(0x7f74cc06bb00 legacy=0x7f74cc0b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.408+0000 7f74d1ce2640 10 monclient: _finish_hunting 0 2024-01-20T11:18:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.408+0000 7f74d1ce2640 1 monclient: found mon.noname-c 2024-01-20T11:18:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.408+0000 7f74d1ce2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.408+0000 7f74d1ce2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.408+0000 7f74d1ce2640 1 -- 172.21.15.110:0/877031340 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc0add10 con 0x7f74cc0af790 2024-01-20T11:18:00.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.410+0000 7f74d1ce2640 1 --2- 172.21.15.110:0/877031340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 0x7f74cc0ae410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009790 tx=0x7f74bc02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd1e850ec8ecf91c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 1 -- 172.21.15.110:0/877031340 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0316f0 con 0x7f74cc0af790 2024-01-20T11:18:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:00.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 10 monclient: dump: 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 10 monclient: _finish_auth 0 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 10 monclient: _check_auth_tickets 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:00.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 1 -- 172.21.15.110:0/877031340 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc031850 con 0x7f74cc0af790 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 1 -- 172.21.15.110:0/877031340 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc031be0 con 0x7f74cc0af790 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d14e1640 10 monclient: dump: 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:00.417 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:00.418 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d4747640 5 monclient: authenticate success, global_id 4726 2024-01-20T11:18:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d4747640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:00.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.414+0000 7f74d4747640 4 set_mon_vals no callback set 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 10 monclient: shutdown 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 1 -- 172.21.15.110:0/877031340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 msgr2=0x7f74cc0ae410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 1 --2- 172.21.15.110:0/877031340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 0x7f74cc0ae410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009790 tx=0x7f74bc02b790 comp rx=0 tx=0).stop 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 1 -- 172.21.15.110:0/877031340 shutdown_connections 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 1 --2- 172.21.15.110:0/877031340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 0x7f74cc0aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 1 --2- 172.21.15.110:0/877031340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 0x7f74cc0ae410 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 1 -- 172.21.15.110:0/877031340 >> 172.21.15.110:0/877031340 conn(0x7f74cc066d60 msgr2=0x7f74cc0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 1 -- 172.21.15.110:0/877031340 shutdown_connections 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.415+0000 7f74d4747640 1 -- 172.21.15.110:0/877031340 wait complete. 2024-01-20T11:18:00.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.416+0000 7f74d4747640 10 monclient: build_initial_monmap 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.416+0000 7f74d4747640 10 monclient: monmap: 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.416+0000 7f74d4747640 1 librados: starting msgr at 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.416+0000 7f74d4747640 1 librados: starting objecter 2024-01-20T11:18:00.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.416+0000 7f74d4747640 1 Processor -- start 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.416+0000 7f74d4747640 1 -- start start 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.416+0000 7f74d4747640 1 librados: setting wanted keys 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.416+0000 7f74d4747640 1 librados: calling monclient init 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.416+0000 7f74d4747640 10 monclient: init 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74cc06bb00 0x7f74cc0707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient(hunting): picked mon.noname-a con 0x7f74cc06bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 0x7f74cc11dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient(hunting): picked mon.noname-c con 0x7f74cc0ae950 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 0x7f74cc1dad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient(hunting): picked mon.noname-b con 0x7f74cc0af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc1d4f10 con 0x7f74cc06bb00 2024-01-20T11:18:00.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc1da810 con 0x7f74cc0af790 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc11eae0 con 0x7f74cc0ae950 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d4747640 10 monclient(hunting): authenticate will time out at 1523.768294s 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d2ce4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 0x7f74cc1dad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d24e3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74cc06bb00 0x7f74cc0707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d2ce4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 0x7f74cc1dad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:56258/0 (socket says 172.21.15.110:56258) 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d2ce4640 1 -- 172.21.15.110:0/1294773023 learned_addr learned my addr 172.21.15.110:0/1294773023 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d2ce4640 10 monclient(hunting): get_auth_request con 0x7f74cc0af790 auth_method 0 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d2ce4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d2ce4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.417+0000 7f74d2ce4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d1ce2640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 0x7f74cc11dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:00.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d24e3640 10 monclient(hunting): get_auth_request con 0x7f74cc06bb00 auth_method 0 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d24e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d24e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d24e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d1ce2640 10 monclient(hunting): get_auth_request con 0x7f74cc0ae950 auth_method 0 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d1ce2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d1ce2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d1ce2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d2ce4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d2ce4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d2ce4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d24e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d24e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d24e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d1ce2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:00.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d1ce2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.418+0000 7f74d1ce2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 10 monclient(hunting): handle_auth_done global_id 4729 payload 824 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 1 -- 172.21.15.110:0/1294773023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 msgr2=0x7f74cc11dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 0x7f74cc11dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 1 -- 172.21.15.110:0/1294773023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74cc06bb00 msgr2=0x7f74cc0707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74cc06bb00 0x7f74cc0707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 10 monclient: _finish_hunting 0 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 1 monclient: found mon.noname-b 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:00.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.421+0000 7f74d2ce4640 1 -- 172.21.15.110:0/1294773023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc0093a0 con 0x7f74cc0af790 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.423+0000 7f74d2ce4640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 0x7f74cc1dad10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f74c800efc0 tx=0x7f74c800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.423+0000 7f74c2ffd640 10 client.?.objecter ms_handle_connect 0x7f74cc0af790 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.423+0000 7f74c2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 1 -- 172.21.15.110:0/1294773023 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c800cd20 con 0x7f74cc0af790 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 10 monclient: dump: 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 1 -- 172.21.15.110:0/1294773023 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c800f040 con 0x7f74cc0af790 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74d4747640 5 monclient: authenticate success, global_id 4729 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74d4747640 10 monclient: _renew_subs 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74d4747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc11dff0 con 0x7f74cc0af790 2024-01-20T11:18:00.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 1 -- 172.21.15.110:0/1294773023 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c8003bd0 con 0x7f74cc0af790 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74c2ffd640 10 monclient: dump: 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:00.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74d0ce0640 4 set_mon_vals no callback set 2024-01-20T11:18:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.426+0000 7f74d4747640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.427+0000 7f74d4747640 10 monclient: _renew_subs 2024-01-20T11:18:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.427+0000 7f74d4747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.427+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc1db250 con 0x7f74cc0af790 2024-01-20T11:18:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.427+0000 7f74d4747640 1 librados: init done 2024-01-20T11:18:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.427+0000 7f74d4747640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.427+0000 7f74d4747640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.427+0000 7f74d4747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:00.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.427+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74cc06c8d0 con 0x7f74cc0af790 2024-01-20T11:18:00.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.446+0000 7f74c2ffd640 1 -- 172.21.15.110:0/1294773023 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f74c8003d30 con 0x7f74cc0af790 2024-01-20T11:18:00.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.446+0000 7f74c2ffd640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f74b006c8b0 0x7f74b006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:00.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.446+0000 7f74d24e3640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f74b006c8b0 0x7f74b006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:00.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.446+0000 7f74d24e3640 10 monclient: get_auth_request con 0x7f74b006c8b0 auth_method 0 2024-01-20T11:18:00.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.447+0000 7f74c2ffd640 10 client.4729.objecter ms_handle_connect 0x7f74b006c8b0 2024-01-20T11:18:00.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.447+0000 7f74d24e3640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f74b006c8b0 0x7f74b006ed70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f74c40017f0 tx=0x7f74c4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:00.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.450+0000 7f74c2ffd640 1 -- 172.21.15.110:0/1294773023 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f74c8059560 con 0x7f74cc0af790 2024-01-20T11:18:00.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.450+0000 7f74c2ffd640 10 client.4729.objecter ms_dispatch 0x7f74cc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:00.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.450+0000 7f74c2ffd640 3 client.4729.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:00.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.450+0000 7f74c2ffd640 3 client.4729.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:00.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.450+0000 7f74c2ffd640 20 client.4729.objecter dump_active .. 0 homeless 2024-01-20T11:18:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.685+0000 7f74c2ffd640 1 -- 172.21.15.110:0/1294773023 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f74c80558a0 con 0x7f74cc0af790 2024-01-20T11:18:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.685+0000 7f74c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.685+0000 7f74c2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.887+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74cc1da9a0 con 0x7f74b006c8b0 2024-01-20T11:18:00.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.890+0000 7f74c2ffd640 1 -- 172.21.15.110:0/1294773023 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64211 (secure 0 0 0) 0x7f74cc1da9a0 con 0x7f74b006c8b0 2024-01-20T11:18:00.891 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:00.892 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 10 librados: watch_flush enter 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 10 librados: watch_flush exit 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 20 client.4729.objecter shutdown clearing up homeless session... 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 10 client.4729.objecter successfully canceled tick 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f74b006c8b0 msgr2=0x7f74b006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f74b006c8b0 0x7f74b006ed70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f74c40017f0 tx=0x7f74c4009040 comp rx=0 tx=0).stop 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 10 monclient: shutdown 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 msgr2=0x7f74cc1dad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 0x7f74cc1dad10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f74c800efc0 tx=0x7f74c800c490 comp rx=0 tx=0).stop 2024-01-20T11:18:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 shutdown_connections 2024-01-20T11:18:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f74b006c8b0 0x7f74b006ed70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74cc0af790 0x7f74cc1dad10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74cc0ae950 0x7f74cc11dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 --2- 172.21.15.110:0/1294773023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74cc06bb00 0x7f74cc0707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 >> 172.21.15.110:0/1294773023 conn(0x7f74cc066d60 msgr2=0x7f74cc0b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 shutdown_connections 2024-01-20T11:18:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 -- 172.21.15.110:0/1294773023 wait complete. 2024-01-20T11:18:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:00.892+0000 7f74d4747640 1 librados: shutdown 2024-01-20T11:18:00.904 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-01-20T11:18:00.742320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001957"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":14.779},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.5800000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.32},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.258}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.858000000000001},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.531000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.6760000000000002},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":13.847}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.503},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":11.627000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.6720000000000006},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":16.774999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":3.9820000000000002},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7279999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":13.026},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.5800000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":6.9370000000000003},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.935}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.973000000000001},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":8.3689999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.284000000000001},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":15.173999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.568},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.132}]}]},{"osd":3,"up_from":21,"seq":90194313254,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.3520000000000003},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.625}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.1899999999999995},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.699}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":8.8420000000000005},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.819000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.972},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.132}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":13.090999999999999},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.481999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.861000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.307}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.0750000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.673}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.33},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.8979999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.555999999999999},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.5139999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.111000000000001},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.034000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.92},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.771000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.2870000000000008},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.887}]}]},{"osd":1,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.7599999999999998},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":11.651}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.510999999999999},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.3869999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":10.106},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.4190000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":14.763},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":14.038}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.1539999999999999},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":13.313000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.8309999999999995},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.105}]}]},{"osd":0,"up_from":17,"seq":73014444069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":8.1210000000000004},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":6.0179999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.5570000000000004},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.59}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":13.191000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.5609999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.025},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7490000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.130000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.362}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.991},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":9.3130000000000006}]}]},{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":6.2779999999999996},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.2450000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.262},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":8.7319999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.8630000000000004},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.609999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":11.878},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.933}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.907999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.238}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.837999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.994999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:00.906 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:01.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.004+0000 7f4dfe57c640 10 monclient: get_monmap_and_config 2024-01-20T11:18:01.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.004+0000 7f4dfe57c640 10 monclient: build_initial_monmap 2024-01-20T11:18:01.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.004+0000 7f4dfe57c640 10 monclient: monmap: 2024-01-20T11:18:01.005 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:01.005 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.005 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:01.005993+0000 2024-01-20T11:18:01.005 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:01.005993+0000 2024-01-20T11:18:01.006 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:01.006 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.006 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:01.006 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:01.006 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:01.006 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:01.006 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:01.006 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 1 Processor -- start 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 1 -- start start 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 10 monclient: init 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4df806bb00 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4df80af790 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4df80aeb20 0x7f4df80aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 10 monclient(hunting): picked mon.noname-c con 0x7f4df80aeb20 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df806bee0 con 0x7f4df80af790 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80ada30 con 0x7f4df806bb00 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df80add10 con 0x7f4df80aeb20 2024-01-20T11:18:01.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.005+0000 7f4dfe57c640 10 monclient(hunting): authenticate will time out at 1524.356919s 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.006+0000 7f4dfcb19640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4df80aeb20 0x7f4df80aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.006+0000 7f4df77fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4df80af790 0x7f4df80ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:41002/0 (socket says 172.21.15.110:41002) 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.006+0000 7f4df77fe640 1 -- 172.21.15.110:0/4231299650 learned_addr learned my addr 172.21.15.110:0/4231299650 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.006+0000 7f4dfcb19640 10 monclient(hunting): get_auth_request con 0x7f4df80aeb20 auth_method 0 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.006+0000 7f4dfcb19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.006+0000 7f4dfcb19640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.006+0000 7f4dfcb19640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.007+0000 7f4dfcb19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.007+0000 7f4dfcb19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:01.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.007+0000 7f4dfcb19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.008+0000 7f4dfcb19640 10 monclient(hunting): handle_auth_done global_id 4732 payload 274 2024-01-20T11:18:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.008+0000 7f4dfcb19640 1 -- 172.21.15.110:0/4231299650 >> v1:172.21.15.134:6789/0 conn(0x7f4df806bb00 legacy=0x7f4df80b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.008+0000 7f4dfcb19640 1 -- 172.21.15.110:0/4231299650 >> v1:172.21.15.110:6789/0 conn(0x7f4df80af790 legacy=0x7f4df80ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.008+0000 7f4dfcb19640 10 monclient: _finish_hunting 0 2024-01-20T11:18:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.008+0000 7f4dfcb19640 1 monclient: found mon.noname-c 2024-01-20T11:18:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.008+0000 7f4dfcb19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.008+0000 7f4dfcb19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:01.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.008+0000 7f4dfcb19640 1 -- 172.21.15.110:0/4231299650 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df80adff0 con 0x7f4df80aeb20 2024-01-20T11:18:01.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.011+0000 7f4dfcb19640 1 --2- 172.21.15.110:0/4231299650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df80aeb20 0x7f4df80aef90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009d90 tx=0x7f4df002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=581bf234c506834c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:01.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 1 -- 172.21.15.110:0/4231299650 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df002cbd0 con 0x7f4df80aeb20 2024-01-20T11:18:01.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 10 monclient: dump: 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:01.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 1 -- 172.21.15.110:0/4231299650 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df002cd30 con 0x7f4df80aeb20 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 5 monclient: authenticate success, global_id 4732 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 4 set_mon_vals no callback set 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 1 -- 172.21.15.110:0/4231299650 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df002cea0 con 0x7f4df80aeb20 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4df6ffd640 10 monclient: dump: 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.018 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 10 monclient: shutdown 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 1 -- 172.21.15.110:0/4231299650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df80aeb20 msgr2=0x7f4df80aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 1 --2- 172.21.15.110:0/4231299650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df80aeb20 0x7f4df80aef90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009d90 tx=0x7f4df002be20 comp rx=0 tx=0).stop 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 1 -- 172.21.15.110:0/4231299650 shutdown_connections 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 1 --2- 172.21.15.110:0/4231299650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df80aeb20 0x7f4df80aef90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 1 -- 172.21.15.110:0/4231299650 >> 172.21.15.110:0/4231299650 conn(0x7f4df8066d60 msgr2=0x7f4df80b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 1 -- 172.21.15.110:0/4231299650 shutdown_connections 2024-01-20T11:18:01.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.015+0000 7f4dfe57c640 1 -- 172.21.15.110:0/4231299650 wait complete. 2024-01-20T11:18:01.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.016+0000 7f4dfe57c640 10 monclient: build_initial_monmap 2024-01-20T11:18:01.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.016+0000 7f4dfe57c640 10 monclient: monmap: 2024-01-20T11:18:01.022 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:01.022 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.016+0000 7f4dfe57c640 1 librados: starting msgr at 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.016+0000 7f4dfe57c640 1 librados: starting objecter 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.016+0000 7f4dfe57c640 1 Processor -- start 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 1 -- start start 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 1 librados: setting wanted keys 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 1 librados: calling monclient init 2024-01-20T11:18:01.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient: init 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df806bb00 0x7f4df80b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient(hunting): picked mon.noname-b con 0x7f4df806bb00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4df80aeb20 0x7f4df81d7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient(hunting): picked mon.noname-a con 0x7f4df80aeb20 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4df80af790 0x7f4df813ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient(hunting): picked mon.noname-c con 0x7f4df80af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df8121020 con 0x7f4df80aeb20 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df8121290 con 0x7f4df806bb00 2024-01-20T11:18:01.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df811f940 con 0x7f4df80af790 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.017+0000 7f4dfe57c640 10 monclient(hunting): authenticate will time out at 1524.368781s 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4df80aeb20 0x7f4df81d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df806bb00 0x7f4df80b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4dfcb19640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4df80af790 0x7f4df813ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df806bb00 0x7f4df80b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:56270/0 (socket says 172.21.15.110:56270) 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4df80aeb20 0x7f4df81d7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45770/0 (socket says 172.21.15.110:45770) 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 1 -- 172.21.15.110:0/16968333 learned_addr learned my addr 172.21.15.110:0/16968333 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 10 monclient(hunting): get_auth_request con 0x7f4df806bb00 auth_method 0 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:01.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df77fe640 10 monclient(hunting): get_auth_request con 0x7f4df80aeb20 auth_method 0 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4dfcb19640 10 monclient(hunting): get_auth_request con 0x7f4df80af790 auth_method 0 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4dfcb19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4dfcb19640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4dfcb19640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.018+0000 7f4df77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.019+0000 7f4dfcb19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:01.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.019+0000 7f4dfcb19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.019+0000 7f4dfcb19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 10 monclient(hunting): handle_auth_done global_id 4735 payload 824 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 1 -- 172.21.15.110:0/16968333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4df80af790 msgr2=0x7f4df813ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4df80af790 0x7f4df813ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 1 -- 172.21.15.110:0/16968333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4df80aeb20 msgr2=0x7f4df81d7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4df80aeb20 0x7f4df81d7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 1 monclient: found mon.noname-b 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:01.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.021+0000 7f4df7fff640 1 -- 172.21.15.110:0/16968333 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0009a70 con 0x7f4df806bb00 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.023+0000 7f4df7fff640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df806bb00 0x7f4df80b4ad0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4de800c410 tx=0x7f4de800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.023+0000 7f4df4ff9640 10 client.?.objecter ms_handle_connect 0x7f4df806bb00 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.023+0000 7f4df4ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 1 -- 172.21.15.110:0/16968333 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de80035f0 con 0x7f4df806bb00 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 10 monclient: dump: 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:01.029 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 1 -- 172.21.15.110:0/16968333 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de8003750 con 0x7f4df806bb00 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 1 -- 172.21.15.110:0/16968333 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de8003b40 con 0x7f4df806bb00 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df4ff9640 10 monclient: dump: 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:01.030 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4dfe57c640 5 monclient: authenticate success, global_id 4735 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4dfe57c640 10 monclient: _renew_subs 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4dfe57c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df811f080 con 0x7f4df806bb00 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.026+0000 7f4df5ffb640 4 set_mon_vals no callback set 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.027+0000 7f4dfe57c640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.027+0000 7f4dfe57c640 10 monclient: _renew_subs 2024-01-20T11:18:01.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.027+0000 7f4dfe57c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:01.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.027+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df811f690 con 0x7f4df806bb00 2024-01-20T11:18:01.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.027+0000 7f4dfe57c640 1 librados: init done 2024-01-20T11:18:01.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.027+0000 7f4dfe57c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:01.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.027+0000 7f4dfe57c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:01.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.027+0000 7f4dfe57c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:01.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.027+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df811f2a0 con 0x7f4df806bb00 2024-01-20T11:18:01.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.269+0000 7f4df4ff9640 1 -- 172.21.15.110:0/16968333 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4de8003ca0 con 0x7f4df806bb00 2024-01-20T11:18:01.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.269+0000 7f4df4ff9640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4de406c860 0x7f4de406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.269+0000 7f4df77fe640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4de406c860 0x7f4de406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.269+0000 7f4df77fe640 10 monclient: get_auth_request con 0x7f4de406c860 auth_method 0 2024-01-20T11:18:01.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.270+0000 7f4df77fe640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4de406c860 0x7f4de406ed20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4df80b5150 tx=0x7f4dec007d00 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:01.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.270+0000 7f4df4ff9640 10 client.4735.objecter ms_handle_connect 0x7f4de406c860 2024-01-20T11:18:01.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.270+0000 7f4df4ff9640 1 -- 172.21.15.110:0/16968333 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4de80010c0 con 0x7f4df806bb00 2024-01-20T11:18:01.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.270+0000 7f4df4ff9640 10 client.4735.objecter ms_dispatch 0x7f4df8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:01.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.270+0000 7f4df4ff9640 3 client.4735.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:01.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.270+0000 7f4df4ff9640 3 client.4735.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:01.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.270+0000 7f4df4ff9640 20 client.4735.objecter dump_active .. 0 homeless 2024-01-20T11:18:01.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.281+0000 7f4df4ff9640 1 -- 172.21.15.110:0/16968333 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4de8053950 con 0x7f4df806bb00 2024-01-20T11:18:01.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.281+0000 7f4df4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:01.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.281+0000 7f4df4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:01.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.446+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4df806d380 con 0x7f4de406c860 2024-01-20T11:18:01.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.449+0000 7f4df4ff9640 1 -- 172.21.15.110:0/16968333 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64211 (secure 0 0 0) 0x7f4dec006c40 con 0x7f4de406c860 2024-01-20T11:18:01.450 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:01.451 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 10 librados: watch_flush enter 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 10 librados: watch_flush exit 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 20 client.4735.objecter shutdown clearing up homeless session... 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 10 client.4735.objecter successfully canceled tick 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4de406c860 msgr2=0x7f4de406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4de406c860 0x7f4de406ed20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4df80b5150 tx=0x7f4dec007d00 comp rx=0 tx=0).stop 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 10 monclient: shutdown 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df806bb00 msgr2=0x7f4df80b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df806bb00 0x7f4df80b4ad0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4de800c410 tx=0x7f4de800c8d0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 shutdown_connections 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4de406c860 0x7f4de406ed20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4df80af790 0x7f4df813ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4df80aeb20 0x7f4df81d7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 --2- 172.21.15.110:0/16968333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4df806bb00 0x7f4df80b4ad0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 >> 172.21.15.110:0/16968333 conn(0x7f4df8066d60 msgr2=0x7f4df8144fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:01.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 shutdown_connections 2024-01-20T11:18:01.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 -- 172.21.15.110:0/16968333 wait complete. 2024-01-20T11:18:01.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.451+0000 7f4dfe57c640 1 librados: shutdown 2024-01-20T11:18:01.462 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-01-20T11:18:00.742320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001957"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":14.779},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.5800000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.32},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.258}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.858000000000001},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.531000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.6760000000000002},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":13.847}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.503},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":11.627000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.6720000000000006},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":16.774999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":3.9820000000000002},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7279999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":13.026},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.5800000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":6.9370000000000003},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.935}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.973000000000001},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":8.3689999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.284000000000001},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":15.173999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.568},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.132}]}]},{"osd":3,"up_from":21,"seq":90194313254,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.3520000000000003},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.625}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.1899999999999995},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.699}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":8.8420000000000005},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.819000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.972},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.132}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":13.090999999999999},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.481999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.861000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.307}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.0750000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.673}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.33},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.8979999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.555999999999999},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.5139999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.111000000000001},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.034000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.92},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.771000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.2870000000000008},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.887}]}]},{"osd":1,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.7599999999999998},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":11.651}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.510999999999999},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.3869999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":10.106},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.4190000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":14.763},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":14.038}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.1539999999999999},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":13.313000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.8309999999999995},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.105}]}]},{"osd":0,"up_from":17,"seq":73014444069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":8.1210000000000004},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":6.0179999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.5570000000000004},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.59}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":13.191000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.5609999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.025},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7490000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.130000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.362}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.991},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":9.3130000000000006}]}]},{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":6.2779999999999996},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.2450000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.262},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":8.7319999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.8630000000000004},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.609999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":11.878},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.933}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.907999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.238}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.837999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.994999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:01.464 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:01.464 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:01.464 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:01.464 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:01.465 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:01.465 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:01.465 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:01.465 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:01.465 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:01.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.563+0000 7f2f30101640 10 monclient: get_monmap_and_config 2024-01-20T11:18:01.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.563+0000 7f2f30101640 10 monclient: build_initial_monmap 2024-01-20T11:18:01.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.563+0000 7f2f30101640 10 monclient: monmap: 2024-01-20T11:18:01.564 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:01.564 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.564 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:01.565119+0000 2024-01-20T11:18:01.565 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:01.565119+0000 2024-01-20T11:18:01.565 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:01.565 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.565 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:01.565 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:01.565 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:01.565 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:01.565 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:01.565 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 1 Processor -- start 2024-01-20T11:18:01.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 1 -- start start 2024-01-20T11:18:01.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient: init 2024-01-20T11:18:01.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f280a96f0 0x7f2f2813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient(hunting): picked mon.noname-b con 0x7f2f280a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2f2813fa10 0x7f2f281423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient(hunting): picked mon.noname-a con 0x7f2f2813fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f280a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f280b25c0 con 0x7f2f280a9ad0 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f280aaea0 con 0x7f2f2813fa10 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280ab0a0 con 0x7f2f280a96f0 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.564+0000 7f2f30101640 10 monclient(hunting): authenticate will time out at 1524.916074s 2024-01-20T11:18:01.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2d69c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2f2813fa10 0x7f2f281423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2de9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f280a96f0 0x7f2f2813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2e69e640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2f280a9ad0 0x7f2f28142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:56662/0 (socket says 172.21.15.110:56662) 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2e69e640 1 -- 172.21.15.110:0/3861593564 learned_addr learned my addr 172.21.15.110:0/3861593564 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2d69c640 10 monclient(hunting): get_auth_request con 0x7f2f2813fa10 auth_method 0 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2d69c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2d69c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2d69c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2de9d640 10 monclient(hunting): get_auth_request con 0x7f2f280a96f0 auth_method 0 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2de9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2de9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.565+0000 7f2f2de9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.566+0000 7f2f2d69c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.566+0000 7f2f2d69c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:01.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.566+0000 7f2f2d69c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.566+0000 7f2f2de9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.566+0000 7f2f2de9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.566+0000 7f2f2de9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2de9d640 10 monclient(hunting): handle_auth_done global_id 4547 payload 274 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2de9d640 1 -- 172.21.15.110:0/3861593564 >> v2:172.21.15.110:3300/0 conn(0x7f2f2813fa10 msgr2=0x7f2f281423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2de9d640 1 --2- 172.21.15.110:0/3861593564 >> v2:172.21.15.110:3300/0 conn(0x7f2f2813fa10 0x7f2f281423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2de9d640 1 -- 172.21.15.110:0/3861593564 >> v1:172.21.15.134:6789/0 conn(0x7f2f280a9ad0 legacy=0x7f2f28142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2de9d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2de9d640 1 monclient: found mon.noname-b 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2ce9b640 1 -- 172.21.15.110:0/3861593564 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308706097 0 0) 0x7f2f280b25c0 con 0x7f2f280a9ad0 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2de9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2de9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2de9d640 1 -- 172.21.15.110:0/3861593564 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f280b3fb0 con 0x7f2f280a96f0 2024-01-20T11:18:01.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2d69c640 1 --2- 172.21.15.110:0/3861593564 >> v2:172.21.15.110:3300/0 conn(0x7f2f2813fa10 0x7f2f281423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:18:01.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.568+0000 7f2f2ce9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:18:01.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.570+0000 7f2f2de9d640 1 --2- 172.21.15.110:0/3861593564 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f280a96f0 0x7f2f2813f4d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2f240097a0 tx=0x7f2f2402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18a348ccad744d2f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 1 -- 172.21.15.110:0/3861593564 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2402e070 con 0x7f2f280a96f0 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 10 monclient: dump: 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:01.575 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 10 monclient: _finish_auth 0 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 10 monclient: _check_auth_tickets 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 1 -- 172.21.15.110:0/3861593564 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f240315a0 con 0x7f2f280a96f0 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 1 -- 172.21.15.110:0/3861593564 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24031930 con 0x7f2f280a96f0 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 5 monclient: authenticate success, global_id 4547 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 4 set_mon_vals no callback set 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f2ce9b640 10 monclient: dump: 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:01.576 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.577 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.577 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.577 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:01.577 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.577 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:01.577 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:01.577 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:01.577 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 10 monclient: shutdown 2024-01-20T11:18:01.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 1 -- 172.21.15.110:0/3861593564 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f280a96f0 msgr2=0x7f2f2813f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 1 --2- 172.21.15.110:0/3861593564 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f280a96f0 0x7f2f2813f4d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2f240097a0 tx=0x7f2f2402b840 comp rx=0 tx=0).stop 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 1 -- 172.21.15.110:0/3861593564 shutdown_connections 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 1 --2- 172.21.15.110:0/3861593564 >> v2:172.21.15.110:3300/0 conn(0x7f2f2813fa10 0x7f2f281423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 1 --2- 172.21.15.110:0/3861593564 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f280a96f0 0x7f2f2813f4d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.574+0000 7f2f30101640 1 -- 172.21.15.110:0/3861593564 >> 172.21.15.110:0/3861593564 conn(0x7f2f280b7bc0 msgr2=0x7f2f280b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.575+0000 7f2f30101640 1 -- 172.21.15.110:0/3861593564 shutdown_connections 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.575+0000 7f2f30101640 1 -- 172.21.15.110:0/3861593564 wait complete. 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.575+0000 7f2f30101640 10 monclient: build_initial_monmap 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.575+0000 7f2f30101640 10 monclient: monmap: 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:01.581 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 librados: starting msgr at 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 librados: starting objecter 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 Processor -- start 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 -- start start 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 librados: setting wanted keys 2024-01-20T11:18:01.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 librados: calling monclient init 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient: init 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f280a96f0 0x7f2f28136770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient(hunting): picked mon.noname-a con 0x7f2f280a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f280a9ad0 0x7f2f2813e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient(hunting): picked mon.noname-b con 0x7f2f280a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f28143300 0x7f2f281da470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient(hunting): picked mon.noname-c con 0x7f2f28143300 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2813ecc0 con 0x7f2f280a96f0 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f28144880 con 0x7f2f280a9ad0 2024-01-20T11:18:01.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f28144b60 con 0x7f2f28143300 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.576+0000 7f2f30101640 10 monclient(hunting): authenticate will time out at 1524.927873s 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2d69c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f280a9ad0 0x7f2f2813e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2e69e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f28143300 0x7f2f281da470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2de9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f280a96f0 0x7f2f28136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2d69c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f280a9ad0 0x7f2f2813e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:56272/0 (socket says 172.21.15.110:56272) 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2d69c640 1 -- 172.21.15.110:0/2244598878 learned_addr learned my addr 172.21.15.110:0/2244598878 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2d69c640 10 monclient(hunting): get_auth_request con 0x7f2f280a9ad0 auth_method 0 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2d69c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2d69c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2d69c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:01.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2e69e640 10 monclient(hunting): get_auth_request con 0x7f2f28143300 auth_method 0 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2e69e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2e69e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2e69e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2de9d640 10 monclient(hunting): get_auth_request con 0x7f2f280a96f0 auth_method 0 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2de9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2de9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.577+0000 7f2f2de9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.578+0000 7f2f2d69c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.578+0000 7f2f2d69c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.578+0000 7f2f2d69c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.578+0000 7f2f2e69e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:01.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.578+0000 7f2f2e69e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:01.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.578+0000 7f2f2e69e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:01.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.578+0000 7f2f2de9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:01.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.578+0000 7f2f2de9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:01.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.578+0000 7f2f2de9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 10 monclient(hunting): handle_auth_done global_id 4741 payload 824 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 1 -- 172.21.15.110:0/2244598878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f28143300 msgr2=0x7f2f281da470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f28143300 0x7f2f281da470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 1 -- 172.21.15.110:0/2244598878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f280a96f0 msgr2=0x7f2f28136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f280a96f0 0x7f2f28136770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 10 monclient: _finish_hunting 0 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 1 monclient: found mon.noname-b 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.581+0000 7f2f2d69c640 1 -- 172.21.15.110:0/2244598878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f18009400 con 0x7f2f280a9ad0 2024-01-20T11:18:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.583+0000 7f2f2d69c640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f280a9ad0 0x7f2f2813e780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800e8e0 tx=0x7f2f1800eda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:01.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.583+0000 7f2f1e7fc640 10 client.?.objecter ms_handle_connect 0x7f2f280a9ad0 2024-01-20T11:18:01.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.584+0000 7f2f1e7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:01.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 1 -- 172.21.15.110:0/2244598878 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1800c940 con 0x7f2f280a9ad0 2024-01-20T11:18:01.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:01.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:01.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 10 monclient: dump: 2024-01-20T11:18:01.589 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:01.589 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 1 -- 172.21.15.110:0/2244598878 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1800caa0 con 0x7f2f280a9ad0 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 1 -- 172.21.15.110:0/2244598878 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1800ce30 con 0x7f2f280a9ad0 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1e7fc640 10 monclient: dump: 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f30101640 5 monclient: authenticate success, global_id 4741 2024-01-20T11:18:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f30101640 10 monclient: _renew_subs 2024-01-20T11:18:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f30101640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f24009400 con 0x7f2f280a9ad0 2024-01-20T11:18:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f1f7fe640 4 set_mon_vals no callback set 2024-01-20T11:18:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f30101640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f30101640 10 monclient: _renew_subs 2024-01-20T11:18:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f30101640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f281d4a30 con 0x7f2f280a9ad0 2024-01-20T11:18:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.588+0000 7f2f30101640 1 librados: init done 2024-01-20T11:18:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.589+0000 7f2f30101640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.589+0000 7f2f30101640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.589+0000 7f2f30101640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.589+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f28075490 con 0x7f2f280a9ad0 2024-01-20T11:18:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.608+0000 7f2f1e7fc640 1 -- 172.21.15.110:0/2244598878 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2f18013470 con 0x7f2f280a9ad0 2024-01-20T11:18:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.608+0000 7f2f1e7fc640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f0c06c8b0 0x7f2f0c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.609+0000 7f2f2de9d640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f0c06c8b0 0x7f2f0c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.609+0000 7f2f2de9d640 10 monclient: get_auth_request con 0x7f2f0c06c8b0 auth_method 0 2024-01-20T11:18:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.609+0000 7f2f1e7fc640 10 client.4741.objecter ms_handle_connect 0x7f2f0c06c8b0 2024-01-20T11:18:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.609+0000 7f2f2de9d640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f0c06c8b0 0x7f2f0c06ed70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2f24036000 tx=0x7f2f2403d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.610+0000 7f2f1e7fc640 1 -- 172.21.15.110:0/2244598878 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2f18058c40 con 0x7f2f280a9ad0 2024-01-20T11:18:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.610+0000 7f2f1e7fc640 10 client.4741.objecter ms_dispatch 0x7f2f28000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.610+0000 7f2f1e7fc640 3 client.4741.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.610+0000 7f2f1e7fc640 3 client.4741.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.610+0000 7f2f1e7fc640 20 client.4741.objecter dump_active .. 0 homeless 2024-01-20T11:18:01.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.843+0000 7f2f1e7fc640 1 -- 172.21.15.110:0/2244598878 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2f18054f80 con 0x7f2f280a9ad0 2024-01-20T11:18:01.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.843+0000 7f2f1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:01.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:01.843+0000 7f2f1e7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:02.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.088+0000 7f2f30101640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:02.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.088+0000 7f2f30101640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:02.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.088+0000 7f2f30101640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:02.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.088+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2f281449c0 con 0x7f2f280a9ad0 2024-01-20T11:18:02.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.093+0000 7f2f1e7fc640 1 -- 172.21.15.110:0/2244598878 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f2f18021520 con 0x7f2f280a9ad0 2024-01-20T11:18:02.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.093+0000 7f2f1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:02.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.093+0000 7f2f1e7fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:02.095 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:02.095 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":216,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:02.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.095+0000 7f2f30101640 10 librados: watch_flush enter 2024-01-20T11:18:02.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 10 librados: watch_flush exit 2024-01-20T11:18:02.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 20 client.4741.objecter shutdown clearing up homeless session... 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 10 client.4741.objecter successfully canceled tick 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f0c06c8b0 msgr2=0x7f2f0c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f0c06c8b0 0x7f2f0c06ed70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2f24036000 tx=0x7f2f2403d040 comp rx=0 tx=0).stop 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 10 monclient: shutdown 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f280a9ad0 msgr2=0x7f2f2813e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f280a9ad0 0x7f2f2813e780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800e8e0 tx=0x7f2f1800eda0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 shutdown_connections 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2f0c06c8b0 0x7f2f0c06ed70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f28143300 0x7f2f281da470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f280a9ad0 0x7f2f2813e780 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 --2- 172.21.15.110:0/2244598878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f280a96f0 0x7f2f28136770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 >> 172.21.15.110:0/2244598878 conn(0x7f2f280b7bc0 msgr2=0x7f2f280b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:02.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 shutdown_connections 2024-01-20T11:18:02.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 -- 172.21.15.110:0/2244598878 wait complete. 2024-01-20T11:18:02.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.096+0000 7f2f30101640 1 librados: shutdown 2024-01-20T11:18:02.107 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:02.107 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.205+0000 7ff4f030f640 10 monclient: get_monmap_and_config 2024-01-20T11:18:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.205+0000 7ff4f030f640 10 monclient: build_initial_monmap 2024-01-20T11:18:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.205+0000 7ff4f030f640 10 monclient: monmap: 2024-01-20T11:18:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:02.206 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:02.207124+0000 2024-01-20T11:18:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:02.207124+0000 2024-01-20T11:18:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:02.207 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:02.207 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:02.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 1 Processor -- start 2024-01-20T11:18:02.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 1 -- start start 2024-01-20T11:18:02.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 10 monclient: init 2024-01-20T11:18:02.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:02.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4e806a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8145c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e8072330 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4e813ed20 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e80b1010 con 0x7ff4e813ed20 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e806e480 con 0x7ff4e806a860 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e80afe00 con 0x7ff4e8072330 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.206+0000 7ff4f030f640 10 monclient(hunting): authenticate will time out at 1525.558060s 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.207+0000 7ff4ed8aa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.207+0000 7ff4ee0ab640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff4e806a860 0x7ff4e8145520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:56670/0 (socket says 172.21.15.110:56670) 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.207+0000 7ff4ee0ab640 1 -- 172.21.15.110:0/2710325107 learned_addr learned my addr 172.21.15.110:0/2710325107 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:02.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.207+0000 7ff4ed8aa640 10 monclient(hunting): get_auth_request con 0x7ff4e8072330 auth_method 0 2024-01-20T11:18:02.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.207+0000 7ff4ed8aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:02.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.207+0000 7ff4ed8aa640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:02.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.207+0000 7ff4ed8aa640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:02.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.208+0000 7ff4ed8aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:02.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.208+0000 7ff4ed8aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:02.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.208+0000 7ff4ed8aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:02.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 1 -- 172.21.15.110:0/2710325107 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308228047 0 0) 0x7ff4e806e480 con 0x7ff4e806a860 2024-01-20T11:18:02.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 10 monclient(hunting): _negotiate 2024-01-20T11:18:02.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:02.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 10 monclient(hunting): my global_id is 4744 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 1 -- 172.21.15.110:0/2710325107 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d4003520 con 0x7ff4e806a860 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 1 -- 172.21.15.110:0/2710325107 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165719572 0 0) 0x7ff4e80b1010 con 0x7ff4e813ed20 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 10 monclient(hunting): _negotiate 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 10 monclient(hunting): my global_id is 4674 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.209+0000 7ff4ed0a9640 1 -- 172.21.15.110:0/2710325107 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e806e480 con 0x7ff4e813ed20 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.210+0000 7ff4ed8aa640 10 monclient(hunting): handle_auth_done global_id 4553 payload 274 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.210+0000 7ff4ed8aa640 1 -- 172.21.15.110:0/2710325107 >> v1:172.21.15.134:6789/0 conn(0x7ff4e806a860 legacy=0x7ff4e8145520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.210+0000 7ff4ed8aa640 1 -- 172.21.15.110:0/2710325107 >> v1:172.21.15.110:6789/0 conn(0x7ff4e813ed20 legacy=0x7ff4e813f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.210+0000 7ff4ed8aa640 10 monclient: _finish_hunting 0 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.210+0000 7ff4ed8aa640 1 monclient: found mon.noname-b 2024-01-20T11:18:02.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.210+0000 7ff4ed8aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:02.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.210+0000 7ff4ed8aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:02.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.210+0000 7ff4ed8aa640 1 -- 172.21.15.110:0/2710325107 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e806e760 con 0x7ff4e8072330 2024-01-20T11:18:02.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.211+0000 7ff4ed8aa640 1 --2- 172.21.15.110:0/2710325107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8145c30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80097d0 tx=0x7ff4d802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4401694d614c239c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:02.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 1 -- 172.21.15.110:0/2710325107 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8031890 con 0x7ff4e8072330 2024-01-20T11:18:02.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:02.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:02.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 10 monclient: dump: 2024-01-20T11:18:02.217 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:02.217 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:02.217 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:02.217 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:02.217 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 10 monclient: _finish_auth 0 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 1 -- 172.21.15.110:0/2710325107 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d80319f0 con 0x7ff4e8072330 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4f030f640 5 monclient: authenticate success, global_id 4553 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4f030f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4f030f640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:02.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4f030f640 4 set_mon_vals no callback set 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 1 -- 172.21.15.110:0/2710325107 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8031d80 con 0x7ff4e8072330 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.215+0000 7ff4ed0a9640 10 monclient: dump: 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:02.219 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.216+0000 7ff4f030f640 10 monclient: shutdown 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.216+0000 7ff4f030f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.216+0000 7ff4f030f640 1 -- 172.21.15.110:0/2710325107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 msgr2=0x7ff4e8145c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.216+0000 7ff4f030f640 1 --2- 172.21.15.110:0/2710325107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8145c30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80097d0 tx=0x7ff4d802b930 comp rx=0 tx=0).stop 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.216+0000 7ff4f030f640 1 -- 172.21.15.110:0/2710325107 shutdown_connections 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.216+0000 7ff4f030f640 1 --2- 172.21.15.110:0/2710325107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8145c30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.216+0000 7ff4f030f640 1 -- 172.21.15.110:0/2710325107 >> 172.21.15.110:0/2710325107 conn(0x7ff4e80b47d0 msgr2=0x7ff4e80b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.216+0000 7ff4f030f640 1 -- 172.21.15.110:0/2710325107 shutdown_connections 2024-01-20T11:18:02.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.216+0000 7ff4f030f640 1 -- 172.21.15.110:0/2710325107 wait complete. 2024-01-20T11:18:02.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 10 monclient: build_initial_monmap 2024-01-20T11:18:02.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 10 monclient: monmap: 2024-01-20T11:18:02.223 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 1 librados: starting msgr at 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 1 librados: starting objecter 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 1 Processor -- start 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 1 -- start start 2024-01-20T11:18:02.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 1 librados: setting wanted keys 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 1 librados: calling monclient init 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 10 monclient: init 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.217+0000 7ff4f030f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e806a860 0x7ff4e8140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 10 monclient(hunting): picked mon.noname-a con 0x7ff4e806a860 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8144170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 10 monclient(hunting): picked mon.noname-c con 0x7ff4e8072330 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e813ed20 0x7ff4e8135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e813ed20 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e8142960 con 0x7ff4e806a860 2024-01-20T11:18:02.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:02.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e8141ca0 con 0x7ff4e813ed20 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e8142b90 con 0x7ff4e8072330 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4f030f640 10 monclient(hunting): authenticate will time out at 1525.569223s 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ed8aa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee8ac640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e813ed20 0x7ff4e8135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee0ab640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e806a860 0x7ff4e8140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ed8aa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8144170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:44006/0 (socket says 172.21.15.110:44006) 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ed8aa640 1 -- 172.21.15.110:0/2993469961 learned_addr learned my addr 172.21.15.110:0/2993469961 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ed8aa640 10 monclient(hunting): get_auth_request con 0x7ff4e8072330 auth_method 0 2024-01-20T11:18:02.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ed8aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ed8aa640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ed8aa640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee0ab640 10 monclient(hunting): get_auth_request con 0x7ff4e806a860 auth_method 0 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee0ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee0ab640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee0ab640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee8ac640 10 monclient(hunting): get_auth_request con 0x7ff4e813ed20 auth_method 0 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee8ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee8ac640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.218+0000 7ff4ee8ac640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.219+0000 7ff4ee8ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.219+0000 7ff4ee8ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:02.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.219+0000 7ff4ee8ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.219+0000 7ff4ee0ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.219+0000 7ff4ee0ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.219+0000 7ff4ee0ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.219+0000 7ff4ed8aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.219+0000 7ff4ed8aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.219+0000 7ff4ed8aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.221+0000 7ff4ee8ac640 10 monclient(hunting): handle_auth_done global_id 4747 payload 824 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.221+0000 7ff4ee8ac640 1 -- 172.21.15.110:0/2993469961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 msgr2=0x7ff4e8144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.221+0000 7ff4ee8ac640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.221+0000 7ff4ee8ac640 1 -- 172.21.15.110:0/2993469961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e806a860 msgr2=0x7ff4e8140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:02.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.221+0000 7ff4ee8ac640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e806a860 0x7ff4e8140750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.221+0000 7ff4ee8ac640 10 monclient: _finish_hunting 0 2024-01-20T11:18:02.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.221+0000 7ff4ee8ac640 1 monclient: found mon.noname-b 2024-01-20T11:18:02.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.222+0000 7ff4ee8ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:02.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.222+0000 7ff4ee8ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:02.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.222+0000 7ff4ee8ac640 1 -- 172.21.15.110:0/2993469961 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8009400 con 0x7ff4e813ed20 2024-01-20T11:18:02.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.224+0000 7ff4ee8ac640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e813ed20 0x7ff4e8135ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c730 tx=0x7ff4e000cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:02.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.224+0000 7ff4deffd640 10 client.?.objecter ms_handle_connect 0x7ff4e813ed20 2024-01-20T11:18:02.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.224+0000 7ff4deffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:02.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 1 -- 172.21.15.110:0/2993469961 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0003b70 con 0x7ff4e813ed20 2024-01-20T11:18:02.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:02.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 10 monclient: dump: 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:02.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 1 -- 172.21.15.110:0/2993469961 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e0003cf0 con 0x7ff4e813ed20 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 1 -- 172.21.15.110:0/2993469961 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0003ea0 con 0x7ff4e813ed20 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4deffd640 10 monclient: dump: 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:02.233 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4f030f640 5 monclient: authenticate success, global_id 4747 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4f030f640 10 monclient: _renew_subs 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4f030f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e81446b0 con 0x7ff4e813ed20 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4ec8a8640 4 set_mon_vals no callback set 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4f030f640 10 client.4747.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4f030f640 10 monclient: _renew_subs 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4f030f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e8142260 con 0x7ff4e813ed20 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.227+0000 7ff4f030f640 1 librados: init done 2024-01-20T11:18:02.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.228+0000 7ff4f030f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:02.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.228+0000 7ff4f030f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:02.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.228+0000 7ff4f030f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:02.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.228+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e8075490 con 0x7ff4e813ed20 2024-01-20T11:18:02.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.466+0000 7ff4deffd640 1 -- 172.21.15.110:0/2993469961 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff4e0005520 con 0x7ff4e813ed20 2024-01-20T11:18:02.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.466+0000 7ff4deffd640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c860 0x7ff4cc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:02.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.466+0000 7ff4ee0ab640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c860 0x7ff4cc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:02.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.466+0000 7ff4ee0ab640 10 monclient: get_auth_request con 0x7ff4cc06c860 auth_method 0 2024-01-20T11:18:02.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.467+0000 7ff4ee0ab640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c860 0x7ff4cc06ed20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4000f60 tx=0x7ff4e4007d70 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:02.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.467+0000 7ff4deffd640 10 client.4747.objecter ms_handle_connect 0x7ff4cc06c860 2024-01-20T11:18:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.469+0000 7ff4deffd640 1 -- 172.21.15.110:0/2993469961 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff4e0058bb0 con 0x7ff4e813ed20 2024-01-20T11:18:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.469+0000 7ff4deffd640 10 client.4747.objecter ms_dispatch 0x7ff4e8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.469+0000 7ff4deffd640 3 client.4747.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.469+0000 7ff4deffd640 3 client.4747.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.469+0000 7ff4deffd640 20 client.4747.objecter dump_active .. 0 homeless 2024-01-20T11:18:02.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.478+0000 7ff4deffd640 1 -- 172.21.15.110:0/2993469961 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff4e0054ef0 con 0x7ff4e813ed20 2024-01-20T11:18:02.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.478+0000 7ff4deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:02.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.478+0000 7ff4deffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:02.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.645+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4e81362a0 con 0x7ff4cc06c860 2024-01-20T11:18:02.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.648+0000 7ff4deffd640 1 -- 172.21.15.110:0/2993469961 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64211 (secure 0 0 0) 0x7ff4e40014f0 con 0x7ff4cc06c860 2024-01-20T11:18:02.649 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:02.649 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:02.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.649+0000 7ff4f030f640 10 librados: watch_flush enter 2024-01-20T11:18:02.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.649+0000 7ff4f030f640 10 librados: watch_flush exit 2024-01-20T11:18:02.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 20 client.4747.objecter shutdown clearing up homeless session... 2024-01-20T11:18:02.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 10 client.4747.objecter successfully canceled tick 2024-01-20T11:18:02.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c860 msgr2=0x7ff4cc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c860 0x7ff4cc06ed20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4000f60 tx=0x7ff4e4007d70 comp rx=0 tx=0).stop 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 10 monclient: shutdown 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e813ed20 msgr2=0x7ff4e8135ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e813ed20 0x7ff4e8135ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c730 tx=0x7ff4e000cbf0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 shutdown_connections 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c860 0x7ff4cc06ed20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e813ed20 0x7ff4e8135ca0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e8072330 0x7ff4e8144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 --2- 172.21.15.110:0/2993469961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e806a860 0x7ff4e8140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:02.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 >> 172.21.15.110:0/2993469961 conn(0x7ff4e80b47d0 msgr2=0x7ff4e813f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:02.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 shutdown_connections 2024-01-20T11:18:02.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 -- 172.21.15.110:0/2993469961 wait complete. 2024-01-20T11:18:02.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:02.650+0000 7ff4f030f640 1 librados: shutdown 2024-01-20T11:18:02.661 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-01-20T11:18:00.742320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001957"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":14.779},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.5800000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.32},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.258}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.858000000000001},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.531000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.6760000000000002},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":13.847}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.503},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":11.627000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.6720000000000006},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":16.774999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":3.9820000000000002},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7279999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":13.026},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.5800000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":6.9370000000000003},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.935}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.973000000000001},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":8.3689999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.284000000000001},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":15.173999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.568},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.132}]}]},{"osd":3,"up_from":21,"seq":90194313254,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.3520000000000003},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.625}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.1899999999999995},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.699}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":8.8420000000000005},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.819000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.972},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.132}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":13.090999999999999},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.481999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.861000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.307}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.0750000000000002},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":6.673}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.33},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.8979999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.555999999999999},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.5139999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.111000000000001},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.034000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.92},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.771000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.2870000000000008},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.887}]}]},{"osd":1,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.7599999999999998},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":11.651}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.510999999999999},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.3869999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":10.106},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.4190000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":14.763},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":14.038}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.1539999999999999},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":13.313000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.8309999999999995},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.105}]}]},{"osd":0,"up_from":17,"seq":73014444069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":8.1210000000000004},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":6.0179999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.5570000000000004},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.59}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":13.191000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.5609999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.025},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7490000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.130000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.362}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.991},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":9.3130000000000006}]}]},{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":6.2779999999999996},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.2450000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.262},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":8.7319999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.8630000000000004},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.609999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":11.878},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.933}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.907999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.238}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.837999999999999},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.994999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:05.665 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.768+0000 7fee1cf5a640 10 monclient: get_monmap_and_config 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.768+0000 7fee1cf5a640 10 monclient: build_initial_monmap 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.768+0000 7fee1cf5a640 10 monclient: monmap: 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:05.769889+0000 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:05.769889+0000 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:05.770 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:05.770 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:05.770 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:05.770 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:05.770 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.768+0000 7fee1cf5a640 1 Processor -- start 2024-01-20T11:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.768+0000 7fee1cf5a640 1 -- start start 2024-01-20T11:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.768+0000 7fee1cf5a640 10 monclient: init 2024-01-20T11:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-01-20T11:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fee180a96f0 0x7fee1813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 10 monclient(hunting): picked mon.noname-c con 0x7fee180a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee1813fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a9ad0 0x7fee18142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 10 monclient(hunting): picked mon.noname-b con 0x7fee180a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee180b25c0 con 0x7fee1813fa10 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee180aaea0 con 0x7fee180a96f0 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee180ab0a0 con 0x7fee180a9ad0 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.769+0000 7fee1cf5a640 10 monclient(hunting): authenticate will time out at 1529.120816s 2024-01-20T11:18:05.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.770+0000 7fee16d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a9ad0 0x7fee18142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.770+0000 7fee1659c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fee180a96f0 0x7fee1813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.770+0000 7fee1659c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fee180a96f0 0x7fee1813f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:56282/0 (socket says 172.21.15.110:56282) 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.770+0000 7fee1659c640 1 -- 172.21.15.110:0/3726527161 learned_addr learned my addr 172.21.15.110:0/3726527161 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.770+0000 7fee1659c640 10 monclient(hunting): get_auth_request con 0x7fee180a96f0 auth_method 0 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.770+0000 7fee1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.770+0000 7fee1659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.770+0000 7fee1659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.771+0000 7fee1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.771+0000 7fee1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.771+0000 7fee1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.772+0000 7fee16d9d640 10 monclient(hunting): get_auth_request con 0x7fee180a9ad0 auth_method 0 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.772+0000 7fee16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.772+0000 7fee16d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.772+0000 7fee16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.772+0000 7fee16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.772+0000 7fee16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.772+0000 7fee16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.774+0000 7fee1659c640 10 monclient(hunting): handle_auth_done global_id 4750 payload 274 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.774+0000 7fee1659c640 1 -- 172.21.15.110:0/3726527161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a9ad0 msgr2=0x7fee18142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.774+0000 7fee1659c640 1 --2- 172.21.15.110:0/3726527161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a9ad0 0x7fee18142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.774+0000 7fee1659c640 1 -- 172.21.15.110:0/3726527161 >> v1:172.21.15.134:6789/0 conn(0x7fee1813fa10 legacy=0x7fee181423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.774+0000 7fee1659c640 10 monclient: _finish_hunting 0 2024-01-20T11:18:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.774+0000 7fee1659c640 1 monclient: found mon.noname-c 2024-01-20T11:18:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.774+0000 7fee1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.774+0000 7fee1659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.774+0000 7fee1659c640 1 -- 172.21.15.110:0/3726527161 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee180b3fb0 con 0x7fee180a96f0 2024-01-20T11:18:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.776+0000 7fee1659c640 1 --2- 172.21.15.110:0/3726527161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee180a96f0 0x7fee1813f4d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fee10009870 tx=0x7fee1002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=355ddda1f58cb0de server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:05.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 1 -- 172.21.15.110:0/3726527161 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10031640 con 0x7fee180a96f0 2024-01-20T11:18:05.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:05.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:05.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 10 monclient: dump: 2024-01-20T11:18:05.784 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 10 monclient: _finish_auth 0 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 10 monclient: _check_auth_tickets 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 1 -- 172.21.15.110:0/3726527161 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee100317a0 con 0x7fee180a96f0 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1cf5a640 5 monclient: authenticate success, global_id 4750 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1cf5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1cf5a640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 1 -- 172.21.15.110:0/3726527161 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10031b30 con 0x7fee180a96f0 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1cf5a640 4 set_mon_vals no callback set 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.783+0000 7fee1559a640 10 monclient: dump: 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:05.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 10 monclient: shutdown 2024-01-20T11:18:05.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:05.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 1 -- 172.21.15.110:0/3726527161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee180a96f0 msgr2=0x7fee1813f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 1 --2- 172.21.15.110:0/3726527161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee180a96f0 0x7fee1813f4d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fee10009870 tx=0x7fee1002b8f0 comp rx=0 tx=0).stop 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 1 -- 172.21.15.110:0/3726527161 shutdown_connections 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 1 --2- 172.21.15.110:0/3726527161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a9ad0 0x7fee18142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 1 --2- 172.21.15.110:0/3726527161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee180a96f0 0x7fee1813f4d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 1 -- 172.21.15.110:0/3726527161 >> 172.21.15.110:0/3726527161 conn(0x7fee180b7bc0 msgr2=0x7fee180b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 1 -- 172.21.15.110:0/3726527161 shutdown_connections 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.784+0000 7fee1cf5a640 1 -- 172.21.15.110:0/3726527161 wait complete. 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.785+0000 7fee1cf5a640 10 monclient: build_initial_monmap 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.785+0000 7fee1cf5a640 10 monclient: monmap: 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.785+0000 7fee1cf5a640 1 librados: starting msgr at 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.785+0000 7fee1cf5a640 1 librados: starting objecter 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.785+0000 7fee1cf5a640 1 Processor -- start 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.785+0000 7fee1cf5a640 1 -- start start 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.785+0000 7fee1cf5a640 1 librados: setting wanted keys 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.785+0000 7fee1cf5a640 1 librados: calling monclient init 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.785+0000 7fee1cf5a640 10 monclient: init 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a96f0 0x7fee18134d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient(hunting): picked mon.noname-c con 0x7fee180a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee180a9ad0 0x7fee1812ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient(hunting): picked mon.noname-a con 0x7fee180a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee181373b0 0x7fee1812e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient(hunting): picked mon.noname-b con 0x7fee181373b0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee18136060 con 0x7fee180a9ad0 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee18134530 con 0x7fee181373b0 2024-01-20T11:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee18137000 con 0x7fee180a96f0 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1cf5a640 10 monclient(hunting): authenticate will time out at 1529.137589s 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee16d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee181373b0 0x7fee1812e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee1659c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a96f0 0x7fee18134d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.786+0000 7fee15d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee180a9ad0 0x7fee1812ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee16d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee181373b0 0x7fee1812e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:56298/0 (socket says 172.21.15.110:56298) 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee16d9d640 1 -- 172.21.15.110:0/2289780871 learned_addr learned my addr 172.21.15.110:0/2289780871 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee16d9d640 10 monclient(hunting): get_auth_request con 0x7fee181373b0 auth_method 0 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee16d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee1659c640 10 monclient(hunting): get_auth_request con 0x7fee180a96f0 auth_method 0 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee1659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee1659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee15d9b640 10 monclient(hunting): get_auth_request con 0x7fee180a9ad0 auth_method 0 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee15d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.787+0000 7fee1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.788+0000 7fee1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.788+0000 7fee15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.788+0000 7fee15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.788+0000 7fee15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 10 monclient(hunting): handle_auth_done global_id 4756 payload 824 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 1 -- 172.21.15.110:0/2289780871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a96f0 msgr2=0x7fee18134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a96f0 0x7fee18134d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 1 -- 172.21.15.110:0/2289780871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee180a9ad0 msgr2=0x7fee1812ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee180a9ad0 0x7fee1812ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 1 monclient: found mon.noname-b 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.790+0000 7fee16d9d640 1 -- 172.21.15.110:0/2289780871 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee100094d0 con 0x7fee181373b0 2024-01-20T11:18:05.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.792+0000 7fee16d9d640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee181373b0 0x7fee1812e310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fee0800efc0 tx=0x7fee0800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.792+0000 7fee06ffd640 10 client.?.objecter ms_handle_connect 0x7fee181373b0 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.792+0000 7fee06ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 1 -- 172.21.15.110:0/2289780871 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08014480 con 0x7fee181373b0 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 10 monclient: dump: 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 1 -- 172.21.15.110:0/2289780871 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0800f040 con 0x7fee181373b0 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 1 -- 172.21.15.110:0/2289780871 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08014820 con 0x7fee181373b0 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee1cf5a640 5 monclient: authenticate success, global_id 4756 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee1cf5a640 10 monclient: _renew_subs 2024-01-20T11:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee1cf5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee18135490 con 0x7fee181373b0 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee14d99640 4 set_mon_vals no callback set 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee06ffd640 10 monclient: dump: 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee1cf5a640 10 client.4756.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee1cf5a640 10 monclient: _renew_subs 2024-01-20T11:18:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee1cf5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee18135d70 con 0x7fee181373b0 2024-01-20T11:18:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.796+0000 7fee1cf5a640 1 librados: init done 2024-01-20T11:18:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.797+0000 7fee1cf5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.797+0000 7fee1cf5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.797+0000 7fee1cf5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:05.797+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee18075490 con 0x7fee181373b0 2024-01-20T11:18:06.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.054+0000 7fee06ffd640 1 -- 172.21.15.110:0/2289780871 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fee08014980 con 0x7fee181373b0 2024-01-20T11:18:06.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.054+0000 7fee06ffd640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fedf806c8b0 0x7fedf806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:06.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.055+0000 7fee1659c640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fedf806c8b0 0x7fedf806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:06.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.055+0000 7fee1659c640 10 monclient: get_auth_request con 0x7fedf806c8b0 auth_method 0 2024-01-20T11:18:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.055+0000 7fee1659c640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fedf806c8b0 0x7fedf806ed70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fee10036000 tx=0x7fee1003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.055+0000 7fee06ffd640 10 client.4756.objecter ms_handle_connect 0x7fedf806c8b0 2024-01-20T11:18:06.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.058+0000 7fee06ffd640 1 -- 172.21.15.110:0/2289780871 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fee080595b0 con 0x7fee181373b0 2024-01-20T11:18:06.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.058+0000 7fee06ffd640 10 client.4756.objecter ms_dispatch 0x7fee18000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:06.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.058+0000 7fee06ffd640 3 client.4756.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:06.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.058+0000 7fee06ffd640 3 client.4756.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:06.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.058+0000 7fee06ffd640 20 client.4756.objecter dump_active .. 0 homeless 2024-01-20T11:18:06.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.068+0000 7fee06ffd640 1 -- 172.21.15.110:0/2289780871 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fee080558f0 con 0x7fee181373b0 2024-01-20T11:18:06.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.068+0000 7fee06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:06.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.068+0000 7fee06ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:06.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.236+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee18134670 con 0x7fedf806c8b0 2024-01-20T11:18:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.239+0000 7fee06ffd640 1 -- 172.21.15.110:0/2289780871 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64212 (secure 0 0 0) 0x7fee18134670 con 0x7fedf806c8b0 2024-01-20T11:18:06.240 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:06.241 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:06.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.240+0000 7fee1cf5a640 10 librados: watch_flush enter 2024-01-20T11:18:06.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.240+0000 7fee1cf5a640 10 librados: watch_flush exit 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.240+0000 7fee1cf5a640 20 client.4756.objecter shutdown clearing up homeless session... 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.240+0000 7fee1cf5a640 10 client.4756.objecter successfully canceled tick 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fedf806c8b0 msgr2=0x7fedf806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fedf806c8b0 0x7fedf806ed70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fee10036000 tx=0x7fee1003c040 comp rx=0 tx=0).stop 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 10 monclient: shutdown 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee181373b0 msgr2=0x7fee1812e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee181373b0 0x7fee1812e310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fee0800efc0 tx=0x7fee0800c490 comp rx=0 tx=0).stop 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 shutdown_connections 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fedf806c8b0 0x7fedf806ed70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee181373b0 0x7fee1812e310 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee180a9ad0 0x7fee1812ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 --2- 172.21.15.110:0/2289780871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee180a96f0 0x7fee18134d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 >> 172.21.15.110:0/2289780871 conn(0x7fee180b7bc0 msgr2=0x7fee18130ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 shutdown_connections 2024-01-20T11:18:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 -- 172.21.15.110:0/2289780871 wait complete. 2024-01-20T11:18:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.241+0000 7fee1cf5a640 1 librados: shutdown 2024-01-20T11:18:06.252 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-01-20T11:18:04.742972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001989"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.7039999999999997},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.5979999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.960000000000001},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.33}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.958},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.9920000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.7729999999999997},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":16.594000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.856999999999999},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":14.144}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.595000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.43}]}]},{"osd":7,"up_from":21,"seq":90194313254,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.8890000000000002},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.052}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.8499999999999996},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.673}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.612},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.6219999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.300000000000001},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.196}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.238},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":9.3469999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.5},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.996}]}]},{"osd":3,"up_from":21,"seq":90194313255,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.3119999999999994},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.984}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":11.579000000000001},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.0739999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":16.199999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.871}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.231999999999999},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":18.117000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":9.7400000000000002},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":13.446999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":14.037000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.199999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.012},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":5.9909999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.4610000000000003},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.8869999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.737},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.0709999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.582000000000001},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.5709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":10.304},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.648}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.946999999999999},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.340999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":13.178000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":7.3869999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.622},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":14.939}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.116},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.7270000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.8460000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.731}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.0649999999999995},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":10.429}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.173},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.032}]}]},{"osd":0,"up_from":17,"seq":73014444070,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":8.7539999999999996},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.4749999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.2539999999999996},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.4179999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":11.603999999999999},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.131}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.193},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":5.9480000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.089},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.226000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.044},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":8.0589999999999993}]}]},{"osd":6,"up_from":15,"seq":64424509478,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":7.6310000000000002},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.1749999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":10.678000000000001},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":12.268000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.9420000000000002},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.5589999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.731999999999999},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":16.033000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.117000000000001},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.972}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.221},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.315}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:06.254 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.355+0000 7f17d0846640 10 monclient: get_monmap_and_config 2024-01-20T11:18:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.355+0000 7f17d0846640 10 monclient: build_initial_monmap 2024-01-20T11:18:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.355+0000 7f17d0846640 10 monclient: monmap: 2024-01-20T11:18:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:06.356841+0000 2024-01-20T11:18:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:06.356841+0000 2024-01-20T11:18:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:06.357 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.355+0000 7f17d0846640 1 Processor -- start 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 1 -- start start 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient: init 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient(hunting): picked mon.noname-b con 0x7f17cc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f17cc13fa10 0x7f17cc1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient(hunting): picked mon.noname-c con 0x7f17cc13fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17cc0a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17cc0b25c0 con 0x7f17cc0a9ad0 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17cc0aaea0 con 0x7f17cc13fa10 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc0ab0a0 con 0x7f17cc0a96f0 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.356+0000 7f17d0846640 10 monclient(hunting): authenticate will time out at 1529.707801s 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.357+0000 7f17ca59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.357+0000 7f17ca59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:44028/0 (socket says 172.21.15.110:44028) 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.357+0000 7f17ca59c640 1 -- 172.21.15.110:0/3349423505 learned_addr learned my addr 172.21.15.110:0/3349423505 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.357+0000 7f17ca59c640 10 monclient(hunting): get_auth_request con 0x7f17cc0a96f0 auth_method 0 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.357+0000 7f17ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.357+0000 7f17ca59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.357+0000 7f17ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.358+0000 7f17ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:06.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.358+0000 7f17ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:06.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.358+0000 7f17ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:06.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.359+0000 7f17ca59c640 10 monclient(hunting): handle_auth_done global_id 4565 payload 274 2024-01-20T11:18:06.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.359+0000 7f17ca59c640 1 -- 172.21.15.110:0/3349423505 >> v2:172.21.15.134:3300/0 conn(0x7f17cc13fa10 msgr2=0x7f17cc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:06.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.359+0000 7f17ca59c640 1 --2- 172.21.15.110:0/3349423505 >> v2:172.21.15.134:3300/0 conn(0x7f17cc13fa10 0x7f17cc1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.359+0000 7f17ca59c640 1 -- 172.21.15.110:0/3349423505 >> v1:172.21.15.134:6789/0 conn(0x7f17cc0a9ad0 legacy=0x7f17cc142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:06.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.359+0000 7f17ca59c640 10 monclient: _finish_hunting 0 2024-01-20T11:18:06.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.359+0000 7f17ca59c640 1 monclient: found mon.noname-b 2024-01-20T11:18:06.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.359+0000 7f17ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:06.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.359+0000 7f17ca59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:06.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.359+0000 7f17ca59c640 1 -- 172.21.15.110:0/3349423505 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc0b3fb0 con 0x7f17cc0a96f0 2024-01-20T11:18:06.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.362+0000 7f17ca59c640 1 --2- 172.21.15.110:0/3349423505 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc13f4d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009870 tx=0x7f17bc02b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b55201016052af7 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 1 -- 172.21.15.110:0/3349423505 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0317e0 con 0x7f17cc0a96f0 2024-01-20T11:18:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 10 monclient: dump: 2024-01-20T11:18:06.416 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 10 monclient: _finish_auth 0 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 1 -- 172.21.15.110:0/3349423505 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc031940 con 0x7f17cc0a96f0 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:06.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 1 -- 172.21.15.110:0/3349423505 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc031cd0 con 0x7f17cc0a96f0 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17d0846640 5 monclient: authenticate success, global_id 4565 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17d0846640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17d0846640 4 set_mon_vals no callback set 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.415+0000 7f17c9d9b640 10 monclient: dump: 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:06.418 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:06.419 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 10 monclient: shutdown 2024-01-20T11:18:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 1 -- 172.21.15.110:0/3349423505 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 msgr2=0x7f17cc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 1 --2- 172.21.15.110:0/3349423505 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc13f4d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009870 tx=0x7f17bc02b880 comp rx=0 tx=0).stop 2024-01-20T11:18:06.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 1 -- 172.21.15.110:0/3349423505 shutdown_connections 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 1 --2- 172.21.15.110:0/3349423505 >> v2:172.21.15.134:3300/0 conn(0x7f17cc13fa10 0x7f17cc1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 1 --2- 172.21.15.110:0/3349423505 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc13f4d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 1 -- 172.21.15.110:0/3349423505 >> 172.21.15.110:0/3349423505 conn(0x7f17cc0b7bc0 msgr2=0x7f17cc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 1 -- 172.21.15.110:0/3349423505 shutdown_connections 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.416+0000 7f17d0846640 1 -- 172.21.15.110:0/3349423505 wait complete. 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.417+0000 7f17d0846640 10 monclient: build_initial_monmap 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.417+0000 7f17d0846640 10 monclient: monmap: 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:06.422 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.417+0000 7f17d0846640 1 librados: starting msgr at 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.417+0000 7f17d0846640 1 librados: starting objecter 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.417+0000 7f17d0846640 1 Processor -- start 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.417+0000 7f17d0846640 1 -- start start 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.417+0000 7f17d0846640 1 librados: setting wanted keys 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.417+0000 7f17d0846640 1 librados: calling monclient init 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.417+0000 7f17d0846640 10 monclient: init 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient(hunting): picked mon.noname-c con 0x7f17cc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17cc0a9ad0 0x7f17cc145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:06.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient(hunting): picked mon.noname-b con 0x7f17cc0a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17cc145500 0x7f17cc138180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient(hunting): picked mon.noname-a con 0x7f17cc145500 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc1d7180 con 0x7f17cc145500 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc1d7460 con 0x7f17cc0a9ad0 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc1d78f0 con 0x7f17cc0a96f0 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.418+0000 7f17d0846640 10 monclient(hunting): authenticate will time out at 1529.769797s 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17ca59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17cad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17cc145500 0x7f17cc138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17ca59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc143300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:44030/0 (socket says 172.21.15.110:44030) 2024-01-20T11:18:06.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17ca59c640 1 -- 172.21.15.110:0/1375938298 learned_addr learned my addr 172.21.15.110:0/1375938298 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17ca59c640 10 monclient(hunting): get_auth_request con 0x7f17cc0a96f0 auth_method 0 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17ca59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17cad9d640 10 monclient(hunting): get_auth_request con 0x7f17cc145500 auth_method 0 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17cad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.419+0000 7f17ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.420+0000 7f17ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.420+0000 7f17ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.420+0000 7f17cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.420+0000 7f17cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:06.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.420+0000 7f17cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.421+0000 7f17c3fff640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17cc0a9ad0 0x7f17cc145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17c3fff640 10 monclient(hunting): get_auth_request con 0x7f17cc0a9ad0 auth_method 0 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17c3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 10 monclient(hunting): handle_auth_done global_id 4683 payload 824 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 1 -- 172.21.15.110:0/1375938298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 msgr2=0x7f17cc143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 1 -- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17cc0a9ad0 msgr2=0x7f17cc145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17cc0a9ad0 0x7f17cc145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:06.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 1 monclient: found mon.noname-a 2024-01-20T11:18:06.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:06.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:06.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17cad9d640 1 -- 172.21.15.110:0/1375938298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc0094d0 con 0x7f17cc145500 2024-01-20T11:18:06.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.422+0000 7f17ca59c640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc143300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:18:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.439+0000 7f17cad9d640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17cc145500 0x7f17cc138180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f17c400c7b0 tx=0x7f17c400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.439+0000 7f17c2ffd640 10 client.?.objecter ms_handle_connect 0x7f17cc145500 2024-01-20T11:18:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.439+0000 7f17c2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 1 -- 172.21.15.110:0/1375938298 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4003b70 con 0x7f17cc145500 2024-01-20T11:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 10 monclient: dump: 2024-01-20T11:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 1 -- 172.21.15.110:0/1375938298 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c4003cf0 con 0x7f17cc145500 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17d0846640 5 monclient: authenticate success, global_id 4683 2024-01-20T11:18:06.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17d0846640 10 monclient: _renew_subs 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17d0846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17cc1d7b70 con 0x7f17cc145500 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 1 -- 172.21.15.110:0/1375938298 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c40052b0 con 0x7f17cc145500 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c2ffd640 10 monclient: dump: 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:06.447 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:06.448 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17c959a640 4 set_mon_vals no callback set 2024-01-20T11:18:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17d0846640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17d0846640 10 monclient: _renew_subs 2024-01-20T11:18:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17d0846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17cc1d8080 con 0x7f17cc145500 2024-01-20T11:18:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.444+0000 7f17d0846640 1 librados: init done 2024-01-20T11:18:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.445+0000 7f17d0846640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:06.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.445+0000 7f17d0846640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:06.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.445+0000 7f17d0846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:06.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.445+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17cc1d7610 con 0x7f17cc145500 2024-01-20T11:18:06.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.463+0000 7f17c2ffd640 1 -- 172.21.15.110:0/1375938298 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f17c4005410 con 0x7f17cc145500 2024-01-20T11:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.464+0000 7f17c2ffd640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f17ac06c8b0 0x7f17ac06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.464+0000 7f17ca59c640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f17ac06c8b0 0x7f17ac06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.464+0000 7f17ca59c640 10 monclient: get_auth_request con 0x7f17ac06c8b0 auth_method 0 2024-01-20T11:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.464+0000 7f17c2ffd640 10 client.4683.objecter ms_handle_connect 0x7f17ac06c8b0 2024-01-20T11:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.464+0000 7f17ca59c640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f17ac06c8b0 0x7f17ac06ed70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009870 tx=0x7f17bc03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.466+0000 7f17c2ffd640 1 -- 172.21.15.110:0/1375938298 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f17c4057b50 con 0x7f17cc145500 2024-01-20T11:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.466+0000 7f17c2ffd640 10 client.4683.objecter ms_dispatch 0x7f17cc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.466+0000 7f17c2ffd640 3 client.4683.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.466+0000 7f17c2ffd640 3 client.4683.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.466+0000 7f17c2ffd640 20 client.4683.objecter dump_active .. 0 homeless 2024-01-20T11:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.700+0000 7f17c2ffd640 1 -- 172.21.15.110:0/1375938298 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f17c4053e90 con 0x7f17cc145500 2024-01-20T11:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.700+0000 7f17c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.700+0000 7f17c2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:06.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.870+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17cc1d8080 con 0x7f17ac06c8b0 2024-01-20T11:18:06.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.873+0000 7f17c2ffd640 1 -- 172.21.15.110:0/1375938298 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64188 (secure 0 0 0) 0x7f17cc1d8080 con 0x7f17ac06c8b0 2024-01-20T11:18:06.874 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 10 librados: watch_flush enter 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 10 librados: watch_flush exit 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 20 client.4683.objecter shutdown clearing up homeless session... 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 10 client.4683.objecter successfully canceled tick 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f17ac06c8b0 msgr2=0x7f17ac06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f17ac06c8b0 0x7f17ac06ed70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009870 tx=0x7f17bc03d040 comp rx=0 tx=0).stop 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 10 monclient: shutdown 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17cc145500 msgr2=0x7f17cc138180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17cc145500 0x7f17cc138180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f17c400c7b0 tx=0x7f17c400cc70 comp rx=0 tx=0).stop 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 shutdown_connections 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f17ac06c8b0 0x7f17ac06ed70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17cc145500 0x7f17cc138180 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17cc0a9ad0 0x7f17cc145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 --2- 172.21.15.110:0/1375938298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17cc0a96f0 0x7f17cc143300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 >> 172.21.15.110:0/1375938298 conn(0x7f17cc0b7bc0 msgr2=0x7f17cc13c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 shutdown_connections 2024-01-20T11:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 -- 172.21.15.110:0/1375938298 wait complete. 2024-01-20T11:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.875+0000 7f17d0846640 1 librados: shutdown 2024-01-20T11:18:06.886 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-01-20T11:18:06.743295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001956"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.7039999999999997},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.5979999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.960000000000001},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.33}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.958},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.9920000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.7729999999999997},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":16.594000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.856999999999999},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":14.144}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.595000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.43}]}]},{"osd":7,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2270000000000003},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4350000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7289999999999992},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.1639999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.263999999999999},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.599}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.025},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.891}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.583},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.301}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.863},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.554}]}]},{"osd":3,"up_from":21,"seq":90194313256,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.7780000000000005},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":4.8810000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":15.708},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.811999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":7.4119999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.430999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.778},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.138999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":8.0920000000000005},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.522}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":16.5},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.763}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.012},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":5.9909999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.4610000000000003},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.8869999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.737},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.0709999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.582000000000001},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.5709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":10.304},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.648}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.946999999999999},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.340999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":13.178000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":7.3869999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.622},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":14.939}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.116},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.7270000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.8460000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.731}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.0649999999999995},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":10.429}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.173},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.032}]}]},{"osd":0,"up_from":17,"seq":73014444070,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":8.7539999999999996},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.4749999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.2539999999999996},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.4179999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":11.603999999999999},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.131}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.193},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":5.9480000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.089},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.226000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.044},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":8.0589999999999993}]}]},{"osd":6,"up_from":15,"seq":64424509479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.7140000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.4180000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.702999999999999},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":15.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6749999999999998},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.087999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.73},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.0679999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.9329999999999998},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.266999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.678000000000001},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.503}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:06.888 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:06.888 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:06.888 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:06.888 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:06.888 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:06.888 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:06.889 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:06.889 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:06.889 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.985+0000 7efc889f5640 10 monclient: get_monmap_and_config 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.985+0000 7efc889f5640 10 monclient: build_initial_monmap 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.985+0000 7efc889f5640 10 monclient: monmap: 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:06.986859+0000 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:06.986859+0000 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:06.987 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.985+0000 7efc889f5640 1 Processor -- start 2024-01-20T11:18:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 1 -- start start 2024-01-20T11:18:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient: init 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efc8406bb00 0x7efc840b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient(hunting): picked mon.noname-a con 0x7efc8406bb00 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc840af790 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efc840b0900 0x7efc84145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient(hunting): picked mon.noname-c con 0x7efc840b0900 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8406c220 con 0x7efc840af790 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8406bee0 con 0x7efc8406bb00 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc840ada30 con 0x7efc840b0900 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.986+0000 7efc889f5640 10 monclient(hunting): authenticate will time out at 1530.337822s 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc8259c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efc8406bb00 0x7efc840b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc82d9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efc840b0900 0x7efc84145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc82d9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efc840b0900 0x7efc84145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:56320/0 (socket says 172.21.15.110:56320) 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc82d9d640 1 -- 172.21.15.110:0/186335915 learned_addr learned my addr 172.21.15.110:0/186335915 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc82d9d640 10 monclient(hunting): get_auth_request con 0x7efc840b0900 auth_method 0 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc82d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc8259c640 10 monclient(hunting): get_auth_request con 0x7efc8406bb00 auth_method 0 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc8259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.987+0000 7efc8259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.988+0000 7efc82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.988+0000 7efc82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.988+0000 7efc82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:06.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.988+0000 7efc8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:06.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.988+0000 7efc8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:06.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.988+0000 7efc8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:06.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.989+0000 7efc82d9d640 10 monclient(hunting): handle_auth_done global_id 4762 payload 274 2024-01-20T11:18:06.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.989+0000 7efc82d9d640 1 -- 172.21.15.110:0/186335915 >> v2:172.21.15.110:3300/0 conn(0x7efc8406bb00 msgr2=0x7efc840b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:06.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.989+0000 7efc82d9d640 1 --2- 172.21.15.110:0/186335915 >> v2:172.21.15.110:3300/0 conn(0x7efc8406bb00 0x7efc840b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.989+0000 7efc82d9d640 1 -- 172.21.15.110:0/186335915 >> v1:172.21.15.110:6789/0 conn(0x7efc840af790 legacy=0x7efc84145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.989+0000 7efc82d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.989+0000 7efc82d9d640 1 monclient: found mon.noname-c 2024-01-20T11:18:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.989+0000 7efc82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.989+0000 7efc82d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.989+0000 7efc82d9d640 1 -- 172.21.15.110:0/186335915 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc840add10 con 0x7efc840b0900 2024-01-20T11:18:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.991+0000 7efc82d9d640 1 --2- 172.21.15.110:0/186335915 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840b0900 0x7efc84145e80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc840b1fb0 tx=0x7efc7c02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=118f77f47ddb1dd7 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:06.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.994+0000 7efc81d9b640 1 -- 172.21.15.110:0/186335915 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c02ca80 con 0x7efc840b0900 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 10 monclient: dump: 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 10 monclient: _finish_auth 0 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 1 -- 172.21.15.110:0/186335915 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7c02cbe0 con 0x7efc840b0900 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 5 monclient: authenticate success, global_id 4762 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 1 -- 172.21.15.110:0/186335915 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c0343f0 con 0x7efc840b0900 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 4 set_mon_vals no callback set 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc81d9b640 10 monclient: dump: 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 10 monclient: shutdown 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 1 -- 172.21.15.110:0/186335915 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840b0900 msgr2=0x7efc84145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 1 --2- 172.21.15.110:0/186335915 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840b0900 0x7efc84145e80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc840b1fb0 tx=0x7efc7c02bee0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 1 -- 172.21.15.110:0/186335915 shutdown_connections 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 1 --2- 172.21.15.110:0/186335915 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840b0900 0x7efc84145e80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 1 --2- 172.21.15.110:0/186335915 >> v2:172.21.15.110:3300/0 conn(0x7efc8406bb00 0x7efc840b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 1 -- 172.21.15.110:0/186335915 >> 172.21.15.110:0/186335915 conn(0x7efc84066d60 msgr2=0x7efc840b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 1 -- 172.21.15.110:0/186335915 shutdown_connections 2024-01-20T11:18:07.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.995+0000 7efc889f5640 1 -- 172.21.15.110:0/186335915 wait complete. 2024-01-20T11:18:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.996+0000 7efc889f5640 10 monclient: build_initial_monmap 2024-01-20T11:18:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.996+0000 7efc889f5640 10 monclient: monmap: 2024-01-20T11:18:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.996+0000 7efc889f5640 1 librados: starting msgr at 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.996+0000 7efc889f5640 1 librados: starting objecter 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.996+0000 7efc889f5640 1 Processor -- start 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.996+0000 7efc889f5640 1 -- start start 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.996+0000 7efc889f5640 1 librados: setting wanted keys 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.996+0000 7efc889f5640 1 librados: calling monclient init 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.996+0000 7efc889f5640 10 monclient: init 2024-01-20T11:18:07.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc8406bb00 0x7efc840b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient(hunting): picked mon.noname-a con 0x7efc8406bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840af790 0x7efc8411d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient(hunting): picked mon.noname-b con 0x7efc840af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc840b0900 0x7efc8413b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient(hunting): picked mon.noname-c con 0x7efc840b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8411de20 con 0x7efc8406bb00 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8411dae0 con 0x7efc840af790 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:07.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc84142100 con 0x7efc840b0900 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc889f5640 10 monclient(hunting): authenticate will time out at 1530.348721s 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.997+0000 7efc8259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc8406bb00 0x7efc840b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc7bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840af790 0x7efc8411d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc82d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc840b0900 0x7efc8413b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc8259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc8406bb00 0x7efc840b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45844/0 (socket says 172.21.15.110:45844) 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc8259c640 1 -- 172.21.15.110:0/718281752 learned_addr learned my addr 172.21.15.110:0/718281752 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc8259c640 10 monclient(hunting): get_auth_request con 0x7efc8406bb00 auth_method 0 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc8259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc8259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc7bfff640 10 monclient(hunting): get_auth_request con 0x7efc840af790 auth_method 0 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:07.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc7bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc82d9d640 10 monclient(hunting): get_auth_request con 0x7efc840b0900 auth_method 0 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc82d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.998+0000 7efc7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.999+0000 7efc82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.999+0000 7efc82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:06.999+0000 7efc82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 10 monclient(hunting): handle_auth_done global_id 4765 payload 824 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 1 -- 172.21.15.110:0/718281752 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc840b0900 msgr2=0x7efc8413b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:07.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc840b0900 0x7efc8413b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 1 -- 172.21.15.110:0/718281752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc8406bb00 msgr2=0x7efc840b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc8406bb00 0x7efc840b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 1 monclient: found mon.noname-b 2024-01-20T11:18:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:07.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.001+0000 7efc7bfff640 1 -- 172.21.15.110:0/718281752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c009b30 con 0x7efc840af790 2024-01-20T11:18:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.003+0000 7efc7bfff640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840af790 0x7efc8411d5a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00c390 tx=0x7efc6c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.003+0000 7efc7affd640 10 client.?.objecter ms_handle_connect 0x7efc840af790 2024-01-20T11:18:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.003+0000 7efc7affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.006+0000 7efc7affd640 1 -- 172.21.15.110:0/718281752 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0035f0 con 0x7efc840af790 2024-01-20T11:18:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.006+0000 7efc7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.006+0000 7efc7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.006+0000 7efc7affd640 10 monclient: dump: 2024-01-20T11:18:07.009 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc7affd640 10 monclient: _finish_auth 0 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc7affd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc7affd640 1 -- 172.21.15.110:0/718281752 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c003750 con 0x7efc840af790 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc889f5640 5 monclient: authenticate success, global_id 4765 2024-01-20T11:18:07.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc889f5640 10 monclient: _renew_subs 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc889f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc84142390 con 0x7efc840af790 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc7affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc7affd640 1 -- 172.21.15.110:0/718281752 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c003b80 con 0x7efc840af790 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc7affd640 10 monclient: dump: 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc8159a640 4 set_mon_vals no callback set 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc889f5640 10 client.4765.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc889f5640 10 monclient: _renew_subs 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc889f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8413c120 con 0x7efc840af790 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.007+0000 7efc889f5640 1 librados: init done 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.008+0000 7efc889f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.008+0000 7efc889f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.008+0000 7efc889f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.008+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8411df90 con 0x7efc840af790 2024-01-20T11:18:07.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.026+0000 7efc7affd640 1 -- 172.21.15.110:0/718281752 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7efc6c003ce0 con 0x7efc840af790 2024-01-20T11:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.027+0000 7efc7affd640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efc6406c8b0 0x7efc6406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.027+0000 7efc8259c640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efc6406c8b0 0x7efc6406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.027+0000 7efc8259c640 10 monclient: get_auth_request con 0x7efc6406c8b0 auth_method 0 2024-01-20T11:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.027+0000 7efc7affd640 10 client.4765.objecter ms_handle_connect 0x7efc6406c8b0 2024-01-20T11:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.027+0000 7efc8259c640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efc6406c8b0 0x7efc6406ed70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efc74009c00 tx=0x7efc74009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:07.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.251+0000 7efc7affd640 1 -- 172.21.15.110:0/718281752 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efc6c057b90 con 0x7efc840af790 2024-01-20T11:18:07.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.251+0000 7efc7affd640 10 client.4765.objecter ms_dispatch 0x7efc84000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:07.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.251+0000 7efc7affd640 3 client.4765.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:07.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.251+0000 7efc7affd640 3 client.4765.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:07.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.251+0000 7efc7affd640 20 client.4765.objecter dump_active .. 0 homeless 2024-01-20T11:18:07.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.262+0000 7efc7affd640 1 -- 172.21.15.110:0/718281752 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7efc6c053ed0 con 0x7efc840af790 2024-01-20T11:18:07.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.262+0000 7efc7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:07.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.262+0000 7efc7affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:07.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.516+0000 7efc889f5640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:07.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.516+0000 7efc889f5640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:07.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.516+0000 7efc889f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:07.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.516+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7efc84075490 con 0x7efc840af790 2024-01-20T11:18:07.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.533+0000 7efc7affd640 1 -- 172.21.15.110:0/718281752 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7efc6c017520 con 0x7efc840af790 2024-01-20T11:18:07.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.533+0000 7efc7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:07.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.533+0000 7efc7affd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:07.534 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:07.534 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":222,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 10 librados: watch_flush enter 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 10 librados: watch_flush exit 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 20 client.4765.objecter shutdown clearing up homeless session... 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 10 client.4765.objecter successfully canceled tick 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efc6406c8b0 msgr2=0x7efc6406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efc6406c8b0 0x7efc6406ed70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efc74009c00 tx=0x7efc74009190 comp rx=0 tx=0).stop 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 10 monclient: shutdown 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840af790 msgr2=0x7efc8411d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840af790 0x7efc8411d5a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00c390 tx=0x7efc6c00c850 comp rx=0 tx=0).stop 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 shutdown_connections 2024-01-20T11:18:07.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efc6406c8b0 0x7efc6406ed70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc840b0900 0x7efc8413b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc840af790 0x7efc8411d5a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 --2- 172.21.15.110:0/718281752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc8406bb00 0x7efc840b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 >> 172.21.15.110:0/718281752 conn(0x7efc84066d60 msgr2=0x7efc84142950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:07.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 shutdown_connections 2024-01-20T11:18:07.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 -- 172.21.15.110:0/718281752 wait complete. 2024-01-20T11:18:07.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.535+0000 7efc889f5640 1 librados: shutdown 2024-01-20T11:18:07.548 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:07.548 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:07.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.647+0000 7f8e60a64640 10 monclient: get_monmap_and_config 2024-01-20T11:18:07.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.647+0000 7f8e60a64640 10 monclient: build_initial_monmap 2024-01-20T11:18:07.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.647+0000 7f8e60a64640 10 monclient: monmap: 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:07.648770+0000 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:07.648770+0000 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.647+0000 7f8e60a64640 1 Processor -- start 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.647+0000 7f8e60a64640 1 -- start start 2024-01-20T11:18:07.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.647+0000 7f8e60a64640 10 monclient: init 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e5c0a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e5c0a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c142af0 0x7f8e5c14d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 10 monclient(hunting): picked mon.noname-b con 0x7f8e5c142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c0b25c0 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c142f50 con 0x7f8e5c0a96f0 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0aaea0 con 0x7f8e5c142af0 2024-01-20T11:18:07.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.648+0000 7f8e60a64640 10 monclient(hunting): authenticate will time out at 1530.999721s 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e5ad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c142af0 0x7f8e5c14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e5a59c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8e5c0a96f0 0x7f8e5c13f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:56706/0 (socket says 172.21.15.110:56706) 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e5a59c640 1 -- 172.21.15.110:0/905145527 learned_addr learned my addr 172.21.15.110:0/905145527 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e59d9b640 1 --1- 172.21.15.110:0/905145527 >> v1:172.21.15.110:6789/0 conn(0x7f8e5c0a9ad0 0x7f8e5c1423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:41036/0 (socket says 172.21.15.110:41036) 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e5ad9d640 10 monclient(hunting): get_auth_request con 0x7f8e5c142af0 auth_method 0 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e5ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e5ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.649+0000 7f8e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:07.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.650+0000 7f8e5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:07.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.651+0000 7f8e5959a640 1 -- 172.21.15.110:0/905145527 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416700738 0 0) 0x7f8e54001490 con 0x7f8e5c0a96f0 2024-01-20T11:18:07.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.651+0000 7f8e5959a640 10 monclient(hunting): _negotiate 2024-01-20T11:18:07.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.651+0000 7f8e5959a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:07.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.651+0000 7f8e5959a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.651+0000 7f8e5959a640 10 monclient(hunting): my global_id is 4768 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.651+0000 7f8e5959a640 1 -- 172.21.15.110:0/905145527 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e44003520 con 0x7f8e5c0a96f0 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.652+0000 7f8e5ad9d640 10 monclient(hunting): handle_auth_done global_id 4574 payload 274 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.652+0000 7f8e5ad9d640 1 -- 172.21.15.110:0/905145527 >> v1:172.21.15.134:6789/0 conn(0x7f8e5c0a96f0 legacy=0x7f8e5c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.652+0000 7f8e5ad9d640 1 -- 172.21.15.110:0/905145527 >> v1:172.21.15.110:6789/0 conn(0x7f8e5c0a9ad0 legacy=0x7f8e5c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.652+0000 7f8e5ad9d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.652+0000 7f8e5ad9d640 1 monclient: found mon.noname-b 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.652+0000 7f8e5ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.652+0000 7f8e5ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.652+0000 7f8e5ad9d640 1 -- 172.21.15.110:0/905145527 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c13fbe0 con 0x7f8e5c142af0 2024-01-20T11:18:07.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.653+0000 7f8e5ad9d640 1 --2- 172.21.15.110:0/905145527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c142af0 0x7f8e5c14d860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0097d0 tx=0x7f8e4c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ce3fecce568191 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 1 -- 172.21.15.110:0/905145527 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c031800 con 0x7f8e5c142af0 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 10 monclient: dump: 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:07.659 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 10 monclient: _finish_auth 0 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 10 monclient: _check_auth_tickets 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 1 -- 172.21.15.110:0/905145527 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c031960 con 0x7f8e5c142af0 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 1 -- 172.21.15.110:0/905145527 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c031cf0 con 0x7f8e5c142af0 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e60a64640 5 monclient: authenticate success, global_id 4574 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e60a64640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e60a64640 4 set_mon_vals no callback set 2024-01-20T11:18:07.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.657+0000 7f8e5959a640 10 monclient: dump: 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 10 monclient: shutdown 2024-01-20T11:18:07.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:07.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 1 -- 172.21.15.110:0/905145527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c142af0 msgr2=0x7f8e5c14d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:07.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 1 --2- 172.21.15.110:0/905145527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c142af0 0x7f8e5c14d860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0097d0 tx=0x7f8e4c02b8a0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 1 -- 172.21.15.110:0/905145527 shutdown_connections 2024-01-20T11:18:07.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 1 --2- 172.21.15.110:0/905145527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c142af0 0x7f8e5c14d860 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 1 -- 172.21.15.110:0/905145527 >> 172.21.15.110:0/905145527 conn(0x7f8e5c0b7bc0 msgr2=0x7f8e5c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:07.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 1 -- 172.21.15.110:0/905145527 shutdown_connections 2024-01-20T11:18:07.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 1 -- 172.21.15.110:0/905145527 wait complete. 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 10 monclient: build_initial_monmap 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.658+0000 7f8e60a64640 10 monclient: monmap: 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 librados: starting msgr at 2024-01-20T11:18:07.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 librados: starting objecter 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 Processor -- start 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 -- start start 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 librados: setting wanted keys 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 librados: calling monclient init 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient: init 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c0a96f0 0x7f8e5c1330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient(hunting): picked mon.noname-c con 0x7f8e5c0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e5c0a9ad0 0x7f8e5c1d7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient(hunting): picked mon.noname-b con 0x7f8e5c0a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e5c142af0 0x7f8e5c12ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient(hunting): picked mon.noname-a con 0x7f8e5c142af0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:07.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c1379d0 con 0x7f8e5c142af0 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c1d7440 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c1d70d0 con 0x7f8e5c0a96f0 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.659+0000 7f8e60a64640 10 monclient(hunting): authenticate will time out at 1531.011125s 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c0a96f0 0x7f8e5c1330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e5c142af0 0x7f8e5c12ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e59d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e5c0a9ad0 0x7f8e5c1d7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c0a96f0 0x7f8e5c1330c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:44044/0 (socket says 172.21.15.110:44044) 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5a59c640 1 -- 172.21.15.110:0/2637428432 learned_addr learned my addr 172.21.15.110:0/2637428432 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5a59c640 10 monclient(hunting): get_auth_request con 0x7f8e5c0a96f0 auth_method 0 2024-01-20T11:18:07.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e59d9b640 10 monclient(hunting): get_auth_request con 0x7f8e5c0a9ad0 auth_method 0 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e59d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5ad9d640 10 monclient(hunting): get_auth_request con 0x7f8e5c142af0 auth_method 0 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.660+0000 7f8e5ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.661+0000 7f8e59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:07.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.661+0000 7f8e59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.661+0000 7f8e59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.661+0000 7f8e5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.661+0000 7f8e5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.661+0000 7f8e5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.661+0000 7f8e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.661+0000 7f8e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.661+0000 7f8e5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 10 monclient(hunting): handle_auth_done global_id 4771 payload 824 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 1 -- 172.21.15.110:0/2637428432 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c0a96f0 msgr2=0x7f8e5c1330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c0a96f0 0x7f8e5c1330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 1 -- 172.21.15.110:0/2637428432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e5c142af0 msgr2=0x7f8e5c12ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e5c142af0 0x7f8e5c12ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:18:07.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 1 monclient: found mon.noname-b 2024-01-20T11:18:07.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:07.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:07.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.663+0000 7f8e59d9b640 1 -- 172.21.15.110:0/2637428432 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c009400 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.666+0000 7f8e59d9b640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e5c0a9ad0 0x7f8e5c1d7960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8e50000f60 tx=0x7f8e5000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:07.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.666+0000 7f8e4affd640 10 client.?.objecter ms_handle_connect 0x7f8e5c0a9ad0 2024-01-20T11:18:07.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.666+0000 7f8e4affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:07.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 1 -- 172.21.15.110:0/2637428432 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500074e0 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 10 monclient: dump: 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 10 monclient: _finish_auth 0 2024-01-20T11:18:07.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 1 -- 172.21.15.110:0/2637428432 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e50007660 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 1 -- 172.21.15.110:0/2637428432 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50004a20 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e4affd640 10 monclient: dump: 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:07.674 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e60a64640 5 monclient: authenticate success, global_id 4771 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e60a64640 10 monclient: _renew_subs 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e60a64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c1d7ea0 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e58d99640 4 set_mon_vals no callback set 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e60a64640 10 client.4771.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e60a64640 10 monclient: _renew_subs 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e60a64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c12f3e0 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.669+0000 7f8e60a64640 1 librados: init done 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.670+0000 7f8e60a64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:07.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.670+0000 7f8e60a64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:07.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.670+0000 7f8e60a64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:07.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.670+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c1d7290 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.689+0000 7f8e4affd640 1 -- 172.21.15.110:0/2637428432 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8e50004b80 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.689+0000 7f8e4affd640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8e3c06c890 0x7f8e3c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:07.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.689+0000 7f8e5a59c640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8e3c06c890 0x7f8e3c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:07.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.689+0000 7f8e5a59c640 10 monclient: get_auth_request con 0x7f8e3c06c890 auth_method 0 2024-01-20T11:18:07.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.690+0000 7f8e5a59c640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8e3c06c890 0x7f8e3c06ed50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e540016f0 tx=0x7f8e54001b60 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:07.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.690+0000 7f8e4affd640 10 client.4771.objecter ms_handle_connect 0x7f8e3c06c890 2024-01-20T11:18:07.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.690+0000 7f8e4affd640 1 -- 172.21.15.110:0/2637428432 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8e50058220 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.690+0000 7f8e4affd640 10 client.4771.objecter ms_dispatch 0x7f8e5c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:07.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.690+0000 7f8e4affd640 3 client.4771.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:07.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.690+0000 7f8e4affd640 3 client.4771.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:07.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.690+0000 7f8e4affd640 20 client.4771.objecter dump_active .. 0 homeless 2024-01-20T11:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.930+0000 7f8e4affd640 1 -- 172.21.15.110:0/2637428432 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8e50054560 con 0x7f8e5c0a9ad0 2024-01-20T11:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.930+0000 7f8e4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:07.930+0000 7f8e4affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:08.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.103+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e5c0b4400 con 0x7f8e3c06c890 2024-01-20T11:18:08.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.107+0000 7f8e4affd640 1 -- 172.21.15.110:0/2637428432 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64188 (secure 0 0 0) 0x7f8e54002050 con 0x7f8e3c06c890 2024-01-20T11:18:08.107 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:08.108 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 10 librados: watch_flush enter 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 10 librados: watch_flush exit 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 20 client.4771.objecter shutdown clearing up homeless session... 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 10 client.4771.objecter successfully canceled tick 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8e3c06c890 msgr2=0x7f8e3c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8e3c06c890 0x7f8e3c06ed50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e540016f0 tx=0x7f8e54001b60 comp rx=0 tx=0).stop 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 10 monclient: shutdown 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e5c0a9ad0 msgr2=0x7f8e5c1d7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e5c0a9ad0 0x7f8e5c1d7960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8e50000f60 tx=0x7f8e5000c800 comp rx=0 tx=0).stop 2024-01-20T11:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 shutdown_connections 2024-01-20T11:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8e3c06c890 0x7f8e3c06ed50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e5c142af0 0x7f8e5c12ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e5c0a9ad0 0x7f8e5c1d7960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 --2- 172.21.15.110:0/2637428432 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e5c0a96f0 0x7f8e5c1330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 >> 172.21.15.110:0/2637428432 conn(0x7f8e5c0b7bc0 msgr2=0x7f8e5c131d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 shutdown_connections 2024-01-20T11:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 -- 172.21.15.110:0/2637428432 wait complete. 2024-01-20T11:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:08.108+0000 7f8e60a64640 1 librados: shutdown 2024-01-20T11:18:08.120 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-01-20T11:18:06.743295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001956"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.7039999999999997},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.5979999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.960000000000001},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.33}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.958},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.9920000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.7729999999999997},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":16.594000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.856999999999999},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":14.144}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.595000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.43}]}]},{"osd":7,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2270000000000003},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4350000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7289999999999992},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.1639999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.263999999999999},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.599}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.025},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.891}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.583},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.301}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.863},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.554}]}]},{"osd":3,"up_from":21,"seq":90194313256,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.7780000000000005},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":4.8810000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":15.708},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.811999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":7.4119999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.430999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.778},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.138999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":8.0920000000000005},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.522}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":16.5},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.763}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.012},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":5.9909999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.4610000000000003},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.8869999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.737},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.0709999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.582000000000001},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.5709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":10.304},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.648}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.946999999999999},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.340999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":13.178000000000001},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":7.3869999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.622},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":14.939}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.116},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.7270000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.8460000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.731}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.0649999999999995},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":10.429}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.173},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.032}]}]},{"osd":0,"up_from":17,"seq":73014444070,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":8.7539999999999996},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.4749999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.2539999999999996},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.4179999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":11.603999999999999},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.131}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.193},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":5.9480000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.089},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.226000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.044},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":8.0589999999999993}]}]},{"osd":6,"up_from":15,"seq":64424509479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.7140000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.4180000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.702999999999999},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":15.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6749999999999998},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.087999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.73},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.0679999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.9329999999999998},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.266999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.678000000000001},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.503}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:11.122 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:11.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.220+0000 7f2b4af9e640 10 monclient: get_monmap_and_config 2024-01-20T11:18:11.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.220+0000 7f2b4af9e640 10 monclient: build_initial_monmap 2024-01-20T11:18:11.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.220+0000 7f2b4af9e640 10 monclient: monmap: 2024-01-20T11:18:11.221 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:11.221 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.221 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:11.221916+0000 2024-01-20T11:18:11.221 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:11.221916+0000 2024-01-20T11:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 1 Processor -- start 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 1 -- start start 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 10 monclient: init 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b4406f920 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b4406fd00 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 0x7f2b4414b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 10 monclient(hunting): picked mon.noname-b con 0x7f2b44144060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b440715b0 con 0x7f2b4406f920 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b44071890 con 0x7f2b4406fd00 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b44071b70 con 0x7f2b44144060 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.221+0000 7f2b4af9e640 10 monclient(hunting): authenticate will time out at 1534.573029s 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.222+0000 7f2b43fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2b4406fd00 0x7f2b44143950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:45698/0 (socket says 172.21.15.110:45698) 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.222+0000 7f2b43fff640 1 -- 172.21.15.110:0/3031872578 learned_addr learned my addr 172.21.15.110:0/3031872578 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.222+0000 7f2b4953b640 1 --2- 172.21.15.110:0/3031872578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 0x7f2b4414b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.222+0000 7f2b4953b640 10 monclient(hunting): get_auth_request con 0x7f2b44144060 auth_method 0 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.222+0000 7f2b4953b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.222+0000 7f2b4953b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.222+0000 7f2b4953b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.223+0000 7f2b4953b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:11.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.223+0000 7f2b4953b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:11.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.223+0000 7f2b4953b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 1 -- 172.21.15.110:0/3031872578 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109397742 0 0) 0x7f2b440715b0 con 0x7f2b4406f920 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 10 monclient(hunting): _negotiate 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 10 monclient(hunting): my global_id is 4698 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 1 -- 172.21.15.110:0/3031872578 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b2c003520 con 0x7f2b4406f920 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 1 -- 172.21.15.110:0/3031872578 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420930018 0 0) 0x7f2b44071890 con 0x7f2b4406fd00 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 10 monclient(hunting): _negotiate 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 10 monclient(hunting): my global_id is 4774 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.224+0000 7f2b437fe640 1 -- 172.21.15.110:0/3031872578 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b440715b0 con 0x7f2b4406fd00 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.225+0000 7f2b4953b640 10 monclient(hunting): handle_auth_done global_id 4580 payload 274 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.225+0000 7f2b4953b640 1 -- 172.21.15.110:0/3031872578 >> v1:172.21.15.134:6789/0 conn(0x7f2b4406fd00 legacy=0x7f2b44143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.225+0000 7f2b4953b640 1 -- 172.21.15.110:0/3031872578 >> v1:172.21.15.110:6789/0 conn(0x7f2b4406f920 legacy=0x7f2b4413d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.225+0000 7f2b4953b640 10 monclient: _finish_hunting 0 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.225+0000 7f2b4953b640 1 monclient: found mon.noname-b 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.225+0000 7f2b4953b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.225+0000 7f2b4953b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.225+0000 7f2b4953b640 1 -- 172.21.15.110:0/3031872578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b44143430 con 0x7f2b44144060 2024-01-20T11:18:11.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.226+0000 7f2b4953b640 1 --2- 172.21.15.110:0/3031872578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 0x7f2b4414b730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c009dc0 tx=0x7f2b3c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ae866e6f3d6b37f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:11.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 1 -- 172.21.15.110:0/3031872578 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c02cc40 con 0x7f2b44144060 2024-01-20T11:18:11.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:11.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:11.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 10 monclient: dump: 2024-01-20T11:18:11.231 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:11.231 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.231 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.231 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 10 monclient: _finish_auth 0 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 10 monclient: _check_auth_tickets 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 1 -- 172.21.15.110:0/3031872578 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c02cda0 con 0x7f2b44144060 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 1 -- 172.21.15.110:0/3031872578 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c034660 con 0x7f2b44144060 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:11.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b437fe640 10 monclient: dump: 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b4af9e640 5 monclient: authenticate success, global_id 4580 2024-01-20T11:18:11.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b4af9e640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:11.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.230+0000 7f2b4af9e640 4 set_mon_vals no callback set 2024-01-20T11:18:11.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.231+0000 7f2b4af9e640 10 monclient: shutdown 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.231+0000 7f2b4af9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.231+0000 7f2b4af9e640 1 -- 172.21.15.110:0/3031872578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 msgr2=0x7f2b4414b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.231+0000 7f2b4af9e640 1 --2- 172.21.15.110:0/3031872578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 0x7f2b4414b730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c009dc0 tx=0x7f2b3c02be90 comp rx=0 tx=0).stop 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.231+0000 7f2b4af9e640 1 -- 172.21.15.110:0/3031872578 shutdown_connections 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.231+0000 7f2b4af9e640 1 --2- 172.21.15.110:0/3031872578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 0x7f2b4414b730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.231+0000 7f2b4af9e640 1 -- 172.21.15.110:0/3031872578 >> 172.21.15.110:0/3031872578 conn(0x7f2b44066d60 msgr2=0x7f2b440b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.231+0000 7f2b4af9e640 1 -- 172.21.15.110:0/3031872578 shutdown_connections 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.231+0000 7f2b4af9e640 1 -- 172.21.15.110:0/3031872578 wait complete. 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.232+0000 7f2b4af9e640 10 monclient: build_initial_monmap 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.232+0000 7f2b4af9e640 10 monclient: monmap: 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:11.239 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.232+0000 7f2b4af9e640 1 librados: starting msgr at 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.232+0000 7f2b4af9e640 1 librados: starting objecter 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.232+0000 7f2b4af9e640 1 Processor -- start 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.232+0000 7f2b4af9e640 1 -- start start 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.232+0000 7f2b4af9e640 1 librados: setting wanted keys 2024-01-20T11:18:11.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.232+0000 7f2b4af9e640 1 librados: calling monclient init 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.232+0000 7f2b4af9e640 10 monclient: init 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b4406f920 0x7f2b441389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient(hunting): picked mon.noname-b con 0x7f2b4406f920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b4406fd00 0x7f2b4413acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient(hunting): picked mon.noname-a con 0x7f2b4406fd00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 0x7f2b441d79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient(hunting): picked mon.noname-c con 0x7f2b44144060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4413ea40 con 0x7f2b4406fd00 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b441d7560 con 0x7f2b4406f920 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:11.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b441d7230 con 0x7f2b44144060 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4af9e640 10 monclient(hunting): authenticate will time out at 1534.584490s 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b4953b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 0x7f2b441d79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b43fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b4406fd00 0x7f2b4413acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.233+0000 7f2b48d3a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b4406f920 0x7f2b441389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b48d3a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b4406f920 0x7f2b441389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49708/0 (socket says 172.21.15.110:49708) 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b48d3a640 1 -- 172.21.15.110:0/1857012347 learned_addr learned my addr 172.21.15.110:0/1857012347 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b48d3a640 10 monclient(hunting): get_auth_request con 0x7f2b4406f920 auth_method 0 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b48d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b48d3a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b48d3a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b4953b640 10 monclient(hunting): get_auth_request con 0x7f2b44144060 auth_method 0 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b4953b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b4953b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b4953b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b43fff640 10 monclient(hunting): get_auth_request con 0x7f2b4406fd00 auth_method 0 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b43fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b43fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b48d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b48d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b48d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b4953b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b4953b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b4953b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.234+0000 7f2b43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 10 monclient(hunting): handle_auth_done global_id 4777 payload 824 2024-01-20T11:18:11.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 1 -- 172.21.15.110:0/1857012347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 msgr2=0x7f2b441d79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 0x7f2b441d79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 1 -- 172.21.15.110:0/1857012347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b4406fd00 msgr2=0x7f2b4413acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b4406fd00 0x7f2b4413acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 10 monclient: _finish_hunting 0 2024-01-20T11:18:11.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 1 monclient: found mon.noname-b 2024-01-20T11:18:11.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:11.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:11.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.237+0000 7f2b48d3a640 1 -- 172.21.15.110:0/1857012347 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c009a70 con 0x7f2b4406f920 2024-01-20T11:18:11.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.239+0000 7f2b48d3a640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b4406f920 0x7f2b441389d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400c460 tx=0x7f2b3400c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:11.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.239+0000 7f2b417fa640 10 client.?.objecter ms_handle_connect 0x7f2b4406f920 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.239+0000 7f2b417fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 1 -- 172.21.15.110:0/1857012347 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b340052d0 con 0x7f2b4406f920 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 10 monclient: dump: 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:11.246 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 10 monclient: _finish_auth 0 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 10 monclient: _check_auth_tickets 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 1 -- 172.21.15.110:0/1857012347 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b34005470 con 0x7f2b4406f920 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b4af9e640 5 monclient: authenticate success, global_id 4777 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 1 -- 172.21.15.110:0/1857012347 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34012930 con 0x7f2b4406f920 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b417fa640 10 monclient: dump: 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.247 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.242+0000 7f2b42ffd640 4 set_mon_vals no callback set 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 10 monclient: _renew_subs 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b44137e50 con 0x7f2b4406f920 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 10 client.4777.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 10 monclient: _renew_subs 2024-01-20T11:18:11.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:11.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b441d8110 con 0x7f2b4406f920 2024-01-20T11:18:11.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 1 librados: init done 2024-01-20T11:18:11.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:11.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:11.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.243+0000 7f2b4af9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:11.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.244+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b44138040 con 0x7f2b4406f920 2024-01-20T11:18:11.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.484+0000 7f2b417fa640 1 -- 172.21.15.110:0/1857012347 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2b34012a90 con 0x7f2b4406f920 2024-01-20T11:18:11.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.485+0000 7f2b417fa640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b3006c860 0x7f2b3006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.485+0000 7f2b43fff640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b3006c860 0x7f2b3006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.485+0000 7f2b43fff640 10 monclient: get_auth_request con 0x7f2b3006c860 auth_method 0 2024-01-20T11:18:11.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.486+0000 7f2b43fff640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b3006c860 0x7f2b3006ed20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2b440af8e0 tx=0x7f2b38009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:11.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.486+0000 7f2b417fa640 10 client.4777.objecter ms_handle_connect 0x7f2b3006c860 2024-01-20T11:18:11.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.490+0000 7f2b417fa640 1 -- 172.21.15.110:0/1857012347 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2b34058830 con 0x7f2b4406f920 2024-01-20T11:18:11.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.490+0000 7f2b417fa640 10 client.4777.objecter ms_dispatch 0x7f2b44000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:11.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.490+0000 7f2b417fa640 3 client.4777.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:11.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.490+0000 7f2b417fa640 3 client.4777.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:11.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.490+0000 7f2b417fa640 20 client.4777.objecter dump_active .. 0 homeless 2024-01-20T11:18:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.502+0000 7f2b417fa640 1 -- 172.21.15.110:0/1857012347 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2b34054b70 con 0x7f2b4406f920 2024-01-20T11:18:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.502+0000 7f2b417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.502+0000 7f2b417fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:11.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.672+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b440b4950 con 0x7f2b3006c860 2024-01-20T11:18:11.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.675+0000 7f2b417fa640 1 -- 172.21.15.110:0/1857012347 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64227 (secure 0 0 0) 0x7f2b440b4950 con 0x7f2b3006c860 2024-01-20T11:18:11.677 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:11.678 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 10 librados: watch_flush enter 2024-01-20T11:18:11.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 10 librados: watch_flush exit 2024-01-20T11:18:11.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 20 client.4777.objecter shutdown clearing up homeless session... 2024-01-20T11:18:11.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 10 client.4777.objecter successfully canceled tick 2024-01-20T11:18:11.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b3006c860 msgr2=0x7f2b3006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b3006c860 0x7f2b3006ed20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2b440af8e0 tx=0x7f2b38009040 comp rx=0 tx=0).stop 2024-01-20T11:18:11.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 10 monclient: shutdown 2024-01-20T11:18:11.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:11.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b4406f920 msgr2=0x7f2b441389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b4406f920 0x7f2b441389d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400c460 tx=0x7f2b3400c920 comp rx=0 tx=0).stop 2024-01-20T11:18:11.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 shutdown_connections 2024-01-20T11:18:11.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2b3006c860 0x7f2b3006ed20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b44144060 0x7f2b441d79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b4406fd00 0x7f2b4413acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 --2- 172.21.15.110:0/1857012347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b4406f920 0x7f2b441389d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 >> 172.21.15.110:0/1857012347 conn(0x7f2b44066d60 msgr2=0x7f2b440b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 shutdown_connections 2024-01-20T11:18:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 -- 172.21.15.110:0/1857012347 wait complete. 2024-01-20T11:18:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.677+0000 7f2b4af9e640 1 librados: shutdown 2024-01-20T11:18:11.689 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-01-20T11:18:10.743950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001945"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.5549999999999997},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.1630000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.061999999999999},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.4979999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.696},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.818}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":17.420999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":13.279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.819000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":10.234}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.300000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":14.381}]}]},{"osd":7,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2270000000000003},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4350000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7289999999999992},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.1639999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.263999999999999},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.599}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.025},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.891}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.583},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.301}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.863},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.554}]}]},{"osd":3,"up_from":21,"seq":90194313256,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.7780000000000005},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":4.8810000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":15.708},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.811999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":7.4119999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.430999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.778},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.138999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":8.0920000000000005},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.522}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":16.5},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.763}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":11.616},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":16.163}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.8460000000000001},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":14.467000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.173},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":17.103999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":7.3869999999999996},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.849}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.1170000000000009},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.5049999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.7230000000000008},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.044}]}]},{"osd":1,"up_from":17,"seq":73014444071,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.596},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":5.8490000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.9000000000000004},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.983000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.769},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.8100000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.4689999999999994},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.555999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.252000000000001},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":7.9009999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.275},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.555}]}]},{"osd":0,"up_from":17,"seq":73014444071,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":8.1310000000000002},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":12.129}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.7519999999999998},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.986000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":10.753},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.9309999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.388},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.542999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":17.184000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":18.251999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.143999999999998},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":12.842000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.7140000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.4180000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.702999999999999},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":15.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6749999999999998},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.087999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.73},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.0679999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.9329999999999998},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.266999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.678000000000001},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.503}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:11.690 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:11.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.789+0000 7f5bd4007640 10 monclient: get_monmap_and_config 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.789+0000 7f5bd4007640 10 monclient: build_initial_monmap 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.789+0000 7f5bd4007640 10 monclient: monmap: 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:11.791033+0000 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:11.791033+0000 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.789+0000 7f5bd4007640 1 Processor -- start 2024-01-20T11:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 1 -- start start 2024-01-20T11:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient: init 2024-01-20T11:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 0x7f5bcc0ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient(hunting): picked mon.noname-b con 0x7f5bcc0b0ba0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bcc0adc30 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5bcc0ac4d0 0x7f5bcc13bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient(hunting): picked mon.noname-c con 0x7f5bcc0ac4d0 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc0ad530 con 0x7f5bcc0adc30 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bcc0af840 con 0x7f5bcc0ac4d0 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0b0f80 con 0x7f5bcc0b0ba0 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:11.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.790+0000 7f5bd4007640 10 monclient(hunting): authenticate will time out at 1535.141972s 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd25a4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5bcc0ac4d0 0x7f5bcc13bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd1da3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 0x7f5bcc0ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd25a4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5bcc0ac4d0 0x7f5bcc13bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49712/0 (socket says 172.21.15.110:49712) 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd25a4640 1 -- 172.21.15.110:0/3934145018 learned_addr learned my addr 172.21.15.110:0/3934145018 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd25a4640 10 monclient(hunting): get_auth_request con 0x7f5bcc0ac4d0 auth_method 0 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd25a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd25a4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd25a4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd1da3640 10 monclient(hunting): get_auth_request con 0x7f5bcc0b0ba0 auth_method 0 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd1da3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd1da3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.791+0000 7f5bd1da3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.792+0000 7f5bd1da3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.792+0000 7f5bd1da3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:11.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.792+0000 7f5bd1da3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:11.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.792+0000 7f5bd25a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:11.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.792+0000 7f5bd25a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:11.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.792+0000 7f5bd25a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:11.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.793+0000 7f5bd1da3640 10 monclient(hunting): handle_auth_done global_id 4586 payload 274 2024-01-20T11:18:11.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.793+0000 7f5bd1da3640 1 -- 172.21.15.110:0/3934145018 >> v2:172.21.15.134:3300/0 conn(0x7f5bcc0ac4d0 msgr2=0x7f5bcc13bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:11.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.793+0000 7f5bd1da3640 1 --2- 172.21.15.110:0/3934145018 >> v2:172.21.15.134:3300/0 conn(0x7f5bcc0ac4d0 0x7f5bcc13bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.793+0000 7f5bd1da3640 1 -- 172.21.15.110:0/3934145018 >> v1:172.21.15.134:6789/0 conn(0x7f5bcc0adc30 legacy=0x7f5bcc0af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.793+0000 7f5bd1da3640 10 monclient: _finish_hunting 0 2024-01-20T11:18:11.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.793+0000 7f5bd1da3640 1 monclient: found mon.noname-b 2024-01-20T11:18:11.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.793+0000 7f5bd1da3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:11.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.793+0000 7f5bd1da3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:11.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.793+0000 7f5bd1da3640 1 -- 172.21.15.110:0/3934145018 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc0ad810 con 0x7f5bcc0b0ba0 2024-01-20T11:18:11.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.795+0000 7f5bd1da3640 1 --2- 172.21.15.110:0/3934145018 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 0x7f5bcc0ab630 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80097a0 tx=0x7f5bc802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5c132e8a6e91c74 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:11.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 1 -- 172.21.15.110:0/3934145018 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc802e070 con 0x7f5bcc0b0ba0 2024-01-20T11:18:11.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:11.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 10 monclient: dump: 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 10 monclient: _finish_auth 0 2024-01-20T11:18:11.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 10 monclient: _check_auth_tickets 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 1 -- 172.21.15.110:0/3934145018 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc80315a0 con 0x7f5bcc0b0ba0 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 5 monclient: authenticate success, global_id 4586 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 1 -- 172.21.15.110:0/3934145018 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8031930 con 0x7f5bcc0b0ba0 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 4 set_mon_vals no callback set 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd0da1640 10 monclient: dump: 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.808 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 10 monclient: shutdown 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 1 -- 172.21.15.110:0/3934145018 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 msgr2=0x7f5bcc0ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 1 --2- 172.21.15.110:0/3934145018 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 0x7f5bcc0ab630 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80097a0 tx=0x7f5bc802b840 comp rx=0 tx=0).stop 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 1 -- 172.21.15.110:0/3934145018 shutdown_connections 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 1 --2- 172.21.15.110:0/3934145018 >> v2:172.21.15.134:3300/0 conn(0x7f5bcc0ac4d0 0x7f5bcc13bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 1 --2- 172.21.15.110:0/3934145018 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 0x7f5bcc0ab630 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.799+0000 7f5bd4007640 1 -- 172.21.15.110:0/3934145018 >> 172.21.15.110:0/3934145018 conn(0x7f5bcc0b3e60 msgr2=0x7f5bcc0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.800+0000 7f5bd4007640 1 -- 172.21.15.110:0/3934145018 shutdown_connections 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.800+0000 7f5bd4007640 1 -- 172.21.15.110:0/3934145018 wait complete. 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.800+0000 7f5bd4007640 10 monclient: build_initial_monmap 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.800+0000 7f5bd4007640 10 monclient: monmap: 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:11.810 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.800+0000 7f5bd4007640 1 librados: starting msgr at 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.800+0000 7f5bd4007640 1 librados: starting objecter 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 Processor -- start 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 -- start start 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 librados: setting wanted keys 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 librados: calling monclient init 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient: init 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bcc0ac4d0 0x7f5bcc0b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient(hunting): picked mon.noname-a con 0x7f5bcc0ac4d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 0x7f5bcc0a9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient(hunting): picked mon.noname-c con 0x7f5bcc0b0ba0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bcc12e7f0 0x7f5bcc12ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient(hunting): picked mon.noname-b con 0x7f5bcc12e7f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0aa180 con 0x7f5bcc0ac4d0 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0a9ea0 con 0x7f5bcc12e7f0 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc13f640 con 0x7f5bcc0b0ba0 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.801+0000 7f5bd4007640 10 monclient(hunting): authenticate will time out at 1535.152882s 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bcc12e7f0 0x7f5bcc12ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd15a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 0x7f5bcc0a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd1da3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bcc0ac4d0 0x7f5bcc0b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bcc12e7f0 0x7f5bcc12ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49720/0 (socket says 172.21.15.110:49720) 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 1 -- 172.21.15.110:0/1858547262 learned_addr learned my addr 172.21.15.110:0/1858547262 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:11.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 10 monclient(hunting): get_auth_request con 0x7f5bcc12e7f0 auth_method 0 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd15a2640 10 monclient(hunting): get_auth_request con 0x7f5bcc0b0ba0 auth_method 0 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd15a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd15a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd15a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd1da3640 10 monclient(hunting): get_auth_request con 0x7f5bcc0ac4d0 auth_method 0 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd1da3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd1da3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd1da3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.802+0000 7f5bd25a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:11.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.803+0000 7f5bd15a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.803+0000 7f5bd15a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.803+0000 7f5bd15a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.803+0000 7f5bd1da3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.803+0000 7f5bd1da3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.803+0000 7f5bd1da3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 10 monclient(hunting): handle_auth_done global_id 4786 payload 824 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 1 -- 172.21.15.110:0/1858547262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 msgr2=0x7f5bcc0a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 0x7f5bcc0a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 1 -- 172.21.15.110:0/1858547262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bcc0ac4d0 msgr2=0x7f5bcc0b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bcc0ac4d0 0x7f5bcc0b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 10 monclient: _finish_hunting 0 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 1 monclient: found mon.noname-b 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:11.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.805+0000 7f5bd25a4640 1 -- 172.21.15.110:0/1858547262 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8009400 con 0x7f5bcc12e7f0 2024-01-20T11:18:11.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.807+0000 7f5bd25a4640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bcc12e7f0 0x7f5bcc12ec40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400efc0 tx=0x7f5bc400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:11.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.807+0000 7f5bc27fc640 10 client.?.objecter ms_handle_connect 0x7f5bcc12e7f0 2024-01-20T11:18:11.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.807+0000 7f5bc27fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:11.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 1 -- 172.21.15.110:0/1858547262 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc400cd20 con 0x7f5bcc12e7f0 2024-01-20T11:18:11.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:11.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:11.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 10 monclient: dump: 2024-01-20T11:18:11.816 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 1 -- 172.21.15.110:0/1858547262 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc400f040 con 0x7f5bcc12e7f0 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:11.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 1 -- 172.21.15.110:0/1858547262 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4003bd0 con 0x7f5bcc12e7f0 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc27fc640 10 monclient: dump: 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bd4007640 5 monclient: authenticate success, global_id 4786 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bc3fff640 4 set_mon_vals no callback set 2024-01-20T11:18:11.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bd4007640 10 monclient: _renew_subs 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bd4007640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bcc1d28c0 con 0x7f5bcc12e7f0 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bd4007640 10 client.4786.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bd4007640 10 monclient: _renew_subs 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bd4007640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bcc1d84b0 con 0x7f5bcc12e7f0 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.811+0000 7f5bd4007640 1 librados: init done 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.812+0000 7f5bd4007640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.812+0000 7f5bd4007640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.812+0000 7f5bd4007640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:11.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.812+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bcc075490 con 0x7f5bcc12e7f0 2024-01-20T11:18:11.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.831+0000 7f5bc27fc640 1 -- 172.21.15.110:0/1858547262 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5bc4003d30 con 0x7f5bcc12e7f0 2024-01-20T11:18:11.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.831+0000 7f5bc27fc640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5bb006c8e0 0x7f5bb006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:11.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.832+0000 7f5bd1da3640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5bb006c8e0 0x7f5bb006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:11.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.832+0000 7f5bd1da3640 10 monclient: get_auth_request con 0x7f5bb006c8e0 auth_method 0 2024-01-20T11:18:11.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.832+0000 7f5bc27fc640 10 client.4786.objecter ms_handle_connect 0x7f5bb006c8e0 2024-01-20T11:18:11.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:11.832+0000 7f5bd1da3640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5bb006c8e0 0x7f5bb006eda0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8036000 tx=0x7f5bc803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.062+0000 7f5bc27fc640 1 -- 172.21.15.110:0/1858547262 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5bc4059520 con 0x7f5bcc12e7f0 2024-01-20T11:18:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.062+0000 7f5bc27fc640 10 client.4786.objecter ms_dispatch 0x7f5bcc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.062+0000 7f5bc27fc640 3 client.4786.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.062+0000 7f5bc27fc640 3 client.4786.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.063+0000 7f5bc27fc640 20 client.4786.objecter dump_active .. 0 homeless 2024-01-20T11:18:12.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.076+0000 7f5bc27fc640 1 -- 172.21.15.110:0/1858547262 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5bc4055860 con 0x7f5bcc12e7f0 2024-01-20T11:18:12.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.076+0000 7f5bc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:12.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.076+0000 7f5bc27fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:12.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.251+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5bcc0aa2f0 con 0x7f5bb006c8e0 2024-01-20T11:18:12.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.254+0000 7f5bc27fc640 1 -- 172.21.15.110:0/1858547262 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64227 (secure 0 0 0) 0x7f5bcc0aa2f0 con 0x7f5bb006c8e0 2024-01-20T11:18:12.255 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:12.255 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:12.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 10 librados: watch_flush enter 2024-01-20T11:18:12.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 10 librados: watch_flush exit 2024-01-20T11:18:12.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 20 client.4786.objecter shutdown clearing up homeless session... 2024-01-20T11:18:12.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 10 client.4786.objecter successfully canceled tick 2024-01-20T11:18:12.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5bb006c8e0 msgr2=0x7f5bb006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5bb006c8e0 0x7f5bb006eda0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8036000 tx=0x7f5bc803d040 comp rx=0 tx=0).stop 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 10 monclient: shutdown 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bcc12e7f0 msgr2=0x7f5bcc12ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.255+0000 7f5bd4007640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bcc12e7f0 0x7f5bcc12ec40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400efc0 tx=0x7f5bc400c490 comp rx=0 tx=0).stop 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.256+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 shutdown_connections 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.256+0000 7f5bd4007640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5bb006c8e0 0x7f5bb006eda0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.256+0000 7f5bd4007640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bcc12e7f0 0x7f5bcc12ec40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.256+0000 7f5bd4007640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bcc0b0ba0 0x7f5bcc0a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.256+0000 7f5bd4007640 1 --2- 172.21.15.110:0/1858547262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bcc0ac4d0 0x7f5bcc0b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.256+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 >> 172.21.15.110:0/1858547262 conn(0x7f5bcc0b3e60 msgr2=0x7f5bcc139f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.256+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 shutdown_connections 2024-01-20T11:18:12.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.256+0000 7f5bd4007640 1 -- 172.21.15.110:0/1858547262 wait complete. 2024-01-20T11:18:12.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.256+0000 7f5bd4007640 1 librados: shutdown 2024-01-20T11:18:12.267 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-01-20T11:18:10.743950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001945"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.5549999999999997},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.1630000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.061999999999999},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.4979999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.696},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.818}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":17.420999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":13.279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.819000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":10.234}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.300000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":14.381}]}]},{"osd":7,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2270000000000003},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4350000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7289999999999992},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.1639999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.263999999999999},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.599}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.025},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.891}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.583},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":11.301}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.863},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.554}]}]},{"osd":3,"up_from":21,"seq":90194313256,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.7780000000000005},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":4.8810000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":15.708},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.811999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":7.4119999999999999},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.430999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.778},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.138999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":8.0920000000000005},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.522}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":16.5},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.763}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":11.616},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":16.163}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.8460000000000001},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":14.467000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.173},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":17.103999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":7.3869999999999996},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.849}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.1170000000000009},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.5049999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.7230000000000008},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.044}]}]},{"osd":1,"up_from":17,"seq":73014444071,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.596},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":5.8490000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.9000000000000004},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.983000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.769},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.8100000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.4689999999999994},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.555999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.252000000000001},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":7.9009999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.275},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.555}]}]},{"osd":0,"up_from":17,"seq":73014444071,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":8.1310000000000002},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":12.129}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.7519999999999998},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.986000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":10.753},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.9309999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.388},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.542999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":17.184000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":18.251999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.143999999999998},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":12.842000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.7140000000000004},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.4180000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.702999999999999},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":15.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6749999999999998},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.087999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.73},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.0679999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.9329999999999998},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.266999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.678000000000001},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.503}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:12.268 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:12.268 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:12.269 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:12.269 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:12.269 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:12.269 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:12.269 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:12.269 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:12.269 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.366+0000 7f0ff2061640 10 monclient: get_monmap_and_config 2024-01-20T11:18:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.366+0000 7f0ff2061640 10 monclient: build_initial_monmap 2024-01-20T11:18:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.366+0000 7f0ff2061640 10 monclient: monmap: 2024-01-20T11:18:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:12.368085+0000 2024-01-20T11:18:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:12.368085+0000 2024-01-20T11:18:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:12.368 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 1 Processor -- start 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 1 -- start start 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient: init 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec145760 0x7f0fec145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient(hunting): picked mon.noname-b con 0x7f0fec145760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0fec0a9d80 0x7f0fec13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient(hunting): picked mon.noname-c con 0x7f0fec0a9d80 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fec13f6e0 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec146250 con 0x7f0fec13f6e0 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fec0ab050 con 0x7f0fec0a9d80 2024-01-20T11:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec13fb90 con 0x7f0fec145760 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.367+0000 7f0ff2061640 10 monclient(hunting): authenticate will time out at 1535.719056s 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.368+0000 7f0feaffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0fec0a9d80 0x7f0fec13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.368+0000 7f0feb7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec145760 0x7f0fec145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.368+0000 7f0feb7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec145760 0x7f0fec145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:36658/0 (socket says 172.21.15.110:36658) 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.368+0000 7f0feb7fe640 1 -- 172.21.15.110:0/916012123 learned_addr learned my addr 172.21.15.110:0/916012123 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.368+0000 7f0feb7fe640 10 monclient(hunting): get_auth_request con 0x7f0fec145760 auth_method 0 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.368+0000 7f0feb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.368+0000 7f0feb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.368+0000 7f0feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.368+0000 7f0feaffd640 10 monclient(hunting): get_auth_request con 0x7f0fec0a9d80 auth_method 0 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.369+0000 7f0feaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.369+0000 7f0feaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.369+0000 7f0feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.369+0000 7f0feb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.369+0000 7f0feb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.369+0000 7f0feb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.369+0000 7f0feaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.369+0000 7f0feaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.369+0000 7f0feaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.370+0000 7f0feb7fe640 10 monclient(hunting): handle_auth_done global_id 4592 payload 274 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.370+0000 7f0feb7fe640 1 -- 172.21.15.110:0/916012123 >> v2:172.21.15.134:3300/0 conn(0x7f0fec0a9d80 msgr2=0x7f0fec13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.370+0000 7f0feb7fe640 1 --2- 172.21.15.110:0/916012123 >> v2:172.21.15.134:3300/0 conn(0x7f0fec0a9d80 0x7f0fec13f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.370+0000 7f0feb7fe640 1 -- 172.21.15.110:0/916012123 >> v1:172.21.15.134:6789/0 conn(0x7f0fec13f6e0 legacy=0x7f0fec0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.370+0000 7f0feb7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.370+0000 7f0feb7fe640 1 monclient: found mon.noname-b 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.370+0000 7f0feb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.370+0000 7f0feb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.370+0000 7f0feb7fe640 1 -- 172.21.15.110:0/916012123 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec0b20c0 con 0x7f0fec145760 2024-01-20T11:18:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.372+0000 7f0feb7fe640 1 --2- 172.21.15.110:0/916012123 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec145760 0x7f0fec145b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009870 tx=0x7f0fdc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87f09a03ea5ee824 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 1 -- 172.21.15.110:0/916012123 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc030840 con 0x7f0fec145760 2024-01-20T11:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 10 monclient: dump: 2024-01-20T11:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 1 -- 172.21.15.110:0/916012123 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fdc0309a0 con 0x7f0fec145760 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0ff2061640 5 monclient: authenticate success, global_id 4592 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0ff2061640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 1 -- 172.21.15.110:0/916012123 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc030d30 con 0x7f0fec145760 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0fea7fc640 10 monclient: dump: 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0ff2061640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.376+0000 7f0ff2061640 4 set_mon_vals no callback set 2024-01-20T11:18:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 10 monclient: shutdown 2024-01-20T11:18:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 1 -- 172.21.15.110:0/916012123 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec145760 msgr2=0x7f0fec145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 1 --2- 172.21.15.110:0/916012123 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec145760 0x7f0fec145b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009870 tx=0x7f0fdc02b980 comp rx=0 tx=0).stop 2024-01-20T11:18:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 1 -- 172.21.15.110:0/916012123 shutdown_connections 2024-01-20T11:18:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 1 --2- 172.21.15.110:0/916012123 >> v2:172.21.15.134:3300/0 conn(0x7f0fec0a9d80 0x7f0fec13f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 1 --2- 172.21.15.110:0/916012123 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec145760 0x7f0fec145b40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 1 -- 172.21.15.110:0/916012123 >> 172.21.15.110:0/916012123 conn(0x7f0fec0663b0 msgr2=0x7f0fec0667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 1 -- 172.21.15.110:0/916012123 shutdown_connections 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.377+0000 7f0ff2061640 1 -- 172.21.15.110:0/916012123 wait complete. 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.378+0000 7f0ff2061640 10 monclient: build_initial_monmap 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.378+0000 7f0ff2061640 10 monclient: monmap: 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:12.384 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.378+0000 7f0ff2061640 1 librados: starting msgr at 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.378+0000 7f0ff2061640 1 librados: starting objecter 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.378+0000 7f0ff2061640 1 Processor -- start 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.378+0000 7f0ff2061640 1 -- start start 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.378+0000 7f0ff2061640 1 librados: setting wanted keys 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.378+0000 7f0ff2061640 1 librados: calling monclient init 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.378+0000 7f0ff2061640 10 monclient: init 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fec0a9d80 0x7f0fec0b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient(hunting): picked mon.noname-a con 0x7f0fec0a9d80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec13f6e0 0x7f0fec139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient(hunting): picked mon.noname-c con 0x7f0fec13f6e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fec145760 0x7f0fec0b7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient(hunting): picked mon.noname-b con 0x7f0fec145760 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec13a680 con 0x7f0fec0a9d80 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec1409f0 con 0x7f0fec145760 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec1da970 con 0x7f0fec13f6e0 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0ff2061640 10 monclient(hunting): authenticate will time out at 1535.730541s 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0feaffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec13f6e0 0x7f0fec139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0febfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fec145760 0x7f0fec0b7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.379+0000 7f0feb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fec0a9d80 0x7f0fec0b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feaffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec13f6e0 0x7f0fec139ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:36662/0 (socket says 172.21.15.110:36662) 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0febfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fec145760 0x7f0fec0b7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49740/0 (socket says 172.21.15.110:49740) 2024-01-20T11:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feaffd640 1 -- 172.21.15.110:0/2291698858 learned_addr learned my addr 172.21.15.110:0/2291698858 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feaffd640 10 monclient(hunting): get_auth_request con 0x7f0fec13f6e0 auth_method 0 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0febfff640 10 monclient(hunting): get_auth_request con 0x7f0fec145760 auth_method 0 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0febfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0febfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0febfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feb7fe640 10 monclient(hunting): get_auth_request con 0x7f0fec0a9d80 auth_method 0 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0febfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0febfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0febfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.380+0000 7f0feb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:12.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 10 monclient(hunting): handle_auth_done global_id 4792 payload 824 2024-01-20T11:18:12.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 1 -- 172.21.15.110:0/2291698858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec13f6e0 msgr2=0x7f0fec139ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:12.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec13f6e0 0x7f0fec139ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 1 -- 172.21.15.110:0/2291698858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fec0a9d80 msgr2=0x7f0fec0b72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:12.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fec0a9d80 0x7f0fec0b72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 1 monclient: found mon.noname-b 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.383+0000 7f0febfff640 1 -- 172.21.15.110:0/2291698858 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc0094d0 con 0x7f0fec145760 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.385+0000 7f0febfff640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fec145760 0x7f0fec0b7bc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400c730 tx=0x7f0fe400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.385+0000 7f0fd3fff640 10 client.?.objecter ms_handle_connect 0x7f0fec145760 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.385+0000 7f0fd3fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 1 -- 172.21.15.110:0/2291698858 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4006d50 con 0x7f0fec145760 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 10 monclient: dump: 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:12.391 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 10 monclient: _finish_auth 0 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 10 monclient: _check_auth_tickets 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 1 -- 172.21.15.110:0/2291698858 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe4003970 con 0x7f0fec145760 2024-01-20T11:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 1 -- 172.21.15.110:0/2291698858 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4003d60 con 0x7f0fec145760 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fd3fff640 10 monclient: dump: 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0ff2061640 5 monclient: authenticate success, global_id 4792 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0ff2061640 10 monclient: _renew_subs 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0ff2061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec13fd70 con 0x7f0fec145760 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.388+0000 7f0fe9ffb640 4 set_mon_vals no callback set 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.389+0000 7f0ff2061640 10 client.4792.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.389+0000 7f0ff2061640 10 monclient: _renew_subs 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.389+0000 7f0ff2061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.389+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec0b82e0 con 0x7f0fec145760 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.389+0000 7f0ff2061640 1 librados: init done 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.389+0000 7f0ff2061640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.389+0000 7f0ff2061640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.389+0000 7f0ff2061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.389+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec13ff90 con 0x7f0fec145760 2024-01-20T11:18:12.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.407+0000 7f0fd3fff640 1 -- 172.21.15.110:0/2291698858 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f0fe40052e0 con 0x7f0fec145760 2024-01-20T11:18:12.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.408+0000 7f0fd3fff640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0fd806c890 0x7f0fd806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:12.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.408+0000 7f0feb7fe640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0fd806c890 0x7f0fd806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:12.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.408+0000 7f0feb7fe640 10 monclient: get_auth_request con 0x7f0fd806c890 auth_method 0 2024-01-20T11:18:12.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.408+0000 7f0fd3fff640 10 client.4792.objecter ms_handle_connect 0x7f0fd806c890 2024-01-20T11:18:12.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.408+0000 7f0feb7fe640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0fd806c890 0x7f0fd806ed50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc036000 tx=0x7f0fdc008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:12.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.411+0000 7f0fd3fff640 1 -- 172.21.15.110:0/2291698858 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0fe4058c00 con 0x7f0fec145760 2024-01-20T11:18:12.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.411+0000 7f0fd3fff640 10 client.4792.objecter ms_dispatch 0x7f0fec000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:12.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.411+0000 7f0fd3fff640 3 client.4792.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:12.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.411+0000 7f0fd3fff640 3 client.4792.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:12.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.411+0000 7f0fd3fff640 20 client.4792.objecter dump_active .. 0 homeless 2024-01-20T11:18:12.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.666+0000 7f0fd3fff640 1 -- 172.21.15.110:0/2291698858 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0fe4054f40 con 0x7f0fec145760 2024-01-20T11:18:12.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.666+0000 7f0fd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:12.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.666+0000 7f0fd3fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:12.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.903+0000 7f0ff2061640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:12.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.903+0000 7f0ff2061640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:12.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.903+0000 7f0ff2061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:12.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.903+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f0fec0b78c0 con 0x7f0fec145760 2024-01-20T11:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.912+0000 7f0fd3fff640 1 -- 172.21.15.110:0/2291698858 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f0fe4058560 con 0x7f0fec145760 2024-01-20T11:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.912+0000 7f0fd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.912+0000 7f0fd3fff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:12.912 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:12.912 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":227,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 10 librados: watch_flush enter 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 10 librados: watch_flush exit 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 20 client.4792.objecter shutdown clearing up homeless session... 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 10 client.4792.objecter successfully canceled tick 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0fd806c890 msgr2=0x7f0fd806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0fd806c890 0x7f0fd806ed50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc036000 tx=0x7f0fdc008030 comp rx=0 tx=0).stop 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 10 monclient: shutdown 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fec145760 msgr2=0x7f0fec0b7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fec145760 0x7f0fec0b7bc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400c730 tx=0x7f0fe400cbf0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 shutdown_connections 2024-01-20T11:18:12.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0fd806c890 0x7f0fd806ed50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0fec145760 0x7f0fec0b7bc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fec13f6e0 0x7f0fec139ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 --2- 172.21.15.110:0/2291698858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fec0a9d80 0x7f0fec0b72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:12.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 >> 172.21.15.110:0/2291698858 conn(0x7f0fec0663b0 msgr2=0x7f0fec1376b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:12.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 shutdown_connections 2024-01-20T11:18:12.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 -- 172.21.15.110:0/2291698858 wait complete. 2024-01-20T11:18:12.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:12.913+0000 7f0ff2061640 1 librados: shutdown 2024-01-20T11:18:12.925 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:12.925 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:13.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.023+0000 7f3fd67c0640 10 monclient: get_monmap_and_config 2024-01-20T11:18:13.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.023+0000 7f3fd67c0640 10 monclient: build_initial_monmap 2024-01-20T11:18:13.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.023+0000 7f3fd67c0640 10 monclient: monmap: 2024-01-20T11:18:13.024 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:13.024748+0000 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:13.024748+0000 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.023+0000 7f3fd67c0640 1 Processor -- start 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.023+0000 7f3fd67c0640 1 -- start start 2024-01-20T11:18:13.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.023+0000 7f3fd67c0640 10 monclient: init 2024-01-20T11:18:13.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:13.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-01-20T11:18:13.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3fd00a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:13.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3fd00a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:13.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd0142af0 0x7f3fd014d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:13.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 10 monclient(hunting): picked mon.noname-b con 0x7f3fd0142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:13.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd00b25c0 con 0x7f3fd00a9ad0 2024-01-20T11:18:13.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd0142f50 con 0x7f3fd00a96f0 2024-01-20T11:18:13.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:13.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd00aaea0 con 0x7f3fd0142af0 2024-01-20T11:18:13.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:13.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.024+0000 7f3fd67c0640 10 monclient(hunting): authenticate will time out at 1536.375748s 2024-01-20T11:18:13.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.025+0000 7f3fcffff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f3fd00a96f0 0x7f3fd013f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:45732/0 (socket says 172.21.15.110:45732) 2024-01-20T11:18:13.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.025+0000 7f3fcffff640 1 -- 172.21.15.110:0/1315075893 learned_addr learned my addr 172.21.15.110:0/1315075893 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:13.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.025+0000 7f3fd4d5d640 1 --2- 172.21.15.110:0/1315075893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd0142af0 0x7f3fd014d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:13.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.025+0000 7f3fd4d5d640 10 monclient(hunting): get_auth_request con 0x7f3fd0142af0 auth_method 0 2024-01-20T11:18:13.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.025+0000 7f3fd4d5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:13.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.025+0000 7f3fd4d5d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:13.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.025+0000 7f3fd4d5d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:13.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.026+0000 7f3fd4d5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:13.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.026+0000 7f3fd4d5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:13.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.026+0000 7f3fd4d5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:13.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 1 -- 172.21.15.110:0/1315075893 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311660621 0 0) 0x7f3fc0001110 con 0x7f3fd00a96f0 2024-01-20T11:18:13.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 10 monclient(hunting): _negotiate 2024-01-20T11:18:13.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:13.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:13.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 10 monclient(hunting): my global_id is 4795 2024-01-20T11:18:13.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 1 -- 172.21.15.110:0/1315075893 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb8003520 con 0x7f3fd00a96f0 2024-01-20T11:18:13.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 1 -- 172.21.15.110:0/1315075893 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583802552 0 0) 0x7f3fd00b25c0 con 0x7f3fd00a9ad0 2024-01-20T11:18:13.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 10 monclient(hunting): _negotiate 2024-01-20T11:18:13.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 10 monclient(hunting): my global_id is 4710 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.027+0000 7f3fceffd640 1 -- 172.21.15.110:0/1315075893 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc0001110 con 0x7f3fd00a9ad0 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.028+0000 7f3fd4d5d640 10 monclient(hunting): handle_auth_done global_id 4598 payload 274 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.028+0000 7f3fd4d5d640 1 -- 172.21.15.110:0/1315075893 >> v1:172.21.15.134:6789/0 conn(0x7f3fd00a96f0 legacy=0x7f3fd013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.028+0000 7f3fd4d5d640 1 -- 172.21.15.110:0/1315075893 >> v1:172.21.15.110:6789/0 conn(0x7f3fd00a9ad0 legacy=0x7f3fd01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.028+0000 7f3fd4d5d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.028+0000 7f3fd4d5d640 1 monclient: found mon.noname-b 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.028+0000 7f3fd4d5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.028+0000 7f3fd4d5d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.028+0000 7f3fd4d5d640 1 -- 172.21.15.110:0/1315075893 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd013fbe0 con 0x7f3fd0142af0 2024-01-20T11:18:13.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.029+0000 7f3fd4d5d640 1 --2- 172.21.15.110:0/1315075893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd0142af0 0x7f3fd014d860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009dc0 tx=0x7f3fc802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=978421f8e996a3f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:13.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 1 -- 172.21.15.110:0/1315075893 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc802cc40 con 0x7f3fd0142af0 2024-01-20T11:18:13.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:13.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:13.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 10 monclient: dump: 2024-01-20T11:18:13.038 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 1 -- 172.21.15.110:0/1315075893 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc802cda0 con 0x7f3fd0142af0 2024-01-20T11:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 1 -- 172.21.15.110:0/1315075893 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8034660 con 0x7f3fd0142af0 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fd67c0640 5 monclient: authenticate success, global_id 4598 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fd67c0640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fd67c0640 4 set_mon_vals no callback set 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.033+0000 7f3fceffd640 10 monclient: dump: 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 10 monclient: shutdown 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 1 -- 172.21.15.110:0/1315075893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd0142af0 msgr2=0x7f3fd014d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 1 --2- 172.21.15.110:0/1315075893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd0142af0 0x7f3fd014d860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009dc0 tx=0x7f3fc802be90 comp rx=0 tx=0).stop 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 1 -- 172.21.15.110:0/1315075893 shutdown_connections 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 1 --2- 172.21.15.110:0/1315075893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd0142af0 0x7f3fd014d860 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 1 -- 172.21.15.110:0/1315075893 >> 172.21.15.110:0/1315075893 conn(0x7f3fd00b7bc0 msgr2=0x7f3fd00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 1 -- 172.21.15.110:0/1315075893 shutdown_connections 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 1 -- 172.21.15.110:0/1315075893 wait complete. 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 10 monclient: build_initial_monmap 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.034+0000 7f3fd67c0640 10 monclient: monmap: 2024-01-20T11:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.035+0000 7f3fd67c0640 1 librados: starting msgr at 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.035+0000 7f3fd67c0640 1 librados: starting objecter 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.035+0000 7f3fd67c0640 1 Processor -- start 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.035+0000 7f3fd67c0640 1 -- start start 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.035+0000 7f3fd67c0640 1 librados: setting wanted keys 2024-01-20T11:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.035+0000 7f3fd67c0640 1 librados: calling monclient init 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.035+0000 7f3fd67c0640 10 monclient: init 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fd00a96f0 0x7f3fd013e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient(hunting): picked mon.noname-b con 0x7f3fd00a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd00a9ad0 0x7f3fd0135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient(hunting): picked mon.noname-c con 0x7f3fd00a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fd0142af0 0x7f3fd01d7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient(hunting): picked mon.noname-a con 0x7f3fd0142af0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd01359b0 con 0x7f3fd0142af0 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:13.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd0135670 con 0x7f3fd00a96f0 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd0134700 con 0x7f3fd00a9ad0 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd67c0640 10 monclient(hunting): authenticate will time out at 1536.387348s 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd00a9ad0 0x7f3fd0135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fd4d5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fd0142af0 0x7f3fd01d7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fcffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fd00a96f0 0x7f3fd013e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd00a9ad0 0x7f3fd0135130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:36690/0 (socket says 172.21.15.110:36690) 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fcf7fe640 1 -- 172.21.15.110:0/708616783 learned_addr learned my addr 172.21.15.110:0/708616783 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fcf7fe640 10 monclient(hunting): get_auth_request con 0x7f3fd00a9ad0 auth_method 0 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.036+0000 7f3fcffff640 10 monclient(hunting): get_auth_request con 0x7f3fd00a96f0 auth_method 0 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fcffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fd4d5d640 10 monclient(hunting): get_auth_request con 0x7f3fd0142af0 auth_method 0 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fd4d5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fd4d5d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fd4d5d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fd4d5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:13.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fd4d5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:13.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.037+0000 7f3fd4d5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.039+0000 7f3fcffff640 10 monclient(hunting): handle_auth_done global_id 4798 payload 824 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.039+0000 7f3fcffff640 1 -- 172.21.15.110:0/708616783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd00a9ad0 msgr2=0x7f3fd0135130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.039+0000 7f3fcffff640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd00a9ad0 0x7f3fd0135130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.039+0000 7f3fcffff640 1 -- 172.21.15.110:0/708616783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fd0142af0 msgr2=0x7f3fd01d7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.039+0000 7f3fcffff640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fd0142af0 0x7f3fd01d7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.039+0000 7f3fcffff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.039+0000 7f3fcffff640 1 monclient: found mon.noname-b 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.040+0000 7f3fcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.040+0000 7f3fcffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:13.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.040+0000 7f3fcffff640 1 -- 172.21.15.110:0/708616783 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc8009a70 con 0x7f3fd00a96f0 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.042+0000 7f3fcffff640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fd00a96f0 0x7f3fd013e320 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3fc000c700 tx=0x7f3fc000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.042+0000 7f3fccff9640 10 client.?.objecter ms_handle_connect 0x7f3fd00a96f0 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.042+0000 7f3fccff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fccff9640 1 -- 172.21.15.110:0/708616783 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0005ab0 con 0x7f3fd00a96f0 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fccff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fccff9640 10 monclient: dump: 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:13.049 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fccff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fccff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fccff9640 1 -- 172.21.15.110:0/708616783 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc0005c10 con 0x7f3fd00a96f0 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fd67c0640 5 monclient: authenticate success, global_id 4798 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fd67c0640 10 monclient: _renew_subs 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fd67c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.045+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd0134930 con 0x7f3fd00a96f0 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fccff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fccff9640 1 -- 172.21.15.110:0/708616783 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0007070 con 0x7f3fd00a96f0 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:13.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fccff9640 10 monclient: dump: 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fce7fc640 4 set_mon_vals no callback set 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fd67c0640 10 client.4798.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fd67c0640 10 monclient: _renew_subs 2024-01-20T11:18:13.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fd67c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:13.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd01d7880 con 0x7f3fd00a96f0 2024-01-20T11:18:13.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fd67c0640 1 librados: init done 2024-01-20T11:18:13.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fd67c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:13.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fd67c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:13.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fd67c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:13.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.046+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd0075490 con 0x7f3fd00a96f0 2024-01-20T11:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.293+0000 7f3fccff9640 1 -- 172.21.15.110:0/708616783 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3fc00071d0 con 0x7f3fd00a96f0 2024-01-20T11:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.294+0000 7f3fccff9640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3fbc06c8b0 0x7f3fbc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.294+0000 7f3fcf7fe640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3fbc06c8b0 0x7f3fbc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.294+0000 7f3fcf7fe640 10 monclient: get_auth_request con 0x7f3fbc06c8b0 auth_method 0 2024-01-20T11:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.294+0000 7f3fcf7fe640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3fbc06c8b0 0x7f3fbc06ed70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00b71e0 tx=0x7f3fc4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.294+0000 7f3fccff9640 10 client.4798.objecter ms_handle_connect 0x7f3fbc06c8b0 2024-01-20T11:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.299+0000 7f3fccff9640 1 -- 172.21.15.110:0/708616783 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3fc0057f00 con 0x7f3fd00a96f0 2024-01-20T11:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.299+0000 7f3fccff9640 10 client.4798.objecter ms_dispatch 0x7f3fd0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.299+0000 7f3fccff9640 3 client.4798.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.299+0000 7f3fccff9640 3 client.4798.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:13.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.299+0000 7f3fccff9640 20 client.4798.objecter dump_active .. 0 homeless 2024-01-20T11:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.309+0000 7f3fccff9640 1 -- 172.21.15.110:0/708616783 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f3fc0054240 con 0x7f3fd00a96f0 2024-01-20T11:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.309+0000 7f3fccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.309+0000 7f3fccff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:13.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.493+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3fd00b4400 con 0x7f3fbc06c8b0 2024-01-20T11:18:13.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.495+0000 7f3fccff9640 1 -- 172.21.15.110:0/708616783 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64215 (secure 0 0 0) 0x7f3fc4001510 con 0x7f3fbc06c8b0 2024-01-20T11:18:13.497 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:13.497 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:13.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 10 librados: watch_flush enter 2024-01-20T11:18:13.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 10 librados: watch_flush exit 2024-01-20T11:18:13.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 20 client.4798.objecter shutdown clearing up homeless session... 2024-01-20T11:18:13.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 10 client.4798.objecter successfully canceled tick 2024-01-20T11:18:13.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3fbc06c8b0 msgr2=0x7f3fbc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3fbc06c8b0 0x7f3fbc06ed70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00b71e0 tx=0x7f3fc4009040 comp rx=0 tx=0).stop 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 10 monclient: shutdown 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fd00a96f0 msgr2=0x7f3fd013e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fd00a96f0 0x7f3fd013e320 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3fc000c700 tx=0x7f3fc000cbc0 comp rx=0 tx=0).stop 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 shutdown_connections 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3fbc06c8b0 0x7f3fbc06ed70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fd0142af0 0x7f3fd01d7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3fd00a9ad0 0x7f3fd0135130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 --2- 172.21.15.110:0/708616783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fd00a96f0 0x7f3fd013e320 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:13.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 >> 172.21.15.110:0/708616783 conn(0x7f3fd00b7bc0 msgr2=0x7f3fd00b68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:13.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.497+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 shutdown_connections 2024-01-20T11:18:13.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.498+0000 7f3fd67c0640 1 -- 172.21.15.110:0/708616783 wait complete. 2024-01-20T11:18:13.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:13.498+0000 7f3fd67c0640 1 librados: shutdown 2024-01-20T11:18:13.509 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-01-20T11:18:12.744271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001950"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.5549999999999997},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.1630000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.061999999999999},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.4979999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.696},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.818}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":17.420999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":13.279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.819000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":10.234}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.300000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":14.381}]}]},{"osd":7,"up_from":21,"seq":90194313256,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":13.869},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4109999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.0229999999999997},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.4670000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":14.795999999999999},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":16.251000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":5.8129999999999997},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":10.430999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.27},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":16.835000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":17.899000000000001},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.817}]}]},{"osd":3,"up_from":21,"seq":90194313257,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":10.746},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.2279999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.718},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.0430000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":13.398},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.103}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.109},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":8.4749999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":9.0830000000000002},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.097}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":16.213000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.417999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":11.616},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":16.163}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.8460000000000001},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":14.467000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.173},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":17.103999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":7.3869999999999996},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.849}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.1170000000000009},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.5049999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.7230000000000008},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.044}]}]},{"osd":1,"up_from":17,"seq":73014444071,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.596},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":5.8490000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.9000000000000004},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.983000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.769},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.8100000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.4689999999999994},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.555999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.252000000000001},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":7.9009999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.275},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.555}]}]},{"osd":0,"up_from":17,"seq":73014444071,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":8.1310000000000002},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":12.129}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.7519999999999998},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.986000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":10.753},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.9309999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.388},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.542999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":17.184000000000001},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":18.251999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.143999999999998},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":12.842000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509480,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":8.8930000000000007,"5min":11.202999999999999,"15min":11.587999999999999},"min":{"1min":4.3399999999999999,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":16.233000000000001,"5min":106.059,"15min":106.059},"last":5.9749999999999996},{"interface":"front","average":{"1min":7.3890000000000002,"5min":13.455,"15min":14.465999999999999},"min":{"1min":3.96,"5min":3.96,"15min":3.96},"max":{"1min":12.115,"5min":157.67699999999999,"15min":157.67699999999999},"last":13.644}]},{"osd":1,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.5190000000000001,"5min":26.353999999999999,"15min":29.16},"min":{"1min":6.3840000000000003,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.199999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.6390000000000002},{"interface":"front","average":{"1min":11.776999999999999,"5min":27.742000000000001,"15min":30.402999999999999},"min":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":14.554,"5min":190.304,"15min":190.304},"last":11.75}]},{"osd":2,"last update":"Sat Jan 20 11:17:13 2024","interfaces":[{"interface":"back","average":{"1min":9.7550000000000008,"5min":14.068,"15min":14.787000000000001},"min":{"1min":5.6369999999999996,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":16.524000000000001,"5min":128.67099999999999,"15min":128.67099999999999},"last":6.5819999999999999},{"interface":"front","average":{"1min":12.553000000000001,"5min":15.445,"15min":15.927},"min":{"1min":7.1390000000000002,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":16.588999999999999,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.82}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":9.5459999999999994},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":15.401}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.373000000000001},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.8420000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.461},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.372}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:16.512 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:16.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.611+0000 7f384359e640 10 monclient: get_monmap_and_config 2024-01-20T11:18:16.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.611+0000 7f384359e640 10 monclient: build_initial_monmap 2024-01-20T11:18:16.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.611+0000 7f384359e640 10 monclient: monmap: 2024-01-20T11:18:16.612 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:16.613187+0000 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:16.613187+0000 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:16.613 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.612+0000 7f384359e640 1 Processor -- start 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.612+0000 7f384359e640 1 -- start start 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.612+0000 7f384359e640 10 monclient: init 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f384406c580 0x7f38440b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 10 monclient(hunting): picked mon.noname-c con 0x7f384406c580 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f384413ef70 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3844145290 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440b3b70 con 0x7f3844145290 2024-01-20T11:18:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440aaac0 con 0x7f384413ef70 2024-01-20T11:18:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384406c960 con 0x7f384406c580 2024-01-20T11:18:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.613+0000 7f384359e640 10 monclient(hunting): authenticate will time out at 1539.964549s 2024-01-20T11:18:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.614+0000 7f3842d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3844145290 0x7f38441456d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53988/0 (socket says 172.21.15.110:53988) 2024-01-20T11:18:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.614+0000 7f3841d9b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f384413ef70 0x7f384413f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:45740/0 (socket says 172.21.15.110:45740) 2024-01-20T11:18:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.614+0000 7f3842d9d640 1 -- 172.21.15.110:0/2927858816 learned_addr learned my addr 172.21.15.110:0/2927858816 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.615+0000 7f384259c640 1 --2- 172.21.15.110:0/2927858816 >> v2:172.21.15.134:3300/0 conn(0x7f384406c580 0x7f38440b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.615+0000 7f384259c640 10 monclient(hunting): get_auth_request con 0x7f384406c580 auth_method 0 2024-01-20T11:18:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.615+0000 7f384259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.615+0000 7f384259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.615+0000 7f384259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.616+0000 7f384159a640 1 -- 172.21.15.110:0/2927858816 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870537235 0 0) 0x7f38440b3b70 con 0x7f3844145290 2024-01-20T11:18:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.616+0000 7f384159a640 10 monclient(hunting): _negotiate 2024-01-20T11:18:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.616+0000 7f384159a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.616+0000 7f384159a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.616+0000 7f384159a640 10 monclient(hunting): my global_id is 4716 2024-01-20T11:18:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.616+0000 7f384159a640 1 -- 172.21.15.110:0/2927858816 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382c003520 con 0x7f3844145290 2024-01-20T11:18:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.617+0000 7f384259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.617+0000 7f384259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.617+0000 7f384259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.619+0000 7f384259c640 10 monclient(hunting): handle_auth_done global_id 4804 payload 274 2024-01-20T11:18:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.619+0000 7f384259c640 1 -- 172.21.15.110:0/2927858816 >> v1:172.21.15.134:6789/0 conn(0x7f384413ef70 legacy=0x7f384413f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.619+0000 7f384259c640 1 -- 172.21.15.110:0/2927858816 >> v1:172.21.15.110:6789/0 conn(0x7f3844145290 legacy=0x7f38441456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.619+0000 7f384259c640 10 monclient: _finish_hunting 0 2024-01-20T11:18:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.619+0000 7f384259c640 1 monclient: found mon.noname-c 2024-01-20T11:18:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.619+0000 7f384259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.619+0000 7f384259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.619+0000 7f384259c640 1 -- 172.21.15.110:0/2927858816 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38440a9b30 con 0x7f384406c580 2024-01-20T11:18:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.623+0000 7f384259c640 1 --2- 172.21.15.110:0/2927858816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f384406c580 0x7f38440b3410 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f38340097a0 tx=0x7f383402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec114d64979e90d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.626+0000 7f384159a640 1 -- 172.21.15.110:0/2927858816 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834030610 con 0x7f384406c580 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.626+0000 7f384159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 10 monclient: dump: 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 10 monclient: _finish_auth 0 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 10 monclient: _check_auth_tickets 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 1 -- 172.21.15.110:0/2927858816 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3834030770 con 0x7f384406c580 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 5 monclient: authenticate success, global_id 4804 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 1 -- 172.21.15.110:0/2927858816 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834030b00 con 0x7f384406c580 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 4 set_mon_vals no callback set 2024-01-20T11:18:16.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384159a640 10 monclient: dump: 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:16.630 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 10 monclient: shutdown 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 1 -- 172.21.15.110:0/2927858816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f384406c580 msgr2=0x7f38440b3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 1 --2- 172.21.15.110:0/2927858816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f384406c580 0x7f38440b3410 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f38340097a0 tx=0x7f383402b960 comp rx=0 tx=0).stop 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 1 -- 172.21.15.110:0/2927858816 shutdown_connections 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 1 --2- 172.21.15.110:0/2927858816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f384406c580 0x7f38440b3410 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.627+0000 7f384359e640 1 -- 172.21.15.110:0/2927858816 >> 172.21.15.110:0/2927858816 conn(0x7f38440b7bf0 msgr2=0x7f38440b8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.628+0000 7f384359e640 1 -- 172.21.15.110:0/2927858816 shutdown_connections 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.628+0000 7f384359e640 1 -- 172.21.15.110:0/2927858816 wait complete. 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.628+0000 7f384359e640 10 monclient: build_initial_monmap 2024-01-20T11:18:16.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.628+0000 7f384359e640 10 monclient: monmap: 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 librados: starting msgr at 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 librados: starting objecter 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 Processor -- start 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 -- start start 2024-01-20T11:18:16.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 librados: setting wanted keys 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 librados: calling monclient init 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient: init 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384406c580 0x7f38441398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient(hunting): picked mon.noname-a con 0x7f384406c580 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384413ef70 0x7f38440ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient(hunting): picked mon.noname-c con 0x7f384413ef70 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3844145290 0x7f38441367f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient(hunting): picked mon.noname-b con 0x7f3844145290 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384413ac10 con 0x7f384406c580 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:16.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440ab790 con 0x7f3844145290 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384413adf0 con 0x7f384413ef70 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.629+0000 7f384359e640 10 monclient(hunting): authenticate will time out at 1539.980947s 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f384259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384406c580 0x7f38441398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3842d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3844145290 0x7f38441367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3841d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384413ef70 0x7f38440ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3842d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3844145290 0x7f38441367f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49776/0 (socket says 172.21.15.110:49776) 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3842d9d640 1 -- 172.21.15.110:0/3685106102 learned_addr learned my addr 172.21.15.110:0/3685106102 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3842d9d640 10 monclient(hunting): get_auth_request con 0x7f3844145290 auth_method 0 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3842d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3842d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3842d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3841d9b640 10 monclient(hunting): get_auth_request con 0x7f384413ef70 auth_method 0 2024-01-20T11:18:16.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3841d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3841d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f3841d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f384259c640 10 monclient(hunting): get_auth_request con 0x7f384406c580 auth_method 0 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f384259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f384259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.630+0000 7f384259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.631+0000 7f3841d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.631+0000 7f3841d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.631+0000 7f3841d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.631+0000 7f384259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.631+0000 7f384259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.631+0000 7f384259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.631+0000 7f3842d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:16.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.631+0000 7f3842d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:16.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.631+0000 7f3842d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:16.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 10 monclient(hunting): handle_auth_done global_id 4604 payload 824 2024-01-20T11:18:16.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 1 -- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3844145290 msgr2=0x7f38441367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:16.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3844145290 0x7f38441367f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 1 -- 172.21.15.110:0/3685106102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384406c580 msgr2=0x7f38441398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384406c580 0x7f38441398d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 1 monclient: found mon.noname-c 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3841d9b640 1 -- 172.21.15.110:0/3685106102 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3834009400 con 0x7f384413ef70 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.633+0000 7f3842d9d640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3844145290 0x7f38441367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.635+0000 7f3841d9b640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384413ef70 0x7f38440ab250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3838000f60 tx=0x7f383800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.635+0000 7f3832ffd640 10 client.?.objecter ms_handle_connect 0x7f384413ef70 2024-01-20T11:18:16.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.635+0000 7f3832ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:16.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 1 -- 172.21.15.110:0/3685106102 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380074e0 con 0x7f384413ef70 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 10 monclient: dump: 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:16.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 1 -- 172.21.15.110:0/3685106102 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3838007660 con 0x7f384413ef70 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 1 -- 172.21.15.110:0/3685106102 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838004a20 con 0x7f384413ef70 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3832ffd640 10 monclient: dump: 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:16.643 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f384359e640 5 monclient: authenticate success, global_id 4604 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f384359e640 10 monclient: _renew_subs 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f384359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f3840d99640 4 set_mon_vals no callback set 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384413b3b0 con 0x7f384413ef70 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f384359e640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f384359e640 10 monclient: _renew_subs 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f384359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3844138ce0 con 0x7f384413ef70 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.639+0000 7f384359e640 1 librados: init done 2024-01-20T11:18:16.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.640+0000 7f384359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:16.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.640+0000 7f384359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:16.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.640+0000 7f384359e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:18:16.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.640+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384413a180 con 0x7f384413ef70 2024-01-20T11:18:16.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.657+0000 7f3832ffd640 1 -- 172.21.15.110:0/3685106102 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3838004b80 con 0x7f384413ef70 2024-01-20T11:18:16.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.658+0000 7f3832ffd640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f382406c860 0x7f382406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:16.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.658+0000 7f384259c640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f382406c860 0x7f382406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:16.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.658+0000 7f384259c640 10 monclient: get_auth_request con 0x7f382406c860 auth_method 0 2024-01-20T11:18:16.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.658+0000 7f384259c640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f382406c860 0x7f382406ed20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3834036000 tx=0x7f3834008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:16.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.658+0000 7f3832ffd640 10 client.4604.objecter ms_handle_connect 0x7f382406c860 2024-01-20T11:18:16.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.660+0000 7f3832ffd640 1 -- 172.21.15.110:0/3685106102 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3838058090 con 0x7f384413ef70 2024-01-20T11:18:16.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.660+0000 7f3832ffd640 10 client.4604.objecter ms_dispatch 0x7f3844000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:16.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.660+0000 7f3832ffd640 3 client.4604.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:16.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.660+0000 7f3832ffd640 3 client.4604.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:16.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.660+0000 7f3832ffd640 20 client.4604.objecter dump_active .. 0 homeless 2024-01-20T11:18:16.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.895+0000 7f3832ffd640 1 -- 172.21.15.110:0/3685106102 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f38380543d0 con 0x7f384413ef70 2024-01-20T11:18:16.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.895+0000 7f3832ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:16.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:16.895+0000 7f3832ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.063+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f384413a180 con 0x7f382406c860 2024-01-20T11:18:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.066+0000 7f3832ffd640 1 -- 172.21.15.110:0/3685106102 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64242 (secure 0 0 0) 0x7f384413a180 con 0x7f382406c860 2024-01-20T11:18:17.068 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.067+0000 7f384359e640 10 librados: watch_flush enter 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.067+0000 7f384359e640 10 librados: watch_flush exit 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.067+0000 7f384359e640 20 client.4604.objecter shutdown clearing up homeless session... 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.067+0000 7f384359e640 10 client.4604.objecter successfully canceled tick 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f382406c860 msgr2=0x7f382406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f382406c860 0x7f382406ed20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3834036000 tx=0x7f3834008030 comp rx=0 tx=0).stop 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 10 monclient: shutdown 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384413ef70 msgr2=0x7f38440ab250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384413ef70 0x7f38440ab250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3838000f60 tx=0x7f383800c800 comp rx=0 tx=0).stop 2024-01-20T11:18:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 shutdown_connections 2024-01-20T11:18:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f382406c860 0x7f382406ed20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3844145290 0x7f38441367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384413ef70 0x7f38440ab250 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 --2- 172.21.15.110:0/3685106102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384406c580 0x7f38441398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 >> 172.21.15.110:0/3685106102 conn(0x7f38440b7bf0 msgr2=0x7f38440b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 shutdown_connections 2024-01-20T11:18:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 -- 172.21.15.110:0/3685106102 wait complete. 2024-01-20T11:18:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.068+0000 7f384359e640 1 librados: shutdown 2024-01-20T11:18:17.079 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-01-20T11:18:16.744949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001977"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280554,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":11.423},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":4.298}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":15.750999999999999},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.3360000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.224},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.829000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.1349999999999998},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":9.6069999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.002000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":12.031000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.0579999999999998},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":13.419}]}]},{"osd":7,"up_from":21,"seq":90194313257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":11.179},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4339999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.0469999999999997},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.592000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":9.8510000000000009},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.2030000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.1310000000000002},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.781000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.472},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":14.180999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.726000000000001},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.805999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313258,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1829999999999998},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.7560000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.246},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.5510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":9.8260000000000005},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.137}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.914999999999999},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.4399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.48},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":14.941000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.715999999999999},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":8.8889999999999993}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444071,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":11.616},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":16.163}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.8460000000000001},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":14.467000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.173},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":17.103999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":7.3869999999999996},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.849}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.1170000000000009},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.5049999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.7230000000000008},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.044}]}]},{"osd":1,"up_from":17,"seq":73014444072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.596},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":5.8490000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.9000000000000004},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.983000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.769},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.8100000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.4689999999999994},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.555999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.252000000000001},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":7.9009999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.275},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.555}]}]},{"osd":0,"up_from":17,"seq":73014444072,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":5.2009999999999996},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.3940000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.0099999999999998},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.449999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":9.4309999999999992},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.308999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.914},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.847}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.167999999999999},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":15.454000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.248999999999999},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":13.563000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":3.4180000000000001},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.2260000000000009}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":14.353999999999999},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":6.0359999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.339},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.6210000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.8680000000000003},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.361000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.327999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.301}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.352},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.398}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:17.081 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:17.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.179+0000 7f322e36c640 10 monclient: get_monmap_and_config 2024-01-20T11:18:17.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.179+0000 7f322e36c640 10 monclient: build_initial_monmap 2024-01-20T11:18:17.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.179+0000 7f322e36c640 10 monclient: monmap: 2024-01-20T11:18:17.180 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:17.180 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.180 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:17.181015+0000 2024-01-20T11:18:17.180 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:17.181015+0000 2024-01-20T11:18:17.180 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:17.181 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.181 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:17.181 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:17.181 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:17.181 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:17.181 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:17.181 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 1 Processor -- start 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 1 -- start start 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 10 monclient: init 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32280b0ba0 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32280adc30 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f32280ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 10 monclient(hunting): picked mon.noname-b con 0x7f32280ac4d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32280ad530 con 0x7f32280b0ba0 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32280ad810 con 0x7f32280adc30 2024-01-20T11:18:17.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32280af840 con 0x7f32280ac4d0 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.180+0000 7f322e36c640 10 monclient(hunting): authenticate will time out at 1540.531978s 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.181+0000 7f322c909640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f32280ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.181+0000 7f32277fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f32280adc30 0x7f322813bf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:45746/0 (socket says 172.21.15.110:45746) 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.181+0000 7f32277fe640 1 -- 172.21.15.110:0/1332611582 learned_addr learned my addr 172.21.15.110:0/1332611582 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.181+0000 7f322c909640 10 monclient(hunting): get_auth_request con 0x7f32280ac4d0 auth_method 0 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.181+0000 7f322c909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.181+0000 7f322c909640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.181+0000 7f322c909640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.182+0000 7f322c909640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.182+0000 7f322c909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:17.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.182+0000 7f322c909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:17.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f3226ffd640 1 -- 172.21.15.110:0/1332611582 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598672933 0 0) 0x7f32280ad810 con 0x7f32280adc30 2024-01-20T11:18:17.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f3226ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:18:17.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f3226ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f3226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f3226ffd640 10 monclient(hunting): my global_id is 4810 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f3226ffd640 1 -- 172.21.15.110:0/1332611582 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3210003520 con 0x7f32280adc30 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f322c909640 10 monclient(hunting): handle_auth_done global_id 4607 payload 274 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f322c909640 1 -- 172.21.15.110:0/1332611582 >> v1:172.21.15.134:6789/0 conn(0x7f32280adc30 legacy=0x7f322813bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f322c909640 1 -- 172.21.15.110:0/1332611582 >> v1:172.21.15.110:6789/0 conn(0x7f32280b0ba0 legacy=0x7f32280af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f322c909640 10 monclient: _finish_hunting 0 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f322c909640 1 monclient: found mon.noname-b 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f322c909640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f322c909640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.183+0000 7f322c909640 1 -- 172.21.15.110:0/1332611582 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32280ab110 con 0x7f32280ac4d0 2024-01-20T11:18:17.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.185+0000 7f322c909640 1 --2- 172.21.15.110:0/1332611582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f32280ab630 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3220009dc0 tx=0x7f322002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=218d5775069dd33 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:17.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 1 -- 172.21.15.110:0/1332611582 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322002cbb0 con 0x7f32280ac4d0 2024-01-20T11:18:17.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 10 monclient: dump: 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:17.192 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 1 -- 172.21.15.110:0/1332611582 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322002cd10 con 0x7f32280ac4d0 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 1 -- 172.21.15.110:0/1332611582 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220034660 con 0x7f32280ac4d0 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f322e36c640 5 monclient: authenticate success, global_id 4607 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f322e36c640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f322e36c640 4 set_mon_vals no callback set 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.189+0000 7f3226ffd640 10 monclient: dump: 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:17.193 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 10 monclient: shutdown 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 1 -- 172.21.15.110:0/1332611582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 msgr2=0x7f32280ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 1 --2- 172.21.15.110:0/1332611582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f32280ab630 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3220009dc0 tx=0x7f322002be00 comp rx=0 tx=0).stop 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 1 -- 172.21.15.110:0/1332611582 shutdown_connections 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 1 --2- 172.21.15.110:0/1332611582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f32280ab630 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 1 -- 172.21.15.110:0/1332611582 >> 172.21.15.110:0/1332611582 conn(0x7f32280b3e60 msgr2=0x7f32280b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 1 -- 172.21.15.110:0/1332611582 shutdown_connections 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 1 -- 172.21.15.110:0/1332611582 wait complete. 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 10 monclient: build_initial_monmap 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.190+0000 7f322e36c640 10 monclient: monmap: 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:17.195 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:17.196 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 1 librados: starting msgr at 2024-01-20T11:18:17.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 1 librados: starting objecter 2024-01-20T11:18:17.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 1 Processor -- start 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 1 -- start start 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 1 librados: setting wanted keys 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 1 librados: calling monclient init 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 10 monclient: init 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f3228070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 10 monclient(hunting): picked mon.noname-c con 0x7f32280ac4d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32280b0ba0 0x7f3228071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.191+0000 7f322e36c640 10 monclient(hunting): picked mon.noname-a con 0x7f32280b0ba0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:17.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f322806e4c0 0x7f32281d30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 10 monclient(hunting): picked mon.noname-b con 0x7f322806e4c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32281d2c50 con 0x7f32280b0ba0 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322813cb30 con 0x7f322806e4c0 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3228071fc0 con 0x7f32280ac4d0 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322e36c640 10 monclient(hunting): authenticate will time out at 1540.543220s 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f3227fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f3228070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322c909640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f322806e4c0 0x7f32281d30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f32277fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32280b0ba0 0x7f3228071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f3227fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f3228070f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:36716/0 (socket says 172.21.15.110:36716) 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f3227fff640 1 -- 172.21.15.110:0/4056590141 learned_addr learned my addr 172.21.15.110:0/4056590141 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f3227fff640 10 monclient(hunting): get_auth_request con 0x7f32280ac4d0 auth_method 0 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f3227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f3227fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f3227fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322c909640 10 monclient(hunting): get_auth_request con 0x7f322806e4c0 auth_method 0 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322c909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322c909640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.192+0000 7f322c909640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f32277fe640 10 monclient(hunting): get_auth_request con 0x7f32280b0ba0 auth_method 0 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f32277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f32277fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f32277fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f322c909640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f322c909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f322c909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f3227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f3227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f3227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f32277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f32277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.193+0000 7f32277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 10 monclient(hunting): handle_auth_done global_id 4813 payload 824 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 1 -- 172.21.15.110:0/4056590141 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 msgr2=0x7f3228070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f3228070f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 1 -- 172.21.15.110:0/4056590141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32280b0ba0 msgr2=0x7f3228071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32280b0ba0 0x7f3228071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 10 monclient: _finish_hunting 0 2024-01-20T11:18:17.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 1 monclient: found mon.noname-b 2024-01-20T11:18:17.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:17.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:17.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.195+0000 7f322c909640 1 -- 172.21.15.110:0/4056590141 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3220009a70 con 0x7f322806e4c0 2024-01-20T11:18:17.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.197+0000 7f322c909640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f322806e4c0 0x7f32281d30e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3220033000 tx=0x7f3220004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:17.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.198+0000 7f3224ff9640 10 client.?.objecter ms_handle_connect 0x7f322806e4c0 2024-01-20T11:18:17.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.198+0000 7f3224ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:17.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 1 -- 172.21.15.110:0/4056590141 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322003e610 con 0x7f322806e4c0 2024-01-20T11:18:17.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:17.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 10 monclient: dump: 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:17.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 1 -- 172.21.15.110:0/4056590141 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322003e770 con 0x7f322806e4c0 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 1 -- 172.21.15.110:0/4056590141 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322003eb00 con 0x7f322806e4c0 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f322e36c640 5 monclient: authenticate success, global_id 4813 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f322e36c640 10 monclient: _renew_subs 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f322e36c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3228072240 con 0x7f322806e4c0 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f3224ff9640 10 monclient: dump: 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f32267fc640 4 set_mon_vals no callback set 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f322e36c640 10 client.4813.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f322e36c640 10 monclient: _renew_subs 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f322e36c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32281d3800 con 0x7f322806e4c0 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.201+0000 7f322e36c640 1 librados: init done 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.202+0000 7f322e36c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:17.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.202+0000 7f322e36c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:17.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.202+0000 7f322e36c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:17.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.202+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32281d39f0 con 0x7f322806e4c0 2024-01-20T11:18:17.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.221+0000 7f3224ff9640 1 -- 172.21.15.110:0/4056590141 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f322003b050 con 0x7f322806e4c0 2024-01-20T11:18:17.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.221+0000 7f3224ff9640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f321406c8b0 0x7f321406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.221+0000 7f3227fff640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f321406c8b0 0x7f321406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.221+0000 7f3227fff640 10 monclient: get_auth_request con 0x7f321406c8b0 auth_method 0 2024-01-20T11:18:17.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.222+0000 7f3224ff9640 10 client.4813.objecter ms_handle_connect 0x7f321406c8b0 2024-01-20T11:18:17.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.222+0000 7f3227fff640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f321406c8b0 0x7f321406ed70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f32180017a0 tx=0x7f3218009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:17.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.452+0000 7f3224ff9640 1 -- 172.21.15.110:0/4056590141 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f322007aee0 con 0x7f322806e4c0 2024-01-20T11:18:17.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.452+0000 7f3224ff9640 10 client.4813.objecter ms_dispatch 0x7f3228000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:17.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.452+0000 7f3224ff9640 3 client.4813.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:17.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.452+0000 7f3224ff9640 3 client.4813.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:17.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.453+0000 7f3224ff9640 20 client.4813.objecter dump_active .. 0 homeless 2024-01-20T11:18:17.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.463+0000 7f3224ff9640 1 -- 172.21.15.110:0/4056590141 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f3220077220 con 0x7f322806e4c0 2024-01-20T11:18:17.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.463+0000 7f3224ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:17.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.463+0000 7f3224ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:17.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.673+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32281d2dc0 con 0x7f321406c8b0 2024-01-20T11:18:17.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.676+0000 7f3224ff9640 1 -- 172.21.15.110:0/4056590141 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64242 (secure 0 0 0) 0x7f32281d2dc0 con 0x7f321406c8b0 2024-01-20T11:18:17.677 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:17.678 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:17.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 10 librados: watch_flush enter 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 10 librados: watch_flush exit 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 20 client.4813.objecter shutdown clearing up homeless session... 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 10 client.4813.objecter successfully canceled tick 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f321406c8b0 msgr2=0x7f321406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f321406c8b0 0x7f321406ed70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f32180017a0 tx=0x7f3218009040 comp rx=0 tx=0).stop 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 10 monclient: shutdown 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f322806e4c0 msgr2=0x7f32281d30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f322806e4c0 0x7f32281d30e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3220033000 tx=0x7f3220004a40 comp rx=0 tx=0).stop 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 shutdown_connections 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f321406c8b0 0x7f321406ed70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f322806e4c0 0x7f32281d30e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32280b0ba0 0x7f3228071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 --2- 172.21.15.110:0/4056590141 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32280ac4d0 0x7f3228070f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 >> 172.21.15.110:0/4056590141 conn(0x7f32280b3e60 msgr2=0x7f32280aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:17.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 shutdown_connections 2024-01-20T11:18:17.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 -- 172.21.15.110:0/4056590141 wait complete. 2024-01-20T11:18:17.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.678+0000 7f322e36c640 1 librados: shutdown 2024-01-20T11:18:17.690 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-01-20T11:18:16.744949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001977"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280554,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":11.423},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":4.298}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":15.750999999999999},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.3360000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.224},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.829000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.1349999999999998},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":9.6069999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.002000000000001},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":12.031000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.0579999999999998},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":13.419}]}]},{"osd":7,"up_from":21,"seq":90194313257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":11.179},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4339999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.0469999999999997},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.592000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":9.8510000000000009},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.2030000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.1310000000000002},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.781000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.472},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":14.180999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.726000000000001},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.805999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313258,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1829999999999998},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.7560000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.246},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.5510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":9.8260000000000005},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.137}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.914999999999999},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.4399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.48},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":14.941000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.715999999999999},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":8.8889999999999993}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444071,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":8.9809999999999999,"5min":10.122,"15min":10.311999999999999},"min":{"1min":5.1470000000000002,"5min":5.1399999999999997,"15min":5.1399999999999997},"max":{"1min":13.239000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":11.616},{"interface":"front","average":{"1min":9.4700000000000006,"5min":8.7870000000000008,"15min":8.673},"min":{"1min":4.8780000000000001,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":15.317,"5min":16.152000000000001,"15min":16.152000000000001},"last":16.163}]},{"osd":1,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":9.1259999999999994,"5min":13.257999999999999,"15min":13.946},"min":{"1min":4.1660000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":20.425999999999998,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.8460000000000001},{"interface":"front","average":{"1min":12.355,"5min":15.932,"15min":16.527999999999999},"min":{"1min":7.8019999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":18.991,"5min":32.414999999999999,"15min":32.414999999999999},"last":14.467000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.173},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":17.103999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":7.3869999999999996},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.849}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:17:15 2024","interfaces":[{"interface":"back","average":{"1min":14.459,"5min":15.752000000000001,"15min":15.968},"min":{"1min":9.8699999999999992,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":20.245999999999999,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.1170000000000009},{"interface":"front","average":{"1min":10.186,"5min":10.795999999999999,"15min":10.898},"min":{"1min":5.9640000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":17.308,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.5049999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.7230000000000008},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":15.044}]}]},{"osd":1,"up_from":17,"seq":73014444072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":8.5069999999999997,"5min":9.4920000000000009,"15min":9.6560000000000006},"min":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":18.547000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.596},{"interface":"front","average":{"1min":9.1159999999999997,"5min":10.300000000000001,"15min":10.497},"min":{"1min":5.6299999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":14.363,"5min":30.439,"15min":30.439},"last":5.8490000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.916,"5min":13.337,"15min":13.407},"min":{"1min":6.5810000000000004,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":23.077999999999999,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.9000000000000004},{"interface":"front","average":{"1min":12.561999999999999,"5min":13.391,"15min":13.529},"min":{"1min":6.1100000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":22.175999999999998,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.983000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.769},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.8100000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.4689999999999994},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.555999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:17:14 2024","interfaces":[{"interface":"back","average":{"1min":12.439,"5min":12.314,"15min":12.292999999999999},"min":{"1min":7.5279999999999996,"5min":7.1139999999999999,"15min":7.1139999999999999},"max":{"1min":27.233000000000001,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.252000000000001},{"interface":"front","average":{"1min":14.5,"5min":15.523999999999999,"15min":15.695},"min":{"1min":8.2620000000000005,"5min":8.2620000000000005,"15min":8.2620000000000005},"max":{"1min":23.158000000000001,"5min":31.634,"15min":31.634},"last":7.9009999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.275},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.555}]}]},{"osd":0,"up_from":17,"seq":73014444072,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":5.2009999999999996},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":7.3940000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.0099999999999998},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.449999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":9.4309999999999992},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.308999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.914},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.847}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":14.167999999999999},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":15.454000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.248999999999999},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":13.563000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":3.4180000000000001},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.2260000000000009}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":14.353999999999999},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":6.0359999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.339},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.6210000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.8680000000000003},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.361000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.327999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.301}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.352},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.398}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:17.691 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:17.691 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:17.691 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:17.692 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:17.692 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:17.692 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:17.692 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:17.692 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:17.692 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:17.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.792+0000 7fcbf8347640 10 monclient: get_monmap_and_config 2024-01-20T11:18:17.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.792+0000 7fcbf8347640 10 monclient: build_initial_monmap 2024-01-20T11:18:17.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.792+0000 7fcbf8347640 10 monclient: monmap: 2024-01-20T11:18:17.793 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:17.793 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.793 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:17.794008+0000 2024-01-20T11:18:17.793 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:17.794008+0000 2024-01-20T11:18:17.794 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:17.794 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.794 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:17.794 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:17.794 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:17.794 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:17.794 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:17.794 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.792+0000 7fcbf8347640 1 Processor -- start 2024-01-20T11:18:17.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 1 -- start start 2024-01-20T11:18:17.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 10 monclient: init 2024-01-20T11:18:17.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:17.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-01-20T11:18:17.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbf006bb00 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:17.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbf00af790 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:17.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 0x7fcbf00aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 10 monclient(hunting): picked mon.noname-b con 0x7fcbf00aeb20 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf006bee0 con 0x7fcbf006bb00 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf00ada30 con 0x7fcbf00af790 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00add10 con 0x7fcbf00aeb20 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.793+0000 7fcbf8347640 10 monclient(hunting): authenticate will time out at 1541.145010s 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.794+0000 7fcbf68e4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 0x7fcbf00aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.794+0000 7fcbf60e3640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcbf006bb00 0x7fcbf00b03c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54004/0 (socket says 172.21.15.110:54004) 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.794+0000 7fcbf60e3640 1 -- 172.21.15.110:0/2241264870 learned_addr learned my addr 172.21.15.110:0/2241264870 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.794+0000 7fcbf68e4640 10 monclient(hunting): get_auth_request con 0x7fcbf00aeb20 auth_method 0 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.794+0000 7fcbf68e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.794+0000 7fcbf68e4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.794+0000 7fcbf68e4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.795+0000 7fcbf68e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:17.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.795+0000 7fcbf68e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:17.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.795+0000 7fcbf68e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.796+0000 7fcbf50e1640 1 -- 172.21.15.110:0/2241264870 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869117257 0 0) 0x7fcbf00ada30 con 0x7fcbf00af790 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.796+0000 7fcbf50e1640 10 monclient(hunting): _negotiate 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.796+0000 7fcbf50e1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.796+0000 7fcbf50e1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.796+0000 7fcbf50e1640 10 monclient(hunting): my global_id is 4816 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.796+0000 7fcbf50e1640 1 -- 172.21.15.110:0/2241264870 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc003520 con 0x7fcbf00af790 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf50e1640 1 -- 172.21.15.110:0/2241264870 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590366637 0 0) 0x7fcbf006bee0 con 0x7fcbf006bb00 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf50e1640 10 monclient(hunting): _negotiate 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf50e1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf50e1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf50e1640 10 monclient(hunting): my global_id is 4728 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf50e1640 1 -- 172.21.15.110:0/2241264870 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf00ada30 con 0x7fcbf006bb00 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf68e4640 10 monclient(hunting): handle_auth_done global_id 4613 payload 274 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf68e4640 1 -- 172.21.15.110:0/2241264870 >> v1:172.21.15.134:6789/0 conn(0x7fcbf00af790 legacy=0x7fcbf00ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf68e4640 1 -- 172.21.15.110:0/2241264870 >> v1:172.21.15.110:6789/0 conn(0x7fcbf006bb00 legacy=0x7fcbf00b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf68e4640 10 monclient: _finish_hunting 0 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf68e4640 1 monclient: found mon.noname-b 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf68e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf68e4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:17.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.797+0000 7fcbf68e4640 1 -- 172.21.15.110:0/2241264870 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00adff0 con 0x7fcbf00aeb20 2024-01-20T11:18:17.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.799+0000 7fcbf68e4640 1 --2- 172.21.15.110:0/2241264870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 0x7fcbf00aefb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009dc0 tx=0x7fcbe802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24da84b1f9121a8e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:17.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 1 -- 172.21.15.110:0/2241264870 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe802cc40 con 0x7fcbf00aeb20 2024-01-20T11:18:17.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:17.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:17.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 10 monclient: dump: 2024-01-20T11:18:17.803 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 10 monclient: _finish_auth 0 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 10 monclient: _check_auth_tickets 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 1 -- 172.21.15.110:0/2241264870 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe802cda0 con 0x7fcbf00aeb20 2024-01-20T11:18:17.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf8347640 5 monclient: authenticate success, global_id 4613 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf8347640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf8347640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 1 -- 172.21.15.110:0/2241264870 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8034660 con 0x7fcbf00aeb20 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf8347640 4 set_mon_vals no callback set 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.802+0000 7fcbf50e1640 10 monclient: dump: 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.805 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:17.806 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:17.806 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:17.806 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.803+0000 7fcbf8347640 10 monclient: shutdown 2024-01-20T11:18:17.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.803+0000 7fcbf8347640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:17.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.803+0000 7fcbf8347640 1 -- 172.21.15.110:0/2241264870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 msgr2=0x7fcbf00aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.803+0000 7fcbf8347640 1 --2- 172.21.15.110:0/2241264870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 0x7fcbf00aefb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009dc0 tx=0x7fcbe802be90 comp rx=0 tx=0).stop 2024-01-20T11:18:17.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.803+0000 7fcbf8347640 1 -- 172.21.15.110:0/2241264870 shutdown_connections 2024-01-20T11:18:17.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.803+0000 7fcbf8347640 1 --2- 172.21.15.110:0/2241264870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 0x7fcbf00aefb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.803+0000 7fcbf8347640 1 -- 172.21.15.110:0/2241264870 >> 172.21.15.110:0/2241264870 conn(0x7fcbf0066d60 msgr2=0x7fcbf00b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.803+0000 7fcbf8347640 1 -- 172.21.15.110:0/2241264870 shutdown_connections 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.803+0000 7fcbf8347640 1 -- 172.21.15.110:0/2241264870 wait complete. 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.804+0000 7fcbf8347640 10 monclient: build_initial_monmap 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.804+0000 7fcbf8347640 10 monclient: monmap: 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:17.810 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.804+0000 7fcbf8347640 1 librados: starting msgr at 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.804+0000 7fcbf8347640 1 librados: starting objecter 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.804+0000 7fcbf8347640 1 Processor -- start 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.804+0000 7fcbf8347640 1 -- start start 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.804+0000 7fcbf8347640 1 librados: setting wanted keys 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.804+0000 7fcbf8347640 1 librados: calling monclient init 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.804+0000 7fcbf8347640 10 monclient: init 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbf006bb00 0x7fcbf01483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient(hunting): picked mon.noname-a con 0x7fcbf006bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 0x7fcbf0148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient(hunting): picked mon.noname-c con 0x7fcbf00aeb20 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:17.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf00af790 0x7fcbf0149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient(hunting): picked mon.noname-b con 0x7fcbf00af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf014a060 con 0x7fcbf006bb00 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf014ac40 con 0x7fcbf00af790 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0147210 con 0x7fcbf00aeb20 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf8347640 10 monclient(hunting): authenticate will time out at 1541.156464s 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf68e4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf00af790 0x7fcbf0149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf58e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 0x7fcbf0148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.805+0000 7fcbf60e3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbf006bb00 0x7fcbf01483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf68e4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf00af790 0x7fcbf0149010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49794/0 (socket says 172.21.15.110:49794) 2024-01-20T11:18:17.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf68e4640 1 -- 172.21.15.110:0/1427445350 learned_addr learned my addr 172.21.15.110:0/1427445350 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf68e4640 10 monclient(hunting): get_auth_request con 0x7fcbf00af790 auth_method 0 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf68e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf68e4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf68e4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf58e2640 10 monclient(hunting): get_auth_request con 0x7fcbf00aeb20 auth_method 0 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf58e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf58e2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf58e2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf60e3640 10 monclient(hunting): get_auth_request con 0x7fcbf006bb00 auth_method 0 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf60e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf60e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf60e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf68e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf68e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:17.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf68e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:17.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf58e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:17.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf58e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:17.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf58e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:17.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf60e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:17.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf60e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:17.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.806+0000 7fcbf60e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 10 monclient(hunting): handle_auth_done global_id 4819 payload 824 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 1 -- 172.21.15.110:0/1427445350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 msgr2=0x7fcbf0148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 0x7fcbf0148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 1 -- 172.21.15.110:0/1427445350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbf006bb00 msgr2=0x7fcbf01483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbf006bb00 0x7fcbf01483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 10 monclient: _finish_hunting 0 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 1 monclient: found mon.noname-b 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.809+0000 7fcbf68e4640 1 -- 172.21.15.110:0/1427445350 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe8009a70 con 0x7fcbf00af790 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.811+0000 7fcbf68e4640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf00af790 0x7fcbf0149010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009ee0 tx=0x7fcbe8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:17.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.811+0000 7fcbe6ffd640 10 client.?.objecter ms_handle_connect 0x7fcbf00af790 2024-01-20T11:18:17.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.811+0000 7fcbe6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 1 -- 172.21.15.110:0/1427445350 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8040650 con 0x7fcbf00af790 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 10 monclient: dump: 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:17.817 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 1 -- 172.21.15.110:0/1427445350 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe80407b0 con 0x7fcbf00af790 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf8347640 5 monclient: authenticate success, global_id 4819 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf8347640 10 monclient: _renew_subs 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf8347640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf014a540 con 0x7fcbf00af790 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 1 -- 172.21.15.110:0/1427445350 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8040b60 con 0x7fcbf00af790 2024-01-20T11:18:17.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbe6ffd640 10 monclient: dump: 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:17.819 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:17.820 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:17.820 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:17.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf48e0640 4 set_mon_vals no callback set 2024-01-20T11:18:17.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf8347640 10 client.4819.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:17.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf8347640 10 monclient: _renew_subs 2024-01-20T11:18:17.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf8347640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:17.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf0149550 con 0x7fcbf00af790 2024-01-20T11:18:17.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.815+0000 7fcbf8347640 1 librados: init done 2024-01-20T11:18:17.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.816+0000 7fcbf8347640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:17.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.816+0000 7fcbf8347640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:17.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.816+0000 7fcbf8347640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:17.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.816+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf0075490 con 0x7fcbf00af790 2024-01-20T11:18:17.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.835+0000 7fcbe6ffd640 1 -- 172.21.15.110:0/1427445350 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fcbe8040cc0 con 0x7fcbf00af790 2024-01-20T11:18:17.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.836+0000 7fcbe6ffd640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcbd406c8b0 0x7fcbd406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:17.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.836+0000 7fcbf60e3640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcbd406c8b0 0x7fcbd406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:17.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.836+0000 7fcbf60e3640 10 monclient: get_auth_request con 0x7fcbd406c8b0 auth_method 0 2024-01-20T11:18:17.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.836+0000 7fcbf60e3640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcbd406c8b0 0x7fcbd406ed70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00011e0 tx=0x7fcbe0007d30 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:17.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.836+0000 7fcbe6ffd640 10 client.4819.objecter ms_handle_connect 0x7fcbd406c8b0 2024-01-20T11:18:17.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.837+0000 7fcbe6ffd640 1 -- 172.21.15.110:0/1427445350 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcbe807b310 con 0x7fcbf00af790 2024-01-20T11:18:17.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.837+0000 7fcbe6ffd640 10 client.4819.objecter ms_dispatch 0x7fcbf0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:17.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.837+0000 7fcbe6ffd640 3 client.4819.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:17.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.837+0000 7fcbe6ffd640 3 client.4819.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:17.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:17.837+0000 7fcbe6ffd640 20 client.4819.objecter dump_active .. 0 homeless 2024-01-20T11:18:18.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.080+0000 7fcbe6ffd640 1 -- 172.21.15.110:0/1427445350 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fcbe8077650 con 0x7fcbf00af790 2024-01-20T11:18:18.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.080+0000 7fcbe6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:18.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.080+0000 7fcbe6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:18.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.315+0000 7fcbf8347640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:18.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.315+0000 7fcbf8347640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:18.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.315+0000 7fcbf8347640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:18.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.315+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fcbf01444d0 con 0x7fcbf00af790 2024-01-20T11:18:18.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.322+0000 7fcbe6ffd640 1 -- 172.21.15.110:0/1427445350 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fcbe8044520 con 0x7fcbf00af790 2024-01-20T11:18:18.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.322+0000 7fcbe6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:18.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.322+0000 7fcbe6ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:18.322 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:18.322 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":232,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:18.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 10 librados: watch_flush enter 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 10 librados: watch_flush exit 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 20 client.4819.objecter shutdown clearing up homeless session... 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 10 client.4819.objecter successfully canceled tick 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcbd406c8b0 msgr2=0x7fcbd406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcbd406c8b0 0x7fcbd406ed70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00011e0 tx=0x7fcbe0007d30 comp rx=0 tx=0).stop 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 10 monclient: shutdown 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf00af790 msgr2=0x7fcbf0149010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf00af790 0x7fcbf0149010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009ee0 tx=0x7fcbe8004a40 comp rx=0 tx=0).stop 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 shutdown_connections 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcbd406c8b0 0x7fcbd406ed70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf00af790 0x7fcbf0149010 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcbf00aeb20 0x7fcbf0148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 --2- 172.21.15.110:0/1427445350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbf006bb00 0x7fcbf01483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 >> 172.21.15.110:0/1427445350 conn(0x7fcbf0066d60 msgr2=0x7fcbf00b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:18.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 shutdown_connections 2024-01-20T11:18:18.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 -- 172.21.15.110:0/1427445350 wait complete. 2024-01-20T11:18:18.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.323+0000 7fcbf8347640 1 librados: shutdown 2024-01-20T11:18:18.335 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:18.335 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.436+0000 7fc5d3df5640 10 monclient: get_monmap_and_config 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.436+0000 7fc5d3df5640 10 monclient: build_initial_monmap 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.436+0000 7fc5d3df5640 10 monclient: monmap: 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:18.437734+0000 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:18.437734+0000 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:18.438 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:18.438 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:18.438 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:18.438 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:18.438 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:18.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.436+0000 7fc5d3df5640 1 Processor -- start 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.436+0000 7fc5d3df5640 1 -- start start 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.436+0000 7fc5d3df5640 10 monclient: init 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 0x7fc5cc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 10 monclient(hunting): picked mon.noname-b con 0x7fc5cc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5cc13fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc5cc0a9ad0 0x7fc5cc142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 10 monclient(hunting): picked mon.noname-a con 0x7fc5cc0a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0b25c0 con 0x7fc5cc13fa10 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5cc0aaea0 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:18.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc0ab0a0 con 0x7fc5cc0a96f0 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.437+0000 7fc5d3df5640 10 monclient(hunting): authenticate will time out at 1541.788698s 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.438+0000 7fc5d1b91640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 0x7fc5cc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.438+0000 7fc5d1b91640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 0x7fc5cc13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:36740/0 (socket says 172.21.15.110:36740) 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.438+0000 7fc5d1b91640 1 -- 172.21.15.110:0/3453569719 learned_addr learned my addr 172.21.15.110:0/3453569719 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.438+0000 7fc5d1b91640 10 monclient(hunting): get_auth_request con 0x7fc5cc0a96f0 auth_method 0 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.438+0000 7fc5d1b91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.438+0000 7fc5d1b91640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.438+0000 7fc5d1b91640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.439+0000 7fc5d1b91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.439+0000 7fc5d1b91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:18.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.439+0000 7fc5d1b91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:18.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.441+0000 7fc5d1b91640 10 monclient(hunting): handle_auth_done global_id 4619 payload 274 2024-01-20T11:18:18.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.441+0000 7fc5d1b91640 1 -- 172.21.15.110:0/3453569719 >> v2:172.21.15.110:3300/0 conn(0x7fc5cc0a9ad0 msgr2=0x7fc5cc142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:18.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.441+0000 7fc5d1b91640 1 --2- 172.21.15.110:0/3453569719 >> v2:172.21.15.110:3300/0 conn(0x7fc5cc0a9ad0 0x7fc5cc142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.441+0000 7fc5d1b91640 1 -- 172.21.15.110:0/3453569719 >> v1:172.21.15.110:6789/0 conn(0x7fc5cc13fa10 legacy=0x7fc5cc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:18.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.441+0000 7fc5d1b91640 10 monclient: _finish_hunting 0 2024-01-20T11:18:18.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.441+0000 7fc5d1b91640 1 monclient: found mon.noname-b 2024-01-20T11:18:18.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.441+0000 7fc5d1b91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:18.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.441+0000 7fc5d1b91640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:18.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.441+0000 7fc5d1b91640 1 -- 172.21.15.110:0/3453569719 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc0b3fb0 con 0x7fc5cc0a96f0 2024-01-20T11:18:18.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.444+0000 7fc5d1b91640 1 --2- 172.21.15.110:0/3453569719 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 0x7fc5cc13f4d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc009870 tx=0x7fc5bc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b56c5e2165812178 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:18.446 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:09.470 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:18.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.446+0000 7fc5d0b8f640 1 -- 172.21.15.110:0/3453569719 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc0317c0 con 0x7fc5cc0a96f0 2024-01-20T11:18:18.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.446+0000 7fc5d0b8f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:18.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.446+0000 7fc5d0b8f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:18.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.446+0000 7fc5d0b8f640 10 monclient: dump: 2024-01-20T11:18:18.447 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:18.447 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.446+0000 7fc5d0b8f640 10 monclient: _finish_auth 0 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.446+0000 7fc5d0b8f640 10 monclient: _check_auth_tickets 2024-01-20T11:18:18.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.446+0000 7fc5d0b8f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:18.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.446+0000 7fc5d3df5640 5 monclient: authenticate success, global_id 4619 2024-01-20T11:18:18.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.446+0000 7fc5d3df5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:18.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.448+0000 7fc5d0b8f640 1 -- 172.21.15.110:0/3453569719 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc031920 con 0x7fc5cc0a96f0 2024-01-20T11:18:18.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.448+0000 7fc5d0b8f640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:18.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.448+0000 7fc5d0b8f640 1 -- 172.21.15.110:0/3453569719 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc031cb0 con 0x7fc5cc0a96f0 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.448+0000 7fc5d0b8f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.448+0000 7fc5d0b8f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.448+0000 7fc5d0b8f640 10 monclient: dump: 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:18.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.448+0000 7fc5d3df5640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:18.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.448+0000 7fc5d3df5640 4 set_mon_vals no callback set 2024-01-20T11:18:18.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 10 monclient: shutdown 2024-01-20T11:18:18.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:18.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 1 -- 172.21.15.110:0/3453569719 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 msgr2=0x7fc5cc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:18.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 1 --2- 172.21.15.110:0/3453569719 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 0x7fc5cc13f4d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc009870 tx=0x7fc5bc02b860 comp rx=0 tx=0).stop 2024-01-20T11:18:18.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 1 -- 172.21.15.110:0/3453569719 shutdown_connections 2024-01-20T11:18:18.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 1 --2- 172.21.15.110:0/3453569719 >> v2:172.21.15.110:3300/0 conn(0x7fc5cc0a9ad0 0x7fc5cc142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 1 --2- 172.21.15.110:0/3453569719 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 0x7fc5cc13f4d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 1 -- 172.21.15.110:0/3453569719 >> 172.21.15.110:0/3453569719 conn(0x7fc5cc0b7bc0 msgr2=0x7fc5cc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 1 -- 172.21.15.110:0/3453569719 shutdown_connections 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.449+0000 7fc5d3df5640 1 -- 172.21.15.110:0/3453569719 wait complete. 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.450+0000 7fc5d3df5640 10 monclient: build_initial_monmap 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.450+0000 7fc5d3df5640 10 monclient: monmap: 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:18.454 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.450+0000 7fc5d3df5640 1 librados: starting msgr at 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.450+0000 7fc5d3df5640 1 librados: starting objecter 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.450+0000 7fc5d3df5640 1 Processor -- start 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.450+0000 7fc5d3df5640 1 -- start start 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.450+0000 7fc5d3df5640 1 librados: setting wanted keys 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.450+0000 7fc5d3df5640 1 librados: calling monclient init 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.450+0000 7fc5d3df5640 10 monclient: init 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:18.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 0x7fc5cc1dad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient(hunting): picked mon.noname-c con 0x7fc5cc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5cc0a9ad0 0x7fc5cc13ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient(hunting): picked mon.noname-a con 0x7fc5cc0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5cc13a180 0x7fc5cc133cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient(hunting): picked mon.noname-b con 0x7fc5cc13a180 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc0abc10 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1da930 con 0x7fc5cc13a180 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:18.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1da5e0 con 0x7fc5cc0a96f0 2024-01-20T11:18:18.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:18.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.451+0000 7fc5d3df5640 10 monclient(hunting): authenticate will time out at 1541.802656s 2024-01-20T11:18:18.457 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:09.479 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:18.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.464+0000 7fc5d1390640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5cc0a9ad0 0x7fc5cc13ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:18.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.469+0000 7fc5d1390640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5cc0a9ad0 0x7fc5cc13ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36034/0 (socket says 172.21.15.110:36034) 2024-01-20T11:18:18.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.469+0000 7fc5d1390640 1 -- 172.21.15.110:0/4110123795 learned_addr learned my addr 172.21.15.110:0/4110123795 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:18.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.469+0000 7fc5d1390640 10 monclient(hunting): get_auth_request con 0x7fc5cc0a9ad0 auth_method 0 2024-01-20T11:18:18.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.469+0000 7fc5d1390640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:18.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.469+0000 7fc5d1390640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:18.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.469+0000 7fc5d1390640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:18.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.471+0000 7fc5d1390640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:18.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.471+0000 7fc5d1390640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:18.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.471+0000 7fc5d1390640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:18.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 10 monclient(hunting): handle_auth_done global_id 4734 payload 824 2024-01-20T11:18:18.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 1 -- 172.21.15.110:0/4110123795 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 msgr2=0x7fc5cc1dad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:18.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 0x7fc5cc1dad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 1 -- 172.21.15.110:0/4110123795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5cc13a180 msgr2=0x7fc5cc133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:18.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5cc13a180 0x7fc5cc133cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 10 monclient: _finish_hunting 0 2024-01-20T11:18:18.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 1 monclient: found mon.noname-a 2024-01-20T11:18:18.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:18.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:18.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.474+0000 7fc5d1390640 1 -- 172.21.15.110:0/4110123795 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc0094d0 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.478+0000 7fc5d1390640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5cc0a9ad0 0x7fc5cc13ad90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800b410 tx=0x7fc5c800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:18.479 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:09.503 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:18.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.478+0000 7fc5c27fc640 10 client.?.objecter ms_handle_connect 0x7fc5cc0a9ad0 2024-01-20T11:18:18.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.478+0000 7fc5c27fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:18.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 1 -- 172.21.15.110:0/4110123795 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8011830 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:18.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:18.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 10 monclient: dump: 2024-01-20T11:18:18.501 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:18.501 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:18.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 1 -- 172.21.15.110:0/4110123795 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c8011990 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 1 -- 172.21.15.110:0/4110123795 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8011d40 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5d3df5640 5 monclient: authenticate success, global_id 4734 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c27fc640 10 monclient: dump: 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5d3df5640 10 monclient: _renew_subs 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5c3fff640 4 set_mon_vals no callback set 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5d3df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc1db290 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5d3df5640 10 client.4734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5d3df5640 10 monclient: _renew_subs 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5d3df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc1380f0 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.500+0000 7fc5d3df5640 1 librados: init done 2024-01-20T11:18:18.506 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:04:09.509 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:18.506 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:04:09.512 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.501+0000 7fc5d3df5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.501+0000 7fc5d3df5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.501+0000 7fc5d3df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:18.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.501+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc1daa70 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.507 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:09.527 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:18.507 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:09.530 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:18.525 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:09.549 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:18.529 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:04:09.535 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:18.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.771+0000 7fc5c27fc640 1 -- 172.21.15.110:0/4110123795 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc5c801a400 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.771+0000 7fc5c27fc640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b006c860 0x7fc5b006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:18.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.771+0000 7fc5d1b91640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b006c860 0x7fc5b006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:18.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.771+0000 7fc5d1b91640 10 monclient: get_auth_request con 0x7fc5b006c860 auth_method 0 2024-01-20T11:18:18.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.772+0000 7fc5c27fc640 10 client.4734.objecter ms_handle_connect 0x7fc5b006c860 2024-01-20T11:18:18.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.772+0000 7fc5d1b91640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b006c860 0x7fc5b006ed20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc036000 tx=0x7fc5bc030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:18.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.802+0000 7fc5c27fc640 1 -- 172.21.15.110:0/4110123795 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc5c800f2e0 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.802+0000 7fc5c27fc640 10 client.4734.objecter ms_dispatch 0x7fc5cc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:18.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.802+0000 7fc5c27fc640 3 client.4734.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:18.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.802+0000 7fc5c27fc640 3 client.4734.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:18.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.802+0000 7fc5c27fc640 20 client.4734.objecter dump_active .. 0 homeless 2024-01-20T11:18:18.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.802+0000 7fc5c27fc640 1 -- 172.21.15.110:0/4110123795 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc5c8054a70 con 0x7fc5cc0a9ad0 2024-01-20T11:18:18.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.802+0000 7fc5c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:18.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.802+0000 7fc5c27fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:18.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.983+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5cc1daa70 con 0x7fc5b006c860 2024-01-20T11:18:18.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.986+0000 7fc5c27fc640 1 -- 172.21.15.110:0/4110123795 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64114 (secure 0 0 0) 0x7fc5cc1daa70 con 0x7fc5b006c860 2024-01-20T11:18:18.987 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:18.987 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:18.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.987+0000 7fc5d3df5640 10 librados: watch_flush enter 2024-01-20T11:18:18.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 10 librados: watch_flush exit 2024-01-20T11:18:18.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 20 client.4734.objecter shutdown clearing up homeless session... 2024-01-20T11:18:18.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 10 client.4734.objecter successfully canceled tick 2024-01-20T11:18:18.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b006c860 msgr2=0x7fc5b006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:18.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b006c860 0x7fc5b006ed20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc036000 tx=0x7fc5bc030000 comp rx=0 tx=0).stop 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 10 monclient: shutdown 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5cc0a9ad0 msgr2=0x7fc5cc13ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5cc0a9ad0 0x7fc5cc13ad90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800b410 tx=0x7fc5c800b8d0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 shutdown_connections 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b006c860 0x7fc5b006ed20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5cc13a180 0x7fc5cc133cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5cc0a9ad0 0x7fc5cc13ad90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 --2- 172.21.15.110:0/4110123795 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5cc0a96f0 0x7fc5cc1dad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 >> 172.21.15.110:0/4110123795 conn(0x7fc5cc0b7bc0 msgr2=0x7fc5cc13c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 shutdown_connections 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 -- 172.21.15.110:0/4110123795 wait complete. 2024-01-20T11:18:18.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:18.988+0000 7fc5d3df5640 1 librados: shutdown 2024-01-20T11:18:18.999 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-01-20T11:18:18.745306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002011"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.4640000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.8869999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.6420000000000003},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.818}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.6229999999999993},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.631}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.4649999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":10.58}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.177},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":16.094000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.906000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":15.224}]}]},{"osd":7,"up_from":21,"seq":90194313257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":11.179},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4339999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.0469999999999997},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.592000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":9.8510000000000009},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.2030000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.1310000000000002},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.781000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.472},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":14.180999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.726000000000001},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.805999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313258,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1829999999999998},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.7560000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.246},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.5510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":9.8260000000000005},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.137}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.914999999999999},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.4399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.48},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":14.941000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.715999999999999},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":8.8889999999999993}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.149},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":7.5}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.8100000000000005},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.867}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.686},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.1159999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.442},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.824}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.666},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.111000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.531000000000001},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":11.103}]}]},{"osd":1,"up_from":17,"seq":73014444073,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.7279999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":10.803000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.667999999999999},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.6749999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.587999999999999},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.5570000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":11.475},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.3159999999999998},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":12.053000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.1210000000000004},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.256}]}]},{"osd":0,"up_from":17,"seq":73014444073,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":5.609},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":8.1280000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.4290000000000003},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.7059999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":10.371},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.888}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.927},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7520000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.225},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.539}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.114000000000001},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":9.343}]}]},{"osd":6,"up_from":15,"seq":64424509481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":3.4180000000000001},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.2260000000000009}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":14.353999999999999},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":6.0359999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.339},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.6210000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.8680000000000003},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.361000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.327999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.301}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.352},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.398}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:22.001 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.105+0000 7f727a383640 10 monclient: get_monmap_and_config 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.105+0000 7f727a383640 10 monclient: build_initial_monmap 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.105+0000 7f727a383640 10 monclient: monmap: 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:22.106728+0000 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:22.106728+0000 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:22.107 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:22.107 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:22.107 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.105+0000 7f727a383640 1 Processor -- start 2024-01-20T11:18:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 1 -- start start 2024-01-20T11:18:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient: init 2024-01-20T11:18:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f72740b4d10 0x7f72740b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient(hunting): picked mon.noname-c con 0x7f72740b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 0x7f72740b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient(hunting): picked mon.noname-b con 0x7f72740b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7274145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72740b2120 con 0x7f7274145980 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72740b50f0 con 0x7f72740b4d10 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72740aade0 con 0x7f72740b17c0 2024-01-20T11:18:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.106+0000 7f727a383640 10 monclient(hunting): authenticate will time out at 1545.457825s 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f7278920640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7274145980 0x7f7274145e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55878/0 (socket says 172.21.15.110:55878) 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f7278920640 1 -- 172.21.15.110:0/2203660823 learned_addr learned my addr 172.21.15.110:0/2203660823 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f7273fff640 1 --2- 172.21.15.110:0/2203660823 >> v2:172.21.15.134:3300/0 conn(0x7f72740b4d10 0x7f72740b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f72737fe640 1 --2- 172.21.15.110:0/2203660823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 0x7f72740b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f7273fff640 10 monclient(hunting): get_auth_request con 0x7f72740b4d10 auth_method 0 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f7273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f7273fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f7273fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f72737fe640 10 monclient(hunting): get_auth_request con 0x7f72740b17c0 auth_method 0 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f72737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f72737fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f72737fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f7273fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.107+0000 7f7273fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.108+0000 7f7273fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.108+0000 7f72737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.108+0000 7f72737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.108+0000 7f72737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.109+0000 7f7273fff640 10 monclient(hunting): handle_auth_done global_id 4822 payload 274 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.109+0000 7f7273fff640 1 -- 172.21.15.110:0/2203660823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 msgr2=0x7f72740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.109+0000 7f7273fff640 1 --2- 172.21.15.110:0/2203660823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 0x7f72740b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.109+0000 7f7273fff640 1 -- 172.21.15.110:0/2203660823 >> v1:172.21.15.110:6789/0 conn(0x7f7274145980 legacy=0x7f7274145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.109+0000 7f7273fff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.109+0000 7f7273fff640 1 monclient: found mon.noname-c 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.109+0000 7f7273fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.109+0000 7f7273fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.109+0000 7f7273fff640 1 -- 172.21.15.110:0/2203660823 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7274145460 con 0x7f72740b4d10 2024-01-20T11:18:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.111+0000 7f7273fff640 1 --2- 172.21.15.110:0/2203660823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72740b4d10 0x7f72740b3da0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f72640097a0 tx=0x7f726402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcd0ccd62d306c7f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:22.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 1 -- 172.21.15.110:0/2203660823 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726402e070 con 0x7f72740b4d10 2024-01-20T11:18:22.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:22.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:22.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 10 monclient: dump: 2024-01-20T11:18:22.116 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:22.116 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.116 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.116 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.116 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 1 -- 172.21.15.110:0/2203660823 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72640315a0 con 0x7f72740b4d10 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 5 monclient: authenticate success, global_id 4822 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:22.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 4 set_mon_vals no callback set 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 1 -- 172.21.15.110:0/2203660823 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7264031930 con 0x7f72740b4d10 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f7272ffd640 10 monclient: dump: 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:22.118 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.119 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:22.119 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:22.119 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:22.119 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 10 monclient: shutdown 2024-01-20T11:18:22.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:22.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 1 -- 172.21.15.110:0/2203660823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72740b4d10 msgr2=0x7f72740b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 1 --2- 172.21.15.110:0/2203660823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72740b4d10 0x7f72740b3da0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f72640097a0 tx=0x7f726402b840 comp rx=0 tx=0).stop 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 1 -- 172.21.15.110:0/2203660823 shutdown_connections 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 1 --2- 172.21.15.110:0/2203660823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 0x7f72740b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 1 --2- 172.21.15.110:0/2203660823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72740b4d10 0x7f72740b3da0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 1 -- 172.21.15.110:0/2203660823 >> 172.21.15.110:0/2203660823 conn(0x7f7274066590 msgr2=0x7f72740669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 1 -- 172.21.15.110:0/2203660823 shutdown_connections 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 1 -- 172.21.15.110:0/2203660823 wait complete. 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 10 monclient: build_initial_monmap 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.116+0000 7f727a383640 10 monclient: monmap: 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:22.124 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 librados: starting msgr at 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 librados: starting objecter 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 Processor -- start 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 -- start start 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 librados: setting wanted keys 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 librados: calling monclient init 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient: init 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:22.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 0x7f72741467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient(hunting): picked mon.noname-c con 0x7f72740b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72740b4d10 0x7f7274140280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient(hunting): picked mon.noname-a con 0x7f72740b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7274145980 0x7f72741407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient(hunting): picked mon.noname-b con 0x7f7274145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7274140d00 con 0x7f72740b4d10 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7274147fa0 con 0x7f7274145980 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7274148280 con 0x7f72740b17c0 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:22.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.117+0000 7f727a383640 10 monclient(hunting): authenticate will time out at 1545.469140s 2024-01-20T11:18:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f72737fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72740b4d10 0x7f7274140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7273fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 0x7f72741467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7278920640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7274145980 0x7f72741407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7278920640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7274145980 0x7f72741407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:52290/0 (socket says 172.21.15.110:52290) 2024-01-20T11:18:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7278920640 1 -- 172.21.15.110:0/1417932872 learned_addr learned my addr 172.21.15.110:0/1417932872 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7278920640 10 monclient(hunting): get_auth_request con 0x7f7274145980 auth_method 0 2024-01-20T11:18:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7278920640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7278920640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7278920640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f72737fe640 10 monclient(hunting): get_auth_request con 0x7f72740b4d10 auth_method 0 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f72737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f72737fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f72737fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7273fff640 10 monclient(hunting): get_auth_request con 0x7f72740b17c0 auth_method 0 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7273fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.118+0000 7f7273fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.119+0000 7f7278920640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.119+0000 7f7278920640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.119+0000 7f7278920640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.119+0000 7f72737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.119+0000 7f72737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.119+0000 7f72737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.119+0000 7f7273fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.119+0000 7f7273fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.119+0000 7f7273fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 10 monclient(hunting): handle_auth_done global_id 4825 payload 824 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 1 -- 172.21.15.110:0/1417932872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 msgr2=0x7f72741467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 0x7f72741467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 1 -- 172.21.15.110:0/1417932872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72740b4d10 msgr2=0x7f7274140280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72740b4d10 0x7f7274140280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 10 monclient: _finish_hunting 0 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 1 monclient: found mon.noname-b 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:22.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:22.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.121+0000 7f7278920640 1 -- 172.21.15.110:0/1417932872 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7264009400 con 0x7f7274145980 2024-01-20T11:18:22.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.125+0000 7f7278920640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7274145980 0x7f72741407c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f726c00caa0 tx=0x7f726c00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:22.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.125+0000 7f7270ff9640 10 client.?.objecter ms_handle_connect 0x7f7274145980 2024-01-20T11:18:22.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.125+0000 7f7270ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:22.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 1 -- 172.21.15.110:0/1417932872 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c00ab40 con 0x7f7274145980 2024-01-20T11:18:22.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:22.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:22.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 10 monclient: dump: 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:22.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 1 -- 172.21.15.110:0/1417932872 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c00aca0 con 0x7f7274145980 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f727a383640 5 monclient: authenticate success, global_id 4825 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f727a383640 10 monclient: _renew_subs 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f727a383640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7274148560 con 0x7f7274145980 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 1 -- 172.21.15.110:0/1417932872 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c012600 con 0x7f7274145980 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7270ff9640 10 monclient: dump: 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.133 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f7271ffb640 4 set_mon_vals no callback set 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f727a383640 10 client.4825.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f727a383640 10 monclient: _renew_subs 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f727a383640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7274146d00 con 0x7f7274145980 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.127+0000 7f727a383640 1 librados: init done 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.128+0000 7f727a383640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.128+0000 7f727a383640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.128+0000 7f727a383640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:22.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.128+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7274075490 con 0x7f7274145980 2024-01-20T11:18:22.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.379+0000 7f7270ff9640 1 -- 172.21.15.110:0/1417932872 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f726c007070 con 0x7f7274145980 2024-01-20T11:18:22.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.379+0000 7f7270ff9640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f726006c860 0x7f726006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.379+0000 7f7273fff640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f726006c860 0x7f726006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.379+0000 7f7273fff640 10 monclient: get_auth_request con 0x7f726006c860 auth_method 0 2024-01-20T11:18:22.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.379+0000 7f7273fff640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f726006c860 0x7f726006ed20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7264036000 tx=0x7f726403d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:22.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.379+0000 7f7270ff9640 10 client.4825.objecter ms_handle_connect 0x7f726006c860 2024-01-20T11:18:22.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.381+0000 7f7270ff9640 1 -- 172.21.15.110:0/1417932872 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f726c058490 con 0x7f7274145980 2024-01-20T11:18:22.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.381+0000 7f7270ff9640 10 client.4825.objecter ms_dispatch 0x7f7274000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:22.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.381+0000 7f7270ff9640 3 client.4825.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:22.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.381+0000 7f7270ff9640 3 client.4825.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:22.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.381+0000 7f7270ff9640 20 client.4825.objecter dump_active .. 0 homeless 2024-01-20T11:18:22.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.391+0000 7f7270ff9640 1 -- 172.21.15.110:0/1417932872 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f726c0547d0 con 0x7f7274145980 2024-01-20T11:18:22.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.391+0000 7f7270ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:22.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.391+0000 7f7270ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:22.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.560+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7274143b20 con 0x7f726006c860 2024-01-20T11:18:22.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.564+0000 7f7270ff9640 1 -- 172.21.15.110:0/1417932872 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64114 (secure 0 0 0) 0x7f7274143b20 con 0x7f726006c860 2024-01-20T11:18:22.565 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:22.565 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:22.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 10 librados: watch_flush enter 2024-01-20T11:18:22.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 10 librados: watch_flush exit 2024-01-20T11:18:22.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 20 client.4825.objecter shutdown clearing up homeless session... 2024-01-20T11:18:22.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 10 client.4825.objecter successfully canceled tick 2024-01-20T11:18:22.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f726006c860 msgr2=0x7f726006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f726006c860 0x7f726006ed20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7264036000 tx=0x7f726403d040 comp rx=0 tx=0).stop 2024-01-20T11:18:22.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 10 monclient: shutdown 2024-01-20T11:18:22.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7274145980 msgr2=0x7f72741407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7274145980 0x7f72741407c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f726c00caa0 tx=0x7f726c00cf60 comp rx=0 tx=0).stop 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 shutdown_connections 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f726006c860 0x7f726006ed20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7274145980 0x7f72741407c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72740b4d10 0x7f7274140280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 --2- 172.21.15.110:0/1417932872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72740b17c0 0x7f72741467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 >> 172.21.15.110:0/1417932872 conn(0x7f7274066590 msgr2=0x7f72740b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 shutdown_connections 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 -- 172.21.15.110:0/1417932872 wait complete. 2024-01-20T11:18:22.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.565+0000 7f727a383640 1 librados: shutdown 2024-01-20T11:18:22.577 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-01-20T11:18:20.745724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002115"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.4640000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.8869999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.6420000000000003},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.818}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.6229999999999993},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.631}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.4649999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":10.58}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.177},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":16.094000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.906000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":15.224}]}]},{"osd":7,"up_from":21,"seq":90194313257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":11.179},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.4339999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.0469999999999997},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.592000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":9.8510000000000009},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.2030000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.1310000000000002},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.781000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.472},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":14.180999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.726000000000001},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.805999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313258,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1829999999999998},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.7560000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.246},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.5510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":9.8260000000000005},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.137}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.914999999999999},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.4399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":10.48},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":14.941000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":15.715999999999999},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":8.8889999999999993}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.149},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":7.5}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.8100000000000005},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.867}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.686},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.1159999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.442},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.824}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.666},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.111000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.531000000000001},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":11.103}]}]},{"osd":1,"up_from":17,"seq":73014444073,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.7279999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":10.803000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.667999999999999},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.6749999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.587999999999999},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.5570000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":11.475},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.3159999999999998},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":12.053000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.1210000000000004},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.256}]}]},{"osd":0,"up_from":17,"seq":73014444073,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":5.609},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":8.1280000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.4290000000000003},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.7059999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":10.371},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.888}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.927},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7520000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.225},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.539}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.114000000000001},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":9.343}]}]},{"osd":6,"up_from":15,"seq":64424509481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":3.4180000000000001},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.2260000000000009}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":14.353999999999999},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":6.0359999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.339},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.6210000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.8680000000000003},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.361000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.327999999999999},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.301}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.352},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.398}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:22.579 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:22.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.678+0000 7f9be3015640 10 monclient: get_monmap_and_config 2024-01-20T11:18:22.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.678+0000 7f9be3015640 10 monclient: build_initial_monmap 2024-01-20T11:18:22.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.678+0000 7f9be3015640 10 monclient: monmap: 2024-01-20T11:18:22.678 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:22.678 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.678 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:22.678664+0000 2024-01-20T11:18:22.678 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:22.678664+0000 2024-01-20T11:18:22.678 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:22.678 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.679 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:22.679 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:22.679 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:22.679 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:22.679 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:22.679 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.678+0000 7f9be3015640 1 Processor -- start 2024-01-20T11:18:22.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.678+0000 7f9be3015640 1 -- start start 2024-01-20T11:18:22.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.678+0000 7f9be3015640 10 monclient: init 2024-01-20T11:18:22.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:22.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-01-20T11:18:22.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9bdc0a96f0 0x7f9bdc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 10 monclient(hunting): picked mon.noname-a con 0x7f9bdc0a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9bdc13fa10 0x7f9bdc1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 10 monclient(hunting): picked mon.noname-c con 0x7f9bdc13fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9bdc0a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bdc0b25c0 con 0x7f9bdc0a9ad0 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bdc0aaea0 con 0x7f9bdc0a96f0 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bdc0ab0a0 con 0x7f9bdc13fa10 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.679+0000 7f9be3015640 10 monclient(hunting): authenticate will time out at 1546.029922s 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.680+0000 7f9bdbfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9bdc13fa10 0x7f9bdc1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.680+0000 7f9be15b2640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9bdc0a9ad0 0x7f9bdc142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:46918/0 (socket says 172.21.15.110:46918) 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.680+0000 7f9be15b2640 1 -- 172.21.15.110:0/1324204094 learned_addr learned my addr 172.21.15.110:0/1324204094 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.680+0000 7f9be0db1640 1 --2- 172.21.15.110:0/1324204094 >> v2:172.21.15.110:3300/0 conn(0x7f9bdc0a96f0 0x7f9bdc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.681+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request con 0x7f9bdc13fa10 auth_method 0 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.681+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.681+0000 7f9bdbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.681+0000 7f9bdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.681+0000 7f9be0db1640 10 monclient(hunting): get_auth_request con 0x7f9bdc0a96f0 auth_method 0 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.681+0000 7f9be0db1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.681+0000 7f9be0db1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.681+0000 7f9be0db1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.682+0000 7f9be0db1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.682+0000 7f9be0db1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:22.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.682+0000 7f9be0db1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:22.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.683+0000 7f9be0db1640 10 monclient(hunting): handle_auth_done global_id 4743 payload 274 2024-01-20T11:18:22.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.683+0000 7f9be0db1640 1 -- 172.21.15.110:0/1324204094 >> v2:172.21.15.134:3300/0 conn(0x7f9bdc13fa10 msgr2=0x7f9bdc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:22.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.683+0000 7f9be0db1640 1 --2- 172.21.15.110:0/1324204094 >> v2:172.21.15.134:3300/0 conn(0x7f9bdc13fa10 0x7f9bdc1423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.683+0000 7f9be0db1640 1 -- 172.21.15.110:0/1324204094 >> v1:172.21.15.134:6789/0 conn(0x7f9bdc0a9ad0 legacy=0x7f9bdc142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.683+0000 7f9be0db1640 10 monclient: _finish_hunting 0 2024-01-20T11:18:22.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.683+0000 7f9be0db1640 1 monclient: found mon.noname-a 2024-01-20T11:18:22.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.683+0000 7f9be0db1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:22.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.683+0000 7f9be0db1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:22.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.683+0000 7f9be0db1640 1 -- 172.21.15.110:0/1324204094 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc0b3fb0 con 0x7f9bdc0a96f0 2024-01-20T11:18:22.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.685+0000 7f9be0db1640 1 --2- 172.21.15.110:0/1324204094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bdc0a96f0 0x7f9bdc13f4d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc06d0f0 tx=0x7f9bd402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f12988e1f727d39 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:22.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.687+0000 7f9bdb7fe640 1 -- 172.21.15.110:0/1324204094 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd402cbf0 con 0x7f9bdc0a96f0 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.687+0000 7f9bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.688+0000 7f9bdb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.688+0000 7f9bdb7fe640 10 monclient: dump: 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:22.688 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:22.689 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:22.689 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.688+0000 7f9bdb7fe640 10 monclient: _finish_auth 0 2024-01-20T11:18:22.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.688+0000 7f9bdb7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:18:22.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.688+0000 7f9bdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:22.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.688+0000 7f9be3015640 5 monclient: authenticate success, global_id 4743 2024-01-20T11:18:22.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.688+0000 7f9be3015640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:22.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.689+0000 7f9bdb7fe640 1 -- 172.21.15.110:0/1324204094 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd402cd50 con 0x7f9bdc0a96f0 2024-01-20T11:18:22.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.689+0000 7f9bdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:22.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.689+0000 7f9bdb7fe640 1 -- 172.21.15.110:0/1324204094 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4034660 con 0x7f9bdc0a96f0 2024-01-20T11:18:22.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.689+0000 7f9bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:22.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.689+0000 7f9bdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:22.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.689+0000 7f9bdb7fe640 10 monclient: dump: 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:22.691 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.689+0000 7f9be3015640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.689+0000 7f9be3015640 4 set_mon_vals no callback set 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 10 monclient: shutdown 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 1 -- 172.21.15.110:0/1324204094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bdc0a96f0 msgr2=0x7f9bdc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 1 --2- 172.21.15.110:0/1324204094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bdc0a96f0 0x7f9bdc13f4d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc06d0f0 tx=0x7f9bd402be40 comp rx=0 tx=0).stop 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 1 -- 172.21.15.110:0/1324204094 shutdown_connections 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 1 --2- 172.21.15.110:0/1324204094 >> v2:172.21.15.134:3300/0 conn(0x7f9bdc13fa10 0x7f9bdc1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 1 --2- 172.21.15.110:0/1324204094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bdc0a96f0 0x7f9bdc13f4d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 1 -- 172.21.15.110:0/1324204094 >> 172.21.15.110:0/1324204094 conn(0x7f9bdc0b7bc0 msgr2=0x7f9bdc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 1 -- 172.21.15.110:0/1324204094 shutdown_connections 2024-01-20T11:18:22.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.690+0000 7f9be3015640 1 -- 172.21.15.110:0/1324204094 wait complete. 2024-01-20T11:18:22.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.691+0000 7f9be3015640 10 monclient: build_initial_monmap 2024-01-20T11:18:22.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.691+0000 7f9be3015640 10 monclient: monmap: 2024-01-20T11:18:22.696 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:22.696 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:22.697 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:22.697 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:22.697 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.691+0000 7f9be3015640 1 librados: starting msgr at 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.691+0000 7f9be3015640 1 librados: starting objecter 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.691+0000 7f9be3015640 1 Processor -- start 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.691+0000 7f9be3015640 1 -- start start 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.691+0000 7f9be3015640 1 librados: setting wanted keys 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.691+0000 7f9be3015640 1 librados: calling monclient init 2024-01-20T11:18:22.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.691+0000 7f9be3015640 10 monclient: init 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bdc0a96f0 0x7f9bdc1d75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient(hunting): picked mon.noname-c con 0x7f9bdc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bdc0a9ad0 0x7f9bdc1d7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient(hunting): picked mon.noname-a con 0x7f9bdc0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bdc137e50 0x7f9bdc13a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient(hunting): picked mon.noname-b con 0x7f9bdc137e50 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc139fa0 con 0x7f9bdc0a9ad0 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc1d70a0 con 0x7f9bdc137e50 2024-01-20T11:18:22.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc13bd40 con 0x7f9bdc0a96f0 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be3015640 10 monclient(hunting): authenticate will time out at 1546.042629s 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bdc0a9ad0 0x7f9bdc1d7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.692+0000 7f9be0db1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bdc0a96f0 0x7f9bdc1d75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bdc137e50 0x7f9bdc13a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bdc137e50 0x7f9bdc13a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:52310/0 (socket says 172.21.15.110:52310) 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 1 -- 172.21.15.110:0/1165651656 learned_addr learned my addr 172.21.15.110:0/1165651656 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 10 monclient(hunting): get_auth_request con 0x7f9bdc137e50 auth_method 0 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request con 0x7f9bdc0a9ad0 auth_method 0 2024-01-20T11:18:22.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9bdbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9bdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be0db1640 10 monclient(hunting): get_auth_request con 0x7f9bdc0a96f0 auth_method 0 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be0db1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be0db1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be0db1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9be15b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:22.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.693+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.694+0000 7f9be0db1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.694+0000 7f9be0db1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.694+0000 7f9be0db1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 10 monclient(hunting): handle_auth_done global_id 4834 payload 824 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 1 -- 172.21.15.110:0/1165651656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bdc0a96f0 msgr2=0x7f9bdc1d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bdc0a96f0 0x7f9bdc1d75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 1 -- 172.21.15.110:0/1165651656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bdc0a9ad0 msgr2=0x7f9bdc1d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bdc0a9ad0 0x7f9bdc1d7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 10 monclient: _finish_hunting 0 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 1 monclient: found mon.noname-b 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:22.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.696+0000 7f9be15b2640 1 -- 172.21.15.110:0/1165651656 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd4009a70 con 0x7f9bdc137e50 2024-01-20T11:18:22.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.698+0000 7f9be15b2640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bdc137e50 0x7f9bdc13a980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00c430 tx=0x7f9bcc00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:22.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.698+0000 7f9bd97fa640 10 client.?.objecter ms_handle_connect 0x7f9bdc137e50 2024-01-20T11:18:22.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.698+0000 7f9bd97fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:22.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.701+0000 7f9bd97fa640 1 -- 172.21.15.110:0/1165651656 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc005ce0 con 0x7f9bdc137e50 2024-01-20T11:18:22.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.701+0000 7f9bd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:22.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.701+0000 7f9bd97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:22.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.701+0000 7f9bd97fa640 10 monclient: dump: 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.701+0000 7f9bd97fa640 10 monclient: _finish_auth 0 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.701+0000 7f9bd97fa640 10 monclient: _check_auth_tickets 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.701+0000 7f9bd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.701+0000 7f9bd97fa640 1 -- 172.21.15.110:0/1165651656 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc005e40 con 0x7f9bdc137e50 2024-01-20T11:18:22.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9be3015640 5 monclient: authenticate success, global_id 4834 2024-01-20T11:18:22.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9be3015640 10 monclient: _renew_subs 2024-01-20T11:18:22.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9be3015640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:22.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bdc13aec0 con 0x7f9bdc137e50 2024-01-20T11:18:22.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9bd97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:22.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9bd97fa640 1 -- 172.21.15.110:0/1165651656 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc0072d0 con 0x7f9bdc137e50 2024-01-20T11:18:22.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9bd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:22.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9bd97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:22.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9bd97fa640 10 monclient: dump: 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9bdaffd640 4 set_mon_vals no callback set 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9be3015640 10 client.4834.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9be3015640 10 monclient: _renew_subs 2024-01-20T11:18:22.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9be3015640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:22.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bdc13b3d0 con 0x7f9bdc137e50 2024-01-20T11:18:22.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.702+0000 7f9be3015640 1 librados: init done 2024-01-20T11:18:22.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.703+0000 7f9be3015640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:22.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.703+0000 7f9be3015640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:22.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.703+0000 7f9be3015640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:22.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.703+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bdc1d72c0 con 0x7f9bdc137e50 2024-01-20T11:18:22.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.721+0000 7f9bd97fa640 1 -- 172.21.15.110:0/1165651656 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9bcc007430 con 0x7f9bdc137e50 2024-01-20T11:18:22.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.721+0000 7f9bd97fa640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bc806c8e0 0x7f9bc806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:22.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.722+0000 7f9be0db1640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bc806c8e0 0x7f9bc806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:22.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.722+0000 7f9be0db1640 10 monclient: get_auth_request con 0x7f9bc806c8e0 auth_method 0 2024-01-20T11:18:22.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.722+0000 7f9be0db1640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bc806c8e0 0x7f9bc806eda0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4036000 tx=0x7f9bd4030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:22.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.722+0000 7f9bd97fa640 10 client.4834.objecter ms_handle_connect 0x7f9bc806c8e0 2024-01-20T11:18:22.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.723+0000 7f9bd97fa640 1 -- 172.21.15.110:0/1165651656 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9bcc057c40 con 0x7f9bdc137e50 2024-01-20T11:18:22.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.723+0000 7f9bd97fa640 10 client.4834.objecter ms_dispatch 0x7f9bdc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:22.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.723+0000 7f9bd97fa640 3 client.4834.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:22.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.723+0000 7f9bd97fa640 3 client.4834.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:22.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.723+0000 7f9bd97fa640 20 client.4834.objecter dump_active .. 0 homeless 2024-01-20T11:18:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.963+0000 7f9bd97fa640 1 -- 172.21.15.110:0/1165651656 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9bcc053f80 con 0x7f9bdc137e50 2024-01-20T11:18:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.963+0000 7f9bd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:22.963+0000 7f9bd97fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:23.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.136+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bdc075490 con 0x7f9bc806c8e0 2024-01-20T11:18:23.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.139+0000 7f9bd97fa640 1 -- 172.21.15.110:0/1165651656 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64052 (secure 0 0 0) 0x7f9bdc075490 con 0x7f9bc806c8e0 2024-01-20T11:18:23.139 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:23.140 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:23.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 10 librados: watch_flush enter 2024-01-20T11:18:23.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 10 librados: watch_flush exit 2024-01-20T11:18:23.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 20 client.4834.objecter shutdown clearing up homeless session... 2024-01-20T11:18:23.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 10 client.4834.objecter successfully canceled tick 2024-01-20T11:18:23.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bc806c8e0 msgr2=0x7f9bc806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bc806c8e0 0x7f9bc806eda0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4036000 tx=0x7f9bd4030000 comp rx=0 tx=0).stop 2024-01-20T11:18:23.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 10 monclient: shutdown 2024-01-20T11:18:23.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bdc137e50 msgr2=0x7f9bdc13a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bdc137e50 0x7f9bdc13a980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00c430 tx=0x7f9bcc00c8f0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 shutdown_connections 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9bc806c8e0 0x7f9bc806eda0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bdc137e50 0x7f9bdc13a980 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bdc0a9ad0 0x7f9bdc1d7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 --2- 172.21.15.110:0/1165651656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bdc0a96f0 0x7f9bdc1d75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 >> 172.21.15.110:0/1165651656 conn(0x7f9bdc0b7bc0 msgr2=0x7f9bdc0b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 shutdown_connections 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 -- 172.21.15.110:0/1165651656 wait complete. 2024-01-20T11:18:23.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.141+0000 7f9be3015640 1 librados: shutdown 2024-01-20T11:18:23.153 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-01-20T11:18:22.746043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002093"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.4640000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.8869999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.6420000000000003},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.818}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.6229999999999993},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.631}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.4649999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":10.58}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.177},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":16.094000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.906000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":15.224}]}]},{"osd":7,"up_from":21,"seq":90194313258,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.9550000000000001},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.052}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.9380000000000006},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.2219999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":9.8130000000000006},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":15.975}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.945},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.991}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.686999999999999},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":7.2169999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.94},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.997}]}]},{"osd":3,"up_from":21,"seq":90194313259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.54},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.2290000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9860000000000007},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":17.704000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":11.539},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":6.8840000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.778},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.332000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":9.6609999999999996},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":15.587}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":14.313000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.271000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.149},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":7.5}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.8100000000000005},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.867}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.686},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.1159999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.442},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.824}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.666},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.111000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.531000000000001},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":11.103}]}]},{"osd":1,"up_from":17,"seq":73014444073,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.7279999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":10.803000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.667999999999999},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.6749999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.587999999999999},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.5570000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":11.475},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.3159999999999998},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":12.053000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.1210000000000004},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.256}]}]},{"osd":0,"up_from":17,"seq":73014444073,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":5.609},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":8.1280000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.4290000000000003},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.7059999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":10.371},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.888}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.927},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7520000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.225},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.539}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.114000000000001},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":9.343}]}]},{"osd":6,"up_from":15,"seq":64424509482,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":9.2949999999999999},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":12.449999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.9349999999999996},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":5.7839999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.6170000000000009},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.396000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":7.1429999999999998},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.182}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.878},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":10.827999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.071},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.993}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:23.154 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:23.154 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:23.154 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:23.155 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:23.155 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:23.155 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:23.155 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:23.155 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:23.155 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:23.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.252+0000 7f1f444d4640 10 monclient: get_monmap_and_config 2024-01-20T11:18:23.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.252+0000 7f1f444d4640 10 monclient: build_initial_monmap 2024-01-20T11:18:23.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.252+0000 7f1f444d4640 10 monclient: monmap: 2024-01-20T11:18:23.252 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:23.253272+0000 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:23.253272+0000 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:23.253 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.253+0000 7f1f444d4640 1 Processor -- start 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.253+0000 7f1f444d4640 1 -- start start 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.253+0000 7f1f444d4640 10 monclient: init 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f3c145760 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 10 monclient(hunting): picked mon.noname-b con 0x7f1f3c0a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1f3c13f8b0 0x7f1f3c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 10 monclient(hunting): picked mon.noname-a con 0x7f1f3c13f8b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c146250 con 0x7f1f3c145760 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c0ab050 con 0x7f1f3c13f8b0 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0b20c0 con 0x7f1f3c0a9d80 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f444d4640 10 monclient(hunting): authenticate will time out at 1546.604256s 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f41a6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.254+0000 7f1f42a71640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1f3c13f8b0 0x7f1f3c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f41a6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45320/0 (socket says 172.21.15.110:45320) 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f41a6f640 1 -- 172.21.15.110:0/273082598 learned_addr learned my addr 172.21.15.110:0/273082598 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f41a6f640 10 monclient(hunting): get_auth_request con 0x7f1f3c0a9d80 auth_method 0 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f41a6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f41a6f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f41a6f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f42a71640 10 monclient(hunting): get_auth_request con 0x7f1f3c13f8b0 auth_method 0 2024-01-20T11:18:23.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f42a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f42a71640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f42a71640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f41a6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f41a6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.255+0000 7f1f41a6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.256+0000 7f1f42a71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.256+0000 7f1f42a71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.256+0000 7f1f42a71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.257+0000 7f1f41a6f640 10 monclient(hunting): handle_auth_done global_id 4631 payload 274 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.257+0000 7f1f41a6f640 1 -- 172.21.15.110:0/273082598 >> v2:172.21.15.110:3300/0 conn(0x7f1f3c13f8b0 msgr2=0x7f1f3c0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.257+0000 7f1f41a6f640 1 --2- 172.21.15.110:0/273082598 >> v2:172.21.15.110:3300/0 conn(0x7f1f3c13f8b0 0x7f1f3c0b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.257+0000 7f1f41a6f640 1 -- 172.21.15.110:0/273082598 >> v1:172.21.15.110:6789/0 conn(0x7f1f3c145760 legacy=0x7f1f3c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.257+0000 7f1f41a6f640 10 monclient: _finish_hunting 0 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.257+0000 7f1f41a6f640 1 monclient: found mon.noname-b 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.257+0000 7f1f41a6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:23.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.257+0000 7f1f41a6f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:23.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.257+0000 7f1f41a6f640 1 -- 172.21.15.110:0/273082598 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c0b2bb0 con 0x7f1f3c0a9d80 2024-01-20T11:18:23.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.259+0000 7f1f41a6f640 1 --2- 172.21.15.110:0/273082598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c145b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009870 tx=0x7f1f3802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27a7a3ba3f3e2dbd server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:23.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 1 -- 172.21.15.110:0/273082598 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38030840 con 0x7f1f3c0a9d80 2024-01-20T11:18:23.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:23.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:23.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 10 monclient: dump: 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 10 monclient: _finish_auth 0 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 10 monclient: _check_auth_tickets 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 1 -- 172.21.15.110:0/273082598 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f380309a0 con 0x7f1f3c0a9d80 2024-01-20T11:18:23.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f444d4640 5 monclient: authenticate success, global_id 4631 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f444d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f444d4640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f444d4640 4 set_mon_vals no callback set 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 1 -- 172.21.15.110:0/273082598 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38030d30 con 0x7f1f3c0a9d80 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.263+0000 7f1f4126e640 10 monclient: dump: 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:23.265 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:23.266 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:23.266 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 10 monclient: shutdown 2024-01-20T11:18:23.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:23.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 1 -- 172.21.15.110:0/273082598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 msgr2=0x7f1f3c145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 1 --2- 172.21.15.110:0/273082598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c145b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009870 tx=0x7f1f3802b980 comp rx=0 tx=0).stop 2024-01-20T11:18:23.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 1 -- 172.21.15.110:0/273082598 shutdown_connections 2024-01-20T11:18:23.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 1 --2- 172.21.15.110:0/273082598 >> v2:172.21.15.110:3300/0 conn(0x7f1f3c13f8b0 0x7f1f3c0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 1 --2- 172.21.15.110:0/273082598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c145b40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 1 -- 172.21.15.110:0/273082598 >> 172.21.15.110:0/273082598 conn(0x7f1f3c0663b0 msgr2=0x7f1f3c0667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:23.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 1 -- 172.21.15.110:0/273082598 shutdown_connections 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.264+0000 7f1f444d4640 1 -- 172.21.15.110:0/273082598 wait complete. 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.265+0000 7f1f444d4640 10 monclient: build_initial_monmap 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.265+0000 7f1f444d4640 10 monclient: monmap: 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.265+0000 7f1f444d4640 1 librados: starting msgr at 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.265+0000 7f1f444d4640 1 librados: starting objecter 2024-01-20T11:18:23.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.265+0000 7f1f444d4640 1 Processor -- start 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.265+0000 7f1f444d4640 1 -- start start 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.265+0000 7f1f444d4640 1 librados: setting wanted keys 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.265+0000 7f1f444d4640 1 librados: calling monclient init 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.265+0000 7f1f444d4640 10 monclient: init 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient(hunting): picked mon.noname-c con 0x7f1f3c0a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f3c13f8b0 0x7f1f3c136450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient(hunting): picked mon.noname-a con 0x7f1f3c13f8b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f3c145760 0x7f1f3c1413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient(hunting): picked mon.noname-b con 0x7f1f3c145760 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c136990 con 0x7f1f3c13f8b0 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c137bc0 con 0x7f1f3c145760 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0b58c0 con 0x7f1f3c0a9d80 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f444d4640 10 monclient(hunting): authenticate will time out at 1546.616486s 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f42270640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f41a6f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f3c13f8b0 0x7f1f3c136450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.266+0000 7f1f42a71640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f3c145760 0x7f1f3c1413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f41a6f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f3c13f8b0 0x7f1f3c136450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60344/0 (socket says 172.21.15.110:60344) 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42270640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c135d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45328/0 (socket says 172.21.15.110:45328) 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f41a6f640 1 -- 172.21.15.110:0/3537332569 learned_addr learned my addr 172.21.15.110:0/3537332569 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f41a6f640 10 monclient(hunting): get_auth_request con 0x7f1f3c13f8b0 auth_method 0 2024-01-20T11:18:23.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f41a6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f41a6f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f41a6f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42270640 10 monclient(hunting): get_auth_request con 0x7f1f3c0a9d80 auth_method 0 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42270640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42270640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42270640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42a71640 10 monclient(hunting): get_auth_request con 0x7f1f3c145760 auth_method 0 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42a71640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42a71640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42a71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42a71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42a71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f41a6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f41a6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f41a6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42270640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42270640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.267+0000 7f1f42270640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 10 monclient(hunting): handle_auth_done global_id 4837 payload 824 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 1 -- 172.21.15.110:0/3537332569 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 msgr2=0x7f1f3c135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c135d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 1 -- 172.21.15.110:0/3537332569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f3c13f8b0 msgr2=0x7f1f3c136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f3c13f8b0 0x7f1f3c136450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 10 monclient: _finish_hunting 0 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 1 monclient: found mon.noname-b 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.270+0000 7f1f42a71640 1 -- 172.21.15.110:0/3537332569 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f380094d0 con 0x7f1f3c145760 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.273+0000 7f1f42a71640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f3c145760 0x7f1f3c1413a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400ef90 tx=0x7f1f3400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.273+0000 7f1f32ffd640 10 client.?.objecter ms_handle_connect 0x7f1f3c145760 2024-01-20T11:18:23.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.273+0000 7f1f32ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f32ffd640 1 -- 172.21.15.110:0/3537332569 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f34009c00 con 0x7f1f3c145760 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f32ffd640 10 monclient: dump: 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:23.277 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f32ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f32ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f32ffd640 1 -- 172.21.15.110:0/3537332569 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3400f040 con 0x7f1f3c145760 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f444d4640 5 monclient: authenticate success, global_id 4837 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.276+0000 7f1f32ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f444d4640 10 monclient: _renew_subs 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f444d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f32ffd640 1 -- 172.21.15.110:0/3537332569 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f340146c0 con 0x7f1f3c145760 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c136c40 con 0x7f1f3c145760 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f40a6d640 4 set_mon_vals no callback set 2024-01-20T11:18:23.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f32ffd640 10 monclient: dump: 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f444d4640 10 client.4837.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f444d4640 10 monclient: _renew_subs 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f444d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c141ac0 con 0x7f1f3c145760 2024-01-20T11:18:23.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.277+0000 7f1f444d4640 1 librados: init done 2024-01-20T11:18:23.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.278+0000 7f1f444d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:23.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.278+0000 7f1f444d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:23.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.278+0000 7f1f444d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:23.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.278+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c136e80 con 0x7f1f3c145760 2024-01-20T11:18:23.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.296+0000 7f1f32ffd640 1 -- 172.21.15.110:0/3537332569 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1f34014820 con 0x7f1f3c145760 2024-01-20T11:18:23.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.297+0000 7f1f32ffd640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f2006c8b0 0x7f1f2006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.297+0000 7f1f42270640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f2006c8b0 0x7f1f2006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.297+0000 7f1f42270640 10 monclient: get_auth_request con 0x7f1f2006c8b0 auth_method 0 2024-01-20T11:18:23.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.297+0000 7f1f42270640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f2006c8b0 0x7f1f2006ed70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0016e0 tx=0x7f1f2c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:23.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.297+0000 7f1f32ffd640 10 client.4837.objecter ms_handle_connect 0x7f1f2006c8b0 2024-01-20T11:18:23.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.544+0000 7f1f32ffd640 1 -- 172.21.15.110:0/3537332569 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1f340593a0 con 0x7f1f3c145760 2024-01-20T11:18:23.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.544+0000 7f1f32ffd640 10 client.4837.objecter ms_dispatch 0x7f1f3c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:23.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.544+0000 7f1f32ffd640 3 client.4837.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:23.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.544+0000 7f1f32ffd640 3 client.4837.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:23.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.545+0000 7f1f32ffd640 20 client.4837.objecter dump_active .. 0 homeless 2024-01-20T11:18:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.555+0000 7f1f32ffd640 1 -- 172.21.15.110:0/3537332569 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1f340556e0 con 0x7f1f3c145760 2024-01-20T11:18:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.555+0000 7f1f32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.555+0000 7f1f32ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.803+0000 7f1f444d4640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.803+0000 7f1f444d4640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.803+0000 7f1f444d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.803+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1f3c136e80 con 0x7f1f3c145760 2024-01-20T11:18:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.809+0000 7f1f32ffd640 1 -- 172.21.15.110:0/3537332569 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f1f3c136e80 con 0x7f1f3c145760 2024-01-20T11:18:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.810+0000 7f1f32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.810+0000 7f1f32ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:23.810 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:23.810 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":238,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.811+0000 7f1f444d4640 10 librados: watch_flush enter 2024-01-20T11:18:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 10 librados: watch_flush exit 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 20 client.4837.objecter shutdown clearing up homeless session... 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 10 client.4837.objecter successfully canceled tick 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f2006c8b0 msgr2=0x7f1f2006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f2006c8b0 0x7f1f2006ed70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0016e0 tx=0x7f1f2c009040 comp rx=0 tx=0).stop 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 10 monclient: shutdown 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f3c145760 msgr2=0x7f1f3c1413a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f3c145760 0x7f1f3c1413a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400ef90 tx=0x7f1f3400c490 comp rx=0 tx=0).stop 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 shutdown_connections 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f2006c8b0 0x7f1f2006ed70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f3c145760 0x7f1f3c1413a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f3c13f8b0 0x7f1f3c136450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 --2- 172.21.15.110:0/3537332569 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f3c0a9d80 0x7f1f3c135d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 >> 172.21.15.110:0/3537332569 conn(0x7f1f3c0663b0 msgr2=0x7f1f3c0b3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 shutdown_connections 2024-01-20T11:18:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 -- 172.21.15.110:0/3537332569 wait complete. 2024-01-20T11:18:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.812+0000 7f1f444d4640 1 librados: shutdown 2024-01-20T11:18:23.823 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:23.823 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:23.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.923+0000 7fc422807640 10 monclient: get_monmap_and_config 2024-01-20T11:18:23.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.923+0000 7fc422807640 10 monclient: build_initial_monmap 2024-01-20T11:18:23.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.923+0000 7fc422807640 10 monclient: monmap: 2024-01-20T11:18:23.923 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:23.924171+0000 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:23.924171+0000 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:23.924 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 1 Processor -- start 2024-01-20T11:18:23.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 1 -- start start 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient: init 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc41c06bb00 0x7fc41c0b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient(hunting): picked mon.noname-c con 0x7fc41c06bb00 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc41c0af790 0x7fc41c145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient(hunting): picked mon.noname-a con 0x7fc41c0af790 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0b0900 0x7fc41c145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient(hunting): picked mon.noname-b con 0x7fc41c0b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c06bee0 con 0x7fc41c0af790 2024-01-20T11:18:23.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c0ada30 con 0x7fc41c06bb00 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0add10 con 0x7fc41c0b0900 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.924+0000 7fc422807640 10 monclient(hunting): authenticate will time out at 1547.275017s 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc420da4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0b0900 0x7fc41c145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc41b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc41c0af790 0x7fc41c145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc41bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc41c06bb00 0x7fc41c0b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc420da4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0b0900 0x7fc41c145cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45332/0 (socket says 172.21.15.110:45332) 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc41b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc41c0af790 0x7fc41c145770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60354/0 (socket says 172.21.15.110:60354) 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc420da4640 1 -- 172.21.15.110:0/2979144176 learned_addr learned my addr 172.21.15.110:0/2979144176 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc420da4640 10 monclient(hunting): get_auth_request con 0x7fc41c0b0900 auth_method 0 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc420da4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc420da4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc420da4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc41b7fe640 10 monclient(hunting): get_auth_request con 0x7fc41c0af790 auth_method 0 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc41b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc41b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.925+0000 7fc41b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41bfff640 10 monclient(hunting): get_auth_request con 0x7fc41c06bb00 auth_method 0 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc420da4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc420da4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc420da4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.926+0000 7fc41bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 10 monclient(hunting): handle_auth_done global_id 4637 payload 274 2024-01-20T11:18:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 1 -- 172.21.15.110:0/2979144176 >> v2:172.21.15.134:3300/0 conn(0x7fc41c06bb00 msgr2=0x7fc41c0b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:23.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 1 --2- 172.21.15.110:0/2979144176 >> v2:172.21.15.134:3300/0 conn(0x7fc41c06bb00 0x7fc41c0b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 1 -- 172.21.15.110:0/2979144176 >> v2:172.21.15.110:3300/0 conn(0x7fc41c0af790 msgr2=0x7fc41c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 1 --2- 172.21.15.110:0/2979144176 >> v2:172.21.15.110:3300/0 conn(0x7fc41c0af790 0x7fc41c145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 10 monclient: _finish_hunting 0 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 1 monclient: found mon.noname-b 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.927+0000 7fc420da4640 1 -- 172.21.15.110:0/2979144176 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0adff0 con 0x7fc41c0b0900 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.929+0000 7fc420da4640 1 --2- 172.21.15.110:0/2979144176 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0b0900 0x7fc41c145cb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0b6630 tx=0x7fc41402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb50cadd9d2d5d91 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.931+0000 7fc41affd640 1 -- 172.21.15.110:0/2979144176 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41402c9e0 con 0x7fc41c0b0900 2024-01-20T11:18:23.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.931+0000 7fc41affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.931+0000 7fc41affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.931+0000 7fc41affd640 10 monclient: dump: 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.931+0000 7fc41affd640 10 monclient: _finish_auth 0 2024-01-20T11:18:23.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.931+0000 7fc41affd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:23.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.931+0000 7fc41affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:23.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.931+0000 7fc422807640 5 monclient: authenticate success, global_id 4637 2024-01-20T11:18:23.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.931+0000 7fc422807640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:23.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc41affd640 1 -- 172.21.15.110:0/2979144176 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41402cb40 con 0x7fc41c0b0900 2024-01-20T11:18:23.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc41affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc41affd640 1 -- 172.21.15.110:0/2979144176 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4140343f0 con 0x7fc41c0b0900 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 4 set_mon_vals no callback set 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc41affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc41affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc41affd640 10 monclient: dump: 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.935 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 10 monclient: shutdown 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 1 -- 172.21.15.110:0/2979144176 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0b0900 msgr2=0x7fc41c145cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 1 --2- 172.21.15.110:0/2979144176 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0b0900 0x7fc41c145cb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0b6630 tx=0x7fc41402be40 comp rx=0 tx=0).stop 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 1 -- 172.21.15.110:0/2979144176 shutdown_connections 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 1 --2- 172.21.15.110:0/2979144176 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0b0900 0x7fc41c145cb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 1 --2- 172.21.15.110:0/2979144176 >> v2:172.21.15.110:3300/0 conn(0x7fc41c0af790 0x7fc41c145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 1 --2- 172.21.15.110:0/2979144176 >> v2:172.21.15.134:3300/0 conn(0x7fc41c06bb00 0x7fc41c0b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 1 -- 172.21.15.110:0/2979144176 >> 172.21.15.110:0/2979144176 conn(0x7fc41c066d60 msgr2=0x7fc41c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:23.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.933+0000 7fc422807640 1 -- 172.21.15.110:0/2979144176 shutdown_connections 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.934+0000 7fc422807640 1 -- 172.21.15.110:0/2979144176 wait complete. 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.934+0000 7fc422807640 10 monclient: build_initial_monmap 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.934+0000 7fc422807640 10 monclient: monmap: 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.934+0000 7fc422807640 1 librados: starting msgr at 2024-01-20T11:18:23.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.934+0000 7fc422807640 1 librados: starting objecter 2024-01-20T11:18:23.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 Processor -- start 2024-01-20T11:18:23.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 -- start start 2024-01-20T11:18:23.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 librados: setting wanted keys 2024-01-20T11:18:23.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 librados: calling monclient init 2024-01-20T11:18:23.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient: init 2024-01-20T11:18:23.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:23.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41c06bb00 0x7fc41c141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient(hunting): picked mon.noname-a con 0x7fc41c06bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0af790 0x7fc41c0b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient(hunting): picked mon.noname-c con 0x7fc41c0af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41c0b0900 0x7fc41c1d6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient(hunting): picked mon.noname-b con 0x7fc41c0b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c11e870 con 0x7fc41c06bb00 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c11ea70 con 0x7fc41c0b0900 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c11ed50 con 0x7fc41c0af790 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.935+0000 7fc422807640 10 monclient(hunting): authenticate will time out at 1547.286024s 2024-01-20T11:18:23.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0af790 0x7fc41c0b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc420da4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41c0b0900 0x7fc41c1d6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41c06bb00 0x7fc41c141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0af790 0x7fc41c0b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45348/0 (socket says 172.21.15.110:45348) 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41b7fe640 1 -- 172.21.15.110:0/2992437431 learned_addr learned my addr 172.21.15.110:0/2992437431 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41b7fe640 10 monclient(hunting): get_auth_request con 0x7fc41c0af790 auth_method 0 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc420da4640 10 monclient(hunting): get_auth_request con 0x7fc41c0b0900 auth_method 0 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc420da4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc420da4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc420da4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41bfff640 10 monclient(hunting): get_auth_request con 0x7fc41c06bb00 auth_method 0 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.936+0000 7fc41bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.937+0000 7fc420da4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.937+0000 7fc420da4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.937+0000 7fc420da4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.937+0000 7fc41b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.937+0000 7fc41b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.937+0000 7fc41b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.937+0000 7fc41bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.937+0000 7fc41bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.937+0000 7fc41bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 10 monclient(hunting): handle_auth_done global_id 4843 payload 824 2024-01-20T11:18:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 1 -- 172.21.15.110:0/2992437431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0af790 msgr2=0x7fc41c0b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0af790 0x7fc41c0b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 1 -- 172.21.15.110:0/2992437431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41c06bb00 msgr2=0x7fc41c141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:23.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41c06bb00 0x7fc41c141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:23.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 10 monclient: _finish_hunting 0 2024-01-20T11:18:23.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 1 monclient: found mon.noname-b 2024-01-20T11:18:23.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:23.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:23.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.939+0000 7fc420da4640 1 -- 172.21.15.110:0/2992437431 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414009b30 con 0x7fc41c0b0900 2024-01-20T11:18:23.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.941+0000 7fc420da4640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41c0b0900 0x7fc41c1d6f90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0ade50 tx=0x7fc414004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:23.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.941+0000 7fc418ff9640 10 client.?.objecter ms_handle_connect 0x7fc41c0b0900 2024-01-20T11:18:23.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.942+0000 7fc418ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 1 -- 172.21.15.110:0/2992437431 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414040670 con 0x7fc41c0b0900 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 10 monclient: dump: 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 1 -- 172.21.15.110:0/2992437431 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4140407d0 con 0x7fc41c0b0900 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 1 -- 172.21.15.110:0/2992437431 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414040bc0 con 0x7fc41c0b0900 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc422807640 5 monclient: authenticate success, global_id 4843 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc422807640 10 monclient: _renew_subs 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc422807640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41c11d960 con 0x7fc41c0b0900 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc41a7fc640 4 set_mon_vals no callback set 2024-01-20T11:18:23.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc418ff9640 10 monclient: dump: 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc422807640 10 client.4843.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:23.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc422807640 10 monclient: _renew_subs 2024-01-20T11:18:23.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc422807640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:23.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c11dc60 con 0x7fc41c0b0900 2024-01-20T11:18:23.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.945+0000 7fc422807640 1 librados: init done 2024-01-20T11:18:23.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.946+0000 7fc422807640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:23.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.946+0000 7fc422807640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:23.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.946+0000 7fc422807640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:23.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:23.946+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c0b5860 con 0x7fc41c0b0900 2024-01-20T11:18:24.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.204+0000 7fc418ff9640 1 -- 172.21.15.110:0/2992437431 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc414040d20 con 0x7fc41c0b0900 2024-01-20T11:18:24.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.204+0000 7fc418ff9640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc40806c8b0 0x7fc40806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:24.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.204+0000 7fc41bfff640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc40806c8b0 0x7fc40806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:24.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.204+0000 7fc41bfff640 10 monclient: get_auth_request con 0x7fc40806c8b0 auth_method 0 2024-01-20T11:18:24.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.205+0000 7fc41bfff640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc40806c8b0 0x7fc40806ed70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009b40 tx=0x7fc40c009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:24.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.205+0000 7fc418ff9640 10 client.4843.objecter ms_handle_connect 0x7fc40806c8b0 2024-01-20T11:18:24.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.207+0000 7fc418ff9640 1 -- 172.21.15.110:0/2992437431 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc41407b400 con 0x7fc41c0b0900 2024-01-20T11:18:24.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.207+0000 7fc418ff9640 10 client.4843.objecter ms_dispatch 0x7fc41c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:24.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.207+0000 7fc418ff9640 3 client.4843.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:24.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.207+0000 7fc418ff9640 3 client.4843.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:24.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.207+0000 7fc418ff9640 20 client.4843.objecter dump_active .. 0 homeless 2024-01-20T11:18:24.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.218+0000 7fc418ff9640 1 -- 172.21.15.110:0/2992437431 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc414077740 con 0x7fc41c0b0900 2024-01-20T11:18:24.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.218+0000 7fc418ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:24.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.218+0000 7fc418ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:24.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.382+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc41c1461f0 con 0x7fc40806c8b0 2024-01-20T11:18:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.385+0000 7fc418ff9640 1 -- 172.21.15.110:0/2992437431 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64052 (secure 0 0 0) 0x7fc40c011a60 con 0x7fc40806c8b0 2024-01-20T11:18:24.385 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:24.386 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 10 librados: watch_flush enter 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 10 librados: watch_flush exit 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 20 client.4843.objecter shutdown clearing up homeless session... 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 10 client.4843.objecter successfully canceled tick 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc40806c8b0 msgr2=0x7fc40806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc40806c8b0 0x7fc40806ed70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009b40 tx=0x7fc40c009190 comp rx=0 tx=0).stop 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 10 monclient: shutdown 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41c0b0900 msgr2=0x7fc41c1d6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41c0b0900 0x7fc41c1d6f90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0ade50 tx=0x7fc414004a40 comp rx=0 tx=0).stop 2024-01-20T11:18:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 shutdown_connections 2024-01-20T11:18:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc40806c8b0 0x7fc40806ed70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41c0b0900 0x7fc41c1d6f90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc41c0af790 0x7fc41c0b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 --2- 172.21.15.110:0/2992437431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41c06bb00 0x7fc41c141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 >> 172.21.15.110:0/2992437431 conn(0x7fc41c066d60 msgr2=0x7fc41c1428e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 shutdown_connections 2024-01-20T11:18:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 -- 172.21.15.110:0/2992437431 wait complete. 2024-01-20T11:18:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:24.387+0000 7fc422807640 1 librados: shutdown 2024-01-20T11:18:24.397 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-01-20T11:18:22.746043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002093"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.4640000000000004},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.8869999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.6420000000000003},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.818}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.6229999999999993},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.631}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.4649999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":10.58}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.177},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":16.094000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.906000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":15.224}]}]},{"osd":7,"up_from":21,"seq":90194313258,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.9550000000000001},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.052}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.9380000000000006},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.2219999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":9.8130000000000006},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":15.975}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.945},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.991}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.686999999999999},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":7.2169999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.94},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.997}]}]},{"osd":3,"up_from":21,"seq":90194313259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.54},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.2290000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9860000000000007},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":17.704000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":11.539},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":6.8840000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.778},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.332000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":9.6609999999999996},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":15.587}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":14.313000000000001},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.271000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.149},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":7.5}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.8100000000000005},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.867}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.686},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.1159999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.442},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.824}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.666},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.111000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.531000000000001},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":11.103}]}]},{"osd":1,"up_from":17,"seq":73014444073,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.7279999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":10.803000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.667999999999999},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.6749999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":14.710000000000001,"5min":24.565999999999999,"15min":26.209},"min":{"1min":9.3330000000000002,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":31.931000000000001,"5min":259.483,"15min":259.483},"last":14.587999999999999},{"interface":"front","average":{"1min":11.055999999999999,"5min":19.523,"15min":20.934000000000001},"min":{"1min":6.0860000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":31.634,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.5570000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":11.475},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.3159999999999998},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":12.053000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:21 2024","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":28.591999999999999,"15min":31.047999999999998},"min":{"1min":8.9039999999999999,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":28.774000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.1210000000000004},{"interface":"front","average":{"1min":16.062000000000001,"5min":29.241,"15min":31.437000000000001},"min":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":11.374000000000001},"max":{"1min":26.094000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.256}]}]},{"osd":0,"up_from":17,"seq":73014444073,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":9.3699999999999992,"5min":14.177,"15min":14.978},"min":{"1min":5.4509999999999996,"5min":4.9400000000000004,"15min":4.9400000000000004},"max":{"1min":17.498999999999999,"5min":114.843,"15min":114.843},"last":5.609},{"interface":"front","average":{"1min":9.0310000000000006,"5min":14.859,"15min":15.83},"min":{"1min":5.5759999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":13.433999999999999,"5min":121.044,"15min":121.044},"last":8.1280000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":7.6580000000000004,"5min":27.138000000000002,"15min":30.384},"min":{"1min":5.984,"5min":5.3079999999999998,"15min":5.3079999999999998},"max":{"1min":14.548999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.4290000000000003},{"interface":"front","average":{"1min":11.236000000000001,"5min":29.050000000000001,"15min":32.018999999999998},"min":{"1min":6.7430000000000003,"5min":6.7430000000000003,"15min":6.7430000000000003},"max":{"1min":15.898999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.7059999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":10.371},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.888}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.927},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7520000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:16 2024","interfaces":[{"interface":"back","average":{"1min":13.737,"5min":31.661999999999999,"15min":34.649999999999999},"min":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":18.321000000000002,"5min":375.733,"15min":375.733},"last":13.225},{"interface":"front","average":{"1min":13.366,"5min":29.463999999999999,"15min":32.146999999999998},"min":{"1min":9.6110000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":15.999000000000001,"5min":374.654,"15min":374.654},"last":12.539}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.114000000000001},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":9.343}]}]},{"osd":6,"up_from":15,"seq":64424509482,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":9.2949999999999999},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":12.449999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.9349999999999996},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":5.7839999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.6170000000000009},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.396000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":14.022,"5min":15.833,"15min":16.135000000000002},"min":{"1min":8.9990000000000006,"5min":7.1870000000000003,"15min":7.1870000000000003},"max":{"1min":17.619,"5min":81.373999999999995,"15min":81.373999999999995},"last":7.1429999999999998},{"interface":"front","average":{"1min":12.282999999999999,"5min":15.798999999999999,"15min":16.385000000000002},"min":{"1min":7.6319999999999997,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":19.268999999999998,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.182}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.878},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":10.827999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:22 2024","interfaces":[{"interface":"back","average":{"1min":15.228999999999999,"5min":16.449000000000002,"15min":16.652000000000001},"min":{"1min":9.9700000000000006,"5min":9.9700000000000006,"15min":9.9700000000000006},"max":{"1min":18.527999999999999,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.071},{"interface":"front","average":{"1min":15.112,"5min":16,"15min":16.148},"min":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"max":{"1min":18.998999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.993}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:27.400 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.500+0000 7fdcb0ba4640 10 monclient: get_monmap_and_config 2024-01-20T11:18:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.500+0000 7fdcb0ba4640 10 monclient: build_initial_monmap 2024-01-20T11:18:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.500+0000 7fdcb0ba4640 10 monclient: monmap: 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:27.501414+0000 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:27.501414+0000 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:27.501 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.501+0000 7fdcb0ba4640 1 Processor -- start 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.501+0000 7fdcb0ba4640 1 -- start start 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.501+0000 7fdcb0ba4640 10 monclient: init 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdcac06f920 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdcac06fd00 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdcac144060 0x7fdcac14b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 10 monclient(hunting): picked mon.noname-a con 0x7fdcac144060 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac0715b0 con 0x7fdcac06fd00 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac071890 con 0x7fdcac06f920 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:27.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcac071b70 con 0x7fdcac144060 2024-01-20T11:18:27.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:27.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcb0ba4640 10 monclient(hunting): authenticate will time out at 1550.852515s 2024-01-20T11:18:27.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcaad9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdcac144060 0x7fdcac14b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:27.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.502+0000 7fdcaa59c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fdcac06f920 0x7fdcac13d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:46930/0 (socket says 172.21.15.110:46930) 2024-01-20T11:18:27.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.503+0000 7fdcaa59c640 1 -- 172.21.15.110:0/2484772521 learned_addr learned my addr 172.21.15.110:0/2484772521 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:27.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.503+0000 7fdcaad9d640 10 monclient(hunting): get_auth_request con 0x7fdcac144060 auth_method 0 2024-01-20T11:18:27.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.503+0000 7fdcaad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:27.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.503+0000 7fdcaad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:27.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.503+0000 7fdcaad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:27.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.504+0000 7fdcaad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:27.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.504+0000 7fdcaad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.504+0000 7fdcaad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdca959a640 1 -- 172.21.15.110:0/2484772521 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64126226 0 0) 0x7fdcac071890 con 0x7fdcac06f920 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdca959a640 10 monclient(hunting): _negotiate 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdca959a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdca959a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdca959a640 10 monclient(hunting): my global_id is 4846 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdca959a640 1 -- 172.21.15.110:0/2484772521 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc94003520 con 0x7fdcac06f920 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdcaad9d640 10 monclient(hunting): handle_auth_done global_id 4761 payload 274 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdcaad9d640 1 -- 172.21.15.110:0/2484772521 >> v1:172.21.15.134:6789/0 conn(0x7fdcac06f920 legacy=0x7fdcac13d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdcaad9d640 1 -- 172.21.15.110:0/2484772521 >> v1:172.21.15.110:6789/0 conn(0x7fdcac06fd00 legacy=0x7fdcac143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.505+0000 7fdcaad9d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.506+0000 7fdcaad9d640 1 monclient: found mon.noname-a 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.506+0000 7fdcaad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.506+0000 7fdcaad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:27.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.506+0000 7fdcaad9d640 1 -- 172.21.15.110:0/2484772521 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac143430 con 0x7fdcac144060 2024-01-20T11:18:27.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.509+0000 7fdcaad9d640 1 --2- 172.21.15.110:0/2484772521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 0x7fdcac14b730 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdca40097a0 tx=0x7fdca402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab5667debe4743f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:27.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 1 -- 172.21.15.110:0/2484772521 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca4031830 con 0x7fdcac144060 2024-01-20T11:18:27.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:27.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:27.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 10 monclient: dump: 2024-01-20T11:18:27.513 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:27.513 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 10 monclient: _finish_auth 0 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 10 monclient: _check_auth_tickets 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 1 -- 172.21.15.110:0/2484772521 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca4031990 con 0x7fdcac144060 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdcb0ba4640 5 monclient: authenticate success, global_id 4761 2024-01-20T11:18:27.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdcb0ba4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdcb0ba4640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdcb0ba4640 4 set_mon_vals no callback set 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 1 -- 172.21.15.110:0/2484772521 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca4031d20 con 0x7fdcac144060 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.513+0000 7fdca959a640 10 monclient: dump: 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:27.515 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:27.516 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:27.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.514+0000 7fdcb0ba4640 10 monclient: shutdown 2024-01-20T11:18:27.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.514+0000 7fdcb0ba4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:27.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.514+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2484772521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 msgr2=0x7fdcac14b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.514+0000 7fdcb0ba4640 1 --2- 172.21.15.110:0/2484772521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 0x7fdcac14b730 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdca40097a0 tx=0x7fdca402b8d0 comp rx=0 tx=0).stop 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.514+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2484772521 shutdown_connections 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.514+0000 7fdcb0ba4640 1 --2- 172.21.15.110:0/2484772521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 0x7fdcac14b730 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.514+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2484772521 >> 172.21.15.110:0/2484772521 conn(0x7fdcac066d60 msgr2=0x7fdcac0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.514+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2484772521 shutdown_connections 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.514+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2484772521 wait complete. 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.515+0000 7fdcb0ba4640 10 monclient: build_initial_monmap 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.515+0000 7fdcb0ba4640 10 monclient: monmap: 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:27.520 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.515+0000 7fdcb0ba4640 1 librados: starting msgr at 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.515+0000 7fdcb0ba4640 1 librados: starting objecter 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.515+0000 7fdcb0ba4640 1 Processor -- start 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.515+0000 7fdcb0ba4640 1 -- start start 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.515+0000 7fdcb0ba4640 1 librados: setting wanted keys 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.515+0000 7fdcb0ba4640 1 librados: calling monclient init 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.515+0000 7fdcb0ba4640 10 monclient: init 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcac06f920 0x7fdcac136270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:27.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient(hunting): picked mon.noname-c con 0x7fdcac06f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcac06fd00 0x7fdcac1d5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient(hunting): picked mon.noname-b con 0x7fdcac06fd00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 0x7fdcac1d5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient(hunting): picked mon.noname-a con 0x7fdcac144060 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac1d5120 con 0x7fdcac144060 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac13f800 con 0x7fdcac06fd00 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac13e710 con 0x7fdcac06f920 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcb0ba4640 10 monclient(hunting): authenticate will time out at 1550.866507s 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcaad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 0x7fdcac1d5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdca9d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcac06fd00 0x7fdcac1d5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:27.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.516+0000 7fdcaa59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcac06f920 0x7fdcac136270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 0x7fdcac1d5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60372/0 (socket says 172.21.15.110:60372) 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaad9d640 1 -- 172.21.15.110:0/2243592135 learned_addr learned my addr 172.21.15.110:0/2243592135 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaad9d640 10 monclient(hunting): get_auth_request con 0x7fdcac144060 auth_method 0 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdca9d9b640 10 monclient(hunting): get_auth_request con 0x7fdcac06fd00 auth_method 0 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdca9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdca9d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdca9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaa59c640 10 monclient(hunting): get_auth_request con 0x7fdcac06f920 auth_method 0 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaa59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:27.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaa59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:27.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdca9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:27.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdca9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:27.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdca9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:27.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:27.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:27.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:27.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:27.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:27.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.517+0000 7fdcaa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:27.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 10 monclient(hunting): handle_auth_done global_id 4849 payload 824 2024-01-20T11:18:27.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 1 -- 172.21.15.110:0/2243592135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcac06f920 msgr2=0x7fdcac136270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:27.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcac06f920 0x7fdcac136270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:27.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 1 -- 172.21.15.110:0/2243592135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 msgr2=0x7fdcac1d5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:27.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 0x7fdcac1d5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:27.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:18:27.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 1 monclient: found mon.noname-b 2024-01-20T11:18:27.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:27.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:27.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdca9d9b640 1 -- 172.21.15.110:0/2243592135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca4009400 con 0x7fdcac06fd00 2024-01-20T11:18:27.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.520+0000 7fdcaad9d640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 0x7fdcac1d5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:18:27.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.523+0000 7fdca9d9b640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcac06fd00 0x7fdcac1d5600 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdca000c3c0 tx=0x7fdca000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:27.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.523+0000 7fdc9affd640 10 client.?.objecter ms_handle_connect 0x7fdcac06fd00 2024-01-20T11:18:27.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.523+0000 7fdc9affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 1 -- 172.21.15.110:0/2243592135 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca00035f0 con 0x7fdcac06fd00 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 10 monclient: dump: 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:27.527 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 10 monclient: _finish_auth 0 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 1 -- 172.21.15.110:0/2243592135 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca0003750 con 0x7fdcac06fd00 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 1 -- 172.21.15.110:0/2243592135 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0003b80 con 0x7fdcac06fd00 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdc9affd640 10 monclient: dump: 2024-01-20T11:18:27.528 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdcb0ba4640 5 monclient: authenticate success, global_id 4849 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdcb0ba4640 10 monclient: _renew_subs 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdcb0ba4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcac133950 con 0x7fdcac06fd00 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.526+0000 7fdca8d99640 4 set_mon_vals no callback set 2024-01-20T11:18:27.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.527+0000 7fdcb0ba4640 10 client.4849.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:27.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.527+0000 7fdcb0ba4640 10 monclient: _renew_subs 2024-01-20T11:18:27.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.527+0000 7fdcb0ba4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:27.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.527+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcac13e3a0 con 0x7fdcac06fd00 2024-01-20T11:18:27.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.527+0000 7fdcb0ba4640 1 librados: init done 2024-01-20T11:18:27.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.527+0000 7fdcb0ba4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:27.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.527+0000 7fdcb0ba4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:27.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.527+0000 7fdcb0ba4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:27.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.527+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcac075490 con 0x7fdcac06fd00 2024-01-20T11:18:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.545+0000 7fdc9affd640 1 -- 172.21.15.110:0/2243592135 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdca0003ce0 con 0x7fdcac06fd00 2024-01-20T11:18:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.546+0000 7fdc9affd640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdc8c06c8b0 0x7fdc8c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.546+0000 7fdcaa59c640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdc8c06c8b0 0x7fdc8c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.546+0000 7fdcaa59c640 10 monclient: get_auth_request con 0x7fdc8c06c8b0 auth_method 0 2024-01-20T11:18:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.546+0000 7fdc9affd640 10 client.4849.objecter ms_handle_connect 0x7fdc8c06c8b0 2024-01-20T11:18:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.546+0000 7fdcaa59c640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdc8c06c8b0 0x7fdc8c06ed70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c001560 tx=0x7fdc9c001a90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:27.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.776+0000 7fdc9affd640 1 -- 172.21.15.110:0/2243592135 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdca0057b30 con 0x7fdcac06fd00 2024-01-20T11:18:27.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.776+0000 7fdc9affd640 10 client.4849.objecter ms_dispatch 0x7fdcac000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:27.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.776+0000 7fdc9affd640 3 client.4849.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:27.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.776+0000 7fdc9affd640 3 client.4849.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:27.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.777+0000 7fdc9affd640 20 client.4849.objecter dump_active .. 0 homeless 2024-01-20T11:18:27.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.790+0000 7fdc9affd640 1 -- 172.21.15.110:0/2243592135 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fdca0053e70 con 0x7fdcac06fd00 2024-01-20T11:18:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.790+0000 7fdc9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.790+0000 7fdc9affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:27.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.973+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdcac14bc70 con 0x7fdc8c06c8b0 2024-01-20T11:18:27.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.976+0000 7fdc9affd640 1 -- 172.21.15.110:0/2243592135 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63979 (secure 0 0 0) 0x7fdc9c001f60 con 0x7fdc8c06c8b0 2024-01-20T11:18:27.976 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:27.977 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:27.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 10 librados: watch_flush enter 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 10 librados: watch_flush exit 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 20 client.4849.objecter shutdown clearing up homeless session... 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 10 client.4849.objecter successfully canceled tick 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdc8c06c8b0 msgr2=0x7fdc8c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdc8c06c8b0 0x7fdc8c06ed70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c001560 tx=0x7fdc9c001a90 comp rx=0 tx=0).stop 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 10 monclient: shutdown 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcac06fd00 msgr2=0x7fdcac1d5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcac06fd00 0x7fdcac1d5600 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdca000c3c0 tx=0x7fdca000c880 comp rx=0 tx=0).stop 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 shutdown_connections 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdc8c06c8b0 0x7fdc8c06ed70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcac144060 0x7fdcac1d5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:27.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcac06fd00 0x7fdcac1d5600 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:27.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 --2- 172.21.15.110:0/2243592135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcac06f920 0x7fdcac136270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:27.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 >> 172.21.15.110:0/2243592135 conn(0x7fdcac066d60 msgr2=0x7fdcac0b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:27.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 shutdown_connections 2024-01-20T11:18:27.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 -- 172.21.15.110:0/2243592135 wait complete. 2024-01-20T11:18:27.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:27.978+0000 7fdcb0ba4640 1 librados: shutdown 2024-01-20T11:18:27.988 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-01-20T11:18:26.746721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002110"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.6890000000000001},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.3150000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":6.2549999999999999},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.427}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.147},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.518000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.462},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":16.052}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":16.849},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":13.574}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":15.234},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.731999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2809999999999997},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.7000000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.7089999999999996},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.257}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.397},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.797000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.8739999999999997},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.964}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.680999999999999},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":8.6419999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.632999999999999},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.534000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4690000000000003},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.3879999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.1799999999999997},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.894}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":13.050000000000001},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.8160000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.555},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.757999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":15.343999999999999},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.515000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":14.949},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.568999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444073,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":10.766999999999999},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":3.9630000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":15.331},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.364000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.467000000000001},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.7549999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":11.898},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":7.4269999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":9.0980000000000008},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.795999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.9969999999999999},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":13.137}]}]},{"osd":1,"up_from":17,"seq":73014444074,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":14.347},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":8.0060000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.891},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":15.398999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":16.975999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.384}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.5220000000000002},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.198}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":10.699},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":12.042999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.8640000000000008},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.404}]}]},{"osd":0,"up_from":17,"seq":73014444074,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":12.619999999999999},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":4.4400000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.5609999999999999},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.445}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":13.782999999999999},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.7949999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.846},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.5199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":15.648},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":12.035}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.079000000000001},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":10.233000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":4.9859999999999998},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.758}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.9379999999999997},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":9.3260000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.7029999999999994},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.464}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.163},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.388999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.134},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.803000000000001},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.987}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:27.990 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:28.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.090+0000 7fa7f30b4640 10 monclient: get_monmap_and_config 2024-01-20T11:18:28.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.090+0000 7fa7f30b4640 10 monclient: build_initial_monmap 2024-01-20T11:18:28.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.090+0000 7fa7f30b4640 10 monclient: monmap: 2024-01-20T11:18:28.090 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:28.090 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.090 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:28.091095+0000 2024-01-20T11:18:28.090 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:28.091095+0000 2024-01-20T11:18:28.091 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:28.091 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.091 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:28.091 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:28.091 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:28.091 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:28.091 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:28.091 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 1 Processor -- start 2024-01-20T11:18:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 1 -- start start 2024-01-20T11:18:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient: init 2024-01-20T11:18:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec06f920 0x7fa7ec06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ec06f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa7ec13d230 0x7fa7ec13d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient(hunting): picked mon.noname-c con 0x7fa7ec13d230 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7ec143950 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec0715b0 con 0x7fa7ec143950 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7ec071890 con 0x7fa7ec13d230 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec071b70 con 0x7fa7ec06f920 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.091+0000 7fa7f30b4640 10 monclient(hunting): authenticate will time out at 1551.442053s 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7f0e50640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec06f920 0x7fa7ec06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7ebfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa7ec13d230 0x7fa7ec13d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7f0e50640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec06f920 0x7fa7ec06fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45362/0 (socket says 172.21.15.110:45362) 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7ebfff640 10 monclient(hunting): get_auth_request con 0x7fa7ec13d230 auth_method 0 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7f0e50640 1 -- 172.21.15.110:0/1108311652 learned_addr learned my addr 172.21.15.110:0/1108311652 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7ebfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7f0e50640 10 monclient(hunting): get_auth_request con 0x7fa7ec06f920 auth_method 0 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7f0e50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7f0e50640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.092+0000 7fa7f0e50640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.093+0000 7fa7f0e50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.093+0000 7fa7f0e50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.093+0000 7fa7f0e50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7f0e50640 10 monclient(hunting): handle_auth_done global_id 4646 payload 274 2024-01-20T11:18:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7f0e50640 1 -- 172.21.15.110:0/1108311652 >> v2:172.21.15.134:3300/0 conn(0x7fa7ec13d230 msgr2=0x7fa7ec13d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7f0e50640 1 --2- 172.21.15.110:0/1108311652 >> v2:172.21.15.134:3300/0 conn(0x7fa7ec13d230 0x7fa7ec13d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7f0e50640 1 -- 172.21.15.110:0/1108311652 >> v1:172.21.15.134:6789/0 conn(0x7fa7ec143950 legacy=0x7fa7ec13fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7f0e50640 10 monclient: _finish_hunting 0 2024-01-20T11:18:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7f0e50640 1 monclient: found mon.noname-b 2024-01-20T11:18:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7f0e50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7f0e50640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.094+0000 7fa7f0e50640 1 -- 172.21.15.110:0/1108311652 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec143430 con 0x7fa7ec06f920 2024-01-20T11:18:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.096+0000 7fa7f0e50640 1 --2- 172.21.15.110:0/1108311652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec06f920 0x7fa7ec06fd00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009870 tx=0x7fa7e002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afc677ecdebfb300 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 1 -- 172.21.15.110:0/1108311652 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0031640 con 0x7fa7ec06f920 2024-01-20T11:18:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 10 monclient: dump: 2024-01-20T11:18:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 10 monclient: _finish_auth 0 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 1 -- 172.21.15.110:0/1108311652 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e00317a0 con 0x7fa7ec06f920 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7f30b4640 5 monclient: authenticate success, global_id 4646 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7f30b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7f30b4640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7f30b4640 4 set_mon_vals no callback set 2024-01-20T11:18:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 1 -- 172.21.15.110:0/1108311652 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0031b30 con 0x7fa7ec06f920 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.100+0000 7fa7eb7fe640 10 monclient: dump: 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:28.102 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 10 monclient: shutdown 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 1 -- 172.21.15.110:0/1108311652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec06f920 msgr2=0x7fa7ec06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 1 --2- 172.21.15.110:0/1108311652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec06f920 0x7fa7ec06fd00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009870 tx=0x7fa7e002b8f0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 1 -- 172.21.15.110:0/1108311652 shutdown_connections 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 1 --2- 172.21.15.110:0/1108311652 >> v2:172.21.15.134:3300/0 conn(0x7fa7ec13d230 0x7fa7ec13d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 1 --2- 172.21.15.110:0/1108311652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec06f920 0x7fa7ec06fd00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 1 -- 172.21.15.110:0/1108311652 >> 172.21.15.110:0/1108311652 conn(0x7fa7ec066d60 msgr2=0x7fa7ec0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 1 -- 172.21.15.110:0/1108311652 shutdown_connections 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 1 -- 172.21.15.110:0/1108311652 wait complete. 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 10 monclient: build_initial_monmap 2024-01-20T11:18:28.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.101+0000 7fa7f30b4640 10 monclient: monmap: 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.102+0000 7fa7f30b4640 1 librados: starting msgr at 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.102+0000 7fa7f30b4640 1 librados: starting objecter 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.102+0000 7fa7f30b4640 1 Processor -- start 2024-01-20T11:18:28.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.102+0000 7fa7f30b4640 1 -- start start 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.102+0000 7fa7f30b4640 1 librados: setting wanted keys 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.102+0000 7fa7f30b4640 1 librados: calling monclient init 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.102+0000 7fa7f30b4640 10 monclient: init 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec06f920 0x7fa7ec0ae750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient(hunting): picked mon.noname-a con 0x7fa7ec06f920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec13d230 0x7fa7ec1d69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient(hunting): picked mon.noname-c con 0x7fa7ec13d230 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7ec143950 0x7fa7ec136b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ec143950 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec0aa1a0 con 0x7fa7ec06f920 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec1d7420 con 0x7fa7ec143950 2024-01-20T11:18:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec1d6f20 con 0x7fa7ec13d230 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f30b4640 10 monclient(hunting): authenticate will time out at 1551.453291s 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7ebfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec13d230 0x7fa7ec1d69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f0e50640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec06f920 0x7fa7ec0ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f1651640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7ec143950 0x7fa7ec136b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f1651640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7ec143950 0x7fa7ec136b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:52360/0 (socket says 172.21.15.110:52360) 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f1651640 1 -- 172.21.15.110:0/500046596 learned_addr learned my addr 172.21.15.110:0/500046596 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f1651640 10 monclient(hunting): get_auth_request con 0x7fa7ec143950 auth_method 0 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f1651640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f1651640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7f1651640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.103+0000 7fa7ebfff640 10 monclient(hunting): get_auth_request con 0x7fa7ec13d230 auth_method 0 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7ebfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f0e50640 10 monclient(hunting): get_auth_request con 0x7fa7ec06f920 auth_method 0 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f0e50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f0e50640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f0e50640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f1651640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f1651640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f1651640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f0e50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f0e50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.104+0000 7fa7f0e50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 10 monclient(hunting): handle_auth_done global_id 4855 payload 824 2024-01-20T11:18:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 1 -- 172.21.15.110:0/500046596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec13d230 msgr2=0x7fa7ec1d69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec13d230 0x7fa7ec1d69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 1 -- 172.21.15.110:0/500046596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec06f920 msgr2=0x7fa7ec0ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec06f920 0x7fa7ec0ae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 10 monclient: _finish_hunting 0 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 1 monclient: found mon.noname-b 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.107+0000 7fa7f1651640 1 -- 172.21.15.110:0/500046596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00094d0 con 0x7fa7ec143950 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.109+0000 7fa7f1651640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7ec143950 0x7fa7ec136b10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400c7a0 tx=0x7fa7e400cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.109+0000 7fa7e97fa640 10 client.?.objecter ms_handle_connect 0x7fa7ec143950 2024-01-20T11:18:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.109+0000 7fa7e97fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 1 -- 172.21.15.110:0/500046596 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4003b70 con 0x7fa7ec143950 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 10 monclient: dump: 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:28.114 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 10 monclient: _finish_auth 0 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 10 monclient: _check_auth_tickets 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 1 -- 172.21.15.110:0/500046596 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e4003cf0 con 0x7fa7ec143950 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7f30b4640 5 monclient: authenticate success, global_id 4855 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7f30b4640 10 monclient: _renew_subs 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7f30b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ec1d71a0 con 0x7fa7ec143950 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 1 -- 172.21.15.110:0/500046596 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4003ea0 con 0x7fa7ec143950 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7e97fa640 10 monclient: dump: 2024-01-20T11:18:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7eaffd640 4 set_mon_vals no callback set 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7f30b4640 10 client.4855.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7f30b4640 10 monclient: _renew_subs 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7f30b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec137230 con 0x7fa7ec143950 2024-01-20T11:18:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.113+0000 7fa7f30b4640 1 librados: init done 2024-01-20T11:18:28.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.114+0000 7fa7f30b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:28.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.114+0000 7fa7f30b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:28.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.114+0000 7fa7f30b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:28.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.114+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ec075490 con 0x7fa7ec143950 2024-01-20T11:18:28.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.132+0000 7fa7e97fa640 1 -- 172.21.15.110:0/500046596 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa7e400aa90 con 0x7fa7ec143950 2024-01-20T11:18:28.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.132+0000 7fa7e97fa640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa7d006c8e0 0x7fa7d006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.132+0000 7fa7f0e50640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa7d006c8e0 0x7fa7d006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.132+0000 7fa7f0e50640 10 monclient: get_auth_request con 0x7fa7d006c8e0 auth_method 0 2024-01-20T11:18:28.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.133+0000 7fa7e97fa640 10 client.4855.objecter ms_handle_connect 0x7fa7d006c8e0 2024-01-20T11:18:28.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.133+0000 7fa7f0e50640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa7d006c8e0 0x7fa7d006eda0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0036000 tx=0x7fa7e003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.372+0000 7fa7e97fa640 1 -- 172.21.15.110:0/500046596 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa7e40589c0 con 0x7fa7ec143950 2024-01-20T11:18:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.372+0000 7fa7e97fa640 10 client.4855.objecter ms_dispatch 0x7fa7ec000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.372+0000 7fa7e97fa640 3 client.4855.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.372+0000 7fa7e97fa640 3 client.4855.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.372+0000 7fa7e97fa640 20 client.4855.objecter dump_active .. 0 homeless 2024-01-20T11:18:28.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.384+0000 7fa7e97fa640 1 -- 172.21.15.110:0/500046596 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa7e4054d00 con 0x7fa7ec143950 2024-01-20T11:18:28.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.384+0000 7fa7e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:28.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.384+0000 7fa7e97fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:28.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.559+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7ec0aa310 con 0x7fa7d006c8e0 2024-01-20T11:18:28.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.562+0000 7fa7e97fa640 1 -- 172.21.15.110:0/500046596 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63979 (secure 0 0 0) 0x7fa7ec0aa310 con 0x7fa7d006c8e0 2024-01-20T11:18:28.563 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:28.563 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:28.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 10 librados: watch_flush enter 2024-01-20T11:18:28.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 10 librados: watch_flush exit 2024-01-20T11:18:28.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 20 client.4855.objecter shutdown clearing up homeless session... 2024-01-20T11:18:28.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 10 client.4855.objecter successfully canceled tick 2024-01-20T11:18:28.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa7d006c8e0 msgr2=0x7fa7d006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa7d006c8e0 0x7fa7d006eda0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0036000 tx=0x7fa7e003c040 comp rx=0 tx=0).stop 2024-01-20T11:18:28.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 10 monclient: shutdown 2024-01-20T11:18:28.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:28.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7ec143950 msgr2=0x7fa7ec136b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7ec143950 0x7fa7ec136b10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400c7a0 tx=0x7fa7e400cc60 comp rx=0 tx=0).stop 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 shutdown_connections 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa7d006c8e0 0x7fa7d006eda0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7ec143950 0x7fa7ec136b10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec13d230 0x7fa7ec1d69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 --2- 172.21.15.110:0/500046596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec06f920 0x7fa7ec0ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 >> 172.21.15.110:0/500046596 conn(0x7fa7ec066d60 msgr2=0x7fa7ec0b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 shutdown_connections 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 -- 172.21.15.110:0/500046596 wait complete. 2024-01-20T11:18:28.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.564+0000 7fa7f30b4640 1 librados: shutdown 2024-01-20T11:18:28.575 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-01-20T11:18:26.746721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002110"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.6890000000000001},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.3150000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":6.2549999999999999},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.427}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.147},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.518000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.462},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":16.052}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":16.849},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":13.574}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":15.234},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":17.731999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2809999999999997},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.7000000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.7089999999999996},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.257}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.397},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.797000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.8739999999999997},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.964}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.680999999999999},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":8.6419999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.632999999999999},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.534000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4690000000000003},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.3879999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.1799999999999997},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.894}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":13.050000000000001},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.8160000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.555},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.757999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":15.343999999999999},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.515000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":14.949},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.568999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444073,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":10.766999999999999},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":3.9630000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":15.331},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.364000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.932,"5min":16.798999999999999,"15min":17.443000000000001},"min":{"1min":8.8620000000000001,"5min":8.8620000000000001,"15min":8.8620000000000001},"max":{"1min":17.277999999999999,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.467000000000001},{"interface":"front","average":{"1min":11.145,"5min":14.260999999999999,"15min":14.781000000000001},"min":{"1min":6.1299999999999999,"5min":6.1299999999999999,"15min":6.1299999999999999},"max":{"1min":16.324999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.7549999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":11.898},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":7.4269999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":9.0980000000000008},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.795999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.468,"5min":13.567,"15min":13.75},"min":{"1min":7.8680000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":16.945,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.9969999999999999},{"interface":"front","average":{"1min":15.800000000000001,"5min":17.632000000000001,"15min":17.937999999999999},"min":{"1min":11.859,"5min":11.430999999999999,"15min":11.430999999999999},"max":{"1min":19.43,"5min":28.148,"15min":28.148},"last":13.137}]}]},{"osd":1,"up_from":17,"seq":73014444074,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":14.347},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":8.0060000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.891},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":15.398999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":16.975999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.384}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":9.5220000000000002},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.198}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":10.699},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":12.042999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.8640000000000008},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.404}]}]},{"osd":0,"up_from":17,"seq":73014444074,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":12.619999999999999},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":4.4400000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.5609999999999999},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.445}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":13.782999999999999},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.7949999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.846},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.5199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":15.648},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":12.035}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.079000000000001},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":10.233000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":4.9859999999999998},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.758}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.9379999999999997},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":9.3260000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.7029999999999994},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.464}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.163},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.388999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.134},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.803000000000001},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.987}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:28.576 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:28.576 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:28.576 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:28.577 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:28.577 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:28.577 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:28.577 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:28.577 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:28.577 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.675+0000 7fee45341640 10 monclient: get_monmap_and_config 2024-01-20T11:18:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.675+0000 7fee45341640 10 monclient: build_initial_monmap 2024-01-20T11:18:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.675+0000 7fee45341640 10 monclient: monmap: 2024-01-20T11:18:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:28.675953+0000 2024-01-20T11:18:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:28.675953+0000 2024-01-20T11:18:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:28.676 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 1 Processor -- start 2024-01-20T11:18:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 1 -- start start 2024-01-20T11:18:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient: init 2024-01-20T11:18:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee400a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fee400a9ad0 0x7fee401423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient(hunting): picked mon.noname-c con 0x7fee400a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee40142920 0x7fee4014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient(hunting): picked mon.noname-b con 0x7fee40142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee400b25c0 con 0x7fee400a96f0 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee400aaea0 con 0x7fee400a9ad0 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4013fbe0 con 0x7fee40142920 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.676+0000 7fee45341640 10 monclient(hunting): authenticate will time out at 1552.027147s 2024-01-20T11:18:28.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee37fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fee400a9ad0 0x7fee401423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee3f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee40142920 0x7fee4014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee37fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fee400a9ad0 0x7fee401423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:52362/0 (socket says 172.21.15.110:52362) 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee37fff640 1 -- 172.21.15.110:0/3055324889 learned_addr learned my addr 172.21.15.110:0/3055324889 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee37fff640 10 monclient(hunting): get_auth_request con 0x7fee400a9ad0 auth_method 0 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee37fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee37fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee3f7fe640 10 monclient(hunting): get_auth_request con 0x7fee40142920 auth_method 0 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee3f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.677+0000 7fee3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.678+0000 7fee37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.678+0000 7fee37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.678+0000 7fee37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.678+0000 7fee3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.678+0000 7fee3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.678+0000 7fee3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.680+0000 7fee37fff640 10 monclient(hunting): handle_auth_done global_id 4858 payload 274 2024-01-20T11:18:28.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.680+0000 7fee37fff640 1 -- 172.21.15.110:0/3055324889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee40142920 msgr2=0x7fee4014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.680+0000 7fee37fff640 1 --2- 172.21.15.110:0/3055324889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee40142920 0x7fee4014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.680+0000 7fee37fff640 1 -- 172.21.15.110:0/3055324889 >> v1:172.21.15.110:6789/0 conn(0x7fee400a96f0 legacy=0x7fee4013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.680+0000 7fee37fff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.680+0000 7fee37fff640 1 monclient: found mon.noname-c 2024-01-20T11:18:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.680+0000 7fee37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.680+0000 7fee37fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.680+0000 7fee37fff640 1 -- 172.21.15.110:0/3055324889 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee400ab0a0 con 0x7fee400a9ad0 2024-01-20T11:18:28.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.681+0000 7fee37fff640 1 --2- 172.21.15.110:0/3055324889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a9ad0 0x7fee401423e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fee28009870 tx=0x7fee2802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b37477aa78e99c0d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:28.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 1 -- 172.21.15.110:0/3055324889 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28031640 con 0x7fee400a9ad0 2024-01-20T11:18:28.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:28.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:28.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 10 monclient: dump: 2024-01-20T11:18:28.685 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:28.685 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 1 -- 172.21.15.110:0/3055324889 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee280317a0 con 0x7fee400a9ad0 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 1 -- 172.21.15.110:0/3055324889 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28031b30 con 0x7fee400a9ad0 2024-01-20T11:18:28.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee3e7fc640 10 monclient: dump: 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee45341640 5 monclient: authenticate success, global_id 4858 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee45341640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:28.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.685+0000 7fee45341640 4 set_mon_vals no callback set 2024-01-20T11:18:28.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 10 monclient: shutdown 2024-01-20T11:18:28.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:28.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 1 -- 172.21.15.110:0/3055324889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a9ad0 msgr2=0x7fee401423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 1 --2- 172.21.15.110:0/3055324889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a9ad0 0x7fee401423e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fee28009870 tx=0x7fee2802b8f0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 1 -- 172.21.15.110:0/3055324889 shutdown_connections 2024-01-20T11:18:28.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 1 --2- 172.21.15.110:0/3055324889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee40142920 0x7fee4014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 1 --2- 172.21.15.110:0/3055324889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a9ad0 0x7fee401423e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 1 -- 172.21.15.110:0/3055324889 >> 172.21.15.110:0/3055324889 conn(0x7fee400b7bc0 msgr2=0x7fee400b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:28.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 1 -- 172.21.15.110:0/3055324889 shutdown_connections 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.686+0000 7fee45341640 1 -- 172.21.15.110:0/3055324889 wait complete. 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.687+0000 7fee45341640 10 monclient: build_initial_monmap 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.687+0000 7fee45341640 10 monclient: monmap: 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:28.692 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.687+0000 7fee45341640 1 librados: starting msgr at 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.687+0000 7fee45341640 1 librados: starting objecter 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.687+0000 7fee45341640 1 Processor -- start 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.687+0000 7fee45341640 1 -- start start 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.687+0000 7fee45341640 1 librados: setting wanted keys 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.687+0000 7fee45341640 1 librados: calling monclient init 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.687+0000 7fee45341640 10 monclient: init 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a96f0 0x7fee40138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient(hunting): picked mon.noname-b con 0x7fee400a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee400a9ad0 0x7fee401d7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient(hunting): picked mon.noname-c con 0x7fee400a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:28.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee40142920 0x7fee400af280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient(hunting): picked mon.noname-a con 0x7fee40142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee400b0930 con 0x7fee40142920 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee400b1390 con 0x7fee400a96f0 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee401d7d90 con 0x7fee400a9ad0 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee45341640 10 monclient(hunting): authenticate will time out at 1552.038304s 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee37fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee400a9ad0 0x7fee401d7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee3effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a96f0 0x7fee40138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee3f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee40142920 0x7fee400af280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee37fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee400a9ad0 0x7fee401d7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45400/0 (socket says 172.21.15.110:45400) 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee37fff640 1 -- 172.21.15.110:0/1956080280 learned_addr learned my addr 172.21.15.110:0/1956080280 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:28.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee37fff640 10 monclient(hunting): get_auth_request con 0x7fee400a9ad0 auth_method 0 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee37fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee37fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee3effd640 10 monclient(hunting): get_auth_request con 0x7fee400a96f0 auth_method 0 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee3effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.688+0000 7fee3effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3f7fe640 10 monclient(hunting): get_auth_request con 0x7fee40142920 auth_method 0 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:28.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:28.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.689+0000 7fee3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 10 monclient(hunting): handle_auth_done global_id 4861 payload 824 2024-01-20T11:18:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 1 -- 172.21.15.110:0/1956080280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee400a9ad0 msgr2=0x7fee401d7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee400a9ad0 0x7fee401d7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 1 -- 172.21.15.110:0/1956080280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee40142920 msgr2=0x7fee400af280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee40142920 0x7fee400af280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 10 monclient: _finish_hunting 0 2024-01-20T11:18:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 1 monclient: found mon.noname-b 2024-01-20T11:18:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:28.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.692+0000 7fee3effd640 1 -- 172.21.15.110:0/1956080280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee280094d0 con 0x7fee400a96f0 2024-01-20T11:18:28.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.694+0000 7fee3effd640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a96f0 0x7fee40138ee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fee3000c340 tx=0x7fee3000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:28.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.694+0000 7fee377fe640 10 client.?.objecter ms_handle_connect 0x7fee400a96f0 2024-01-20T11:18:28.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.694+0000 7fee377fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 1 -- 172.21.15.110:0/1956080280 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300052d0 con 0x7fee400a96f0 2024-01-20T11:18:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 10 monclient: dump: 2024-01-20T11:18:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.699 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 10 monclient: _finish_auth 0 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 10 monclient: _check_auth_tickets 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 1 -- 172.21.15.110:0/1956080280 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee30005450 con 0x7fee400a96f0 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee45341640 5 monclient: authenticate success, global_id 4861 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 1 -- 172.21.15.110:0/1956080280 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee30007260 con 0x7fee400a96f0 2024-01-20T11:18:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee377fe640 10 monclient: dump: 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee45341640 10 monclient: _renew_subs 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee3dffb640 4 set_mon_vals no callback set 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee45341640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee400b1660 con 0x7fee400a96f0 2024-01-20T11:18:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee45341640 10 client.4861.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee45341640 10 monclient: _renew_subs 2024-01-20T11:18:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee45341640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee400af7c0 con 0x7fee400a96f0 2024-01-20T11:18:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.698+0000 7fee45341640 1 librados: init done 2024-01-20T11:18:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.699+0000 7fee45341640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.699+0000 7fee45341640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.699+0000 7fee45341640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.699+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee40075490 con 0x7fee400a96f0 2024-01-20T11:18:28.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.719+0000 7fee377fe640 1 -- 172.21.15.110:0/1956080280 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fee300073c0 con 0x7fee400a96f0 2024-01-20T11:18:28.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.719+0000 7fee377fe640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee2006c8b0 0x7fee2006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:28.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.719+0000 7fee37fff640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee2006c8b0 0x7fee2006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:28.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.719+0000 7fee37fff640 10 monclient: get_auth_request con 0x7fee2006c8b0 auth_method 0 2024-01-20T11:18:28.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.720+0000 7fee377fe640 10 client.4861.objecter ms_handle_connect 0x7fee2006c8b0 2024-01-20T11:18:28.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.720+0000 7fee37fff640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee2006c8b0 0x7fee2006ed70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fee2802b930 tx=0x7fee2803c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:28.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.956+0000 7fee377fe640 1 -- 172.21.15.110:0/1956080280 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fee30057dc0 con 0x7fee400a96f0 2024-01-20T11:18:28.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.956+0000 7fee377fe640 10 client.4861.objecter ms_dispatch 0x7fee40000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:28.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.956+0000 7fee377fe640 3 client.4861.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:28.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.956+0000 7fee377fe640 3 client.4861.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:28.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.956+0000 7fee377fe640 20 client.4861.objecter dump_active .. 0 homeless 2024-01-20T11:18:28.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.967+0000 7fee377fe640 1 -- 172.21.15.110:0/1956080280 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fee30054100 con 0x7fee400a96f0 2024-01-20T11:18:28.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.967+0000 7fee377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:28.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:28.967+0000 7fee377fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:29.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.204+0000 7fee45341640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:29.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.204+0000 7fee45341640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:29.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.204+0000 7fee45341640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:29.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.204+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fee400b66a0 con 0x7fee400a96f0 2024-01-20T11:18:29.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.210+0000 7fee377fe640 1 -- 172.21.15.110:0/1956080280 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fee30057720 con 0x7fee400a96f0 2024-01-20T11:18:29.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.210+0000 7fee377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:29.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.210+0000 7fee377fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:29.210 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:29.210 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":243,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-20T11:16:30.727603+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:29.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 10 librados: watch_flush enter 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 10 librados: watch_flush exit 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 20 client.4861.objecter shutdown clearing up homeless session... 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 10 client.4861.objecter successfully canceled tick 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee2006c8b0 msgr2=0x7fee2006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee2006c8b0 0x7fee2006ed70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fee2802b930 tx=0x7fee2803c040 comp rx=0 tx=0).stop 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 10 monclient: shutdown 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a96f0 msgr2=0x7fee40138ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a96f0 0x7fee40138ee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fee3000c340 tx=0x7fee3000c800 comp rx=0 tx=0).stop 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 shutdown_connections 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fee2006c8b0 0x7fee2006ed70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee40142920 0x7fee400af280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee400a9ad0 0x7fee401d7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 --2- 172.21.15.110:0/1956080280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee400a96f0 0x7fee40138ee0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 >> 172.21.15.110:0/1956080280 conn(0x7fee400b7bc0 msgr2=0x7fee400b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:29.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 shutdown_connections 2024-01-20T11:18:29.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 -- 172.21.15.110:0/1956080280 wait complete. 2024-01-20T11:18:29.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.212+0000 7fee45341640 1 librados: shutdown 2024-01-20T11:18:29.223 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:29.223 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.323+0000 7f8048d01640 10 monclient: get_monmap_and_config 2024-01-20T11:18:29.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.323+0000 7f8048d01640 10 monclient: build_initial_monmap 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.323+0000 7f8048d01640 10 monclient: monmap: 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:29.324492+0000 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:29.324492+0000 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:29.324 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:29.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.324+0000 7f8048d01640 1 Processor -- start 2024-01-20T11:18:29.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.324+0000 7f8048d01640 1 -- start start 2024-01-20T11:18:29.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.324+0000 7f8048d01640 10 monclient: init 2024-01-20T11:18:29.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8044066680 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f804406dcf0 0x7f8044143270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 10 monclient(hunting): picked mon.noname-c con 0x7f804406dcf0 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f804413d610 0x7f80441437b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 10 monclient(hunting): picked mon.noname-a con 0x7f804413d610 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804406e2f0 con 0x7f8044066680 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8044072140 con 0x7f804413d610 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8044072420 con 0x7f804406dcf0 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8048d01640 10 monclient(hunting): authenticate will time out at 1552.675520s 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.325+0000 7f8042d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f804413d610 0x7f80441437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:29.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f8041d9b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f804406dcf0 0x7f8044143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f804259c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8044066680 0x7f804413cf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55928/0 (socket says 172.21.15.110:55928) 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f804259c640 1 -- 172.21.15.110:0/1669098904 learned_addr learned my addr 172.21.15.110:0/1669098904 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f8042d9d640 10 monclient(hunting): get_auth_request con 0x7f804413d610 auth_method 0 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f8042d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f8042d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f8042d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f8041d9b640 10 monclient(hunting): get_auth_request con 0x7f804406dcf0 auth_method 0 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f8041d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f8041d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.326+0000 7f8041d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.327+0000 7f8041d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.327+0000 7f8041d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:29.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.327+0000 7f8041d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:29.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.327+0000 7f8042d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:29.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.327+0000 7f8042d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:29.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.327+0000 7f8042d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.328+0000 7f8041d9b640 10 monclient(hunting): handle_auth_done global_id 4864 payload 274 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.328+0000 7f8041d9b640 1 -- 172.21.15.110:0/1669098904 >> v2:172.21.15.110:3300/0 conn(0x7f804413d610 msgr2=0x7f80441437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.328+0000 7f8041d9b640 1 --2- 172.21.15.110:0/1669098904 >> v2:172.21.15.110:3300/0 conn(0x7f804413d610 0x7f80441437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.328+0000 7f8041d9b640 1 -- 172.21.15.110:0/1669098904 >> v1:172.21.15.110:6789/0 conn(0x7f8044066680 legacy=0x7f804413cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.328+0000 7f8041d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.328+0000 7f8041d9b640 1 monclient: found mon.noname-c 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.328+0000 7f8041d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.328+0000 7f8041d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.328+0000 7f8041d9b640 1 -- 172.21.15.110:0/1669098904 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8044072700 con 0x7f804406dcf0 2024-01-20T11:18:29.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.331+0000 7f8041d9b640 1 --2- 172.21.15.110:0/1669098904 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f804406dcf0 0x7f8044143270 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80380097a0 tx=0x7f803802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f02962dd9991b502 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:29.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.334+0000 7f804159a640 1 -- 172.21.15.110:0/1669098904 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038031830 con 0x7f804406dcf0 2024-01-20T11:18:29.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:29.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:29.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 10 monclient: dump: 2024-01-20T11:18:29.335 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:29.335 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:29.335 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:29.335 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:29.335 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 10 monclient: _finish_auth 0 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 10 monclient: _check_auth_tickets 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 1 -- 172.21.15.110:0/1669098904 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8038031990 con 0x7f804406dcf0 2024-01-20T11:18:29.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 5 monclient: authenticate success, global_id 4864 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 4 set_mon_vals no callback set 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 1 -- 172.21.15.110:0/1669098904 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038031d20 con 0x7f804406dcf0 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f804159a640 10 monclient: dump: 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:29.337 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:29.338 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:29.338 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:29.338 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:29.338 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:29.338 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:29.338 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:29.338 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:29.338 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 10 monclient: shutdown 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 1 -- 172.21.15.110:0/1669098904 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f804406dcf0 msgr2=0x7f8044143270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 1 --2- 172.21.15.110:0/1669098904 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f804406dcf0 0x7f8044143270 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80380097a0 tx=0x7f803802b8d0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 1 -- 172.21.15.110:0/1669098904 shutdown_connections 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 1 --2- 172.21.15.110:0/1669098904 >> v2:172.21.15.110:3300/0 conn(0x7f804413d610 0x7f80441437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 1 --2- 172.21.15.110:0/1669098904 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f804406dcf0 0x7f8044143270 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.335+0000 7f8048d01640 1 -- 172.21.15.110:0/1669098904 >> 172.21.15.110:0/1669098904 conn(0x7f80440b09e0 msgr2=0x7f80440b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.336+0000 7f8048d01640 1 -- 172.21.15.110:0/1669098904 shutdown_connections 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.336+0000 7f8048d01640 1 -- 172.21.15.110:0/1669098904 wait complete. 2024-01-20T11:18:29.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.337+0000 7f8048d01640 10 monclient: build_initial_monmap 2024-01-20T11:18:29.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.337+0000 7f8048d01640 10 monclient: monmap: 2024-01-20T11:18:29.343 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:29.343 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:29.343 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:29.343 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:29.343 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:29.343 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:29.343 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:29.343 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.337+0000 7f8048d01640 1 librados: starting msgr at 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.337+0000 7f8048d01640 1 librados: starting objecter 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.337+0000 7f8048d01640 1 Processor -- start 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.337+0000 7f8048d01640 1 -- start start 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.337+0000 7f8048d01640 1 librados: setting wanted keys 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.337+0000 7f8048d01640 1 librados: calling monclient init 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.337+0000 7f8048d01640 10 monclient: init 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:18:29.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8044066680 0x7f80440ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient(hunting): picked mon.noname-c con 0x7f8044066680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f804406dcf0 0x7f80440acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient(hunting): picked mon.noname-b con 0x7f804406dcf0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f804413d610 0x7f80440a9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient(hunting): picked mon.noname-a con 0x7f804413d610 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80440aaab0 con 0x7f804413d610 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8044139600 con 0x7f804406dcf0 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80441398e0 con 0x7f8044066680 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:29.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.338+0000 7f8048d01640 10 monclient(hunting): authenticate will time out at 1552.688825s 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8041d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f804406dcf0 0x7f80440acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f804259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8044066680 0x7f80440ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8042d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f804413d610 0x7f80440a9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8042d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f804413d610 0x7f80440a9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60392/0 (socket says 172.21.15.110:60392) 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8042d9d640 1 -- 172.21.15.110:0/3433106741 learned_addr learned my addr 172.21.15.110:0/3433106741 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8042d9d640 10 monclient(hunting): get_auth_request con 0x7f804413d610 auth_method 0 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8042d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8042d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8042d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8041d9b640 10 monclient(hunting): get_auth_request con 0x7f804406dcf0 auth_method 0 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8041d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8041d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:29.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f8041d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f804259c640 10 monclient(hunting): get_auth_request con 0x7f8044066680 auth_method 0 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f804259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f804259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.339+0000 7f804259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.340+0000 7f8042d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.340+0000 7f8042d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.340+0000 7f8042d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.340+0000 7f8041d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.340+0000 7f8041d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.340+0000 7f8041d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.340+0000 7f804259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.340+0000 7f804259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:29.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.340+0000 7f804259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:29.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.342+0000 7f8042d9d640 10 monclient(hunting): handle_auth_done global_id 4785 payload 824 2024-01-20T11:18:29.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.342+0000 7f8042d9d640 1 -- 172.21.15.110:0/3433106741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8044066680 msgr2=0x7f80440ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:29.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.342+0000 7f8042d9d640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8044066680 0x7f80440ac480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.342+0000 7f8042d9d640 1 -- 172.21.15.110:0/3433106741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f804406dcf0 msgr2=0x7f80440acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:29.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.342+0000 7f8042d9d640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f804406dcf0 0x7f80440acc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.342+0000 7f8042d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.343+0000 7f8042d9d640 1 monclient: found mon.noname-a 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.343+0000 7f8042d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.343+0000 7f8042d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.343+0000 7f8042d9d640 1 -- 172.21.15.110:0/3433106741 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8038009400 con 0x7f804413d610 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.345+0000 7f8042d9d640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f804413d610 0x7f80440a9480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f803c00e7f0 tx=0x7f803c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.345+0000 7f8032ffd640 10 client.?.objecter ms_handle_connect 0x7f804413d610 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.345+0000 7f8032ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 1 -- 172.21.15.110:0/3433106741 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c009510 con 0x7f804413d610 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 10 monclient: dump: 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 1 -- 172.21.15.110:0/3433106741 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803c00ee40 con 0x7f804413d610 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 1 -- 172.21.15.110:0/3433106741 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c00cb80 con 0x7f804413d610 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8048d01640 5 monclient: authenticate success, global_id 4785 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8048d01640 10 monclient: _renew_subs 2024-01-20T11:18:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8048d01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8044139bc0 con 0x7f804413d610 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8040d99640 4 set_mon_vals no callback set 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8032ffd640 10 monclient: dump: 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:29.352 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:29.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8048d01640 10 client.4785.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:29.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.348+0000 7f8048d01640 10 monclient: _renew_subs 2024-01-20T11:18:29.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.349+0000 7f8048d01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:29.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.349+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80440a99c0 con 0x7f804413d610 2024-01-20T11:18:29.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.349+0000 7f8048d01640 1 librados: init done 2024-01-20T11:18:29.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.349+0000 7f8048d01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:29.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.349+0000 7f8048d01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:29.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.349+0000 7f8048d01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:29.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.349+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80440a9bb0 con 0x7f804413d610 2024-01-20T11:18:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.593+0000 7f8032ffd640 1 -- 172.21.15.110:0/3433106741 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f803c00cce0 con 0x7f804413d610 2024-01-20T11:18:29.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.594+0000 7f8032ffd640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f802406c860 0x7f802406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.594+0000 7f804259c640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f802406c860 0x7f802406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.594+0000 7f804259c640 10 monclient: get_auth_request con 0x7f802406c860 auth_method 0 2024-01-20T11:18:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.594+0000 7f8032ffd640 10 client.4785.objecter ms_handle_connect 0x7f802406c860 2024-01-20T11:18:29.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.594+0000 7f804259c640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f802406c860 0x7f802406ed20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f80340012e0 tx=0x7f8034001a30 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.610+0000 7f8032ffd640 1 -- 172.21.15.110:0/3433106741 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f803c058a60 con 0x7f804413d610 2024-01-20T11:18:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.610+0000 7f8032ffd640 10 client.4785.objecter ms_dispatch 0x7f8044000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.610+0000 7f8032ffd640 3 client.4785.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.610+0000 7f8032ffd640 3 client.4785.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.610+0000 7f8032ffd640 20 client.4785.objecter dump_active .. 0 homeless 2024-01-20T11:18:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.611+0000 7f8032ffd640 1 -- 172.21.15.110:0/3433106741 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f803c054da0 con 0x7f804413d610 2024-01-20T11:18:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.611+0000 7f8032ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:29.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.611+0000 7f8032ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:29.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.790+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f804406c110 con 0x7f802406c860 2024-01-20T11:18:29.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.793+0000 7f8032ffd640 1 -- 172.21.15.110:0/3433106741 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64062 (secure 0 0 0) 0x7f804406c110 con 0x7f802406c860 2024-01-20T11:18:29.793 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:29.793 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:29.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.794+0000 7f8048d01640 10 librados: watch_flush enter 2024-01-20T11:18:29.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.794+0000 7f8048d01640 10 librados: watch_flush exit 2024-01-20T11:18:29.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.794+0000 7f8048d01640 20 client.4785.objecter shutdown clearing up homeless session... 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 10 client.4785.objecter successfully canceled tick 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f802406c860 msgr2=0x7f802406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f802406c860 0x7f802406ed20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f80340012e0 tx=0x7f8034001a30 comp rx=0 tx=0).stop 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 10 monclient: shutdown 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f804413d610 msgr2=0x7f80440a9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f804413d610 0x7f80440a9480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f803c00e7f0 tx=0x7f803c00ebf0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 shutdown_connections 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f802406c860 0x7f802406ed20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f804413d610 0x7f80440a9480 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f804406dcf0 0x7f80440acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 --2- 172.21.15.110:0/3433106741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8044066680 0x7f80440ac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:29.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 >> 172.21.15.110:0/3433106741 conn(0x7f80440b09e0 msgr2=0x7f8044135fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:29.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 shutdown_connections 2024-01-20T11:18:29.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 -- 172.21.15.110:0/3433106741 wait complete. 2024-01-20T11:18:29.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:29.795+0000 7f8048d01640 1 librados: shutdown 2024-01-20T11:18:29.805 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-01-20T11:18:28.747055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002106"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.2520000000000007},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.0330000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.702},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":15.276999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.888},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":17.042999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.8399999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":18.138000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":19.931999999999999},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":16.187999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.704000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":21.094000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.1229999999999993,"5min":9.9309999999999992,"15min":10.233000000000001},"min":{"1min":5.2830000000000004,"5min":4.5570000000000004,"15min":4.5570000000000004},"max":{"1min":14.531000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2809999999999997},{"interface":"front","average":{"1min":10.147,"5min":13.276,"15min":13.798},"min":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":16.138999999999999,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.7000000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":9.1880000000000006,"5min":10.359,"15min":10.554},"min":{"1min":5.9370000000000003,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":16.992000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.7089999999999996},{"interface":"front","average":{"1min":11.41,"5min":13.077999999999999,"15min":13.356},"min":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.975,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.257}]},{"osd":2,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":13.516999999999999,"5min":16.699999999999999,"15min":17.23},"min":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"max":{"1min":18.154,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.397},{"interface":"front","average":{"1min":12.1,"5min":16.148,"15min":16.823},"min":{"1min":6.4320000000000004,"5min":6.4320000000000004,"15min":6.4320000000000004},"max":{"1min":19.795999999999999,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.797000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":12.779,"5min":14.929,"15min":15.287000000000001},"min":{"1min":9.5540000000000003,"5min":6.5519999999999996,"15min":6.5519999999999996},"max":{"1min":18.934999999999999,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.8739999999999997},{"interface":"front","average":{"1min":16.143999999999998,"5min":17.175999999999998,"15min":17.347999999999999},"min":{"1min":11.132,"5min":10.83,"15min":10.83},"max":{"1min":32.773000000000003,"5min":49.829999999999998,"15min":49.829999999999998},"last":11.964}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.526999999999999,"5min":17.495000000000001,"15min":17.989000000000001},"min":{"1min":10.172000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"max":{"1min":27.498000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.680999999999999},{"interface":"front","average":{"1min":14.551,"5min":17.562999999999999,"15min":18.065000000000001},"min":{"1min":9.0939999999999994,"5min":8.6799999999999997,"15min":8.6799999999999997},"max":{"1min":20.239999999999998,"5min":92.613,"15min":92.613},"last":8.6419999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":16.324000000000002,"5min":18.675999999999998,"15min":19.068000000000001},"min":{"1min":11.015000000000001,"5min":10.353,"15min":10.353},"max":{"1min":29.834,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.632999999999999},{"interface":"front","average":{"1min":17.103999999999999,"5min":19.027999999999999,"15min":19.347999999999999},"min":{"1min":13.105,"5min":13.105,"15min":13.105},"max":{"1min":25.001999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.534000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":8.5730000000000004,"5min":15.654999999999999,"15min":16.835000000000001},"min":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.877000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4690000000000003},{"interface":"front","average":{"1min":7.6429999999999998,"5min":13.755000000000001,"15min":14.773999999999999},"min":{"1min":4.9610000000000003,"5min":4.9610000000000003,"15min":4.9610000000000003},"max":{"1min":13.547000000000001,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.3879999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.738,"5min":15.446,"15min":16.23},"min":{"1min":8.1449999999999996,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":18.771000000000001,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.1799999999999997},{"interface":"front","average":{"1min":12.977,"5min":19.004999999999999,"15min":20.010000000000002},"min":{"1min":7.4210000000000003,"5min":7.1550000000000002,"15min":7.1550000000000002},"max":{"1min":19.655000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.894}]},{"osd":2,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":10.311,"5min":17.417000000000002,"15min":18.600999999999999},"min":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":15.436999999999999,"5min":145.553,"15min":145.553},"last":13.050000000000001},{"interface":"front","average":{"1min":12.074999999999999,"5min":17.635000000000002,"15min":18.562000000000001},"min":{"1min":7.1310000000000002,"5min":7.1310000000000002,"15min":7.1310000000000002},"max":{"1min":16.994,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.8160000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":24.591999999999999,"15min":26.373999999999999},"min":{"1min":8.1549999999999994,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":20.449000000000002,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.555},{"interface":"front","average":{"1min":13.609,"5min":25.298999999999999,"15min":27.247},"min":{"1min":9.1259999999999994,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":17.876000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.757999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":11.946999999999999,"5min":19.710999999999999,"15min":21.004999999999999},"min":{"1min":6.2000000000000002,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":17.997,"5min":146.744,"15min":146.744},"last":15.343999999999999},{"interface":"front","average":{"1min":14.364000000000001,"5min":20.535,"15min":21.562999999999999},"min":{"1min":10.906000000000001,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":19.210000000000001,"5min":150.542,"15min":150.542},"last":12.515000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":16.109000000000002,"5min":22.550000000000001,"15min":23.623999999999999},"min":{"1min":12.263,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":21.661999999999999,"5min":159.179,"15min":159.179},"last":14.949},{"interface":"front","average":{"1min":14.851000000000001,"5min":22.513999999999999,"15min":23.791},"min":{"1min":10.081,"5min":10.081,"15min":10.081},"max":{"1min":19.161999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.568999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444074,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.5800000000000001},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":6.6500000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.5},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.282}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.196999999999999},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.2260000000000009}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.516999999999999},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.9760000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.936999999999999},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.545999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.926},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.206}]}]},{"osd":1,"up_from":17,"seq":73014444075,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.0170000000000003},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":9.3599999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.589},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.061}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":14.548999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.5460000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":11.233000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":5.8780000000000001},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":11.840999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":7.5960000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.853}]}]},{"osd":0,"up_from":17,"seq":73014444075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":5.7320000000000002},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":8.5410000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.1849999999999996},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":16.954000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":11.819000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.8520000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.881},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":14.417}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":10.965},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":13.478999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.445},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":10.083}]}]},{"osd":6,"up_from":15,"seq":64424509483,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":4.9859999999999998},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.758}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.9379999999999997},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":9.3260000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.7029999999999994},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.464}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.163},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.388999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":13.458,"5min":16.666,"15min":17.201000000000001},"min":{"1min":8.5890000000000004,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":20.404,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.134},{"interface":"front","average":{"1min":12.99,"5min":16.606999999999999,"15min":17.210000000000001},"min":{"1min":8.452,"5min":8.016,"15min":8.016},"max":{"1min":18.748000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.803000000000001},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.987}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:32.807 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:32.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.914+0000 7fa108421640 10 monclient: get_monmap_and_config 2024-01-20T11:18:32.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.914+0000 7fa108421640 10 monclient: build_initial_monmap 2024-01-20T11:18:32.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.914+0000 7fa108421640 10 monclient: monmap: 2024-01-20T11:18:32.915 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:32.915 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:32.915 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:32.914740+0000 2024-01-20T11:18:32.915 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:32.914740+0000 2024-01-20T11:18:32.915 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:32.916 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:32.916 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:32.916 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:32.916 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:32.916 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:32.916 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:32.917 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:32.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.914+0000 7fa108421640 1 Processor -- start 2024-01-20T11:18:32.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.914+0000 7fa108421640 1 -- start start 2024-01-20T11:18:32.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.914+0000 7fa108421640 10 monclient: init 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1000b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 10 monclient(hunting): picked mon.noname-b con 0x7fa1000b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa1000b17c0 0x7fa1000b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 10 monclient(hunting): picked mon.noname-a con 0x7fa1000b17c0 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa100145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1000b2120 con 0x7fa100145980 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1000b50f0 con 0x7fa1000b17c0 2024-01-20T11:18:32.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000aade0 con 0x7fa1000b4d10 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.915+0000 7fa108421640 10 monclient(hunting): authenticate will time out at 1556.265677s 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.916+0000 7fa1061bd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1000b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.916+0000 7fa1069be640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa100145980 0x7fa100145e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:36550/0 (socket says 172.21.15.110:36550) 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.916+0000 7fa1069be640 1 -- 172.21.15.110:0/998851964 learned_addr learned my addr 172.21.15.110:0/998851964 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.916+0000 7fa1061bd640 10 monclient(hunting): get_auth_request con 0x7fa1000b4d10 auth_method 0 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.916+0000 7fa1061bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.916+0000 7fa1061bd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.916+0000 7fa1061bd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.916+0000 7fa1061bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.916+0000 7fa1061bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.917+0000 7fa1061bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:32.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.918+0000 7fa1061bd640 10 monclient(hunting): handle_auth_done global_id 4661 payload 274 2024-01-20T11:18:32.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.918+0000 7fa1061bd640 1 -- 172.21.15.110:0/998851964 >> v2:172.21.15.110:3300/0 conn(0x7fa1000b17c0 msgr2=0x7fa1000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:32.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.918+0000 7fa1061bd640 1 --2- 172.21.15.110:0/998851964 >> v2:172.21.15.110:3300/0 conn(0x7fa1000b17c0 0x7fa1000b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:32.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.918+0000 7fa1061bd640 1 -- 172.21.15.110:0/998851964 >> v1:172.21.15.134:6789/0 conn(0x7fa100145980 legacy=0x7fa100145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:32.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.918+0000 7fa1061bd640 10 monclient: _finish_hunting 0 2024-01-20T11:18:32.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.918+0000 7fa1061bd640 1 monclient: found mon.noname-b 2024-01-20T11:18:32.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.918+0000 7fa1061bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:32.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.918+0000 7fa1061bd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:32.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.918+0000 7fa1061bd640 1 -- 172.21.15.110:0/998851964 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100145460 con 0x7fa1000b4d10 2024-01-20T11:18:32.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.920+0000 7fa1061bd640 1 --2- 172.21.15.110:0/998851964 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1000b3da0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc0097a0 tx=0x7fa0fc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=691fdda531a334c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 1 -- 172.21.15.110:0/998851964 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc02e070 con 0x7fa1000b4d10 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 10 monclient: dump: 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:32.925 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 10 monclient: _finish_auth 0 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 10 monclient: _check_auth_tickets 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 1 -- 172.21.15.110:0/998851964 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc0315a0 con 0x7fa1000b4d10 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa108421640 5 monclient: authenticate success, global_id 4661 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa108421640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa108421640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 1 -- 172.21.15.110:0/998851964 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc031930 con 0x7fa1000b4d10 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa108421640 4 set_mon_vals no callback set 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa1051bb640 10 monclient: dump: 2024-01-20T11:18:32.926 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa108421640 10 monclient: shutdown 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa108421640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa108421640 1 -- 172.21.15.110:0/998851964 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 msgr2=0x7fa1000b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.924+0000 7fa108421640 1 --2- 172.21.15.110:0/998851964 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1000b3da0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc0097a0 tx=0x7fa0fc02b840 comp rx=0 tx=0).stop 2024-01-20T11:18:32.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.925+0000 7fa108421640 1 -- 172.21.15.110:0/998851964 shutdown_connections 2024-01-20T11:18:32.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.925+0000 7fa108421640 1 --2- 172.21.15.110:0/998851964 >> v2:172.21.15.110:3300/0 conn(0x7fa1000b17c0 0x7fa1000b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:32.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.925+0000 7fa108421640 1 --2- 172.21.15.110:0/998851964 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1000b3da0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:32.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.925+0000 7fa108421640 1 -- 172.21.15.110:0/998851964 >> 172.21.15.110:0/998851964 conn(0x7fa100066590 msgr2=0x7fa1000669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:32.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.925+0000 7fa108421640 1 -- 172.21.15.110:0/998851964 shutdown_connections 2024-01-20T11:18:32.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.925+0000 7fa108421640 1 -- 172.21.15.110:0/998851964 wait complete. 2024-01-20T11:18:32.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.925+0000 7fa108421640 10 monclient: build_initial_monmap 2024-01-20T11:18:32.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.925+0000 7fa108421640 10 monclient: monmap: 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 librados: starting msgr at 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 librados: starting objecter 2024-01-20T11:18:32.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 Processor -- start 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 -- start start 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 librados: setting wanted keys 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 librados: calling monclient init 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient: init 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1000b17c0 0x7fa1001d9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient(hunting): picked mon.noname-a con 0x7fa1000b17c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1001da4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient(hunting): picked mon.noname-c con 0x7fa1000b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa100145980 0x7fa1001daee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient(hunting): picked mon.noname-b con 0x7fa100145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:32.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa100146880 con 0x7fa1000b17c0 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1001daa30 con 0x7fa100145980 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1001480c0 con 0x7fa1000b4d10 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.926+0000 7fa108421640 10 monclient(hunting): authenticate will time out at 1556.276985s 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1059bc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1001da4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1069be640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa100145980 0x7fa1001daee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1059bc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1001da4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43562/0 (socket says 172.21.15.110:43562) 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1059bc640 1 -- 172.21.15.110:0/1657219982 learned_addr learned my addr 172.21.15.110:0/1657219982 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1059bc640 10 monclient(hunting): get_auth_request con 0x7fa1000b4d10 auth_method 0 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1059bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1059bc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:32.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1059bc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1069be640 10 monclient(hunting): get_auth_request con 0x7fa100145980 auth_method 0 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1069be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1069be640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.927+0000 7fa1069be640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.928+0000 7fa1059bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.928+0000 7fa1059bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.928+0000 7fa1059bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.928+0000 7fa1069be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.928+0000 7fa1069be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.928+0000 7fa1069be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 10 monclient(hunting): handle_auth_done global_id 4664 payload 824 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 1 -- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa100145980 msgr2=0x7fa1001daee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa100145980 0x7fa1001daee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:32.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 1 -- 172.21.15.110:0/1657219982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1000b17c0 msgr2=0x7fa1001d9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:32.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1000b17c0 0x7fa1001d9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:32.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 10 monclient: _finish_hunting 0 2024-01-20T11:18:32.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 1 monclient: found mon.noname-c 2024-01-20T11:18:32.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:32.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:18:32.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1059bc640 1 -- 172.21.15.110:0/1657219982 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc009400 con 0x7fa1000b4d10 2024-01-20T11:18:32.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.930+0000 7fa1069be640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa100145980 0x7fa1001daee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:18:32.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.933+0000 7fa1059bc640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1001da4f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000b4d0 tx=0x7fa0f000b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:32.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.933+0000 7fa0f6ffd640 10 client.?.objecter ms_handle_connect 0x7fa1000b4d10 2024-01-20T11:18:32.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.933+0000 7fa0f6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:32.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa0f6ffd640 1 -- 172.21.15.110:0/1657219982 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0010870 con 0x7fa1000b4d10 2024-01-20T11:18:32.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa0f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:32.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa0f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:32.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa0f6ffd640 10 monclient: dump: 2024-01-20T11:18:32.939 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:32.939 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:32.939 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:32.939 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:32.939 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:32.939 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:32.939 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:32.939 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:32.940 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:32.940 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:32.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa0f6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:32.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa0f6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:32.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa0f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:32.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa108421640 5 monclient: authenticate success, global_id 4664 2024-01-20T11:18:32.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa108421640 10 monclient: _renew_subs 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa108421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1001483a0 con 0x7fa1000b4d10 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa108421640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa108421640 10 monclient: _renew_subs 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa108421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10013ee10 con 0x7fa1000b4d10 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.936+0000 7fa108421640 1 librados: init done 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa0f6ffd640 1 -- 172.21.15.110:0/1657219982 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f00092d0 con 0x7fa1000b4d10 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa0f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa0f6ffd640 1 -- 172.21.15.110:0/1657219982 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0010bd0 con 0x7fa1000b4d10 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa0f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa0f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa0f6ffd640 10 monclient: dump: 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa1049ba640 4 set_mon_vals no callback set 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa108421640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa108421640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa108421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:18:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.937+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e8005080 con 0x7fa1000b4d10 2024-01-20T11:18:32.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.958+0000 7fa0f6ffd640 1 -- 172.21.15.110:0/1657219982 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa0f0010d30 con 0x7fa1000b4d10 2024-01-20T11:18:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.958+0000 7fa0f6ffd640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0e406c860 0x7fa0e406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.958+0000 7fa1061bd640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0e406c860 0x7fa0e406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.958+0000 7fa1061bd640 10 monclient: get_auth_request con 0x7fa0e406c860 auth_method 0 2024-01-20T11:18:32.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.959+0000 7fa1061bd640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0e406c860 0x7fa0e406ed20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc036000 tx=0x7fa0fc030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:32.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:32.959+0000 7fa0f6ffd640 10 client.4664.objecter ms_handle_connect 0x7fa0e406c860 2024-01-20T11:18:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.196+0000 7fa0f6ffd640 1 -- 172.21.15.110:0/1657219982 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa0f0058770 con 0x7fa1000b4d10 2024-01-20T11:18:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.196+0000 7fa0f6ffd640 10 client.4664.objecter ms_dispatch 0x7fa100000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.196+0000 7fa0f6ffd640 3 client.4664.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.196+0000 7fa0f6ffd640 3 client.4664.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.196+0000 7fa0f6ffd640 20 client.4664.objecter dump_active .. 0 homeless 2024-01-20T11:18:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.196+0000 7fa0f6ffd640 1 -- 172.21.15.110:0/1657219982 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa0f0058540 con 0x7fa1000b4d10 2024-01-20T11:18:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.196+0000 7fa0f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:33.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.196+0000 7fa0f6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:33.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.363+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0e8000bc0 con 0x7fa0e406c860 2024-01-20T11:18:33.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.366+0000 7fa0f6ffd640 1 -- 172.21.15.110:0/1657219982 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63962 (secure 0 0 0) 0x7fa0e8000bc0 con 0x7fa0e406c860 2024-01-20T11:18:33.367 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:33.367 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:33.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 10 librados: watch_flush enter 2024-01-20T11:18:33.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 10 librados: watch_flush exit 2024-01-20T11:18:33.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 20 client.4664.objecter shutdown clearing up homeless session... 2024-01-20T11:18:33.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 10 client.4664.objecter successfully canceled tick 2024-01-20T11:18:33.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0e406c860 msgr2=0x7fa0e406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:33.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0e406c860 0x7fa0e406ed20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc036000 tx=0x7fa0fc030000 comp rx=0 tx=0).stop 2024-01-20T11:18:33.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 10 monclient: shutdown 2024-01-20T11:18:33.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 msgr2=0x7fa1001da4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1001da4f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000b4d0 tx=0x7fa0f000b990 comp rx=0 tx=0).stop 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 shutdown_connections 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa0e406c860 0x7fa0e406ed20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa100145980 0x7fa1001daee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa1000b4d10 0x7fa1001da4f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 --2- 172.21.15.110:0/1657219982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1000b17c0 0x7fa1001d9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 >> 172.21.15.110:0/1657219982 conn(0x7fa100066590 msgr2=0x7fa10013b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 shutdown_connections 2024-01-20T11:18:33.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 -- 172.21.15.110:0/1657219982 wait complete. 2024-01-20T11:18:33.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.368+0000 7fa108421640 1 librados: shutdown 2024-01-20T11:18:33.379 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-01-20T11:18:32.747709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001984"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.2520000000000007},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.0330000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.702},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":15.276999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.888},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":17.042999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.8399999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":18.138000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":19.931999999999999},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":16.187999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.704000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":21.094000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6189999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.311999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7270000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.476000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.399},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.9350000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.3550000000000004},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.343}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.2769999999999992},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":13.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.105},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.0419999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":3.9260000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.3689999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.7080000000000002},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":8}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":9.0869999999999997},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.467000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.061},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.673999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":9.6699999999999999},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":12.334}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":14.768000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.749000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444074,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.5800000000000001},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":6.6500000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.5},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.282}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.196999999999999},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.2260000000000009}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.516999999999999},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.9760000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.936999999999999},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.545999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.926},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.206}]}]},{"osd":1,"up_from":17,"seq":73014444075,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.0170000000000003},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":9.3599999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.589},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.061}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":14.548999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.5460000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":11.233000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":5.8780000000000001},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":11.840999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":7.5960000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.853}]}]},{"osd":0,"up_from":17,"seq":73014444075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":5.7320000000000002},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":8.5410000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.1849999999999996},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":16.954000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":11.819000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.8520000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.881},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":14.417}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":10.965},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":13.478999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.445},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":10.083}]}]},{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":9.4930000000000003},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.0660000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.4480000000000004},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":14.113}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.163},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.34}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.085000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":18.533000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.756},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.76}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.269},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.654}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:33.380 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.482+0000 7f477c434640 10 monclient: get_monmap_and_config 2024-01-20T11:18:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.482+0000 7f477c434640 10 monclient: build_initial_monmap 2024-01-20T11:18:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.482+0000 7f477c434640 10 monclient: monmap: 2024-01-20T11:18:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:33.483163+0000 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:33.483163+0000 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:33.483 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:33.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 1 Processor -- start 2024-01-20T11:18:33.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 1 -- start start 2024-01-20T11:18:33.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient: init 2024-01-20T11:18:33.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:33.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-01-20T11:18:33.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f47740a96f0 0x7f477413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:33.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient(hunting): picked mon.noname-c con 0x7f47740a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:33.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f477413fa10 0x7f47741423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient(hunting): picked mon.noname-a con 0x7f477413fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47740a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47740b25c0 con 0x7f47740a9ad0 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47740aaea0 con 0x7f477413fa10 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47740ab0a0 con 0x7f47740a96f0 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.483+0000 7f477c434640 10 monclient(hunting): authenticate will time out at 1556.834147s 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f47799cf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f477413fa10 0x7f47741423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f477a1d0640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f47740a96f0 0x7f477413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f47799cf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f477413fa10 0x7f47741423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39366/0 (socket says 172.21.15.110:39366) 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f47799cf640 1 -- 172.21.15.110:0/2640669331 learned_addr learned my addr 172.21.15.110:0/2640669331 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:33.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f47799cf640 10 monclient(hunting): get_auth_request con 0x7f477413fa10 auth_method 0 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f47799cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f47799cf640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f47799cf640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f477a1d0640 10 monclient(hunting): get_auth_request con 0x7f47740a96f0 auth_method 0 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f477a1d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f477a1d0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.484+0000 7f477a1d0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.485+0000 7f477a1d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.485+0000 7f477a1d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.485+0000 7f477a1d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.485+0000 7f47799cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.485+0000 7f47799cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:33.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.485+0000 7f47799cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:33.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.486+0000 7f477a1d0640 10 monclient(hunting): handle_auth_done global_id 4873 payload 274 2024-01-20T11:18:33.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.486+0000 7f477a1d0640 1 -- 172.21.15.110:0/2640669331 >> v2:172.21.15.110:3300/0 conn(0x7f477413fa10 msgr2=0x7f47741423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:33.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.486+0000 7f477a1d0640 1 --2- 172.21.15.110:0/2640669331 >> v2:172.21.15.110:3300/0 conn(0x7f477413fa10 0x7f47741423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.486+0000 7f477a1d0640 1 -- 172.21.15.110:0/2640669331 >> v1:172.21.15.110:6789/0 conn(0x7f47740a9ad0 legacy=0x7f4774142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:33.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.486+0000 7f477a1d0640 10 monclient: _finish_hunting 0 2024-01-20T11:18:33.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.486+0000 7f477a1d0640 1 monclient: found mon.noname-c 2024-01-20T11:18:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.486+0000 7f477a1d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.486+0000 7f477a1d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.486+0000 7f477a1d0640 1 -- 172.21.15.110:0/2640669331 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47740b3fb0 con 0x7f47740a96f0 2024-01-20T11:18:33.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.488+0000 7f477a1d0640 1 --2- 172.21.15.110:0/2640669331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 0x7f477413f4d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f47740b5720 tx=0x7f477002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da1a1ba767b62520 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:33.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.490+0000 7f47791ce640 1 -- 172.21.15.110:0/2640669331 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477002e070 con 0x7f47740a96f0 2024-01-20T11:18:33.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.490+0000 7f47791ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:33.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.490+0000 7f47791ce640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:33.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.490+0000 7f47791ce640 10 monclient: dump: 2024-01-20T11:18:33.490 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.490+0000 7f47791ce640 10 monclient: _finish_auth 0 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.490+0000 7f47791ce640 10 monclient: _check_auth_tickets 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.490+0000 7f47791ce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.490+0000 7f477c434640 5 monclient: authenticate success, global_id 4873 2024-01-20T11:18:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.490+0000 7f477c434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f47791ce640 1 -- 172.21.15.110:0/2640669331 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47700315a0 con 0x7f47740a96f0 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f47791ce640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f47791ce640 1 -- 172.21.15.110:0/2640669331 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770031930 con 0x7f47740a96f0 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f477c434640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f477c434640 4 set_mon_vals no callback set 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f47791ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f47791ce640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f47791ce640 10 monclient: dump: 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:33.493 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f477c434640 10 monclient: shutdown 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f477c434640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f477c434640 1 -- 172.21.15.110:0/2640669331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 msgr2=0x7f477413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.492+0000 7f477c434640 1 --2- 172.21.15.110:0/2640669331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 0x7f477413f4d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f47740b5720 tx=0x7f477002b840 comp rx=0 tx=0).stop 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 1 -- 172.21.15.110:0/2640669331 shutdown_connections 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 1 --2- 172.21.15.110:0/2640669331 >> v2:172.21.15.110:3300/0 conn(0x7f477413fa10 0x7f47741423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 1 --2- 172.21.15.110:0/2640669331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 0x7f477413f4d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 1 -- 172.21.15.110:0/2640669331 >> 172.21.15.110:0/2640669331 conn(0x7f47740b7bc0 msgr2=0x7f47740b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:33.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 1 -- 172.21.15.110:0/2640669331 shutdown_connections 2024-01-20T11:18:33.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 1 -- 172.21.15.110:0/2640669331 wait complete. 2024-01-20T11:18:33.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 10 monclient: build_initial_monmap 2024-01-20T11:18:33.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 10 monclient: monmap: 2024-01-20T11:18:33.497 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:33.497 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:33.497 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:33.497 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:33.497 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:33.497 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:33.497 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 1 librados: starting msgr at 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.493+0000 7f477c434640 1 librados: starting objecter 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 Processor -- start 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 -- start start 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 librados: setting wanted keys 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 librados: calling monclient init 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient: init 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 0x7f47741db020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient(hunting): picked mon.noname-b con 0x7f47740a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:33.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47740a9ad0 0x7f4774138010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient(hunting): picked mon.noname-c con 0x7f47740a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47741da5e0 0x7f477413aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient(hunting): picked mon.noname-a con 0x7f47741da5e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477413a4d0 con 0x7f47741da5e0 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47741dab00 con 0x7f47740a96f0 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4774138550 con 0x7f47740a9ad0 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.494+0000 7f477c434640 10 monclient(hunting): authenticate will time out at 1556.844913s 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a1d0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 0x7f47741db020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a9d1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47741da5e0 0x7f477413aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:33.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a1d0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 0x7f47741db020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46364/0 (socket says 172.21.15.110:46364) 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a1d0640 1 -- 172.21.15.110:0/321236077 learned_addr learned my addr 172.21.15.110:0/321236077 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a1d0640 10 monclient(hunting): get_auth_request con 0x7f47740a96f0 auth_method 0 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a1d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a1d0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a1d0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a9d1640 10 monclient(hunting): get_auth_request con 0x7f47741da5e0 auth_method 0 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a9d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a9d1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.495+0000 7f477a9d1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.496+0000 7f477a1d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.496+0000 7f477a1d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.496+0000 7f477a1d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.496+0000 7f477a9d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:33.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.496+0000 7f477a9d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:33.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.496+0000 7f477a9d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:33.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 10 monclient(hunting): handle_auth_done global_id 4876 payload 824 2024-01-20T11:18:33.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 1 -- 172.21.15.110:0/321236077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47740a9ad0 msgr2=0x7f4774138010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47740a9ad0 0x7f4774138010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 1 -- 172.21.15.110:0/321236077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47741da5e0 msgr2=0x7f477413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47741da5e0 0x7f477413aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 10 monclient: _finish_hunting 0 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 1 monclient: found mon.noname-b 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.498+0000 7f477a1d0640 1 -- 172.21.15.110:0/321236077 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4770009400 con 0x7f47740a96f0 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.500+0000 7f477a1d0640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 0x7f47741db020 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4770034040 tx=0x7f47700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.500+0000 7f476affd640 10 client.?.objecter ms_handle_connect 0x7f47740a96f0 2024-01-20T11:18:33.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.500+0000 7f476affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:33.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.503+0000 7f476affd640 1 -- 172.21.15.110:0/321236077 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770032840 con 0x7f47740a96f0 2024-01-20T11:18:33.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.503+0000 7f476affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:33.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.503+0000 7f476affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.503+0000 7f476affd640 10 monclient: dump: 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f476affd640 10 monclient: _finish_auth 0 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f476affd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:33.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f476affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f476affd640 1 -- 172.21.15.110:0/321236077 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47700329a0 con 0x7f47740a96f0 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 5 monclient: authenticate success, global_id 4876 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 10 monclient: _renew_subs 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477413b400 con 0x7f47740a96f0 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f476affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f476affd640 1 -- 172.21.15.110:0/321236077 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770032d50 con 0x7f47740a96f0 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f476affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f476affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f476affd640 10 monclient: dump: 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:33.506 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f47789cd640 4 set_mon_vals no callback set 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 10 client.4876.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 10 monclient: _renew_subs 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477413b8b0 con 0x7f47740a96f0 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 1 librados: init done 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:33.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:33.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.504+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47741dad20 con 0x7f47740a96f0 2024-01-20T11:18:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.745+0000 7f476affd640 1 -- 172.21.15.110:0/321236077 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f477004e430 con 0x7f47740a96f0 2024-01-20T11:18:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.746+0000 7f476affd640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f475806c8b0 0x7f475806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.746+0000 7f47799cf640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f475806c8b0 0x7f475806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.746+0000 7f47799cf640 10 monclient: get_auth_request con 0x7f475806c8b0 auth_method 0 2024-01-20T11:18:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.747+0000 7f476affd640 10 client.4876.objecter ms_handle_connect 0x7f475806c8b0 2024-01-20T11:18:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.747+0000 7f47799cf640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f475806c8b0 0x7f475806ed70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4764006f40 tx=0x7f4764006e90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.748+0000 7f476affd640 1 -- 172.21.15.110:0/321236077 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f477007a540 con 0x7f47740a96f0 2024-01-20T11:18:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.748+0000 7f476affd640 10 client.4876.objecter ms_dispatch 0x7f4774000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.748+0000 7f476affd640 3 client.4876.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.748+0000 7f476affd640 3 client.4876.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.748+0000 7f476affd640 20 client.4876.objecter dump_active .. 0 homeless 2024-01-20T11:18:33.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.758+0000 7f476affd640 1 -- 172.21.15.110:0/321236077 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4770076180 con 0x7f47740a96f0 2024-01-20T11:18:33.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.758+0000 7f476affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:33.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.758+0000 7f476affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.927+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f477413b8b0 con 0x7f475806c8b0 2024-01-20T11:18:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.930+0000 7f476affd640 1 -- 172.21.15.110:0/321236077 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63962 (secure 0 0 0) 0x7f477413b8b0 con 0x7f475806c8b0 2024-01-20T11:18:33.930 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:33.931 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:33.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 10 librados: watch_flush enter 2024-01-20T11:18:33.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 10 librados: watch_flush exit 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 20 client.4876.objecter shutdown clearing up homeless session... 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 10 client.4876.objecter successfully canceled tick 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f475806c8b0 msgr2=0x7f475806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f475806c8b0 0x7f475806ed70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4764006f40 tx=0x7f4764006e90 comp rx=0 tx=0).stop 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 10 monclient: shutdown 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 msgr2=0x7f47741db020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 0x7f47741db020 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4770034040 tx=0x7f47700046c0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 shutdown_connections 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f475806c8b0 0x7f475806ed70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47741da5e0 0x7f477413aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47740a9ad0 0x7f4774138010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 --2- 172.21.15.110:0/321236077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47740a96f0 0x7f47741db020 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:33.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 >> 172.21.15.110:0/321236077 conn(0x7f47740b7bc0 msgr2=0x7f4774137630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:33.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 shutdown_connections 2024-01-20T11:18:33.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 -- 172.21.15.110:0/321236077 wait complete. 2024-01-20T11:18:33.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:33.932+0000 7f477c434640 1 librados: shutdown 2024-01-20T11:18:33.943 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-01-20T11:18:32.747709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001984"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.6989999999999998,"5min":14.026999999999999,"15min":14.914999999999999},"min":{"1min":4.8230000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":21.064,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.2520000000000007},{"interface":"front","average":{"1min":8.6340000000000003,"5min":15.138999999999999,"15min":16.222999999999999},"min":{"1min":5.0899999999999999,"5min":5.0899999999999999,"15min":5.0899999999999999},"max":{"1min":24.245000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.0330000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":10.289,"5min":16.742999999999999,"15min":17.818999999999999},"min":{"1min":6.0010000000000003,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":26.027000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.702},{"interface":"front","average":{"1min":11.813000000000001,"5min":16.007999999999999,"15min":16.707000000000001},"min":{"1min":8.702,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":22.533999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":15.276999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":12.507,"5min":17.959,"15min":18.867000000000001},"min":{"1min":10.128,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":15.529999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.888},{"interface":"front","average":{"1min":14.816000000000001,"5min":17.544,"15min":17.998000000000001},"min":{"1min":9.8859999999999992,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":28.405999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":17.042999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":20.294,"15min":21.370000000000001},"min":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":55.868000000000002,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.8399999999999999},{"interface":"front","average":{"1min":16.853999999999999,"5min":22.25,"15min":23.149999999999999},"min":{"1min":9.1489999999999991,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":52.960000000000001,"5min":100.633,"15min":100.633},"last":18.138000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":21.858000000000001,"15min":22.628},"min":{"1min":13.683999999999999,"5min":8.1129999999999995,"15min":8.1129999999999995},"max":{"1min":30,"5min":131.70599999999999,"15min":131.70599999999999},"last":19.931999999999999},{"interface":"front","average":{"1min":15.545,"5min":20.52,"15min":21.349},"min":{"1min":9.8390000000000004,"5min":9.8390000000000004,"15min":9.8390000000000004},"max":{"1min":23.395,"5min":107.962,"15min":107.962},"last":16.187999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:25 2024","interfaces":[{"interface":"back","average":{"1min":13.875999999999999,"5min":19,"15min":19.853999999999999},"min":{"1min":6.7939999999999996,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":29.283999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.704000000000001},{"interface":"front","average":{"1min":18.585999999999999,"5min":22.754000000000001,"15min":23.448},"min":{"1min":12.696,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":29.736999999999998,"5min":129.828,"15min":129.828},"last":21.094000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6189999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.311999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7270000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.476000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.399},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.9350000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.3550000000000004},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.343}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.2769999999999992},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":13.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.105},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.0419999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":3.9260000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.3689999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.7080000000000002},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":8}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":9.0869999999999997},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.467000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.061},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.673999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":9.6699999999999999},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":12.334}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":14.768000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.749000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444074,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.5800000000000001},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":6.6500000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.5},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.282}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.196999999999999},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.2260000000000009}]},{"osd":4,"last update":"Sat Jan 20 11:17:29 2024","interfaces":[{"interface":"back","average":{"1min":14.819000000000001,"5min":17.382000000000001,"15min":17.809000000000001},"min":{"1min":11.119999999999999,"5min":11.119999999999999,"15min":11.119999999999999},"max":{"1min":18.715,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.516999999999999},{"interface":"front","average":{"1min":12.5,"5min":14.381,"15min":14.695},"min":{"1min":7.9059999999999997,"5min":7.9059999999999997,"15min":7.9059999999999997},"max":{"1min":17.716000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.9760000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.936999999999999},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.545999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.926},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.206}]}]},{"osd":1,"up_from":17,"seq":73014444075,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.0170000000000003},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":9.3599999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.589},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.061}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":14.548999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.5460000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:27 2024","interfaces":[{"interface":"back","average":{"1min":12.975,"5min":14.029999999999999,"15min":14.206},"min":{"1min":8.3019999999999996,"5min":8.3019999999999996,"15min":8.3019999999999996},"max":{"1min":25.541,"5min":25.541,"15min":25.541},"last":11.233000000000001},{"interface":"front","average":{"1min":15.824999999999999,"5min":15.435,"15min":15.369999999999999},"min":{"1min":10.593,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":30.204999999999998,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":5.8780000000000001},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":11.840999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":7.5960000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.853}]}]},{"osd":0,"up_from":17,"seq":73014444075,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":5.7320000000000002},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":8.5410000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.1849999999999996},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":16.954000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.678000000000001,"5min":26.533999999999999,"15min":28.844000000000001},"min":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":8.4429999999999996},"max":{"1min":17.588999999999999,"5min":358.738,"15min":358.738},"last":11.819000000000001},{"interface":"front","average":{"1min":10.871,"5min":12.778,"15min":13.096},"min":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"max":{"1min":15.702,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.8520000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":12.882999999999999,"5min":28.587,"15min":31.204999999999998},"min":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.193000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.881},{"interface":"front","average":{"1min":10.766,"5min":25.878,"15min":28.396000000000001},"min":{"1min":7.157,"5min":5.96,"15min":5.96},"max":{"1min":19.41,"5min":372.70800000000003,"15min":372.70800000000003},"last":14.417}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":10.965},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":13.478999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:17:28 2024","interfaces":[{"interface":"back","average":{"1min":15.170999999999999,"5min":17.059000000000001,"15min":17.373999999999999},"min":{"1min":11.207000000000001,"5min":11.207000000000001,"15min":11.207000000000001},"max":{"1min":18.779,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.445},{"interface":"front","average":{"1min":12.720000000000001,"5min":12.515000000000001,"15min":12.481},"min":{"1min":8.7629999999999999,"5min":8.3279999999999994,"15min":8.3279999999999994},"max":{"1min":17.597999999999999,"5min":21.529,"15min":21.529},"last":10.083}]}]},{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":9.4930000000000003},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.0660000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.4480000000000004},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":14.113}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.163},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.34}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.085000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":18.533000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.756},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.76}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.269},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.654}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:33.944 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:33.945 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:33.945 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:33.945 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:33.945 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:33.945 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:33.945 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:33.945 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:33.945 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:34.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.044+0000 7f7536773640 10 monclient: get_monmap_and_config 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.044+0000 7f7536773640 10 monclient: build_initial_monmap 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.044+0000 7f7536773640 10 monclient: monmap: 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:34.045486+0000 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:34.045486+0000 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:34.045 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.045+0000 7f7536773640 1 Processor -- start 2024-01-20T11:18:34.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.045+0000 7f7536773640 1 -- start start 2024-01-20T11:18:34.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.045+0000 7f7536773640 10 monclient: init 2024-01-20T11:18:34.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:34.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7530071c60 0x7f7530142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 10 monclient(hunting): picked mon.noname-a con 0x7f7530071c60 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75300ac540 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75300b0930 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75300ad610 con 0x7f75300ac540 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75301426c0 con 0x7f75300b0930 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7530072040 con 0x7f7530071c60 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7536773640 10 monclient(hunting): authenticate will time out at 1557.396467s 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f752ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7530071c60 0x7f7530142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7534d10640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f75300b0930 0x7f7530149510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:36560/0 (socket says 172.21.15.110:36560) 2024-01-20T11:18:34.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.046+0000 7f7534d10640 1 -- 172.21.15.110:0/3085675247 learned_addr learned my addr 172.21.15.110:0/3085675247 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:34.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.047+0000 7f752ffff640 10 monclient(hunting): get_auth_request con 0x7f7530071c60 auth_method 0 2024-01-20T11:18:34.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.047+0000 7f752ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:34.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.047+0000 7f752ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.047+0000 7f752ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.048+0000 7f752ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:34.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.048+0000 7f752ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:34.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.048+0000 7f752ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:34.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.049+0000 7f752effd640 1 -- 172.21.15.110:0/3085675247 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080944021 0 0) 0x7f75301426c0 con 0x7f75300b0930 2024-01-20T11:18:34.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.049+0000 7f752effd640 10 monclient(hunting): _negotiate 2024-01-20T11:18:34.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.049+0000 7f752effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.049+0000 7f752effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.049+0000 7f752effd640 10 monclient(hunting): my global_id is 4879 2024-01-20T11:18:34.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.049+0000 7f752effd640 1 -- 172.21.15.110:0/3085675247 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7518005730 con 0x7f75300b0930 2024-01-20T11:18:34.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.051+0000 7f752ffff640 10 monclient(hunting): handle_auth_done global_id 4794 payload 274 2024-01-20T11:18:34.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.051+0000 7f752ffff640 1 -- 172.21.15.110:0/3085675247 >> v1:172.21.15.134:6789/0 conn(0x7f75300b0930 legacy=0x7f7530149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.051+0000 7f752ffff640 1 -- 172.21.15.110:0/3085675247 >> v1:172.21.15.110:6789/0 conn(0x7f75300ac540 legacy=0x7f75300ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.051+0000 7f752ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:34.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.051+0000 7f752ffff640 1 monclient: found mon.noname-a 2024-01-20T11:18:34.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.051+0000 7f752ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:34.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.051+0000 7f752ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:34.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.051+0000 7f752ffff640 1 -- 172.21.15.110:0/3085675247 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75300721b0 con 0x7f7530071c60 2024-01-20T11:18:34.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.054+0000 7f752ffff640 1 --2- 172.21.15.110:0/3085675247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7530071c60 0x7f7530142180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f75200097a0 tx=0x7f752002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae58324b8764220 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:34.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 1 -- 172.21.15.110:0/3085675247 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520030610 con 0x7f7530071c60 2024-01-20T11:18:34.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 10 monclient: dump: 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 10 monclient: _finish_auth 0 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 1 -- 172.21.15.110:0/3085675247 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7520030770 con 0x7f7530071c60 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 1 -- 172.21.15.110:0/3085675247 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520030b00 con 0x7f7530071c60 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f752effd640 10 monclient: dump: 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:34.060 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.061 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:34.061 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:34.061 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:34.061 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f7536773640 5 monclient: authenticate success, global_id 4794 2024-01-20T11:18:34.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f7536773640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:34.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.058+0000 7f7536773640 4 set_mon_vals no callback set 2024-01-20T11:18:34.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.059+0000 7f7536773640 10 monclient: shutdown 2024-01-20T11:18:34.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.059+0000 7f7536773640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:34.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.059+0000 7f7536773640 1 -- 172.21.15.110:0/3085675247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7530071c60 msgr2=0x7f7530142180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.059+0000 7f7536773640 1 --2- 172.21.15.110:0/3085675247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7530071c60 0x7f7530142180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f75200097a0 tx=0x7f752002b960 comp rx=0 tx=0).stop 2024-01-20T11:18:34.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.059+0000 7f7536773640 1 -- 172.21.15.110:0/3085675247 shutdown_connections 2024-01-20T11:18:34.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.059+0000 7f7536773640 1 --2- 172.21.15.110:0/3085675247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7530071c60 0x7f7530142180 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.059+0000 7f7536773640 1 -- 172.21.15.110:0/3085675247 >> 172.21.15.110:0/3085675247 conn(0x7f75300b3e60 msgr2=0x7f75300b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:34.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.059+0000 7f7536773640 1 -- 172.21.15.110:0/3085675247 shutdown_connections 2024-01-20T11:18:34.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.059+0000 7f7536773640 1 -- 172.21.15.110:0/3085675247 wait complete. 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.060+0000 7f7536773640 10 monclient: build_initial_monmap 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.060+0000 7f7536773640 10 monclient: monmap: 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.060+0000 7f7536773640 1 librados: starting msgr at 2024-01-20T11:18:34.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.060+0000 7f7536773640 1 librados: starting objecter 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.060+0000 7f7536773640 1 Processor -- start 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.060+0000 7f7536773640 1 -- start start 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.060+0000 7f7536773640 1 librados: setting wanted keys 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.060+0000 7f7536773640 1 librados: calling monclient init 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.060+0000 7f7536773640 10 monclient: init 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530071c60 0x7f753013ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient(hunting): picked mon.noname-b con 0x7f7530071c60 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75300ac540 0x7f75300707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient(hunting): picked mon.noname-a con 0x7f75300ac540 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75300b0930 0x7f753013f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient(hunting): picked mon.noname-c con 0x7f75300b0930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:34.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75301d6730 con 0x7f75300ac540 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753013efb0 con 0x7f7530071c60 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75301d6400 con 0x7f75300b0930 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7536773640 10 monclient(hunting): authenticate will time out at 1557.411642s 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f752f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75300ac540 0x7f75300707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f7534d10640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75300b0930 0x7f753013f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.061+0000 7f752ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530071c60 0x7f753013ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530071c60 0x7f753013ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46370/0 (socket says 172.21.15.110:46370) 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752ffff640 1 -- 172.21.15.110:0/1428166402 learned_addr learned my addr 172.21.15.110:0/1428166402 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752ffff640 10 monclient(hunting): get_auth_request con 0x7f7530071c60 auth_method 0 2024-01-20T11:18:34.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752f7fe640 10 monclient(hunting): get_auth_request con 0x7f75300ac540 auth_method 0 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f7534d10640 10 monclient(hunting): get_auth_request con 0x7f75300b0930 auth_method 0 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f7534d10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f7534d10640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f7534d10640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:34.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f752f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f7534d10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f7534d10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.062+0000 7f7534d10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 10 monclient(hunting): handle_auth_done global_id 4882 payload 824 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 1 -- 172.21.15.110:0/1428166402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75300b0930 msgr2=0x7f753013f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75300b0930 0x7f753013f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 1 -- 172.21.15.110:0/1428166402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75300ac540 msgr2=0x7f75300707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75300ac540 0x7f75300707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 1 monclient: found mon.noname-b 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:34.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:34.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.065+0000 7f752ffff640 1 -- 172.21.15.110:0/1428166402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7520009400 con 0x7f7530071c60 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.067+0000 7f752ffff640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530071c60 0x7f753013ca00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7520009890 tx=0x7f7520004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.067+0000 7f752cff9640 10 client.?.objecter ms_handle_connect 0x7f7530071c60 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.067+0000 7f752cff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 1 -- 172.21.15.110:0/1428166402 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752003e040 con 0x7f7530071c60 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 10 monclient: dump: 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.072 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 1 -- 172.21.15.110:0/1428166402 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7520039a40 con 0x7f7530071c60 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 1 -- 172.21.15.110:0/1428166402 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520039df0 con 0x7f7530071c60 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f7536773640 5 monclient: authenticate success, global_id 4882 2024-01-20T11:18:34.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f7536773640 10 monclient: _renew_subs 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f7536773640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753013fc80 con 0x7f7530071c60 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752e7fc640 4 set_mon_vals no callback set 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f752cff9640 10 monclient: dump: 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:34.074 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:34.075 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f7536773640 10 client.4882.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:34.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f7536773640 10 monclient: _renew_subs 2024-01-20T11:18:34.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.071+0000 7f7536773640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:34.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.072+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75301d6df0 con 0x7f7530071c60 2024-01-20T11:18:34.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.072+0000 7f7536773640 1 librados: init done 2024-01-20T11:18:34.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.072+0000 7f7536773640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:34.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.072+0000 7f7536773640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:34.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.072+0000 7f7536773640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:34.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.072+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7530075490 con 0x7f7530071c60 2024-01-20T11:18:34.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.091+0000 7f752cff9640 1 -- 172.21.15.110:0/1428166402 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f75200423e0 con 0x7f7530071c60 2024-01-20T11:18:34.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.092+0000 7f752cff9640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f751c06c8b0 0x7f751c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.092+0000 7f752f7fe640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f751c06c8b0 0x7f751c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.092+0000 7f752f7fe640 10 monclient: get_auth_request con 0x7f751c06c8b0 auth_method 0 2024-01-20T11:18:34.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.092+0000 7f752cff9640 10 client.4882.objecter ms_handle_connect 0x7f751c06c8b0 2024-01-20T11:18:34.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.092+0000 7f752f7fe640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f751c06c8b0 0x7f751c06ed70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f753013ec00 tx=0x7f7524009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:34.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.094+0000 7f752cff9640 1 -- 172.21.15.110:0/1428166402 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f752007a750 con 0x7f7530071c60 2024-01-20T11:18:34.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.094+0000 7f752cff9640 10 client.4882.objecter ms_dispatch 0x7f7530000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:34.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.094+0000 7f752cff9640 3 client.4882.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:34.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.094+0000 7f752cff9640 3 client.4882.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:34.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.094+0000 7f752cff9640 20 client.4882.objecter dump_active .. 0 homeless 2024-01-20T11:18:34.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.335+0000 7f752cff9640 1 -- 172.21.15.110:0/1428166402 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7520076a90 con 0x7f7530071c60 2024-01-20T11:18:34.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.335+0000 7f752cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:34.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.335+0000 7f752cff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:34.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.579+0000 7f7536773640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:34.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.579+0000 7f7536773640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:34.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.579+0000 7f7536773640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:34.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.579+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7530072c50 con 0x7f7530071c60 2024-01-20T11:18:34.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.584+0000 7f752cff9640 1 -- 172.21.15.110:0/1428166402 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f7520034520 con 0x7f7530071c60 2024-01-20T11:18:34.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.584+0000 7f752cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:34.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.584+0000 7f752cff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:34.584 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:34.585 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":249,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:34.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 10 librados: watch_flush enter 2024-01-20T11:18:34.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 10 librados: watch_flush exit 2024-01-20T11:18:34.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 20 client.4882.objecter shutdown clearing up homeless session... 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 10 client.4882.objecter successfully canceled tick 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f751c06c8b0 msgr2=0x7f751c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f751c06c8b0 0x7f751c06ed70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f753013ec00 tx=0x7f7524009040 comp rx=0 tx=0).stop 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 10 monclient: shutdown 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530071c60 msgr2=0x7f753013ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530071c60 0x7f753013ca00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7520009890 tx=0x7f7520004730 comp rx=0 tx=0).stop 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 shutdown_connections 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f751c06c8b0 0x7f751c06ed70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75300b0930 0x7f753013f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75300ac540 0x7f75300707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 --2- 172.21.15.110:0/1428166402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530071c60 0x7f753013ca00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 >> 172.21.15.110:0/1428166402 conn(0x7f75300b3e60 msgr2=0x7f7530073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:34.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 shutdown_connections 2024-01-20T11:18:34.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.586+0000 7f7536773640 1 -- 172.21.15.110:0/1428166402 wait complete. 2024-01-20T11:18:34.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.587+0000 7f7536773640 1 librados: shutdown 2024-01-20T11:18:34.597 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:34.597 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.697+0000 7f6be9ca2640 10 monclient: get_monmap_and_config 2024-01-20T11:18:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.697+0000 7f6be9ca2640 10 monclient: build_initial_monmap 2024-01-20T11:18:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.697+0000 7f6be9ca2640 10 monclient: monmap: 2024-01-20T11:18:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:34.697779+0000 2024-01-20T11:18:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:34.697779+0000 2024-01-20T11:18:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:34.698 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.698 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:34.698 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:34.698 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:34.698 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:34.698 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:34.698 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.697+0000 7f6be9ca2640 1 Processor -- start 2024-01-20T11:18:34.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.697+0000 7f6be9ca2640 1 -- start start 2024-01-20T11:18:34.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.697+0000 7f6be9ca2640 10 monclient: init 2024-01-20T11:18:34.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6be406bb00 0x7f6be40b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 10 monclient(hunting): picked mon.noname-c con 0x7f6be406bb00 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be40af790 0x7f6be4145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 10 monclient(hunting): picked mon.noname-b con 0x7f6be40af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6be40b0900 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be406c220 con 0x7f6be40b0900 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be406bee0 con 0x7f6be406bb00 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be40ada30 con 0x7f6be40af790 2024-01-20T11:18:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.698+0000 7f6be9ca2640 10 monclient(hunting): authenticate will time out at 1558.048748s 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be37fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6be406bb00 0x7f6be40b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be40af790 0x7f6be4145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be40af790 0x7f6be4145770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43588/0 (socket says 172.21.15.110:43588) 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be2ffd640 1 -- 172.21.15.110:0/3737729522 learned_addr learned my addr 172.21.15.110:0/3737729522 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be2ffd640 10 monclient(hunting): get_auth_request con 0x7f6be40af790 auth_method 0 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be37fe640 10 monclient(hunting): get_auth_request con 0x7f6be406bb00 auth_method 0 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.699+0000 7f6be2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:34.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.700+0000 7f6be2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:34.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.700+0000 7f6be2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:34.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.700+0000 7f6be37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:34.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.700+0000 7f6be37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:34.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.700+0000 7f6be37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.701+0000 7f6be2ffd640 10 monclient(hunting): handle_auth_done global_id 4670 payload 274 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.701+0000 7f6be2ffd640 1 -- 172.21.15.110:0/3737729522 >> v2:172.21.15.134:3300/0 conn(0x7f6be406bb00 msgr2=0x7f6be40b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.701+0000 7f6be2ffd640 1 --2- 172.21.15.110:0/3737729522 >> v2:172.21.15.134:3300/0 conn(0x7f6be406bb00 0x7f6be40b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.701+0000 7f6be2ffd640 1 -- 172.21.15.110:0/3737729522 >> v1:172.21.15.134:6789/0 conn(0x7f6be40b0900 legacy=0x7f6be4145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.701+0000 7f6be2ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.701+0000 7f6be2ffd640 1 monclient: found mon.noname-b 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.701+0000 7f6be2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.701+0000 7f6be2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.701+0000 7f6be2ffd640 1 -- 172.21.15.110:0/3737729522 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be40add10 con 0x7f6be40af790 2024-01-20T11:18:34.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.703+0000 7f6be2ffd640 1 --2- 172.21.15.110:0/3737729522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be40af790 0x7f6be4145770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009850 tx=0x7f6bd402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbb4a993f50332a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 1 -- 172.21.15.110:0/3737729522 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4004230 con 0x7f6be40af790 2024-01-20T11:18:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:34.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 10 monclient: dump: 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 1 -- 172.21.15.110:0/3737729522 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd4004390 con 0x7f6be40af790 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 1 -- 172.21.15.110:0/3737729522 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4031830 con 0x7f6be40af790 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be9ca2640 5 monclient: authenticate success, global_id 4670 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be9ca2640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be9ca2640 4 set_mon_vals no callback set 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.706+0000 7f6be27fc640 10 monclient: dump: 2024-01-20T11:18:34.709 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 10 monclient: shutdown 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 1 -- 172.21.15.110:0/3737729522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be40af790 msgr2=0x7f6be4145770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 1 --2- 172.21.15.110:0/3737729522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be40af790 0x7f6be4145770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009850 tx=0x7f6bd402b8d0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 1 -- 172.21.15.110:0/3737729522 shutdown_connections 2024-01-20T11:18:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 1 --2- 172.21.15.110:0/3737729522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be40af790 0x7f6be4145770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 1 --2- 172.21.15.110:0/3737729522 >> v2:172.21.15.134:3300/0 conn(0x7f6be406bb00 0x7f6be40b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 1 -- 172.21.15.110:0/3737729522 >> 172.21.15.110:0/3737729522 conn(0x7f6be4066d60 msgr2=0x7f6be40b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 1 -- 172.21.15.110:0/3737729522 shutdown_connections 2024-01-20T11:18:34.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.707+0000 7f6be9ca2640 1 -- 172.21.15.110:0/3737729522 wait complete. 2024-01-20T11:18:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.708+0000 7f6be9ca2640 10 monclient: build_initial_monmap 2024-01-20T11:18:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.708+0000 7f6be9ca2640 10 monclient: monmap: 2024-01-20T11:18:34.714 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.708+0000 7f6be9ca2640 1 librados: starting msgr at 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.708+0000 7f6be9ca2640 1 librados: starting objecter 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.708+0000 7f6be9ca2640 1 Processor -- start 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.708+0000 7f6be9ca2640 1 -- start start 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.708+0000 7f6be9ca2640 1 librados: setting wanted keys 2024-01-20T11:18:34.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.708+0000 7f6be9ca2640 1 librados: calling monclient init 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.708+0000 7f6be9ca2640 10 monclient: init 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be406bb00 0x7f6be40b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient(hunting): picked mon.noname-c con 0x7f6be406bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6be40af790 0x7f6be411f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient(hunting): picked mon.noname-a con 0x7f6be40af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6be40b0900 0x7f6be41da440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient(hunting): picked mon.noname-b con 0x7f6be40b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be411f560 con 0x7f6be40af790 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:34.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be411edb0 con 0x7f6be40b0900 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be411f080 con 0x7f6be406bb00 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be9ca2640 10 monclient(hunting): authenticate will time out at 1558.059690s 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.709+0000 7f6be37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be406bb00 0x7f6be40b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be3fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6be40b0900 0x7f6be41da440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be2ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6be40af790 0x7f6be411f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be406bb00 0x7f6be40b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43604/0 (socket says 172.21.15.110:43604) 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be3fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6be40b0900 0x7f6be41da440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46402/0 (socket says 172.21.15.110:46402) 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be37fe640 1 -- 172.21.15.110:0/986683439 learned_addr learned my addr 172.21.15.110:0/986683439 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be37fe640 10 monclient(hunting): get_auth_request con 0x7f6be406bb00 auth_method 0 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:34.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be3fff640 10 monclient(hunting): get_auth_request con 0x7f6be40b0900 auth_method 0 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be2ffd640 10 monclient(hunting): get_auth_request con 0x7f6be40af790 auth_method 0 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.710+0000 7f6be37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.711+0000 7f6be2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.711+0000 7f6be2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.711+0000 7f6be2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 10 monclient(hunting): handle_auth_done global_id 4891 payload 824 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 1 -- 172.21.15.110:0/986683439 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be406bb00 msgr2=0x7f6be40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be406bb00 0x7f6be40b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 1 -- 172.21.15.110:0/986683439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6be40af790 msgr2=0x7f6be411f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6be40af790 0x7f6be411f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:34.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 1 monclient: found mon.noname-b 2024-01-20T11:18:34.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:34.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:34.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.713+0000 7f6be3fff640 1 -- 172.21.15.110:0/986683439 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd4009460 con 0x7f6be40b0900 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.715+0000 7f6be3fff640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6be40b0900 0x7f6be41da440 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00c700 tx=0x7f6bdc00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.715+0000 7f6bcbfff640 10 client.?.objecter ms_handle_connect 0x7f6be40b0900 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.715+0000 7f6bcbfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 1 -- 172.21.15.110:0/986683439 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc003b70 con 0x7f6be40b0900 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 10 monclient: dump: 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 10 monclient: _finish_auth 0 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 10 monclient: _check_auth_tickets 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 1 -- 172.21.15.110:0/986683439 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc003cd0 con 0x7f6be40b0900 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 1 -- 172.21.15.110:0/986683439 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc005310 con 0x7f6be40b0900 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6be9ca2640 5 monclient: authenticate success, global_id 4891 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6be9ca2640 10 monclient: _renew_subs 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6be9ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be41da980 con 0x7f6be40b0900 2024-01-20T11:18:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6bcbfff640 10 monclient: dump: 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:34.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.719+0000 7f6be1ffb640 4 set_mon_vals no callback set 2024-01-20T11:18:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.720+0000 7f6be9ca2640 10 client.4891.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.720+0000 7f6be9ca2640 10 monclient: _renew_subs 2024-01-20T11:18:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.720+0000 7f6be9ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.720+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be413e890 con 0x7f6be40b0900 2024-01-20T11:18:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.720+0000 7f6be9ca2640 1 librados: init done 2024-01-20T11:18:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.721+0000 7f6be9ca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.721+0000 7f6be9ca2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.721+0000 7f6be9ca2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:34.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.721+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be411f6d0 con 0x7f6be40b0900 2024-01-20T11:18:34.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.739+0000 7f6bcbfff640 1 -- 172.21.15.110:0/986683439 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6bdc005470 con 0x7f6be40b0900 2024-01-20T11:18:34.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.739+0000 7f6bcbfff640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bd006c8b0 0x7f6bd006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:34.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.739+0000 7f6be37fe640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bd006c8b0 0x7f6bd006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:34.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.739+0000 7f6be37fe640 10 monclient: get_auth_request con 0x7f6bd006c8b0 auth_method 0 2024-01-20T11:18:34.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.740+0000 7f6be37fe640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bd006c8b0 0x7f6bd006ed70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009c00 tx=0x7f6bd8009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:34.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.740+0000 7f6bcbfff640 10 client.4891.objecter ms_handle_connect 0x7f6bd006c8b0 2024-01-20T11:18:34.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.742+0000 7f6bcbfff640 1 -- 172.21.15.110:0/986683439 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6bdc0588b0 con 0x7f6be40b0900 2024-01-20T11:18:34.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.742+0000 7f6bcbfff640 10 client.4891.objecter ms_dispatch 0x7f6be4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:34.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.742+0000 7f6bcbfff640 3 client.4891.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:34.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.742+0000 7f6bcbfff640 3 client.4891.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:34.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.742+0000 7f6bcbfff640 20 client.4891.objecter dump_active .. 0 homeless 2024-01-20T11:18:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.984+0000 7f6bcbfff640 1 -- 172.21.15.110:0/986683439 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6bdc054bf0 con 0x7f6be40b0900 2024-01-20T11:18:34.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.984+0000 7f6bcbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:34.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:34.984+0000 7f6bcbfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:35.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.153+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6be4075490 con 0x7f6bd006c8b0 2024-01-20T11:18:35.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.156+0000 7f6bcbfff640 1 -- 172.21.15.110:0/986683439 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63962 (secure 0 0 0) 0x7f6be4075490 con 0x7f6bd006c8b0 2024-01-20T11:18:35.157 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:35.157 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:35.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 10 librados: watch_flush enter 2024-01-20T11:18:35.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 10 librados: watch_flush exit 2024-01-20T11:18:35.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 20 client.4891.objecter shutdown clearing up homeless session... 2024-01-20T11:18:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 10 client.4891.objecter successfully canceled tick 2024-01-20T11:18:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bd006c8b0 msgr2=0x7f6bd006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bd006c8b0 0x7f6bd006ed70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009c00 tx=0x7f6bd8009190 comp rx=0 tx=0).stop 2024-01-20T11:18:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 10 monclient: shutdown 2024-01-20T11:18:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6be40b0900 msgr2=0x7f6be41da440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6be40b0900 0x7f6be41da440 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00c700 tx=0x7f6bdc00cbc0 comp rx=0 tx=0).stop 2024-01-20T11:18:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 shutdown_connections 2024-01-20T11:18:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bd006c8b0 0x7f6bd006ed70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:35.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6be40b0900 0x7f6be41da440 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:35.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6be40af790 0x7f6be411f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:35.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 --2- 172.21.15.110:0/986683439 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6be406bb00 0x7f6be40b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:35.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 >> 172.21.15.110:0/986683439 conn(0x7f6be4066d60 msgr2=0x7f6be4142970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:35.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 shutdown_connections 2024-01-20T11:18:35.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 -- 172.21.15.110:0/986683439 wait complete. 2024-01-20T11:18:35.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:35.158+0000 7f6be9ca2640 1 librados: shutdown 2024-01-20T11:18:35.169 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-01-20T11:18:34.748053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002009"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.4630000000000001},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.8120000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":5.5469999999999997},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.7699999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.055},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.238}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.712999999999999},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":13.986000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.813000000000001},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":11.792999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.808},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":17.802}]}]},{"osd":7,"up_from":21,"seq":90194313260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6189999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.311999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7270000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.476000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.399},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.9350000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.3550000000000004},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.343}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.2769999999999992},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":13.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.105},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.0419999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":3.9260000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.3689999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.7080000000000002},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":8}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":9.0869999999999997},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.467000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.061},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.673999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":9.6699999999999999},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":12.334}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":14.768000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.749000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444075,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.3280000000000003},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":7.9589999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.5620000000000003},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.9350000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.228},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.2300000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.045},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.6539999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.971},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.657}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.367000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.946999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.7629999999999999},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":5.9329999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.159000000000001},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.798999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.997999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.8780000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":16.736999999999998},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.452999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.859},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":14.709}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":7.96},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.581}]}]},{"osd":0,"up_from":17,"seq":73014444076,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":6.3739999999999997},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":4.9560000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.4109999999999996},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.3159999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":11.720000000000001},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.021000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.131},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":12.877000000000001},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.202}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.416},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":13.510999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":9.4930000000000003},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.0660000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.4480000000000004},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":14.113}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.163},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.34}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.085000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":18.533000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.756},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.76}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.269},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.654}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:38.172 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:38.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.272+0000 7f832a875640 10 monclient: get_monmap_and_config 2024-01-20T11:18:38.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.272+0000 7f832a875640 10 monclient: build_initial_monmap 2024-01-20T11:18:38.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.272+0000 7f832a875640 10 monclient: monmap: 2024-01-20T11:18:38.272 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:38.272 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.273 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:38.273102+0000 2024-01-20T11:18:38.273 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:38.273102+0000 2024-01-20T11:18:38.273 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:38.273 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.273 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:38.273 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:38.274 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:38.274 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:38.274 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:38.274 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 1 Processor -- start 2024-01-20T11:18:38.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 1 -- start start 2024-01-20T11:18:38.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient: init 2024-01-20T11:18:38.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:38.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-01-20T11:18:38.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f832406f920 0x7f832406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient(hunting): picked mon.noname-b con 0x7f832406f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:38.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f832413d230 0x7f832413d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient(hunting): picked mon.noname-c con 0x7f832413d230 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:38.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8324143950 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:38.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83240715b0 con 0x7f8324143950 2024-01-20T11:18:38.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8324071890 con 0x7f832413d230 2024-01-20T11:18:38.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8324071b70 con 0x7f832406f920 2024-01-20T11:18:38.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:38.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.273+0000 7f832a875640 10 monclient(hunting): authenticate will time out at 1561.624044s 2024-01-20T11:18:38.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f8323fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f832406f920 0x7f832406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f8328e12640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8324143950 0x7f832413fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36060/0 (socket says 172.21.15.110:36060) 2024-01-20T11:18:38.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f8328e12640 1 -- 172.21.15.110:0/296404509 learned_addr learned my addr 172.21.15.110:0/296404509 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:38.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f83237fe640 1 --2- 172.21.15.110:0/296404509 >> v2:172.21.15.134:3300/0 conn(0x7f832413d230 0x7f832413d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f8323fff640 10 monclient(hunting): get_auth_request con 0x7f832406f920 auth_method 0 2024-01-20T11:18:38.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f8323fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f8323fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f8323fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f83237fe640 10 monclient(hunting): get_auth_request con 0x7f832413d230 auth_method 0 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f83237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f83237fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.274+0000 7f83237fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.275+0000 7f8323fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.275+0000 7f8323fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.275+0000 7f8323fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.275+0000 7f83237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.275+0000 7f83237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.275+0000 7f83237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f8322ffd640 1 -- 172.21.15.110:0/296404509 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77656504 0 0) 0x7f83240715b0 con 0x7f8324143950 2024-01-20T11:18:38.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f8322ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:18:38.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f8322ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f8322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f8322ffd640 10 monclient(hunting): my global_id is 4806 2024-01-20T11:18:38.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f8322ffd640 1 -- 172.21.15.110:0/296404509 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f830c003520 con 0x7f8324143950 2024-01-20T11:18:38.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f83237fe640 10 monclient(hunting): handle_auth_done global_id 4894 payload 274 2024-01-20T11:18:38.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f83237fe640 1 -- 172.21.15.110:0/296404509 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f832406f920 msgr2=0x7f832406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:38.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f83237fe640 1 --2- 172.21.15.110:0/296404509 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f832406f920 0x7f832406fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f83237fe640 1 -- 172.21.15.110:0/296404509 >> v1:172.21.15.110:6789/0 conn(0x7f8324143950 legacy=0x7f832413fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f83237fe640 10 monclient: _finish_hunting 0 2024-01-20T11:18:38.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f83237fe640 1 monclient: found mon.noname-c 2024-01-20T11:18:38.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f83237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:38.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f83237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:38.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f83237fe640 1 -- 172.21.15.110:0/296404509 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8324143430 con 0x7f832413d230 2024-01-20T11:18:38.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.277+0000 7f8323fff640 1 --2- 172.21.15.110:0/296404509 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f832406f920 0x7f832406fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:18:38.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.279+0000 7f83237fe640 1 --2- 172.21.15.110:0/296404509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832413d230 0x7f832413d650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8318009790 tx=0x7f831802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca67211847c2390a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:38.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 1 -- 172.21.15.110:0/296404509 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318031770 con 0x7f832413d230 2024-01-20T11:18:38.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:38.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:38.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 10 monclient: dump: 2024-01-20T11:18:38.293 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:38.293 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.293 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.293 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.293 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:38.294 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.294 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:38.294 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:38.294 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:38.294 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:38.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:38.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:38.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 1 -- 172.21.15.110:0/296404509 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83180318d0 con 0x7f832413d230 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 1 -- 172.21.15.110:0/296404509 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318031c60 con 0x7f832413d230 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f8322ffd640 10 monclient: dump: 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:38.295 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f832a875640 5 monclient: authenticate success, global_id 4894 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f832a875640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.282+0000 7f832a875640 4 set_mon_vals no callback set 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 10 monclient: shutdown 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 1 -- 172.21.15.110:0/296404509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832413d230 msgr2=0x7f832413d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 1 --2- 172.21.15.110:0/296404509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832413d230 0x7f832413d650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8318009790 tx=0x7f831802b810 comp rx=0 tx=0).stop 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 1 -- 172.21.15.110:0/296404509 shutdown_connections 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 1 --2- 172.21.15.110:0/296404509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832413d230 0x7f832413d650 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 1 --2- 172.21.15.110:0/296404509 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f832406f920 0x7f832406fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 1 -- 172.21.15.110:0/296404509 >> 172.21.15.110:0/296404509 conn(0x7f8324066d60 msgr2=0x7f83240b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 1 -- 172.21.15.110:0/296404509 shutdown_connections 2024-01-20T11:18:38.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.283+0000 7f832a875640 1 -- 172.21.15.110:0/296404509 wait complete. 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.284+0000 7f832a875640 10 monclient: build_initial_monmap 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.284+0000 7f832a875640 10 monclient: monmap: 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.284+0000 7f832a875640 1 librados: starting msgr at 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.284+0000 7f832a875640 1 librados: starting objecter 2024-01-20T11:18:38.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.284+0000 7f832a875640 1 Processor -- start 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.284+0000 7f832a875640 1 -- start start 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.284+0000 7f832a875640 1 librados: setting wanted keys 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.284+0000 7f832a875640 1 librados: calling monclient init 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.284+0000 7f832a875640 10 monclient: init 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832406f920 0x7f8324138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient(hunting): picked mon.noname-b con 0x7f832406f920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f832413d230 0x7f8324142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient(hunting): picked mon.noname-a con 0x7f832413d230 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8324143950 0x7f8324142b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient(hunting): picked mon.noname-c con 0x7f8324143950 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8324143080 con 0x7f832413d230 2024-01-20T11:18:38.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8324139720 con 0x7f832406f920 2024-01-20T11:18:38.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8324141640 con 0x7f8324143950 2024-01-20T11:18:38.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:38.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f832a875640 10 monclient(hunting): authenticate will time out at 1561.635652s 2024-01-20T11:18:38.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f83237fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f832413d230 0x7f8324142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f8328e12640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8324143950 0x7f8324142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.285+0000 7f8323fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832406f920 0x7f8324138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8323fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832406f920 0x7f8324138ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46418/0 (socket says 172.21.15.110:46418) 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8323fff640 1 -- 172.21.15.110:0/2028365125 learned_addr learned my addr 172.21.15.110:0/2028365125 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8323fff640 10 monclient(hunting): get_auth_request con 0x7f832406f920 auth_method 0 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8323fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8323fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8323fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8328e12640 10 monclient(hunting): get_auth_request con 0x7f8324143950 auth_method 0 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8328e12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8328e12640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8328e12640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f83237fe640 10 monclient(hunting): get_auth_request con 0x7f832413d230 auth_method 0 2024-01-20T11:18:38.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f83237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f83237fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f83237fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8323fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8323fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8323fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8328e12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8328e12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f8328e12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f83237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f83237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.286+0000 7f83237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 10 monclient(hunting): handle_auth_done global_id 4897 payload 824 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 1 -- 172.21.15.110:0/2028365125 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8324143950 msgr2=0x7f8324142b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8324143950 0x7f8324142b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 1 -- 172.21.15.110:0/2028365125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f832413d230 msgr2=0x7f8324142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f832413d230 0x7f8324142600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 1 monclient: found mon.noname-b 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.289+0000 7f8323fff640 1 -- 172.21.15.110:0/2028365125 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8314009400 con 0x7f832406f920 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.291+0000 7f8323fff640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832406f920 0x7f8324138ee0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f831400e990 tx=0x7f831400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.291+0000 7f8320ff9640 10 client.?.objecter ms_handle_connect 0x7f832406f920 2024-01-20T11:18:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.291+0000 7f8320ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:38.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 1 -- 172.21.15.110:0/2028365125 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831400c9e0 con 0x7f832406f920 2024-01-20T11:18:38.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:38.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:38.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 10 monclient: dump: 2024-01-20T11:18:38.305 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:38.305 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.305 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.305 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.305 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 1 -- 172.21.15.110:0/2028365125 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831400cb40 con 0x7f832406f920 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 1 -- 172.21.15.110:0/2028365125 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831400ced0 con 0x7f832406f920 2024-01-20T11:18:38.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f832a875640 5 monclient: authenticate success, global_id 4897 2024-01-20T11:18:38.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f832a875640 10 monclient: _renew_subs 2024-01-20T11:18:38.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f832a875640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:38.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83180093a0 con 0x7f832406f920 2024-01-20T11:18:38.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:38.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:38.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f8320ff9640 10 monclient: dump: 2024-01-20T11:18:38.307 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f83227fc640 4 set_mon_vals no callback set 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f832a875640 10 client.4897.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f832a875640 10 monclient: _renew_subs 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f832a875640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:38.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8324141b90 con 0x7f832406f920 2024-01-20T11:18:38.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.295+0000 7f832a875640 1 librados: init done 2024-01-20T11:18:38.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.296+0000 7f832a875640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:38.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.296+0000 7f832a875640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:38.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.296+0000 7f832a875640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:38.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.296+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8324075490 con 0x7f832406f920 2024-01-20T11:18:38.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.314+0000 7f8320ff9640 1 -- 172.21.15.110:0/2028365125 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8314013470 con 0x7f832406f920 2024-01-20T11:18:38.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.314+0000 7f8320ff9640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f831006c890 0x7f831006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.314+0000 7f83237fe640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f831006c890 0x7f831006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.314+0000 7f83237fe640 10 monclient: get_auth_request con 0x7f831006c890 auth_method 0 2024-01-20T11:18:38.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.315+0000 7f83237fe640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f831006c890 0x7f831006ed50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8318036000 tx=0x7f831803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:38.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.315+0000 7f8320ff9640 10 client.4897.objecter ms_handle_connect 0x7f831006c890 2024-01-20T11:18:38.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.316+0000 7f8320ff9640 1 -- 172.21.15.110:0/2028365125 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8314058c10 con 0x7f832406f920 2024-01-20T11:18:38.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.316+0000 7f8320ff9640 10 client.4897.objecter ms_dispatch 0x7f8324000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:38.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.316+0000 7f8320ff9640 3 client.4897.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:38.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.316+0000 7f8320ff9640 3 client.4897.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:38.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.317+0000 7f8320ff9640 20 client.4897.objecter dump_active .. 0 homeless 2024-01-20T11:18:38.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.554+0000 7f8320ff9640 1 -- 172.21.15.110:0/2028365125 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8314054f50 con 0x7f832406f920 2024-01-20T11:18:38.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.554+0000 7f8320ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:38.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.554+0000 7f8320ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:38.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.729+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8324139420 con 0x7f831006c890 2024-01-20T11:18:38.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.732+0000 7f8320ff9640 1 -- 172.21.15.110:0/2028365125 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63968 (secure 0 0 0) 0x7f8324139420 con 0x7f831006c890 2024-01-20T11:18:38.733 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:38.733 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 10 librados: watch_flush enter 2024-01-20T11:18:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 10 librados: watch_flush exit 2024-01-20T11:18:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 20 client.4897.objecter shutdown clearing up homeless session... 2024-01-20T11:18:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 10 client.4897.objecter successfully canceled tick 2024-01-20T11:18:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f831006c890 msgr2=0x7f831006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f831006c890 0x7f831006ed50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8318036000 tx=0x7f831803d040 comp rx=0 tx=0).stop 2024-01-20T11:18:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 10 monclient: shutdown 2024-01-20T11:18:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832406f920 msgr2=0x7f8324138ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832406f920 0x7f8324138ee0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f831400e990 tx=0x7f831400ee50 comp rx=0 tx=0).stop 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 shutdown_connections 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f831006c890 0x7f831006ed50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8324143950 0x7f8324142b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f832413d230 0x7f8324142600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 --2- 172.21.15.110:0/2028365125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f832406f920 0x7f8324138ee0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 >> 172.21.15.110:0/2028365125 conn(0x7f8324066d60 msgr2=0x7f832413a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 shutdown_connections 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.734+0000 7f832a875640 1 -- 172.21.15.110:0/2028365125 wait complete. 2024-01-20T11:18:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.735+0000 7f832a875640 1 librados: shutdown 2024-01-20T11:18:38.745 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-01-20T11:18:36.748381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002010"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.4630000000000001},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.8120000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":5.5469999999999997},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.7699999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.055},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.238}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.712999999999999},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":13.986000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.813000000000001},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":11.792999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.808},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":17.802}]}]},{"osd":7,"up_from":21,"seq":90194313261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6189999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.311999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7270000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.476000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.399},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.9350000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.3550000000000004},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.343}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.2769999999999992},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":13.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.105},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.0419999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":4.8479999999999999},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.867}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.3810000000000002},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.923999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":8.3930000000000007},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.291}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.616},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.282}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":6.5579999999999998},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":14.347}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.269},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.555}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444075,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.3280000000000003},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":7.9589999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.5620000000000003},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.9350000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.228},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.2300000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.045},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.6539999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.971},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.657}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.367000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.946999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.7629999999999999},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":5.9329999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.159000000000001},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.798999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.997999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.8780000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":16.736999999999998},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.452999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.859},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":14.709}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":7.96},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.581}]}]},{"osd":0,"up_from":17,"seq":73014444076,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":6.3739999999999997},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":4.9560000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.4109999999999996},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.3159999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":11.720000000000001},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.021000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.131},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.7400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":12.877000000000001},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.202}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.416},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":13.510999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509485,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":7.7999999999999998},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.3099999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.5129999999999999},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":11.391}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.010999999999999},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":6.5119999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.061999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":16.658000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.640999999999998},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.9529999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.709},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.647}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:38.747 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:38.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.846+0000 7fed77be2640 10 monclient: get_monmap_and_config 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.846+0000 7fed77be2640 10 monclient: build_initial_monmap 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.847+0000 7fed77be2640 10 monclient: monmap: 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:38.847603+0000 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:38.847603+0000 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:38.847 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:38.848 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.847+0000 7fed77be2640 1 Processor -- start 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.847+0000 7fed77be2640 1 -- start start 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.847+0000 7fed77be2640 10 monclient: init 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed70071c60 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700ac540 0x7fed700ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 10 monclient(hunting): picked mon.noname-b con 0x7fed700ac540 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fed700b0930 0x7fed70149510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 10 monclient(hunting): picked mon.noname-c con 0x7fed700b0930 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:38.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed700ad610 con 0x7fed70071c60 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed70072040 con 0x7fed700b0930 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed700aba00 con 0x7fed700ac540 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed77be2640 10 monclient(hunting): authenticate will time out at 1562.198567s 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed7617f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fed700b0930 0x7fed70149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.848+0000 7fed7517d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700ac540 0x7fed700ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7617f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fed700b0930 0x7fed70149510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46434/0 (socket says 172.21.15.110:46434) 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7617f640 1 -- 172.21.15.110:0/4053221776 learned_addr learned my addr 172.21.15.110:0/4053221776 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7617f640 10 monclient(hunting): get_auth_request con 0x7fed700b0930 auth_method 0 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7617f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7617f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7617f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7517d640 10 monclient(hunting): get_auth_request con 0x7fed700ac540 auth_method 0 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7517d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7517d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7517d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7517d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7517d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.849+0000 7fed7517d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.850+0000 7fed7617f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.850+0000 7fed7617f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.850+0000 7fed7617f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.851+0000 7fed7517d640 10 monclient(hunting): handle_auth_done global_id 4682 payload 274 2024-01-20T11:18:38.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.851+0000 7fed7517d640 1 -- 172.21.15.110:0/4053221776 >> v2:172.21.15.134:3300/0 conn(0x7fed700b0930 msgr2=0x7fed70149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:38.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.851+0000 7fed7517d640 1 --2- 172.21.15.110:0/4053221776 >> v2:172.21.15.134:3300/0 conn(0x7fed700b0930 0x7fed70149510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.851+0000 7fed7517d640 1 -- 172.21.15.110:0/4053221776 >> v1:172.21.15.134:6789/0 conn(0x7fed70071c60 legacy=0x7fed70142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.851+0000 7fed7517d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:38.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.851+0000 7fed7517d640 1 monclient: found mon.noname-b 2024-01-20T11:18:38.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.851+0000 7fed7517d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:38.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.851+0000 7fed7517d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:38.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.851+0000 7fed7517d640 1 -- 172.21.15.110:0/4053221776 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed700721b0 con 0x7fed700ac540 2024-01-20T11:18:38.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.853+0000 7fed7517d640 1 --2- 172.21.15.110:0/4053221776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700ac540 0x7fed700ab4c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fed600097a0 tx=0x7fed6002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ae3901898868a97 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 1 -- 172.21.15.110:0/4053221776 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60031830 con 0x7fed700ac540 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 10 monclient: dump: 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:38.857 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 10 monclient: _finish_auth 0 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 10 monclient: _check_auth_tickets 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 1 -- 172.21.15.110:0/4053221776 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed60031990 con 0x7fed700ac540 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 1 -- 172.21.15.110:0/4053221776 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60031d20 con 0x7fed700ac540 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 5 monclient: authenticate success, global_id 4682 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 4 set_mon_vals no callback set 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed7497c640 10 monclient: dump: 2024-01-20T11:18:38.858 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:38.859 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.859 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.859 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.859 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:38.859 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.859 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:38.859 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:38.859 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:38.859 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 10 monclient: shutdown 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 1 -- 172.21.15.110:0/4053221776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700ac540 msgr2=0x7fed700ab4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 1 --2- 172.21.15.110:0/4053221776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700ac540 0x7fed700ab4c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fed600097a0 tx=0x7fed6002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 1 -- 172.21.15.110:0/4053221776 shutdown_connections 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 1 --2- 172.21.15.110:0/4053221776 >> v2:172.21.15.134:3300/0 conn(0x7fed700b0930 0x7fed70149510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 1 --2- 172.21.15.110:0/4053221776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700ac540 0x7fed700ab4c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.857+0000 7fed77be2640 1 -- 172.21.15.110:0/4053221776 >> 172.21.15.110:0/4053221776 conn(0x7fed700b3e60 msgr2=0x7fed700b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.858+0000 7fed77be2640 1 -- 172.21.15.110:0/4053221776 shutdown_connections 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.858+0000 7fed77be2640 1 -- 172.21.15.110:0/4053221776 wait complete. 2024-01-20T11:18:38.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.858+0000 7fed77be2640 10 monclient: build_initial_monmap 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.858+0000 7fed77be2640 10 monclient: monmap: 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 librados: starting msgr at 2024-01-20T11:18:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 librados: starting objecter 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 Processor -- start 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 -- start start 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 librados: setting wanted keys 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 librados: calling monclient init 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient: init 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed70071c60 0x7fed700a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient(hunting): picked mon.noname-b con 0x7fed70071c60 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed700ac540 0x7fed70147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient(hunting): picked mon.noname-a con 0x7fed700ac540 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700b0930 0x7fed701d7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient(hunting): picked mon.noname-c con 0x7fed700b0930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed70147b90 con 0x7fed700ac540 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7006d6a0 con 0x7fed70071c60 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed701485a0 con 0x7fed700b0930 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.859+0000 7fed77be2640 10 monclient(hunting): authenticate will time out at 1562.210146s 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.860+0000 7fed7597e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed70071c60 0x7fed700a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.860+0000 7fed7517d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed700ac540 0x7fed70147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.860+0000 7fed7617f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700b0930 0x7fed701d7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.860+0000 7fed7597e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed70071c60 0x7fed700a9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:46450/0 (socket says 172.21.15.110:46450) 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.860+0000 7fed7597e640 1 -- 172.21.15.110:0/3539750910 learned_addr learned my addr 172.21.15.110:0/3539750910 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.860+0000 7fed7597e640 10 monclient(hunting): get_auth_request con 0x7fed70071c60 auth_method 0 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.860+0000 7fed7597e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.860+0000 7fed7597e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.860+0000 7fed7597e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7617f640 10 monclient(hunting): get_auth_request con 0x7fed700b0930 auth_method 0 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7617f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7617f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7617f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7517d640 10 monclient(hunting): get_auth_request con 0x7fed700ac540 auth_method 0 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7517d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7517d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7517d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7597e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7597e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7597e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7517d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7517d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7517d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7617f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:38.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7617f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:38.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.861+0000 7fed7617f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.863+0000 7fed7597e640 10 monclient(hunting): handle_auth_done global_id 4903 payload 824 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.864+0000 7fed7597e640 1 -- 172.21.15.110:0/3539750910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700b0930 msgr2=0x7fed701d7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.864+0000 7fed7597e640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700b0930 0x7fed701d7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.864+0000 7fed7597e640 1 -- 172.21.15.110:0/3539750910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed700ac540 msgr2=0x7fed70147480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.864+0000 7fed7597e640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed700ac540 0x7fed70147480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.864+0000 7fed7597e640 10 monclient: _finish_hunting 0 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.864+0000 7fed7597e640 1 monclient: found mon.noname-b 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.864+0000 7fed7597e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.864+0000 7fed7597e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.864+0000 7fed7597e640 1 -- 172.21.15.110:0/3539750910 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed60009400 con 0x7fed70071c60 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.866+0000 7fed7597e640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed70071c60 0x7fed700a9760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fed6800c790 tx=0x7fed6800cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:38.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.866+0000 7fed667fc640 10 client.?.objecter ms_handle_connect 0x7fed70071c60 2024-01-20T11:18:38.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.866+0000 7fed667fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 1 -- 172.21.15.110:0/3539750910 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6800cea0 con 0x7fed70071c60 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 10 monclient: dump: 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:38.871 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 1 -- 172.21.15.110:0/3539750910 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68006d50 con 0x7fed70071c60 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed77be2640 5 monclient: authenticate success, global_id 4903 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed77be2640 10 monclient: _renew_subs 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed77be2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed70148820 con 0x7fed70071c60 2024-01-20T11:18:38.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 1 -- 172.21.15.110:0/3539750910 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68003bb0 con 0x7fed70071c60 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed667fc640 10 monclient: dump: 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:38.873 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed67fff640 4 set_mon_vals no callback set 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed77be2640 10 client.4903.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed77be2640 10 monclient: _renew_subs 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed77be2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.869+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed701d7a90 con 0x7fed70071c60 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.870+0000 7fed77be2640 1 librados: init done 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.870+0000 7fed77be2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.870+0000 7fed77be2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.870+0000 7fed77be2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:38.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.870+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed70075490 con 0x7fed70071c60 2024-01-20T11:18:38.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.888+0000 7fed667fc640 1 -- 172.21.15.110:0/3539750910 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fed68003d10 con 0x7fed70071c60 2024-01-20T11:18:38.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.888+0000 7fed667fc640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed5406c8e0 0x7fed5406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:38.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.889+0000 7fed7517d640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed5406c8e0 0x7fed5406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:38.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.889+0000 7fed7517d640 10 monclient: get_auth_request con 0x7fed5406c8e0 auth_method 0 2024-01-20T11:18:38.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.889+0000 7fed7517d640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed5406c8e0 0x7fed5406eda0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fed60036000 tx=0x7fed6003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:38.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.889+0000 7fed667fc640 10 client.4903.objecter ms_handle_connect 0x7fed5406c8e0 2024-01-20T11:18:38.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.890+0000 7fed667fc640 1 -- 172.21.15.110:0/3539750910 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fed68058b20 con 0x7fed70071c60 2024-01-20T11:18:38.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.890+0000 7fed667fc640 10 client.4903.objecter ms_dispatch 0x7fed70000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:38.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.890+0000 7fed667fc640 3 client.4903.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:38.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.890+0000 7fed667fc640 3 client.4903.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:38.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:38.890+0000 7fed667fc640 20 client.4903.objecter dump_active .. 0 homeless 2024-01-20T11:18:39.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.125+0000 7fed667fc640 1 -- 172.21.15.110:0/3539750910 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fed680588f0 con 0x7fed70071c60 2024-01-20T11:18:39.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.125+0000 7fed667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:39.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.125+0000 7fed667fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:39.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.294+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed70072c50 con 0x7fed5406c8e0 2024-01-20T11:18:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.297+0000 7fed667fc640 1 -- 172.21.15.110:0/3539750910 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63903 (secure 0 0 0) 0x7fed70072c50 con 0x7fed5406c8e0 2024-01-20T11:18:39.297 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:39.298 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 10 librados: watch_flush enter 2024-01-20T11:18:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 10 librados: watch_flush exit 2024-01-20T11:18:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 20 client.4903.objecter shutdown clearing up homeless session... 2024-01-20T11:18:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 10 client.4903.objecter successfully canceled tick 2024-01-20T11:18:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed5406c8e0 msgr2=0x7fed5406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed5406c8e0 0x7fed5406eda0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fed60036000 tx=0x7fed6003c040 comp rx=0 tx=0).stop 2024-01-20T11:18:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 10 monclient: shutdown 2024-01-20T11:18:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed70071c60 msgr2=0x7fed700a9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed70071c60 0x7fed700a9760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fed6800c790 tx=0x7fed6800cc50 comp rx=0 tx=0).stop 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 shutdown_connections 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed5406c8e0 0x7fed5406eda0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed700b0930 0x7fed701d7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed700ac540 0x7fed70147480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 --2- 172.21.15.110:0/3539750910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed70071c60 0x7fed700a9760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 >> 172.21.15.110:0/3539750910 conn(0x7fed700b3e60 msgr2=0x7fed700af700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 shutdown_connections 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 -- 172.21.15.110:0/3539750910 wait complete. 2024-01-20T11:18:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.299+0000 7fed77be2640 1 librados: shutdown 2024-01-20T11:18:39.309 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-01-20T11:18:38.748676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001955"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1879999999999997},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.1920000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":4.9640000000000004},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3390000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.196},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.423}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":12.375999999999999},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":14}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.773},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":9.3569999999999993}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.115},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.667999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6189999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.311999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7270000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.476000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.399},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.9350000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.3550000000000004},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.343}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.2769999999999992},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":13.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.105},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.0419999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":4.8479999999999999},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.867}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.3810000000000002},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.923999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":8.3930000000000007},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.291}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.616},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.282}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":6.5579999999999998},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":14.347}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.269},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.555}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.5339999999999998},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":5.7089999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.9640000000000004},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.3689999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.657},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.993}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":16.004000000000001},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.294}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.59},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.952}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.377000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.278}]}]},{"osd":1,"up_from":17,"seq":73014444077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.1189999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":7.8010000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.5640000000000001},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.945}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.541},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.803000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":6.5800000000000001},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.731}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.157},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":8.4250000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.124000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":9.0139999999999993}]}]},{"osd":0,"up_from":17,"seq":73014444077,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":5.9630000000000001},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":9.3599999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.7389999999999999},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.9510000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.813000000000001},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.903}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.273},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.0709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":16.323},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":12.558}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.6},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":13.146000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509485,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":7.7999999999999998},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.3099999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.5129999999999999},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":11.391}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.010999999999999},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":6.5119999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.061999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":16.658000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.640999999999998},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.9529999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.709},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.647}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:39.311 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:39.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:39.311 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:39.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:39.311 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:39.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:39.311 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:39.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:39.312 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:39.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.409+0000 7f525c86f640 10 monclient: get_monmap_and_config 2024-01-20T11:18:39.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.409+0000 7f525c86f640 10 monclient: build_initial_monmap 2024-01-20T11:18:39.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.409+0000 7f525c86f640 10 monclient: monmap: 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:39.410265+0000 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:39.410265+0000 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:39.410 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 1 Processor -- start 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 1 -- start start 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 10 monclient: init 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f525806bb00 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52580af790 0x7f52580ae410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 10 monclient(hunting): picked mon.noname-b con 0x7f52580af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f52580ae950 0x7f52580aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:39.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 10 monclient(hunting): picked mon.noname-a con 0x7f52580ae950 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:39.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52580af320 con 0x7f525806bb00 2024-01-20T11:18:39.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:39.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525806bee0 con 0x7f52580ae950 2024-01-20T11:18:39.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:39.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52580ada30 con 0x7f52580af790 2024-01-20T11:18:39.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.410+0000 7f525c86f640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:39.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.411+0000 7f525c86f640 10 monclient(hunting): authenticate will time out at 1562.761185s 2024-01-20T11:18:39.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.411+0000 7f5256d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f52580ae950 0x7f52580aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:39.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.411+0000 7f524ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52580af790 0x7f52580ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.411+0000 7f524ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52580af790 0x7f52580ae410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43648/0 (socket says 172.21.15.110:43648) 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.411+0000 7f524ffff640 1 -- 172.21.15.110:0/2383132039 learned_addr learned my addr 172.21.15.110:0/2383132039 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.411+0000 7f524ffff640 10 monclient(hunting): get_auth_request con 0x7f52580af790 auth_method 0 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.411+0000 7f524ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.411+0000 7f524ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.411+0000 7f524ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f5256d9d640 10 monclient(hunting): get_auth_request con 0x7f52580ae950 auth_method 0 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f5256d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f5256d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f5256d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f524ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f524ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f524ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:39.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f5256d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:39.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f5256d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:39.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.412+0000 7f5256d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:39.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.414+0000 7f524ffff640 10 monclient(hunting): handle_auth_done global_id 4688 payload 274 2024-01-20T11:18:39.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.414+0000 7f524ffff640 1 -- 172.21.15.110:0/2383132039 >> v2:172.21.15.110:3300/0 conn(0x7f52580ae950 msgr2=0x7f52580aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:39.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.414+0000 7f524ffff640 1 --2- 172.21.15.110:0/2383132039 >> v2:172.21.15.110:3300/0 conn(0x7f52580ae950 0x7f52580aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.414+0000 7f524ffff640 1 -- 172.21.15.110:0/2383132039 >> v1:172.21.15.110:6789/0 conn(0x7f525806bb00 legacy=0x7f52580b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:39.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.414+0000 7f524ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:39.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.414+0000 7f524ffff640 1 monclient: found mon.noname-b 2024-01-20T11:18:39.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.414+0000 7f524ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:39.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.414+0000 7f524ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:39.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.414+0000 7f524ffff640 1 -- 172.21.15.110:0/2383132039 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52580add10 con 0x7f52580af790 2024-01-20T11:18:39.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.416+0000 7f524ffff640 1 --2- 172.21.15.110:0/2383132039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52580af790 0x7f52580ae410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5240009850 tx=0x7f524002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c8dfc3fc920bdfa server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:39.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 1 -- 172.21.15.110:0/2383132039 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5240004230 con 0x7f52580af790 2024-01-20T11:18:39.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:39.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:39.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 10 monclient: dump: 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 10 monclient: _finish_auth 0 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 1 -- 172.21.15.110:0/2383132039 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5240004390 con 0x7f52580af790 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 1 -- 172.21.15.110:0/2383132039 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5240031830 con 0x7f52580af790 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f525c86f640 5 monclient: authenticate success, global_id 4688 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f525c86f640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f525c86f640 4 set_mon_vals no callback set 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.419+0000 7f5255d9b640 10 monclient: dump: 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:39.422 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:39.422 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:39.422 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:39.422 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:39.422 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:39.422 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:39.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 10 monclient: shutdown 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 1 -- 172.21.15.110:0/2383132039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52580af790 msgr2=0x7f52580ae410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 1 --2- 172.21.15.110:0/2383132039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52580af790 0x7f52580ae410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5240009850 tx=0x7f524002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 1 -- 172.21.15.110:0/2383132039 shutdown_connections 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 1 --2- 172.21.15.110:0/2383132039 >> v2:172.21.15.110:3300/0 conn(0x7f52580ae950 0x7f52580aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 1 --2- 172.21.15.110:0/2383132039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f52580af790 0x7f52580ae410 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 1 -- 172.21.15.110:0/2383132039 >> 172.21.15.110:0/2383132039 conn(0x7f5258066d60 msgr2=0x7f52580b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 1 -- 172.21.15.110:0/2383132039 shutdown_connections 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.420+0000 7f525c86f640 1 -- 172.21.15.110:0/2383132039 wait complete. 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.421+0000 7f525c86f640 10 monclient: build_initial_monmap 2024-01-20T11:18:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.421+0000 7f525c86f640 10 monclient: monmap: 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.421+0000 7f525c86f640 1 librados: starting msgr at 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.421+0000 7f525c86f640 1 librados: starting objecter 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 Processor -- start 2024-01-20T11:18:39.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 -- start start 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 librados: setting wanted keys 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 librados: calling monclient init 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient: init 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f525806bb00 0x7f52581da750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient(hunting): picked mon.noname-c con 0x7f525806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52580ae950 0x7f5258120f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient(hunting): picked mon.noname-b con 0x7f52580ae950 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52580af790 0x7f52581214a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient(hunting): picked mon.noname-a con 0x7f52580af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5258120ad0 con 0x7f52580af790 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5258122d00 con 0x7f52580ae950 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525813f4a0 con 0x7f525806bb00 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.422+0000 7f525c86f640 10 monclient(hunting): authenticate will time out at 1562.772860s 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f525806bb00 0x7f52581da750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f524ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52580ae950 0x7f5258120f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f5256d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52580af790 0x7f52581214a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f525806bb00 0x7f52581da750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:53966/0 (socket says 172.21.15.110:53966) 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 1 -- 172.21.15.110:0/1750094719 learned_addr learned my addr 172.21.15.110:0/1750094719 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 10 monclient(hunting): get_auth_request con 0x7f525806bb00 auth_method 0 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f524ffff640 10 monclient(hunting): get_auth_request con 0x7f52580ae950 auth_method 0 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f524ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f524ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f524ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f5256d9d640 10 monclient(hunting): get_auth_request con 0x7f52580af790 auth_method 0 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f5256d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f5256d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f5256d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.423+0000 7f525659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.424+0000 7f524ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.424+0000 7f524ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.424+0000 7f524ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.424+0000 7f5256d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.424+0000 7f5256d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.424+0000 7f5256d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 10 monclient(hunting): handle_auth_done global_id 4906 payload 824 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 1 -- 172.21.15.110:0/1750094719 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f525806bb00 msgr2=0x7f52581da750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f525806bb00 0x7f52581da750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 1 -- 172.21.15.110:0/1750094719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52580af790 msgr2=0x7f52581214a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52580af790 0x7f52581214a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 1 monclient: found mon.noname-b 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f524ffff640 1 -- 172.21.15.110:0/1750094719 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5240009460 con 0x7f52580ae950 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.426+0000 7f525659c640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f525806bb00 0x7f52581da750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.429+0000 7f524ffff640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52580ae950 0x7f5258120f60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5240004000 tx=0x7f52400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.429+0000 7f524effd640 10 client.?.objecter ms_handle_connect 0x7f52580ae950 2024-01-20T11:18:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.429+0000 7f524effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 1 -- 172.21.15.110:0/1750094719 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5240031a40 con 0x7f52580ae950 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 10 monclient: dump: 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 10 monclient: _finish_auth 0 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 1 -- 172.21.15.110:0/1750094719 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5240031ba0 con 0x7f52580ae950 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525c86f640 5 monclient: authenticate success, global_id 4906 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525c86f640 10 monclient: _renew_subs 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525c86f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52581206a0 con 0x7f52580ae950 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 1 -- 172.21.15.110:0/1750094719 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5240005e50 con 0x7f52580ae950 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:39.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f524effd640 10 monclient: dump: 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525559a640 4 set_mon_vals no callback set 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525c86f640 10 client.4906.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525c86f640 10 monclient: _renew_subs 2024-01-20T11:18:39.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525c86f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:39.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5258121bc0 con 0x7f52580ae950 2024-01-20T11:18:39.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.432+0000 7f525c86f640 1 librados: init done 2024-01-20T11:18:39.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.433+0000 7f525c86f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:39.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.433+0000 7f525c86f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:39.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.433+0000 7f525c86f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:39.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.433+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5258075490 con 0x7f52580ae950 2024-01-20T11:18:39.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.683+0000 7f524effd640 1 -- 172.21.15.110:0/1750094719 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5240003610 con 0x7f52580ae950 2024-01-20T11:18:39.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.684+0000 7f524effd640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523806c860 0x7f523806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:39.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.684+0000 7f525659c640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523806c860 0x7f523806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:39.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.684+0000 7f525659c640 10 monclient: get_auth_request con 0x7f523806c860 auth_method 0 2024-01-20T11:18:39.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.685+0000 7f525659c640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523806c860 0x7f523806ed20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5248001c30 tx=0x7f5248009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:39.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.685+0000 7f524effd640 10 client.4906.objecter ms_handle_connect 0x7f523806c860 2024-01-20T11:18:39.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.686+0000 7f524effd640 1 -- 172.21.15.110:0/1750094719 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f524007a8e0 con 0x7f52580ae950 2024-01-20T11:18:39.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.686+0000 7f524effd640 10 client.4906.objecter ms_dispatch 0x7f5258000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:39.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.686+0000 7f524effd640 3 client.4906.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:39.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.686+0000 7f524effd640 3 client.4906.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:39.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.686+0000 7f524effd640 20 client.4906.objecter dump_active .. 0 homeless 2024-01-20T11:18:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.697+0000 7f524effd640 1 -- 172.21.15.110:0/1750094719 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5240076c20 con 0x7f52580ae950 2024-01-20T11:18:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.697+0000 7f524effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.697+0000 7f524effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:39.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.971+0000 7f525c86f640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:39.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.971+0000 7f525c86f640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:39.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.971+0000 7f525c86f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:39.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.971+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5258120890 con 0x7f52580ae950 2024-01-20T11:18:39.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.977+0000 7f524effd640 1 -- 172.21.15.110:0/1750094719 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f524003f020 con 0x7f52580ae950 2024-01-20T11:18:39.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.977+0000 7f524effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:39.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.977+0000 7f524effd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:39.977 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:39.977 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":254,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.978+0000 7f525c86f640 10 librados: watch_flush enter 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 10 librados: watch_flush exit 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 20 client.4906.objecter shutdown clearing up homeless session... 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 10 client.4906.objecter successfully canceled tick 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523806c860 msgr2=0x7f523806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523806c860 0x7f523806ed20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5248001c30 tx=0x7f5248009040 comp rx=0 tx=0).stop 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 10 monclient: shutdown 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52580ae950 msgr2=0x7f5258120f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:39.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52580ae950 0x7f5258120f60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5240004000 tx=0x7f52400046c0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 shutdown_connections 2024-01-20T11:18:39.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f523806c860 0x7f523806ed20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52580af790 0x7f52581214a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52580ae950 0x7f5258120f60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 --2- 172.21.15.110:0/1750094719 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f525806bb00 0x7f52581da750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:39.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 >> 172.21.15.110:0/1750094719 conn(0x7f5258066d60 msgr2=0x7f52580b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:39.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 shutdown_connections 2024-01-20T11:18:39.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 -- 172.21.15.110:0/1750094719 wait complete. 2024-01-20T11:18:39.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:39.979+0000 7f525c86f640 1 librados: shutdown 2024-01-20T11:18:39.990 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:39.990 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:40.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.088+0000 7f7b8fbfc640 10 monclient: get_monmap_and_config 2024-01-20T11:18:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.088+0000 7f7b8fbfc640 10 monclient: build_initial_monmap 2024-01-20T11:18:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.088+0000 7f7b8fbfc640 10 monclient: monmap: 2024-01-20T11:18:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:40.089474+0000 2024-01-20T11:18:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:40.089474+0000 2024-01-20T11:18:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:40.090 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:40.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.089+0000 7f7b8fbfc640 1 Processor -- start 2024-01-20T11:18:40.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.089+0000 7f7b8fbfc640 1 -- start start 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.089+0000 7f7b8fbfc640 10 monclient: init 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7b8806bb00 0x7f7b880b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient(hunting): picked mon.noname-c con 0x7f7b8806bb00 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 0x7f7b88145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient(hunting): picked mon.noname-b con 0x7f7b880af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b880b0900 0x7f7b88145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient(hunting): picked mon.noname-a con 0x7f7b880b0900 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8806bee0 con 0x7f7b880b0900 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b880ada30 con 0x7f7b8806bb00 2024-01-20T11:18:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b880add10 con 0x7f7b880af790 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8fbfc640 10 monclient(hunting): authenticate will time out at 1563.440482s 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8d998640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7b8806bb00 0x7f7b880b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.090+0000 7f7b8d197640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 0x7f7b88145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8e199640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b880b0900 0x7f7b88145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d998640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7b8806bb00 0x7f7b880b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49448/0 (socket says 172.21.15.110:49448) 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d998640 1 -- 172.21.15.110:0/1475156287 learned_addr learned my addr 172.21.15.110:0/1475156287 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d998640 10 monclient(hunting): get_auth_request con 0x7f7b8806bb00 auth_method 0 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d998640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d998640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d998640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d197640 10 monclient(hunting): get_auth_request con 0x7f7b880af790 auth_method 0 2024-01-20T11:18:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d197640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d197640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d197640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8e199640 10 monclient(hunting): get_auth_request con 0x7f7b880b0900 auth_method 0 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8e199640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8e199640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8e199640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d998640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d998640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d998640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d197640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d197640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.091+0000 7f7b8d197640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.092+0000 7f7b8e199640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:40.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.092+0000 7f7b8e199640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.092+0000 7f7b8e199640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 10 monclient(hunting): handle_auth_done global_id 4909 payload 274 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 1 -- 172.21.15.110:0/1475156287 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 msgr2=0x7f7b88145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 1 --2- 172.21.15.110:0/1475156287 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 0x7f7b88145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 1 -- 172.21.15.110:0/1475156287 >> v2:172.21.15.110:3300/0 conn(0x7f7b880b0900 msgr2=0x7f7b88145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 1 --2- 172.21.15.110:0/1475156287 >> v2:172.21.15.110:3300/0 conn(0x7f7b880b0900 0x7f7b88145cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 10 monclient: _finish_hunting 0 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 1 monclient: found mon.noname-c 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:40.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.093+0000 7f7b8d998640 1 -- 172.21.15.110:0/1475156287 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b880adff0 con 0x7f7b8806bb00 2024-01-20T11:18:40.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.095+0000 7f7b8d998640 1 --2- 172.21.15.110:0/1475156287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b8806bb00 0x7f7b880b03c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7b880b6630 tx=0x7f7b8402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea944673aeed3aa4 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:40.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 1 -- 172.21.15.110:0/1475156287 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b840317e0 con 0x7f7b8806bb00 2024-01-20T11:18:40.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:40.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:40.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 10 monclient: dump: 2024-01-20T11:18:40.100 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:40.100 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:40.100 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:40.100 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:40.100 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:40.100 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:40.101 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:40.101 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:40.101 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:40.101 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:40.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 10 monclient: _finish_auth 0 2024-01-20T11:18:40.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 10 monclient: _check_auth_tickets 2024-01-20T11:18:40.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:40.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8fbfc640 5 monclient: authenticate success, global_id 4909 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 1 -- 172.21.15.110:0/1475156287 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b84031940 con 0x7f7b8806bb00 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8fbfc640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8fbfc640 4 set_mon_vals no callback set 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 1 -- 172.21.15.110:0/1475156287 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84031cd0 con 0x7f7b8806bb00 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.099+0000 7f7b8c996640 10 monclient: dump: 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:40.102 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:40.103 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:40.103 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:40.103 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:40.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 10 monclient: shutdown 2024-01-20T11:18:40.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:40.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/1475156287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b8806bb00 msgr2=0x7f7b880b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:40.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/1475156287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b8806bb00 0x7f7b880b03c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7b880b6630 tx=0x7f7b8402b880 comp rx=0 tx=0).stop 2024-01-20T11:18:40.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/1475156287 shutdown_connections 2024-01-20T11:18:40.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/1475156287 >> v2:172.21.15.110:3300/0 conn(0x7f7b880b0900 0x7f7b88145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/1475156287 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 0x7f7b88145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/1475156287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b8806bb00 0x7f7b880b03c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/1475156287 >> 172.21.15.110:0/1475156287 conn(0x7f7b88066d60 msgr2=0x7f7b880b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:40.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/1475156287 shutdown_connections 2024-01-20T11:18:40.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.100+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/1475156287 wait complete. 2024-01-20T11:18:40.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.101+0000 7f7b8fbfc640 10 monclient: build_initial_monmap 2024-01-20T11:18:40.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.101+0000 7f7b8fbfc640 10 monclient: monmap: 2024-01-20T11:18:40.109 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:40.110 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:40.110 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:40.110 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:40.110 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:40.110 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:40.110 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:40.111 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:40.111 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:40.111 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:40.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.101+0000 7f7b8fbfc640 1 librados: starting msgr at 2024-01-20T11:18:40.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.101+0000 7f7b8fbfc640 1 librados: starting objecter 2024-01-20T11:18:40.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.101+0000 7f7b8fbfc640 1 Processor -- start 2024-01-20T11:18:40.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.101+0000 7f7b8fbfc640 1 -- start start 2024-01-20T11:18:40.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.101+0000 7f7b8fbfc640 1 librados: setting wanted keys 2024-01-20T11:18:40.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.101+0000 7f7b8fbfc640 1 librados: calling monclient init 2024-01-20T11:18:40.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.101+0000 7f7b8fbfc640 10 monclient: init 2024-01-20T11:18:40.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:40.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:40.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8806bb00 0x7f7b88070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient(hunting): picked mon.noname-a con 0x7f7b8806bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 0x7f7b8811da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient(hunting): picked mon.noname-c con 0x7f7b880af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b880b0900 0x7f7b881da600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient(hunting): picked mon.noname-b con 0x7f7b880b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b880b5370 con 0x7f7b8806bb00 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8811f0f0 con 0x7f7b880b0900 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b880b4dc0 con 0x7f7b880af790 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.102+0000 7f7b8fbfc640 10 monclient(hunting): authenticate will time out at 1563.452708s 2024-01-20T11:18:40.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b880b0900 0x7f7b881da600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:40.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d998640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8806bb00 0x7f7b88070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:40.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d197640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 0x7f7b8811da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:40.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b880b0900 0x7f7b881da600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49452/0 (socket says 172.21.15.110:49452) 2024-01-20T11:18:40.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 1 -- 172.21.15.110:0/2132519931 learned_addr learned my addr 172.21.15.110:0/2132519931 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:40.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 10 monclient(hunting): get_auth_request con 0x7f7b880b0900 auth_method 0 2024-01-20T11:18:40.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:40.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:40.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d998640 10 monclient(hunting): get_auth_request con 0x7f7b8806bb00 auth_method 0 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d998640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d998640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d998640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d197640 10 monclient(hunting): get_auth_request con 0x7f7b880af790 auth_method 0 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d197640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d197640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8d197640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:40.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:40.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.103+0000 7f7b8e199640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:40.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.104+0000 7f7b8d998640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:40.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.104+0000 7f7b8d998640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:40.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.104+0000 7f7b8d998640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:40.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.104+0000 7f7b8d197640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:40.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.104+0000 7f7b8d197640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:40.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.104+0000 7f7b8d197640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:40.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 10 monclient(hunting): handle_auth_done global_id 4912 payload 824 2024-01-20T11:18:40.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 1 -- 172.21.15.110:0/2132519931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 msgr2=0x7f7b8811da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:40.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 0x7f7b8811da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 1 -- 172.21.15.110:0/2132519931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8806bb00 msgr2=0x7f7b88070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:40.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8806bb00 0x7f7b88070a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 10 monclient: _finish_hunting 0 2024-01-20T11:18:40.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 1 monclient: found mon.noname-b 2024-01-20T11:18:40.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:40.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:40.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.106+0000 7f7b8e199640 1 -- 172.21.15.110:0/2132519931 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b840094d0 con 0x7f7b880b0900 2024-01-20T11:18:40.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.108+0000 7f7b8e199640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b880b0900 0x7f7b881da600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000efc0 tx=0x7f7b8000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:40.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.109+0000 7f7b7e7fc640 10 client.?.objecter ms_handle_connect 0x7f7b880b0900 2024-01-20T11:18:40.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.109+0000 7f7b7e7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:40.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 1 -- 172.21.15.110:0/2132519931 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80009ba0 con 0x7f7b880b0900 2024-01-20T11:18:40.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:40.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 10 monclient: dump: 2024-01-20T11:18:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:40.121 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 1 -- 172.21.15.110:0/2132519931 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8000f040 con 0x7f7b880b0900 2024-01-20T11:18:40.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b8fbfc640 5 monclient: authenticate success, global_id 4912 2024-01-20T11:18:40.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b8fbfc640 10 monclient: _renew_subs 2024-01-20T11:18:40.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b8fbfc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:40.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b880b4f80 con 0x7f7b880b0900 2024-01-20T11:18:40.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:40.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 1 -- 172.21.15.110:0/2132519931 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b800146c0 con 0x7f7b880b0900 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7e7fc640 10 monclient: dump: 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:40.123 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b7ffff640 4 set_mon_vals no callback set 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b8fbfc640 10 client.4912.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b8fbfc640 10 monclient: _renew_subs 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b8fbfc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b881dad20 con 0x7f7b880b0900 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.112+0000 7f7b8fbfc640 1 librados: init done 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.113+0000 7f7b8fbfc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.113+0000 7f7b8fbfc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.113+0000 7f7b8fbfc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.113+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b880b54e0 con 0x7f7b880b0900 2024-01-20T11:18:40.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.364+0000 7f7b7e7fc640 1 -- 172.21.15.110:0/2132519931 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7b80014820 con 0x7f7b880b0900 2024-01-20T11:18:40.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.364+0000 7f7b7e7fc640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7b6c06c860 0x7f7b6c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:40.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.364+0000 7f7b8d998640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7b6c06c860 0x7f7b6c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:40.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.364+0000 7f7b8d998640 10 monclient: get_auth_request con 0x7f7b6c06c860 auth_method 0 2024-01-20T11:18:40.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.365+0000 7f7b8d998640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7b6c06c860 0x7f7b6c06ed20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7b84036000 tx=0x7f7b8403d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:40.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.365+0000 7f7b7e7fc640 10 client.4912.objecter ms_handle_connect 0x7f7b6c06c860 2024-01-20T11:18:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.368+0000 7f7b7e7fc640 1 -- 172.21.15.110:0/2132519931 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7b80059360 con 0x7f7b880b0900 2024-01-20T11:18:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.368+0000 7f7b7e7fc640 10 client.4912.objecter ms_dispatch 0x7f7b88000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:40.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.368+0000 7f7b7e7fc640 3 client.4912.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:40.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.368+0000 7f7b7e7fc640 3 client.4912.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:40.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.369+0000 7f7b7e7fc640 20 client.4912.objecter dump_active .. 0 homeless 2024-01-20T11:18:40.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.380+0000 7f7b7e7fc640 1 -- 172.21.15.110:0/2132519931 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7b800556a0 con 0x7f7b880b0900 2024-01-20T11:18:40.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.380+0000 7f7b7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:40.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.380+0000 7f7b7e7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:40.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.548+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b881461f0 con 0x7f7b6c06c860 2024-01-20T11:18:40.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.551+0000 7f7b7e7fc640 1 -- 172.21.15.110:0/2132519931 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63903 (secure 0 0 0) 0x7f7b84031ab0 con 0x7f7b6c06c860 2024-01-20T11:18:40.552 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:40.552 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:40.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 10 librados: watch_flush enter 2024-01-20T11:18:40.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 10 librados: watch_flush exit 2024-01-20T11:18:40.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 20 client.4912.objecter shutdown clearing up homeless session... 2024-01-20T11:18:40.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 10 client.4912.objecter successfully canceled tick 2024-01-20T11:18:40.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7b6c06c860 msgr2=0x7f7b6c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:40.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7b6c06c860 0x7f7b6c06ed20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7b84036000 tx=0x7f7b8403d040 comp rx=0 tx=0).stop 2024-01-20T11:18:40.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 10 monclient: shutdown 2024-01-20T11:18:40.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b880b0900 msgr2=0x7f7b881da600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b880b0900 0x7f7b881da600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000efc0 tx=0x7f7b8000c490 comp rx=0 tx=0).stop 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 shutdown_connections 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7b6c06c860 0x7f7b6c06ed20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b880b0900 0x7f7b881da600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b880af790 0x7f7b8811da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 --2- 172.21.15.110:0/2132519931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8806bb00 0x7f7b88070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.553+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 >> 172.21.15.110:0/2132519931 conn(0x7f7b88066d60 msgr2=0x7f7b880b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.554+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 shutdown_connections 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.554+0000 7f7b8fbfc640 1 -- 172.21.15.110:0/2132519931 wait complete. 2024-01-20T11:18:40.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:40.554+0000 7f7b8fbfc640 1 librados: shutdown 2024-01-20T11:18:40.564 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-01-20T11:18:38.748676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001955"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1879999999999997},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.1920000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":4.9640000000000004},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3390000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.196},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.423}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":12.375999999999999},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":14}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.773},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":9.3569999999999993}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.115},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.667999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6189999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.311999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7270000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.476000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.399},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.9350000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.3550000000000004},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.343}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.2769999999999992},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":13.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.105},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.0419999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":4.8479999999999999},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.867}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.3810000000000002},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.923999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":8.3930000000000007},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.291}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.616},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.282}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":6.5579999999999998},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":14.347}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.269},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.555}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.5339999999999998},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":5.7089999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.9640000000000004},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.3689999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.657},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.993}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":16.004000000000001},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.294}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.59},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.952}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.377000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.278}]}]},{"osd":1,"up_from":17,"seq":73014444077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.1189999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":7.8010000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.5640000000000001},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.945}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.541},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.803000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":6.5800000000000001},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.731}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.157},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":8.4250000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.124000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":9.0139999999999993}]}]},{"osd":0,"up_from":17,"seq":73014444077,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":5.9630000000000001},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":9.3599999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.7389999999999999},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.9510000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.813000000000001},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.903}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.273},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.0709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":16.323},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":12.558}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.6},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":13.146000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509485,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":7.7999999999999998},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.3099999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.5129999999999999},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":11.391}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.010999999999999},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":6.5119999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.061999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":16.658000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.640999999999998},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.9529999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.709},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.647}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:43.566 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:43.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.666+0000 7f4e01a2b640 10 monclient: get_monmap_and_config 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.666+0000 7f4e01a2b640 10 monclient: build_initial_monmap 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.666+0000 7f4e01a2b640 10 monclient: monmap: 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:43.667505+0000 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:43.667505+0000 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:43.667 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:43.668 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:43.668 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.667+0000 7f4e01a2b640 1 Processor -- start 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.667+0000 7f4e01a2b640 1 -- start start 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.667+0000 7f4e01a2b640 10 monclient: init 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4dfc0b0c00 0x7f4dfc0accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f4dfc0b0c00 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4dfc13d0e0 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4dfc13d520 0x7f4dfc143910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:43.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f4dfc13d520 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc06dcc0 con 0x7f4dfc13d0e0 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc13d9b0 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc0ad450 con 0x7f4dfc13d520 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4e01a2b640 10 monclient(hunting): authenticate will time out at 1567.018454s 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4dfb7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4dfc13d520 0x7f4dfc143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.668+0000 7f4dfaffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4dfc0b0c00 0x7f4dfc0accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfb7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4dfc13d520 0x7f4dfc143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49468/0 (socket says 172.21.15.110:49468) 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfb7fe640 1 -- 172.21.15.110:0/597737371 learned_addr learned my addr 172.21.15.110:0/597737371 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f4dfc13d520 auth_method 0 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:43.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfaffd640 10 monclient(hunting): get_auth_request con 0x7f4dfc0b0c00 auth_method 0 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.669+0000 7f4dfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:43.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.670+0000 7f4dfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:43.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.671+0000 7f4dfb7fe640 10 monclient(hunting): handle_auth_done global_id 4915 payload 274 2024-01-20T11:18:43.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.671+0000 7f4dfb7fe640 1 -- 172.21.15.110:0/597737371 >> v2:172.21.15.110:3300/0 conn(0x7f4dfc0b0c00 msgr2=0x7f4dfc0accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:43.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.671+0000 7f4dfb7fe640 1 --2- 172.21.15.110:0/597737371 >> v2:172.21.15.110:3300/0 conn(0x7f4dfc0b0c00 0x7f4dfc0accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:43.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.671+0000 7f4dfb7fe640 1 -- 172.21.15.110:0/597737371 >> v1:172.21.15.110:6789/0 conn(0x7f4dfc13d0e0 legacy=0x7f4dfc143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:43.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.671+0000 7f4dfb7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:18:43.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.671+0000 7f4dfb7fe640 1 monclient: found mon.noname-c 2024-01-20T11:18:43.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.671+0000 7f4dfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:43.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.671+0000 7f4dfb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:43.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.671+0000 7f4dfb7fe640 1 -- 172.21.15.110:0/597737371 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc0ad8e0 con 0x7f4dfc13d520 2024-01-20T11:18:43.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.673+0000 7f4dfb7fe640 1 --2- 172.21.15.110:0/597737371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc13d520 0x7f4dfc143910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc1442a0 tx=0x7f4df402be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a741e569119727a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:43.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.676+0000 7f4dfa7fc640 1 -- 172.21.15.110:0/597737371 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df402c9f0 con 0x7f4dfc13d520 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 10 monclient: dump: 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:43.677 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 1 -- 172.21.15.110:0/597737371 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df402cb50 con 0x7f4dfc13d520 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 5 monclient: authenticate success, global_id 4915 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 1 -- 172.21.15.110:0/597737371 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df40343f0 con 0x7f4dfc13d520 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4dfa7fc640 10 monclient: dump: 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:43.678 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:43.679 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:43.679 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:43.679 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:43.679 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:43.679 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:43.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:43.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 4 set_mon_vals no callback set 2024-01-20T11:18:43.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 10 monclient: shutdown 2024-01-20T11:18:43.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:43.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 1 -- 172.21.15.110:0/597737371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc13d520 msgr2=0x7f4dfc143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:43.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 1 --2- 172.21.15.110:0/597737371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc13d520 0x7f4dfc143910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc1442a0 tx=0x7f4df402be50 comp rx=0 tx=0).stop 2024-01-20T11:18:43.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 1 -- 172.21.15.110:0/597737371 shutdown_connections 2024-01-20T11:18:43.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 1 --2- 172.21.15.110:0/597737371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc13d520 0x7f4dfc143910 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:43.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 1 --2- 172.21.15.110:0/597737371 >> v2:172.21.15.110:3300/0 conn(0x7f4dfc0b0c00 0x7f4dfc0accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:43.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 1 -- 172.21.15.110:0/597737371 >> 172.21.15.110:0/597737371 conn(0x7f4dfc0b3e60 msgr2=0x7f4dfc0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:43.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 1 -- 172.21.15.110:0/597737371 shutdown_connections 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.677+0000 7f4e01a2b640 1 -- 172.21.15.110:0/597737371 wait complete. 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.678+0000 7f4e01a2b640 10 monclient: build_initial_monmap 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.678+0000 7f4e01a2b640 10 monclient: monmap: 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.678+0000 7f4e01a2b640 1 librados: starting msgr at 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.678+0000 7f4e01a2b640 1 librados: starting objecter 2024-01-20T11:18:43.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.678+0000 7f4e01a2b640 1 Processor -- start 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.678+0000 7f4e01a2b640 1 -- start start 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.678+0000 7f4e01a2b640 1 librados: setting wanted keys 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.678+0000 7f4e01a2b640 1 librados: calling monclient init 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.678+0000 7f4e01a2b640 10 monclient: init 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc0b0c00 0x7f4dfc0ae490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f4dfc0b0c00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc13d0e0 0x7f4dfc141db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f4dfc13d0e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc13d520 0x7f4dfc0ab800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f4dfc13d520 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:43.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc0af480 con 0x7f4dfc13d520 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc0b01b0 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc1422f0 con 0x7f4dfc13d0e0 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4e01a2b640 10 monclient(hunting): authenticate will time out at 1567.029574s 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4dfb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc13d520 0x7f4dfc0ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4dfaffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc0b0c00 0x7f4dfc0ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.679+0000 7f4df3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc13d0e0 0x7f4dfc141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc13d520 0x7f4dfc0ab800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52362/0 (socket says 172.21.15.110:52362) 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfb7fe640 1 -- 172.21.15.110:0/4253151126 learned_addr learned my addr 172.21.15.110:0/4253151126 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f4dfc13d520 auth_method 0 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:43.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfaffd640 10 monclient(hunting): get_auth_request con 0x7f4dfc0b0c00 auth_method 0 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4df3fff640 10 monclient(hunting): get_auth_request con 0x7f4dfc13d0e0 auth_method 0 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4df3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4df3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4dfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4df3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:43.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4df3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:43.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.680+0000 7f4df3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:43.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 10 monclient(hunting): handle_auth_done global_id 4918 payload 824 2024-01-20T11:18:43.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 1 -- 172.21.15.110:0/4253151126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc13d0e0 msgr2=0x7f4dfc141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:43.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc13d0e0 0x7f4dfc141db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:43.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 1 -- 172.21.15.110:0/4253151126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc13d520 msgr2=0x7f4dfc0ab800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:43.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc13d520 0x7f4dfc0ab800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:43.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 10 monclient: _finish_hunting 0 2024-01-20T11:18:43.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 1 monclient: found mon.noname-b 2024-01-20T11:18:43.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:43.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:43.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.683+0000 7f4dfaffd640 1 -- 172.21.15.110:0/4253151126 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df4009b30 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.685+0000 7f4dfaffd640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc0b0c00 0x7f4dfc0ae490 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e850 tx=0x7f4dec00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:43.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.685+0000 7f4df8ff9640 10 client.?.objecter ms_handle_connect 0x7f4dfc0b0c00 2024-01-20T11:18:43.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.685+0000 7f4df8ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:43.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 1 -- 172.21.15.110:0/4253151126 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0095b0 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 10 monclient: dump: 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:43.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 1 -- 172.21.15.110:0/4253151126 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dec00c940 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 1 -- 172.21.15.110:0/4253151126 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec00ccd0 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4e01a2b640 5 monclient: authenticate success, global_id 4918 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df8ff9640 10 monclient: dump: 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:43.691 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4e01a2b640 10 monclient: _renew_subs 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4df9ffb640 4 set_mon_vals no callback set 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4e01a2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc0af680 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4e01a2b640 10 client.4918.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4e01a2b640 10 monclient: _renew_subs 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4e01a2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc0afd80 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.689+0000 7f4e01a2b640 1 librados: init done 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.690+0000 7f4e01a2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.690+0000 7f4e01a2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.690+0000 7f4e01a2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:43.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.690+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc0b0000 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.709+0000 7f4df8ff9640 1 -- 172.21.15.110:0/4253151126 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4dec013470 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.709+0000 7f4df8ff9640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ddc06c8b0 0x7f4ddc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:43.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.709+0000 7f4df3fff640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ddc06c8b0 0x7f4ddc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:43.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.709+0000 7f4df3fff640 10 monclient: get_auth_request con 0x7f4ddc06c8b0 auth_method 0 2024-01-20T11:18:43.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.710+0000 7f4df3fff640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ddc06c8b0 0x7f4ddc06ed70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc1397e0 tx=0x7f4de4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:43.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.710+0000 7f4df8ff9640 10 client.4918.objecter ms_handle_connect 0x7f4ddc06c8b0 2024-01-20T11:18:43.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.934+0000 7f4df8ff9640 1 -- 172.21.15.110:0/4253151126 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4dec00ce30 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.934+0000 7f4df8ff9640 10 client.4918.objecter ms_dispatch 0x7f4dfc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:43.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.934+0000 7f4df8ff9640 3 client.4918.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.934+0000 7f4df8ff9640 3 client.4918.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.934+0000 7f4df8ff9640 20 client.4918.objecter dump_active .. 0 homeless 2024-01-20T11:18:43.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.946+0000 7f4df8ff9640 1 -- 172.21.15.110:0/4253151126 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4dec054e70 con 0x7f4dfc0b0c00 2024-01-20T11:18:43.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.946+0000 7f4df8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:43.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:43.946+0000 7f4df8ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:44.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.112+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4dfc14a550 con 0x7f4ddc06c8b0 2024-01-20T11:18:44.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.115+0000 7f4df8ff9640 1 -- 172.21.15.110:0/4253151126 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63887 (secure 0 0 0) 0x7f4de4001d30 con 0x7f4ddc06c8b0 2024-01-20T11:18:44.115 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:44.115 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:44.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.116+0000 7f4e01a2b640 10 librados: watch_flush enter 2024-01-20T11:18:44.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.116+0000 7f4e01a2b640 10 librados: watch_flush exit 2024-01-20T11:18:44.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.116+0000 7f4e01a2b640 20 client.4918.objecter shutdown clearing up homeless session... 2024-01-20T11:18:44.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.116+0000 7f4e01a2b640 10 client.4918.objecter successfully canceled tick 2024-01-20T11:18:44.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ddc06c8b0 msgr2=0x7f4ddc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ddc06c8b0 0x7f4ddc06ed70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc1397e0 tx=0x7f4de4009040 comp rx=0 tx=0).stop 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 10 monclient: shutdown 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc0b0c00 msgr2=0x7f4dfc0ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc0b0c00 0x7f4dfc0ae490 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e850 tx=0x7f4dec00ed10 comp rx=0 tx=0).stop 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 shutdown_connections 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ddc06c8b0 0x7f4ddc06ed70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc13d520 0x7f4dfc0ab800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc13d0e0 0x7f4dfc141db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 --2- 172.21.15.110:0/4253151126 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4dfc0b0c00 0x7f4dfc0ae490 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 >> 172.21.15.110:0/4253151126 conn(0x7f4dfc0b3e60 msgr2=0x7f4dfc139f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 shutdown_connections 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 -- 172.21.15.110:0/4253151126 wait complete. 2024-01-20T11:18:44.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.117+0000 7f4e01a2b640 1 librados: shutdown 2024-01-20T11:18:44.127 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-01-20T11:18:42.749390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002017"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1879999999999997},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.1920000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":4.9640000000000004},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3390000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.196},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.423}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":12.375999999999999},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":14}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.773},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":9.3569999999999993}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.115},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.667999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.0289999999999999},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.125}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.015000000000001},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":16.920000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.452},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.4179999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.061},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.944000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":7.7519999999999998},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":11.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.220000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.159000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.3490000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6550000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":13.394},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.004}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":11.16},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.268000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.34},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.984}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":9.1750000000000007},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":14.73}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":17.140999999999998},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.292999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.5339999999999998},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":5.7089999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.9640000000000004},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.3689999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.657},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.993}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":16.004000000000001},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.294}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.59},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.952}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.377000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.278}]}]},{"osd":1,"up_from":17,"seq":73014444077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.1189999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":7.8010000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.5640000000000001},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.945}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.541},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.803000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":6.5800000000000001},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.731}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.157},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":8.4250000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.124000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":9.0139999999999993}]}]},{"osd":0,"up_from":17,"seq":73014444077,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":5.9630000000000001},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":9.3599999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.7389999999999999},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.9510000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.813000000000001},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.903}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.273},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.0709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":16.323},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":12.558}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.6},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":13.146000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509486,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":4.415},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.226}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":16.399000000000001},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":17.751000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.343},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.3450000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":9.1969999999999992},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.959}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.542999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.672000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.231999999999999},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.802}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:44.129 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:44.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.227+0000 7facc6e56640 10 monclient: get_monmap_and_config 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.227+0000 7facc6e56640 10 monclient: build_initial_monmap 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.227+0000 7facc6e56640 10 monclient: monmap: 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:44.228567+0000 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:44.228567+0000 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:44.228 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:44.229 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:44.229 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.228+0000 7facc6e56640 1 Processor -- start 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.228+0000 7facc6e56640 1 -- start start 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.228+0000 7facc6e56640 10 monclient: init 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 10 monclient(hunting): picked mon.noname-b con 0x7facc006f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facc013d230 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7facc0144060 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc00715b0 con 0x7facc0144060 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc0071890 con 0x7facc013d230 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc0071b70 con 0x7facc006f920 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:44.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc6e56640 10 monclient(hunting): authenticate will time out at 1567.579555s 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.229+0000 7facc4bf2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facb7fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7facc013d230 0x7facc0143950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:44492/0 (socket says 172.21.15.110:44492) 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facc53f3640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7facc0144060 0x7facc014a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35348/0 (socket says 172.21.15.110:35348) 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facb7fff640 1 -- 172.21.15.110:0/2208965072 learned_addr learned my addr 172.21.15.110:0/2208965072 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facc4bf2640 10 monclient(hunting): get_auth_request con 0x7facc006f920 auth_method 0 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facc4bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facc4bf2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facc4bf2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facc4bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facc4bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:44.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.230+0000 7facc4bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 1 -- 172.21.15.110:0/2208965072 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011323503 0 0) 0x7facc0071890 con 0x7facc013d230 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 10 monclient(hunting): _negotiate 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 10 monclient(hunting): my global_id is 4921 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 1 -- 172.21.15.110:0/2208965072 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca8003520 con 0x7facc013d230 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 1 -- 172.21.15.110:0/2208965072 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520367902 0 0) 0x7facc00715b0 con 0x7facc0144060 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 10 monclient(hunting): _negotiate 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 10 monclient(hunting): my global_id is 4839 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facb77fe640 1 -- 172.21.15.110:0/2208965072 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc0071890 con 0x7facc0144060 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facc4bf2640 10 monclient(hunting): handle_auth_done global_id 4703 payload 274 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facc4bf2640 1 -- 172.21.15.110:0/2208965072 >> v1:172.21.15.134:6789/0 conn(0x7facc013d230 legacy=0x7facc0143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facc4bf2640 1 -- 172.21.15.110:0/2208965072 >> v1:172.21.15.110:6789/0 conn(0x7facc0144060 legacy=0x7facc014a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facc4bf2640 10 monclient: _finish_hunting 0 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facc4bf2640 1 monclient: found mon.noname-b 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facc4bf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facc4bf2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.232+0000 7facc4bf2640 1 -- 172.21.15.110:0/2208965072 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc0143430 con 0x7facc006f920 2024-01-20T11:18:44.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.235+0000 7facc4bf2640 1 --2- 172.21.15.110:0/2208965072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc006fd00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7facbc0097d0 tx=0x7facbc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9033e0583ace40e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:44.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 1 -- 172.21.15.110:0/2208965072 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc031800 con 0x7facc006f920 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 10 monclient: dump: 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 10 monclient: _finish_auth 0 2024-01-20T11:18:44.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 10 monclient: _check_auth_tickets 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 1 -- 172.21.15.110:0/2208965072 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc031960 con 0x7facc006f920 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facc6e56640 5 monclient: authenticate success, global_id 4703 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facc6e56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 1 -- 172.21.15.110:0/2208965072 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc031cf0 con 0x7facc006f920 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facc6e56640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facc6e56640 4 set_mon_vals no callback set 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.238+0000 7facb77fe640 10 monclient: dump: 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.240 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.241 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:44.241 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.241 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:44.241 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:44.241 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:44.241 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.239+0000 7facc6e56640 10 monclient: shutdown 2024-01-20T11:18:44.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.239+0000 7facc6e56640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:44.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.239+0000 7facc6e56640 1 -- 172.21.15.110:0/2208965072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 msgr2=0x7facc006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.239+0000 7facc6e56640 1 --2- 172.21.15.110:0/2208965072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc006fd00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7facbc0097d0 tx=0x7facbc02b8a0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.239+0000 7facc6e56640 1 -- 172.21.15.110:0/2208965072 shutdown_connections 2024-01-20T11:18:44.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.239+0000 7facc6e56640 1 --2- 172.21.15.110:0/2208965072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc006fd00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.239+0000 7facc6e56640 1 -- 172.21.15.110:0/2208965072 >> 172.21.15.110:0/2208965072 conn(0x7facc0066d60 msgr2=0x7facc00b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:44.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.239+0000 7facc6e56640 1 -- 172.21.15.110:0/2208965072 shutdown_connections 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.239+0000 7facc6e56640 1 -- 172.21.15.110:0/2208965072 wait complete. 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.240+0000 7facc6e56640 10 monclient: build_initial_monmap 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.240+0000 7facc6e56640 10 monclient: monmap: 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.240+0000 7facc6e56640 1 librados: starting msgr at 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.240+0000 7facc6e56640 1 librados: starting objecter 2024-01-20T11:18:44.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.240+0000 7facc6e56640 1 Processor -- start 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.240+0000 7facc6e56640 1 -- start start 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.240+0000 7facc6e56640 1 librados: setting wanted keys 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.240+0000 7facc6e56640 1 librados: calling monclient init 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.240+0000 7facc6e56640 10 monclient: init 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc013dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient(hunting): picked mon.noname-c con 0x7facc006f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc013d230 0x7facc0139e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient(hunting): picked mon.noname-a con 0x7facc013d230 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facc0144060 0x7facc0137840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient(hunting): picked mon.noname-b con 0x7facc0144060 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc013efd0 con 0x7facc013d230 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:44.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc013f520 con 0x7facc0144060 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc013f240 con 0x7facc006f920 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc6e56640 10 monclient(hunting): authenticate will time out at 1567.591606s 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.241+0000 7facc4bf2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc013dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facc0144060 0x7facc0137840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facb7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc013d230 0x7facc0139e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facc0144060 0x7facc0137840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49488/0 (socket says 172.21.15.110:49488) 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 1 -- 172.21.15.110:0/1314377985 learned_addr learned my addr 172.21.15.110:0/1314377985 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 10 monclient(hunting): get_auth_request con 0x7facc0144060 auth_method 0 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc4bf2640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc013dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54010/0 (socket says 172.21.15.110:54010) 2024-01-20T11:18:44.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc4bf2640 10 monclient(hunting): get_auth_request con 0x7facc006f920 auth_method 0 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc4bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc4bf2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc4bf2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facb7fff640 10 monclient(hunting): get_auth_request con 0x7facc013d230 auth_method 0 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facb7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc53f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc4bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc4bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facc4bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:44.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:44.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.242+0000 7facb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:44.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 10 monclient(hunting): handle_auth_done global_id 4924 payload 824 2024-01-20T11:18:44.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 1 -- 172.21.15.110:0/1314377985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 msgr2=0x7facc013dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc013dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 1 -- 172.21.15.110:0/1314377985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc013d230 msgr2=0x7facc0139e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc013d230 0x7facc0139e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 10 monclient: _finish_hunting 0 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 1 monclient: found mon.noname-b 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.245+0000 7facc53f3640 1 -- 172.21.15.110:0/1314377985 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc009400 con 0x7facc0144060 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.247+0000 7facc53f3640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facc0144060 0x7facc0137840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7facb800c710 tx=0x7facb800cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.247+0000 7facb57fa640 10 client.?.objecter ms_handle_connect 0x7facc0144060 2024-01-20T11:18:44.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.247+0000 7facb57fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:44.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 1 -- 172.21.15.110:0/1314377985 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb800ce20 con 0x7facc0144060 2024-01-20T11:18:44.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 10 monclient: dump: 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 10 monclient: _finish_auth 0 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 10 monclient: _check_auth_tickets 2024-01-20T11:18:44.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 1 -- 172.21.15.110:0/1314377985 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb800a5e0 con 0x7facc0144060 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 1 -- 172.21.15.110:0/1314377985 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb800a970 con 0x7facc0144060 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facc6e56640 5 monclient: authenticate success, global_id 4924 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facc6e56640 10 monclient: _renew_subs 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facc6e56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc013a550 con 0x7facc0144060 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb6ffd640 4 set_mon_vals no callback set 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facb57fa640 10 monclient: dump: 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.253 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facc6e56640 10 client.4924.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facc6e56640 10 monclient: _renew_subs 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facc6e56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc01428b0 con 0x7facc0144060 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.250+0000 7facc6e56640 1 librados: init done 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.251+0000 7facc6e56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.251+0000 7facc6e56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.251+0000 7facc6e56640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:44.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.251+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc0075490 con 0x7facc0144060 2024-01-20T11:18:44.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.269+0000 7facb57fa640 1 -- 172.21.15.110:0/1314377985 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7facb800aad0 con 0x7facc0144060 2024-01-20T11:18:44.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.269+0000 7facb57fa640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7facac06c8e0 0x7facac06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.270+0000 7facc4bf2640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7facac06c8e0 0x7facac06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.270+0000 7facc4bf2640 10 monclient: get_auth_request con 0x7facac06c8e0 auth_method 0 2024-01-20T11:18:44.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.270+0000 7facc4bf2640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7facac06c8e0 0x7facac06eda0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7facbc036000 tx=0x7facbc03c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:44.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.270+0000 7facb57fa640 10 client.4924.objecter ms_handle_connect 0x7facac06c8e0 2024-01-20T11:18:44.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.495+0000 7facb57fa640 1 -- 172.21.15.110:0/1314377985 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7facb8057e90 con 0x7facc0144060 2024-01-20T11:18:44.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.495+0000 7facb57fa640 10 client.4924.objecter ms_dispatch 0x7facc0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:44.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.495+0000 7facb57fa640 3 client.4924.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:44.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.495+0000 7facb57fa640 3 client.4924.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:44.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.495+0000 7facb57fa640 20 client.4924.objecter dump_active .. 0 homeless 2024-01-20T11:18:44.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.507+0000 7facb57fa640 1 -- 172.21.15.110:0/1314377985 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7facb80541d0 con 0x7facc0144060 2024-01-20T11:18:44.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.507+0000 7facb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:44.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.507+0000 7facb57fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:44.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.701+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7facc0137f60 con 0x7facac06c8e0 2024-01-20T11:18:44.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.704+0000 7facb57fa640 1 -- 172.21.15.110:0/1314377985 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63887 (secure 0 0 0) 0x7facc0137f60 con 0x7facac06c8e0 2024-01-20T11:18:44.704 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:44.704 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:44.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.705+0000 7facc6e56640 10 librados: watch_flush enter 2024-01-20T11:18:44.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.705+0000 7facc6e56640 10 librados: watch_flush exit 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 20 client.4924.objecter shutdown clearing up homeless session... 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 10 client.4924.objecter successfully canceled tick 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7facac06c8e0 msgr2=0x7facac06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7facac06c8e0 0x7facac06eda0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7facbc036000 tx=0x7facbc03c040 comp rx=0 tx=0).stop 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 10 monclient: shutdown 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facc0144060 msgr2=0x7facc0137840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facc0144060 0x7facc0137840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7facb800c710 tx=0x7facb800cbd0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 shutdown_connections 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7facac06c8e0 0x7facac06eda0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facc0144060 0x7facc0137840 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc013d230 0x7facc0139e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 --2- 172.21.15.110:0/1314377985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc006f920 0x7facc013dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 >> 172.21.15.110:0/1314377985 conn(0x7facc0066d60 msgr2=0x7facc0073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:44.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 shutdown_connections 2024-01-20T11:18:44.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 -- 172.21.15.110:0/1314377985 wait complete. 2024-01-20T11:18:44.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.706+0000 7facc6e56640 1 librados: shutdown 2024-01-20T11:18:44.716 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-01-20T11:18:42.749390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002017"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1879999999999997},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.1920000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":4.9640000000000004},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3390000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.196},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.423}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":12.375999999999999},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":14}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.773},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":9.3569999999999993}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.115},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.667999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.0289999999999999},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.125}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.015000000000001},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":16.920000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.452},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.4179999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.061},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.944000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":7.7519999999999998},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":11.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.220000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.159000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.3490000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6550000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":13.394},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.004}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":11.16},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.268000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.34},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.984}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":9.1750000000000007},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":14.73}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":17.140999999999998},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.292999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.5339999999999998},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":5.7089999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.9640000000000004},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.3689999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.657},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.993}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":16.004000000000001},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.294}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.59},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.952}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.377000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.278}]}]},{"osd":1,"up_from":17,"seq":73014444077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.1189999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":7.8010000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.5640000000000001},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.945}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.541},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.803000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":6.5800000000000001},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.731}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.157},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":8.4250000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.124000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":9.0139999999999993}]}]},{"osd":0,"up_from":17,"seq":73014444077,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":5.9630000000000001},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":9.3599999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.7389999999999999},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.9510000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.813000000000001},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.903}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.273},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.0709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":16.323},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":12.558}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.6},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":13.146000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509486,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":4.415},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.226}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":16.399000000000001},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":17.751000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.343},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.3450000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":9.1969999999999992},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.959}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.542999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.672000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.231999999999999},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.802}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:44.718 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:44.718 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:44.718 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:44.718 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:44.718 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:44.718 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:44.718 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:44.718 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:44.719 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:44.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.819+0000 7f5f72d9e640 10 monclient: get_monmap_and_config 2024-01-20T11:18:44.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.819+0000 7f5f72d9e640 10 monclient: build_initial_monmap 2024-01-20T11:18:44.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.819+0000 7f5f72d9e640 10 monclient: monmap: 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:44.820306+0000 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:44.820306+0000 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:44.820 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.820+0000 7f5f72d9e640 1 Processor -- start 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.820+0000 7f5f72d9e640 1 -- start start 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.820+0000 7f5f72d9e640 10 monclient: init 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.820+0000 7f5f72d9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.820+0000 7f5f72d9e640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c06bb00 0x7f5f6c0b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f6c06bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5f6c0af790 0x7f5f6c145800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c0af790 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f6c0b0900 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c0ada30 con 0x7f5f6c0b0900 2024-01-20T11:18:44.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f6c0add10 con 0x7f5f6c0af790 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c0adff0 con 0x7f5f6c06bb00 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f72d9e640 10 monclient(hunting): authenticate will time out at 1568.171246s 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f70b3a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c06bb00 0x7f5f6c0b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f6bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5f6c0af790 0x7f5f6c145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f7133b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f5f6c0b0900 0x7f5f6c145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:44504/0 (socket says 172.21.15.110:44504) 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.821+0000 7f5f7133b640 1 -- 172.21.15.110:0/113356440 learned_addr learned my addr 172.21.15.110:0/113356440 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f6bfff640 10 monclient(hunting): get_auth_request con 0x7f5f6c0af790 auth_method 0 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f6bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f70b3a640 10 monclient(hunting): get_auth_request con 0x7f5f6c06bb00 auth_method 0 2024-01-20T11:18:44.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f70b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:44.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f70b3a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f70b3a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:44.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:44.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:44.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f70b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:44.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f70b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:44.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.822+0000 7f5f70b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:44.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.823+0000 7f5f6b7fe640 1 -- 172.21.15.110:0/113356440 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703395797 0 0) 0x7f5f6c0ada30 con 0x7f5f6c0b0900 2024-01-20T11:18:44.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.823+0000 7f5f6b7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:18:44.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.823+0000 7f5f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.823+0000 7f5f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.823+0000 7f5f6b7fe640 10 monclient(hunting): my global_id is 4927 2024-01-20T11:18:44.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.823+0000 7f5f6b7fe640 1 -- 172.21.15.110:0/113356440 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f5c003520 con 0x7f5f6c0b0900 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.824+0000 7f5f6bfff640 10 monclient(hunting): handle_auth_done global_id 4845 payload 274 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.824+0000 7f5f6bfff640 1 -- 172.21.15.110:0/113356440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c06bb00 msgr2=0x7f5f6c0b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.824+0000 7f5f6bfff640 1 --2- 172.21.15.110:0/113356440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c06bb00 0x7f5f6c0b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.824+0000 7f5f6bfff640 1 -- 172.21.15.110:0/113356440 >> v1:172.21.15.134:6789/0 conn(0x7f5f6c0b0900 legacy=0x7f5f6c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.824+0000 7f5f6bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.824+0000 7f5f6bfff640 1 monclient: found mon.noname-a 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.824+0000 7f5f6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.824+0000 7f5f6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.824+0000 7f5f6bfff640 1 -- 172.21.15.110:0/113356440 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c06c910 con 0x7f5f6c0af790 2024-01-20T11:18:44.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.826+0000 7f5f6bfff640 1 --2- 172.21.15.110:0/113356440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0af790 0x7f5f6c145800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009790 tx=0x7f5f5802b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270a1f0976cf8097 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:44.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 1 -- 172.21.15.110:0/113356440 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f58031770 con 0x7f5f6c0af790 2024-01-20T11:18:44.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:44.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:44.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 10 monclient: dump: 2024-01-20T11:18:44.829 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 1 -- 172.21.15.110:0/113356440 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f580318d0 con 0x7f5f6c0af790 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f72d9e640 5 monclient: authenticate success, global_id 4845 2024-01-20T11:18:44.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f72d9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f72d9e640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 1 -- 172.21.15.110:0/113356440 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f58031c60 con 0x7f5f6c0af790 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f72d9e640 4 set_mon_vals no callback set 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.829+0000 7f5f6b7fe640 10 monclient: dump: 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:44.831 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:44.832 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:44.832 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 10 monclient: shutdown 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 1 -- 172.21.15.110:0/113356440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0af790 msgr2=0x7f5f6c145800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 1 --2- 172.21.15.110:0/113356440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0af790 0x7f5f6c145800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009790 tx=0x7f5f5802b810 comp rx=0 tx=0).stop 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 1 -- 172.21.15.110:0/113356440 shutdown_connections 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 1 --2- 172.21.15.110:0/113356440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0af790 0x7f5f6c145800 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 1 --2- 172.21.15.110:0/113356440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c06bb00 0x7f5f6c0b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 1 -- 172.21.15.110:0/113356440 >> 172.21.15.110:0/113356440 conn(0x7f5f6c066d60 msgr2=0x7f5f6c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 1 -- 172.21.15.110:0/113356440 shutdown_connections 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.830+0000 7f5f72d9e640 1 -- 172.21.15.110:0/113356440 wait complete. 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.831+0000 7f5f72d9e640 10 monclient: build_initial_monmap 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.831+0000 7f5f72d9e640 10 monclient: monmap: 2024-01-20T11:18:44.835 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.831+0000 7f5f72d9e640 1 librados: starting msgr at 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.831+0000 7f5f72d9e640 1 librados: starting objecter 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.831+0000 7f5f72d9e640 1 Processor -- start 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.831+0000 7f5f72d9e640 1 -- start start 2024-01-20T11:18:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.831+0000 7f5f72d9e640 1 librados: setting wanted keys 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.831+0000 7f5f72d9e640 1 librados: calling monclient init 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.831+0000 7f5f72d9e640 10 monclient: init 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f6c06bb00 0x7f5f6c11e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f6c06bb00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c0af790 0x7f5f6c11f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6c0af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0b0900 0x7f5f6c13d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c0b0900 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c11eeb0 con 0x7f5f6c0b0900 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c11ea20 con 0x7f5f6c06bb00 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c11ebe0 con 0x7f5f6c0af790 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f72d9e640 10 monclient(hunting): authenticate will time out at 1568.182399s 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f7133b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0b0900 0x7f5f6c13d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f6bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c0af790 0x7f5f6c11f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.832+0000 7f5f70b3a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f6c06bb00 0x7f5f6c11e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f7133b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0b0900 0x7f5f6c13d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52380/0 (socket says 172.21.15.110:52380) 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f7133b640 1 -- 172.21.15.110:0/1842264612 learned_addr learned my addr 172.21.15.110:0/1842264612 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f7133b640 10 monclient(hunting): get_auth_request con 0x7f5f6c0b0900 auth_method 0 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f7133b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f7133b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f7133b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f70b3a640 10 monclient(hunting): get_auth_request con 0x7f5f6c06bb00 auth_method 0 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f70b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f70b3a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f70b3a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f6bfff640 10 monclient(hunting): get_auth_request con 0x7f5f6c0af790 auth_method 0 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f6bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f70b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f70b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f70b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f7133b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f7133b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f7133b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:44.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.833+0000 7f5f6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:44.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 10 monclient(hunting): handle_auth_done global_id 4930 payload 824 2024-01-20T11:18:44.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 1 -- 172.21.15.110:0/1842264612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c0af790 msgr2=0x7f5f6c11f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c0af790 0x7f5f6c11f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 1 -- 172.21.15.110:0/1842264612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0b0900 msgr2=0x7f5f6c13d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:44.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0b0900 0x7f5f6c13d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:44.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 10 monclient: _finish_hunting 0 2024-01-20T11:18:44.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 1 monclient: found mon.noname-b 2024-01-20T11:18:44.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:44.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:44.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.836+0000 7f5f70b3a640 1 -- 172.21.15.110:0/1842264612 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f580093a0 con 0x7f5f6c06bb00 2024-01-20T11:18:44.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.838+0000 7f5f70b3a640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f6c06bb00 0x7f5f6c11e4e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000e880 tx=0x7f5f6000ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:44.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.838+0000 7f5f697fa640 10 client.?.objecter ms_handle_connect 0x7f5f6c06bb00 2024-01-20T11:18:44.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.838+0000 7f5f697fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.841+0000 7f5f697fa640 1 -- 172.21.15.110:0/1842264612 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6000c940 con 0x7f5f6c06bb00 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.841+0000 7f5f697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 10 monclient: dump: 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.842 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 10 monclient: _finish_auth 0 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 10 monclient: _check_auth_tickets 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 1 -- 172.21.15.110:0/1842264612 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6000caa0 con 0x7f5f6c06bb00 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f72d9e640 5 monclient: authenticate success, global_id 4930 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f72d9e640 10 monclient: _renew_subs 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f72d9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c11f940 con 0x7f5f6c06bb00 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 1 -- 172.21.15.110:0/1842264612 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6000ce50 con 0x7f5f6c06bb00 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:44.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f697fa640 10 monclient: dump: 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f6affd640 4 set_mon_vals no callback set 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f72d9e640 10 client.4930.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:44.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f72d9e640 10 monclient: _renew_subs 2024-01-20T11:18:44.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f72d9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:44.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c13df00 con 0x7f5f6c06bb00 2024-01-20T11:18:44.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.842+0000 7f5f72d9e640 1 librados: init done 2024-01-20T11:18:44.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.843+0000 7f5f72d9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:44.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.843+0000 7f5f72d9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:44.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.843+0000 7f5f72d9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:44.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.843+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c13e1b0 con 0x7f5f6c06bb00 2024-01-20T11:18:44.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.862+0000 7f5f697fa640 1 -- 172.21.15.110:0/1842264612 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5f60013470 con 0x7f5f6c06bb00 2024-01-20T11:18:44.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.862+0000 7f5f697fa640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5f5006c860 0x7f5f5006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:44.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.862+0000 7f5f6bfff640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5f5006c860 0x7f5f5006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:44.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.862+0000 7f5f6bfff640 10 monclient: get_auth_request con 0x7f5f5006c860 auth_method 0 2024-01-20T11:18:44.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.863+0000 7f5f697fa640 10 client.4930.objecter ms_handle_connect 0x7f5f5006c860 2024-01-20T11:18:44.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:44.863+0000 7f5f6bfff640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5f5006c860 0x7f5f5006ed20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5f58036000 tx=0x7f5f5803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.091+0000 7f5f697fa640 1 -- 172.21.15.110:0/1842264612 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5f60058bb0 con 0x7f5f6c06bb00 2024-01-20T11:18:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.091+0000 7f5f697fa640 10 client.4930.objecter ms_dispatch 0x7f5f6c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.091+0000 7f5f697fa640 3 client.4930.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.091+0000 7f5f697fa640 3 client.4930.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.091+0000 7f5f697fa640 20 client.4930.objecter dump_active .. 0 homeless 2024-01-20T11:18:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.101+0000 7f5f697fa640 1 -- 172.21.15.110:0/1842264612 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5f60054ef0 con 0x7f5f6c06bb00 2024-01-20T11:18:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.101+0000 7f5f697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.101+0000 7f5f697fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:45.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.335+0000 7f5f72d9e640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:45.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.335+0000 7f5f72d9e640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:45.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.335+0000 7f5f72d9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:45.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.335+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5f6c075490 con 0x7f5f6c06bb00 2024-01-20T11:18:45.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.341+0000 7f5f697fa640 1 -- 172.21.15.110:0/1842264612 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f5f60021520 con 0x7f5f6c06bb00 2024-01-20T11:18:45.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.341+0000 7f5f697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:45.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.341+0000 7f5f697fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:45.341 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:45.341 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":259,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:45.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 10 librados: watch_flush enter 2024-01-20T11:18:45.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 10 librados: watch_flush exit 2024-01-20T11:18:45.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 20 client.4930.objecter shutdown clearing up homeless session... 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 10 client.4930.objecter successfully canceled tick 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5f5006c860 msgr2=0x7f5f5006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5f5006c860 0x7f5f5006ed20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5f58036000 tx=0x7f5f5803d040 comp rx=0 tx=0).stop 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 10 monclient: shutdown 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f6c06bb00 msgr2=0x7f5f6c11e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f6c06bb00 0x7f5f6c11e4e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000e880 tx=0x7f5f6000ed40 comp rx=0 tx=0).stop 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 shutdown_connections 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5f5006c860 0x7f5f5006ed20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6c0b0900 0x7f5f6c13d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f6c0af790 0x7f5f6c11f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 --2- 172.21.15.110:0/1842264612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f6c06bb00 0x7f5f6c11e4e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 >> 172.21.15.110:0/1842264612 conn(0x7f5f6c066d60 msgr2=0x7f5f6c143e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:45.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 shutdown_connections 2024-01-20T11:18:45.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 -- 172.21.15.110:0/1842264612 wait complete. 2024-01-20T11:18:45.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.343+0000 7f5f72d9e640 1 librados: shutdown 2024-01-20T11:18:45.354 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:45.354 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.453+0000 7fe258108640 10 monclient: get_monmap_and_config 2024-01-20T11:18:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.453+0000 7fe258108640 10 monclient: build_initial_monmap 2024-01-20T11:18:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.453+0000 7fe258108640 10 monclient: monmap: 2024-01-20T11:18:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:45.454034+0000 2024-01-20T11:18:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:45.454034+0000 2024-01-20T11:18:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:45.454 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:45.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.453+0000 7fe258108640 1 Processor -- start 2024-01-20T11:18:45.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 1 -- start start 2024-01-20T11:18:45.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient: init 2024-01-20T11:18:45.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2500b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient(hunting): picked mon.noname-b con 0x7fe25006bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe2500af790 0x7fe250145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient(hunting): picked mon.noname-a con 0x7fe2500af790 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2500b0900 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe25006c220 con 0x7fe2500b0900 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25006bee0 con 0x7fe2500af790 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500ada30 con 0x7fe25006bb00 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.454+0000 7fe258108640 10 monclient(hunting): authenticate will time out at 1568.804959s 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe255ea4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2500b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe2556a3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe2500af790 0x7fe250145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe2556a3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe2500af790 0x7fe250145770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52382/0 (socket says 172.21.15.110:52382) 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe255ea4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2500b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54030/0 (socket says 172.21.15.110:54030) 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe2556a3640 1 -- 172.21.15.110:0/3127681796 learned_addr learned my addr 172.21.15.110:0/3127681796 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe2556a3640 10 monclient(hunting): get_auth_request con 0x7fe2500af790 auth_method 0 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe2556a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe2556a3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe2556a3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe255ea4640 10 monclient(hunting): get_auth_request con 0x7fe25006bb00 auth_method 0 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe255ea4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe255ea4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.455+0000 7fe255ea4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.456+0000 7fe255ea4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.456+0000 7fe255ea4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.456+0000 7fe255ea4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.457+0000 7fe2556a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.457+0000 7fe2556a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.457+0000 7fe2556a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.458+0000 7fe255ea4640 10 monclient(hunting): handle_auth_done global_id 4715 payload 274 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.458+0000 7fe255ea4640 1 -- 172.21.15.110:0/3127681796 >> v2:172.21.15.110:3300/0 conn(0x7fe2500af790 msgr2=0x7fe250145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.458+0000 7fe255ea4640 1 --2- 172.21.15.110:0/3127681796 >> v2:172.21.15.110:3300/0 conn(0x7fe2500af790 0x7fe250145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.458+0000 7fe255ea4640 1 -- 172.21.15.110:0/3127681796 >> v1:172.21.15.110:6789/0 conn(0x7fe2500b0900 legacy=0x7fe250145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.458+0000 7fe255ea4640 10 monclient: _finish_hunting 0 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.458+0000 7fe255ea4640 1 monclient: found mon.noname-b 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.458+0000 7fe255ea4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.458+0000 7fe255ea4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.458+0000 7fe255ea4640 1 -- 172.21.15.110:0/3127681796 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2500add10 con 0x7fe25006bb00 2024-01-20T11:18:45.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.460+0000 7fe255ea4640 1 --2- 172.21.15.110:0/3127681796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2500b03c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe248009d90 tx=0x7fe24802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3fdf0ef79b60024 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 1 -- 172.21.15.110:0/3127681796 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24802cbc0 con 0x7fe25006bb00 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 10 monclient: dump: 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:45.465 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 10 monclient: _finish_auth 0 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 10 monclient: _check_auth_tickets 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 1 -- 172.21.15.110:0/3127681796 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24802cd20 con 0x7fe25006bb00 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe258108640 5 monclient: authenticate success, global_id 4715 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe258108640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe258108640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe258108640 4 set_mon_vals no callback set 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 1 -- 172.21.15.110:0/3127681796 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248034660 con 0x7fe25006bb00 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.464+0000 7fe254ea2640 10 monclient: dump: 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 10 monclient: shutdown 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 1 -- 172.21.15.110:0/3127681796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 msgr2=0x7fe2500b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 1 --2- 172.21.15.110:0/3127681796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2500b03c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe248009d90 tx=0x7fe24802be10 comp rx=0 tx=0).stop 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 1 -- 172.21.15.110:0/3127681796 shutdown_connections 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 1 --2- 172.21.15.110:0/3127681796 >> v2:172.21.15.110:3300/0 conn(0x7fe2500af790 0x7fe250145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 1 --2- 172.21.15.110:0/3127681796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2500b03c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 1 -- 172.21.15.110:0/3127681796 >> 172.21.15.110:0/3127681796 conn(0x7fe250066d60 msgr2=0x7fe2500b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:45.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 1 -- 172.21.15.110:0/3127681796 shutdown_connections 2024-01-20T11:18:45.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 1 -- 172.21.15.110:0/3127681796 wait complete. 2024-01-20T11:18:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 10 monclient: build_initial_monmap 2024-01-20T11:18:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.465+0000 7fe258108640 10 monclient: monmap: 2024-01-20T11:18:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 librados: starting msgr at 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 librados: starting objecter 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 Processor -- start 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 -- start start 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 librados: setting wanted keys 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 librados: calling monclient init 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient: init 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2501416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient(hunting): picked mon.noname-c con 0x7fe25006bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2500af790 0x7fe2501cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient(hunting): picked mon.noname-a con 0x7fe2500af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2500b0900 0x7fe2501cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient(hunting): picked mon.noname-b con 0x7fe2500b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250071380 con 0x7fe2500af790 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2501ccc40 con 0x7fe2500b0900 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2501ccf20 con 0x7fe25006bb00 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.466+0000 7fe258108640 10 monclient(hunting): authenticate will time out at 1568.817101s 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe255ea4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2501416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:45.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2556a3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2500af790 0x7fe2501cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2566a5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2500b0900 0x7fe2501cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe255ea4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2501416b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54042/0 (socket says 172.21.15.110:54042) 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe255ea4640 1 -- 172.21.15.110:0/3521369514 learned_addr learned my addr 172.21.15.110:0/3521369514 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe255ea4640 10 monclient(hunting): get_auth_request con 0x7fe25006bb00 auth_method 0 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe255ea4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe255ea4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe255ea4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2566a5640 10 monclient(hunting): get_auth_request con 0x7fe2500b0900 auth_method 0 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2566a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2566a5640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2566a5640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2556a3640 10 monclient(hunting): get_auth_request con 0x7fe2500af790 auth_method 0 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2556a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2556a3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.467+0000 7fe2556a3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.468+0000 7fe255ea4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.468+0000 7fe255ea4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.468+0000 7fe255ea4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.468+0000 7fe2566a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.468+0000 7fe2566a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.468+0000 7fe2566a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.468+0000 7fe2556a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.468+0000 7fe2556a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.468+0000 7fe2556a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 10 monclient(hunting): handle_auth_done global_id 4933 payload 824 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 1 -- 172.21.15.110:0/3521369514 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 msgr2=0x7fe2501416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2501416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 1 -- 172.21.15.110:0/3521369514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2500af790 msgr2=0x7fe2501cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2500af790 0x7fe2501cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 10 monclient: _finish_hunting 0 2024-01-20T11:18:45.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 1 monclient: found mon.noname-b 2024-01-20T11:18:45.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:45.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:45.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.470+0000 7fe2566a5640 1 -- 172.21.15.110:0/3521369514 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe248009a70 con 0x7fe2500b0900 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.473+0000 7fe2566a5640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2500b0900 0x7fe2501cb7d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00c390 tx=0x7fe24c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.473+0000 7fe2467fc640 10 client.?.objecter ms_handle_connect 0x7fe2500b0900 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.473+0000 7fe2467fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 1 -- 172.21.15.110:0/3521369514 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c0052d0 con 0x7fe2500b0900 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 10 monclient: dump: 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 1 -- 172.21.15.110:0/3521369514 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c005430 con 0x7fe2500b0900 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe258108640 5 monclient: authenticate success, global_id 4933 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe258108640 10 monclient: _renew_subs 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe258108640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2501cbd10 con 0x7fe2500b0900 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 1 -- 172.21.15.110:0/3521369514 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c003830 con 0x7fe2500b0900 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe2467fc640 10 monclient: dump: 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:45.479 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe247fff640 4 set_mon_vals no callback set 2024-01-20T11:18:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe258108640 10 client.4933.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe258108640 10 monclient: _renew_subs 2024-01-20T11:18:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe258108640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25013c820 con 0x7fe2500b0900 2024-01-20T11:18:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.476+0000 7fe258108640 1 librados: init done 2024-01-20T11:18:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.477+0000 7fe258108640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.477+0000 7fe258108640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.477+0000 7fe258108640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.477+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25013ca70 con 0x7fe2500b0900 2024-01-20T11:18:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.497+0000 7fe2467fc640 1 -- 172.21.15.110:0/3521369514 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe24c003990 con 0x7fe2500b0900 2024-01-20T11:18:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.497+0000 7fe2467fc640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe23406c8b0 0x7fe23406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.497+0000 7fe255ea4640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe23406c8b0 0x7fe23406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.497+0000 7fe255ea4640 10 monclient: get_auth_request con 0x7fe23406c8b0 auth_method 0 2024-01-20T11:18:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.498+0000 7fe255ea4640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe23406c8b0 0x7fe23406ed70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe248000f80 tx=0x7fe248030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.498+0000 7fe2467fc640 10 client.4933.objecter ms_handle_connect 0x7fe23406c8b0 2024-01-20T11:18:45.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.725+0000 7fe2467fc640 1 -- 172.21.15.110:0/3521369514 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe24c057b60 con 0x7fe2500b0900 2024-01-20T11:18:45.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.725+0000 7fe2467fc640 10 client.4933.objecter ms_dispatch 0x7fe250000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.725+0000 7fe2467fc640 3 client.4933.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.725+0000 7fe2467fc640 3 client.4933.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.725+0000 7fe2467fc640 20 client.4933.objecter dump_active .. 0 homeless 2024-01-20T11:18:45.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.738+0000 7fe2467fc640 1 -- 172.21.15.110:0/3521369514 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe24c053ea0 con 0x7fe2500b0900 2024-01-20T11:18:45.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.738+0000 7fe2467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:45.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.738+0000 7fe2467fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:45.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.917+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe25006c910 con 0x7fe23406c8b0 2024-01-20T11:18:45.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.920+0000 7fe2467fc640 1 -- 172.21.15.110:0/3521369514 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63903 (secure 0 0 0) 0x7fe24802ce90 con 0x7fe23406c8b0 2024-01-20T11:18:45.921 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:45.921 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:45.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.922+0000 7fe258108640 10 librados: watch_flush enter 2024-01-20T11:18:45.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.922+0000 7fe258108640 10 librados: watch_flush exit 2024-01-20T11:18:45.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.922+0000 7fe258108640 20 client.4933.objecter shutdown clearing up homeless session... 2024-01-20T11:18:45.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.922+0000 7fe258108640 10 client.4933.objecter successfully canceled tick 2024-01-20T11:18:45.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe23406c8b0 msgr2=0x7fe23406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:45.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe23406c8b0 0x7fe23406ed70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe248000f80 tx=0x7fe248030000 comp rx=0 tx=0).stop 2024-01-20T11:18:45.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 10 monclient: shutdown 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2500b0900 msgr2=0x7fe2501cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2500b0900 0x7fe2501cb7d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00c390 tx=0x7fe24c00c850 comp rx=0 tx=0).stop 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 shutdown_connections 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe23406c8b0 0x7fe23406ed70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2500b0900 0x7fe2501cb7d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2500af790 0x7fe2501cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 --2- 172.21.15.110:0/3521369514 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe25006bb00 0x7fe2501416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 >> 172.21.15.110:0/3521369514 conn(0x7fe250066d60 msgr2=0x7fe2500b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 shutdown_connections 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 -- 172.21.15.110:0/3521369514 wait complete. 2024-01-20T11:18:45.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:45.923+0000 7fe258108640 1 librados: shutdown 2024-01-20T11:18:45.937 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-01-20T11:18:44.749713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002004"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280560,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":11.455},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.272}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.0660000000000007},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":5.7830000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.539},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.818}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.4399999999999995},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":15.654999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.710000000000001},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":12.029999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.359},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":16.681000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.0289999999999999},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.125}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.015000000000001},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":16.920000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.452},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.4179999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.061},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.944000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":7.7519999999999998},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":11.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.220000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.159000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.3490000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6550000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":13.394},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.004}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":11.16},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.268000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.34},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.984}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":9.1750000000000007},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":14.73}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":17.140999999999998},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.292999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.9429999999999996},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":7.5549999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.941000000000001},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.507}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.238},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.708}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.597},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.901999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":17.364999999999998},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":16.632000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":18.106000000000002},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":15.195}]}]},{"osd":1,"up_from":17,"seq":73014444078,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.3449999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":6.7439999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.220000000000001},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.268000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.997},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":10.67}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":16.466999999999999},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.678000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.6880000000000006},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":14.948}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.9640000000000004},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.702}]}]},{"osd":0,"up_from":17,"seq":73014444078,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":6.2530000000000001},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":8.1379999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":4.9859999999999998},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.3719999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":9.5530000000000008},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.558999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.339},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":10.702999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":11.359},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":15.097}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.949},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":12.44}]}]},{"osd":6,"up_from":15,"seq":64424509486,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":4.415},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.226}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":16.399000000000001},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":17.751000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.343},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.3450000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":9.1969999999999992},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.959}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.542999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.672000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.231999999999999},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.802}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:48.939 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:49.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.039+0000 7f5e764e7640 10 monclient: get_monmap_and_config 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.039+0000 7f5e764e7640 10 monclient: build_initial_monmap 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.039+0000 7f5e764e7640 10 monclient: monmap: 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:49.039629+0000 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:49.039629+0000 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:49.039 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:49.040 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.039+0000 7f5e764e7640 1 Processor -- start 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.039+0000 7f5e764e7640 1 -- start start 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.039+0000 7f5e764e7640 10 monclient: init 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e7006a860 0x7f5e7013f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 10 monclient(hunting): picked mon.noname-c con 0x7f5e7006a860 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e7006ac40 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7013f990 0x7f5e70145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 10 monclient(hunting): picked mon.noname-b con 0x7f5e7013f990 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e700afd30 con 0x7f5e7006ac40 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e700b01c0 con 0x7f5e7006a860 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700723e0 con 0x7f5e7013f990 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e764e7640 10 monclient(hunting): authenticate will time out at 1572.390570s 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.040+0000 7f5e74a84640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7013f990 0x7f5e70145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e6ffff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e7006a860 0x7f5e7013f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e74a84640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7013f990 0x7f5e70145cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54058/0 (socket says 172.21.15.110:54058) 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e74a84640 1 -- 172.21.15.110:0/240149797 learned_addr learned my addr 172.21.15.110:0/240149797 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e74a84640 10 monclient(hunting): get_auth_request con 0x7f5e7013f990 auth_method 0 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e74a84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e74a84640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e74a84640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e6ffff640 10 monclient(hunting): get_auth_request con 0x7f5e7006a860 auth_method 0 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e6ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:49.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:49.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e74a84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:49.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e74a84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:49.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.041+0000 7f5e74a84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:49.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.042+0000 7f5e6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.042+0000 7f5e6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.042+0000 7f5e6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.043+0000 7f5e74a84640 10 monclient(hunting): handle_auth_done global_id 4721 payload 274 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.043+0000 7f5e74a84640 1 -- 172.21.15.110:0/240149797 >> v2:172.21.15.134:3300/0 conn(0x7f5e7006a860 msgr2=0x7f5e7013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.043+0000 7f5e74a84640 1 --2- 172.21.15.110:0/240149797 >> v2:172.21.15.134:3300/0 conn(0x7f5e7006a860 0x7f5e7013f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.043+0000 7f5e74a84640 1 -- 172.21.15.110:0/240149797 >> v1:172.21.15.134:6789/0 conn(0x7f5e7006ac40 legacy=0x7f5e701455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.043+0000 7f5e74a84640 10 monclient: _finish_hunting 0 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.043+0000 7f5e74a84640 1 monclient: found mon.noname-b 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.043+0000 7f5e74a84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.043+0000 7f5e74a84640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.043+0000 7f5e74a84640 1 -- 172.21.15.110:0/240149797 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e70072550 con 0x7f5e7013f990 2024-01-20T11:18:49.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.046+0000 7f5e74a84640 1 --2- 172.21.15.110:0/240149797 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7013f990 0x7f5e70145cc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009e50 tx=0x7f5e6802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a18d3153f336e53 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 1 -- 172.21.15.110:0/240149797 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6802ca70 con 0x7f5e7013f990 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 10 monclient: dump: 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:49.051 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 10 monclient: _finish_auth 0 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 1 -- 172.21.15.110:0/240149797 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6802cbd0 con 0x7f5e7013f990 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 5 monclient: authenticate success, global_id 4721 2024-01-20T11:18:49.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 4 set_mon_vals no callback set 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 1 -- 172.21.15.110:0/240149797 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e680343f0 con 0x7f5e7013f990 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e6effd640 10 monclient: dump: 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:49.053 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:49.054 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:49.054 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 10 monclient: shutdown 2024-01-20T11:18:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 1 -- 172.21.15.110:0/240149797 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7013f990 msgr2=0x7f5e70145cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 1 --2- 172.21.15.110:0/240149797 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7013f990 0x7f5e70145cc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009e50 tx=0x7f5e6802bed0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 1 -- 172.21.15.110:0/240149797 shutdown_connections 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 1 --2- 172.21.15.110:0/240149797 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7013f990 0x7f5e70145cc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 1 --2- 172.21.15.110:0/240149797 >> v2:172.21.15.134:3300/0 conn(0x7f5e7006a860 0x7f5e7013f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.051+0000 7f5e764e7640 1 -- 172.21.15.110:0/240149797 >> 172.21.15.110:0/240149797 conn(0x7f5e700b47d0 msgr2=0x7f5e700b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.052+0000 7f5e764e7640 1 -- 172.21.15.110:0/240149797 shutdown_connections 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.052+0000 7f5e764e7640 1 -- 172.21.15.110:0/240149797 wait complete. 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.052+0000 7f5e764e7640 10 monclient: build_initial_monmap 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.052+0000 7f5e764e7640 10 monclient: monmap: 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:49.059 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.052+0000 7f5e764e7640 1 librados: starting msgr at 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.052+0000 7f5e764e7640 1 librados: starting objecter 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 Processor -- start 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 -- start start 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 librados: setting wanted keys 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 librados: calling monclient init 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient: init 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7006a860 0x7f5e7006f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient(hunting): picked mon.noname-c con 0x7f5e7006a860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:49.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e7006ac40 0x7f5e70070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient(hunting): picked mon.noname-a con 0x7f5e7006ac40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:49.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7013f990 0x7f5e70070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient(hunting): picked mon.noname-b con 0x7f5e7013f990 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:49.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e701425b0 con 0x7f5e7006ac40 2024-01-20T11:18:49.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70071980 con 0x7f5e7013f990 2024-01-20T11:18:49.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70071c60 con 0x7f5e7006a860 2024-01-20T11:18:49.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:49.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.053+0000 7f5e764e7640 10 monclient(hunting): authenticate will time out at 1572.404028s 2024-01-20T11:18:49.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e74a84640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7013f990 0x7f5e70070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:49.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7006a860 0x7f5e7006f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:49.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e7006ac40 0x7f5e70070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:49.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e74a84640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7013f990 0x7f5e70070790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:49508/0 (socket says 172.21.15.110:49508) 2024-01-20T11:18:49.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e74a84640 1 -- 172.21.15.110:0/2011781851 learned_addr learned my addr 172.21.15.110:0/2011781851 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:49.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e74a84640 10 monclient(hunting): get_auth_request con 0x7f5e7013f990 auth_method 0 2024-01-20T11:18:49.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e74a84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:49.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e74a84640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:49.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e74a84640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:49.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6ffff640 10 monclient(hunting): get_auth_request con 0x7f5e7006a860 auth_method 0 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6f7fe640 10 monclient(hunting): get_auth_request con 0x7f5e7006ac40 auth_method 0 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.054+0000 7f5e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.055+0000 7f5e74a84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.055+0000 7f5e74a84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.055+0000 7f5e74a84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.055+0000 7f5e6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.055+0000 7f5e6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.055+0000 7f5e6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.055+0000 7f5e6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:49.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.055+0000 7f5e6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:49.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.055+0000 7f5e6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:49.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 10 monclient(hunting): handle_auth_done global_id 4939 payload 824 2024-01-20T11:18:49.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 1 -- 172.21.15.110:0/2011781851 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7006a860 msgr2=0x7f5e7006f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7006a860 0x7f5e7006f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 1 -- 172.21.15.110:0/2011781851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e7006ac40 msgr2=0x7f5e70070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e7006ac40 0x7f5e70070250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 10 monclient: _finish_hunting 0 2024-01-20T11:18:49.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 1 monclient: found mon.noname-b 2024-01-20T11:18:49.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:49.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:49.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.057+0000 7f5e74a84640 1 -- 172.21.15.110:0/2011781851 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e68009b30 con 0x7f5e7013f990 2024-01-20T11:18:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.059+0000 7f5e74a84640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7013f990 0x7f5e70070790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009e50 tx=0x7f5e68004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.059+0000 7f5e6cff9640 10 client.?.objecter ms_handle_connect 0x7f5e7013f990 2024-01-20T11:18:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.059+0000 7f5e6cff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 1 -- 172.21.15.110:0/2011781851 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68009510 con 0x7f5e7013f990 2024-01-20T11:18:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 10 monclient: dump: 2024-01-20T11:18:49.071 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:49.071 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.071 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.071 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.071 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:49.071 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.072 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:49.072 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:49.072 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:49.072 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:49.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 1 -- 172.21.15.110:0/2011781851 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6803e450 con 0x7f5e7013f990 2024-01-20T11:18:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 1 -- 172.21.15.110:0/2011781851 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6803e800 con 0x7f5e7013f990 2024-01-20T11:18:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:49.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6cff9640 10 monclient: dump: 2024-01-20T11:18:49.074 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:49.074 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.075 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.075 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.075 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:49.075 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.075 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:49.075 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:49.076 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:49.076 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e764e7640 5 monclient: authenticate success, global_id 4939 2024-01-20T11:18:49.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e764e7640 10 monclient: _renew_subs 2024-01-20T11:18:49.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e764e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:49.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7006fdb0 con 0x7f5e7013f990 2024-01-20T11:18:49.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e6e7fc640 4 set_mon_vals no callback set 2024-01-20T11:18:49.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e764e7640 10 client.4939.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:49.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e764e7640 10 monclient: _renew_subs 2024-01-20T11:18:49.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e764e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:49.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e701468c0 con 0x7f5e7013f990 2024-01-20T11:18:49.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.063+0000 7f5e764e7640 1 librados: init done 2024-01-20T11:18:49.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.064+0000 7f5e764e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:49.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.064+0000 7f5e764e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:49.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.064+0000 7f5e764e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:49.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.064+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e70075490 con 0x7f5e7013f990 2024-01-20T11:18:49.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.310+0000 7f5e6cff9640 1 -- 172.21.15.110:0/2011781851 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5e6803e960 con 0x7f5e7013f990 2024-01-20T11:18:49.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.310+0000 7f5e6cff9640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5e5006c890 0x7f5e5006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.311+0000 7f5e6ffff640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5e5006c890 0x7f5e5006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:49.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.311+0000 7f5e6ffff640 10 monclient: get_auth_request con 0x7f5e5006c890 auth_method 0 2024-01-20T11:18:49.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.311+0000 7f5e6cff9640 10 client.4939.objecter ms_handle_connect 0x7f5e5006c890 2024-01-20T11:18:49.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.311+0000 7f5e6ffff640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5e5006c890 0x7f5e5006ed50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009b40 tx=0x7f5e58009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:49.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.314+0000 7f5e6cff9640 1 -- 172.21.15.110:0/2011781851 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5e6807b670 con 0x7f5e7013f990 2024-01-20T11:18:49.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.314+0000 7f5e6cff9640 10 client.4939.objecter ms_dispatch 0x7f5e70000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:49.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.314+0000 7f5e6cff9640 3 client.4939.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:49.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.314+0000 7f5e6cff9640 3 client.4939.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:49.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.314+0000 7f5e6cff9640 20 client.4939.objecter dump_active .. 0 homeless 2024-01-20T11:18:49.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.325+0000 7f5e6cff9640 1 -- 172.21.15.110:0/2011781851 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5e680779b0 con 0x7f5e7013f990 2024-01-20T11:18:49.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.325+0000 7f5e6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:49.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.325+0000 7f5e6cff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:49.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.494+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e700af2e0 con 0x7f5e5006c890 2024-01-20T11:18:49.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.497+0000 7f5e6cff9640 1 -- 172.21.15.110:0/2011781851 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64043 (secure 0 0 0) 0x7f5e700af2e0 con 0x7f5e5006c890 2024-01-20T11:18:49.497 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:49.498 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 10 librados: watch_flush enter 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 10 librados: watch_flush exit 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 20 client.4939.objecter shutdown clearing up homeless session... 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 10 client.4939.objecter successfully canceled tick 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5e5006c890 msgr2=0x7f5e5006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5e5006c890 0x7f5e5006ed50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009b40 tx=0x7f5e58009190 comp rx=0 tx=0).stop 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 10 monclient: shutdown 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7013f990 msgr2=0x7f5e70070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7013f990 0x7f5e70070790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009e50 tx=0x7f5e68004a40 comp rx=0 tx=0).stop 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 shutdown_connections 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5e5006c890 0x7f5e5006ed50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7013f990 0x7f5e70070790 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e7006ac40 0x7f5e70070250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 --2- 172.21.15.110:0/2011781851 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e7006a860 0x7f5e7006f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 >> 172.21.15.110:0/2011781851 conn(0x7f5e700b47d0 msgr2=0x7f5e700b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:49.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 shutdown_connections 2024-01-20T11:18:49.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 -- 172.21.15.110:0/2011781851 wait complete. 2024-01-20T11:18:49.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.499+0000 7f5e764e7640 1 librados: shutdown 2024-01-20T11:18:49.509 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-01-20T11:18:48.750361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001979"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280561,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.5430000000000001},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.1859999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.2210000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.6050000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.8010000000000002},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.766999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.381},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":14.949}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.816000000000001},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":12.192}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.941000000000001},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.718}]}]},{"osd":7,"up_from":21,"seq":90194313263,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.4420000000000002},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.6120000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":5.7140000000000004},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.914}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":14.071},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.3049999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.516999999999999},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.409000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.727},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.157999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.146000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.085000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313264,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.5970000000000004},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.6420000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.4450000000000003},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.108000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":15.246},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.1460000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.708},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":12.048},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":11.372999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":16.175000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.302}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444078,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.3160000000000007},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":6.0540000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.8659999999999997},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.507}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.9849999999999994},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.571999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.9269999999999996},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.338000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.949999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.319000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":11.193}]}]},{"osd":1,"up_from":17,"seq":73014444079,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":13.891999999999999},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":12.025}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.581},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":15.843999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":8.7159999999999993},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.2640000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":12.640000000000001},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.84}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.1150000000000002},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":14.499000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.9969999999999999},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.422999999999998}]}]},{"osd":0,"up_from":17,"seq":73014444079,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":4.7480000000000002},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":9.8079999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.5359999999999996},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.2210000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.384},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.699999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.082000000000001},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.4190000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":14.773},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.102}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.739000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":7.8200000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509487,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.2649999999999997},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.9130000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.1840000000000002},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":8.9160000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.522},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.151999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.151999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.182}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.244999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":16.100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.9510000000000005},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.093}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:49.511 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:49.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.616+0000 7fb75375e640 10 monclient: get_monmap_and_config 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.616+0000 7fb75375e640 10 monclient: build_initial_monmap 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.616+0000 7fb75375e640 10 monclient: monmap: 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:49.617194+0000 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:49.617194+0000 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:49.618 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 1 Processor -- start 2024-01-20T11:18:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 1 -- start start 2024-01-20T11:18:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 10 monclient: init 2024-01-20T11:18:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-01-20T11:18:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb74c0a96f0 0x7fb74c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 10 monclient(hunting): picked mon.noname-a con 0x7fb74c0a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb74c13fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb74c0a9ad0 0x7fb74c142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.617+0000 7fb75375e640 10 monclient(hunting): picked mon.noname-c con 0x7fb74c0a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:49.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.618+0000 7fb75375e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c0b25c0 con 0x7fb74c13fa10 2024-01-20T11:18:49.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.618+0000 7fb75375e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.618+0000 7fb75375e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c0aaea0 con 0x7fb74c0a96f0 2024-01-20T11:18:49.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.618+0000 7fb75375e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.618+0000 7fb75375e640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c0ab0a0 con 0x7fb74c0a9ad0 2024-01-20T11:18:49.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.618+0000 7fb75375e640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:49.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.618+0000 7fb75375e640 10 monclient(hunting): authenticate will time out at 1572.968218s 2024-01-20T11:18:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.640+0000 7fb751cfb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb74c0a9ad0 0x7fb74c142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.641+0000 7fb750cf9640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fb74c13fa10 0x7fb74c1423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:46770/0 (socket says 172.21.15.110:46770) 2024-01-20T11:18:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.641+0000 7fb750cf9640 1 -- 172.21.15.110:0/1779591246 learned_addr learned my addr 172.21.15.110:0/1779591246 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.641+0000 7fb751cfb640 10 monclient(hunting): get_auth_request con 0x7fb74c0a9ad0 auth_method 0 2024-01-20T11:18:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.641+0000 7fb751cfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.641+0000 7fb751cfb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.641+0000 7fb751cfb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:49.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.642+0000 7fb751cfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:49.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.642+0000 7fb751cfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:49.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.642+0000 7fb751cfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:49.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.643+0000 7fb751cfb640 10 monclient(hunting): handle_auth_done global_id 4942 payload 274 2024-01-20T11:18:49.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.643+0000 7fb751cfb640 1 -- 172.21.15.110:0/1779591246 >> v2:172.21.15.110:3300/0 conn(0x7fb74c0a96f0 msgr2=0x7fb74c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:49.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.643+0000 7fb751cfb640 1 --2- 172.21.15.110:0/1779591246 >> v2:172.21.15.110:3300/0 conn(0x7fb74c0a96f0 0x7fb74c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.643+0000 7fb751cfb640 1 -- 172.21.15.110:0/1779591246 >> v1:172.21.15.134:6789/0 conn(0x7fb74c13fa10 legacy=0x7fb74c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.643+0000 7fb751cfb640 10 monclient: _finish_hunting 0 2024-01-20T11:18:49.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.643+0000 7fb751cfb640 1 monclient: found mon.noname-c 2024-01-20T11:18:49.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.644+0000 7fb751cfb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:49.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.644+0000 7fb751cfb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:49.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.644+0000 7fb751cfb640 1 -- 172.21.15.110:0/1779591246 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0b3fb0 con 0x7fb74c0a9ad0 2024-01-20T11:18:49.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.647+0000 7fb751cfb640 1 --2- 172.21.15.110:0/1779591246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c0a9ad0 0x7fb74c142af0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb74c074d70 tx=0x7fb74402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4203b3f97a2bfcef server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:49.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 1 -- 172.21.15.110:0/1779591246 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74402cbf0 con 0x7fb74c0a9ad0 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 10 monclient: dump: 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:49.652 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 10 monclient: _finish_auth 0 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 10 monclient: _check_auth_tickets 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 1 -- 172.21.15.110:0/1779591246 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74402cd50 con 0x7fb74c0a9ad0 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb75375e640 5 monclient: authenticate success, global_id 4942 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 1 -- 172.21.15.110:0/1779591246 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744034660 con 0x7fb74c0a9ad0 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb75375e640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb75375e640 4 set_mon_vals no callback set 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.651+0000 7fb743fff640 10 monclient: dump: 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.653 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.654 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.654 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:49.654 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.654 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:49.654 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:49.654 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:49.654 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 10 monclient: shutdown 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 1 -- 172.21.15.110:0/1779591246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c0a9ad0 msgr2=0x7fb74c142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 1 --2- 172.21.15.110:0/1779591246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c0a9ad0 0x7fb74c142af0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb74c074d70 tx=0x7fb74402be40 comp rx=0 tx=0).stop 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 1 -- 172.21.15.110:0/1779591246 shutdown_connections 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 1 --2- 172.21.15.110:0/1779591246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c0a9ad0 0x7fb74c142af0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 1 --2- 172.21.15.110:0/1779591246 >> v2:172.21.15.110:3300/0 conn(0x7fb74c0a96f0 0x7fb74c13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 1 -- 172.21.15.110:0/1779591246 >> 172.21.15.110:0/1779591246 conn(0x7fb74c0b7bc0 msgr2=0x7fb74c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 1 -- 172.21.15.110:0/1779591246 shutdown_connections 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.652+0000 7fb75375e640 1 -- 172.21.15.110:0/1779591246 wait complete. 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.653+0000 7fb75375e640 10 monclient: build_initial_monmap 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.653+0000 7fb75375e640 10 monclient: monmap: 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:49.657 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.653+0000 7fb75375e640 1 librados: starting msgr at 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.653+0000 7fb75375e640 1 librados: starting objecter 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.653+0000 7fb75375e640 1 Processor -- start 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.653+0000 7fb75375e640 1 -- start start 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.653+0000 7fb75375e640 1 librados: setting wanted keys 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.653+0000 7fb75375e640 1 librados: calling monclient init 2024-01-20T11:18:49.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.653+0000 7fb75375e640 10 monclient: init 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c0a96f0 0x7fb74c12fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient(hunting): picked mon.noname-c con 0x7fb74c0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c0a9ad0 0x7fb74c13d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient(hunting): picked mon.noname-a con 0x7fb74c0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c13eca0 0x7fb74c13fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient(hunting): picked mon.noname-b con 0x7fb74c13eca0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c134780 con 0x7fb74c0a9ad0 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c13d880 con 0x7fb74c13eca0 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c13e070 con 0x7fb74c0a96f0 2024-01-20T11:18:49.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.654+0000 7fb75375e640 10 monclient(hunting): authenticate will time out at 1573.004737s 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c13eca0 0x7fb74c13fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c13eca0 0x7fb74c13fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:60782/0 (socket says 172.21.15.110:60782) 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 1 -- 172.21.15.110:0/2739456209 learned_addr learned my addr 172.21.15.110:0/2739456209 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 10 monclient(hunting): get_auth_request con 0x7fb74c13eca0 auth_method 0 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:49.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.655+0000 7fb751cfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:49.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 10 monclient(hunting): handle_auth_done global_id 4945 payload 824 2024-01-20T11:18:49.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 1 -- 172.21.15.110:0/2739456209 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c0a96f0 msgr2=0x7fb74c12fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c0a96f0 0x7fb74c12fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 1 -- 172.21.15.110:0/2739456209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c0a9ad0 msgr2=0x7fb74c13d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:49.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c0a9ad0 0x7fb74c13d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:49.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 10 monclient: _finish_hunting 0 2024-01-20T11:18:49.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 1 monclient: found mon.noname-b 2024-01-20T11:18:49.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:49.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:49.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.658+0000 7fb751cfb640 1 -- 172.21.15.110:0/2739456209 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744009a70 con 0x7fb74c13eca0 2024-01-20T11:18:49.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.660+0000 7fb751cfb640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c13eca0 0x7fb74c13fe00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb744009e50 tx=0x7fb744004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:49.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.660+0000 7fb741ffb640 10 client.?.objecter ms_handle_connect 0x7fb74c13eca0 2024-01-20T11:18:49.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.660+0000 7fb741ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:49.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 1 -- 172.21.15.110:0/2739456209 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74403f420 con 0x7fb74c13eca0 2024-01-20T11:18:49.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:49.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:49.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 10 monclient: dump: 2024-01-20T11:18:49.664 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:49.664 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.664 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.664 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.664 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 10 monclient: _finish_auth 0 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 1 -- 172.21.15.110:0/2739456209 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74403f580 con 0x7fb74c13eca0 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 1 -- 172.21.15.110:0/2739456209 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74403f910 con 0x7fb74c13eca0 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb75375e640 5 monclient: authenticate success, global_id 4945 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb75375e640 10 monclient: _renew_subs 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb75375e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c13e2c0 con 0x7fb74c13eca0 2024-01-20T11:18:49.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb7437fe640 4 set_mon_vals no callback set 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb741ffb640 10 monclient: dump: 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb75375e640 10 client.4945.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:49.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb75375e640 10 monclient: _renew_subs 2024-01-20T11:18:49.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb75375e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:49.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c1344a0 con 0x7fb74c13eca0 2024-01-20T11:18:49.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.664+0000 7fb75375e640 1 librados: init done 2024-01-20T11:18:49.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.665+0000 7fb75375e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:49.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.665+0000 7fb75375e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:49.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.665+0000 7fb75375e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:49.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.665+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c13d060 con 0x7fb74c13eca0 2024-01-20T11:18:49.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.684+0000 7fb741ffb640 1 -- 172.21.15.110:0/2739456209 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb74403fa70 con 0x7fb74c13eca0 2024-01-20T11:18:49.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.685+0000 7fb741ffb640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb73006c860 0x7fb73006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:49.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.685+0000 7fb7514fa640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb73006c860 0x7fb73006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:49.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.685+0000 7fb7514fa640 10 monclient: get_auth_request con 0x7fb73006c860 auth_method 0 2024-01-20T11:18:49.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.686+0000 7fb741ffb640 10 client.4945.objecter ms_handle_connect 0x7fb73006c860 2024-01-20T11:18:49.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.686+0000 7fb7514fa640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb73006c860 0x7fb73006ed20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb748009710 tx=0x7fb748006c90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:49.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.687+0000 7fb741ffb640 1 -- 172.21.15.110:0/2739456209 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb74407b7d0 con 0x7fb74c13eca0 2024-01-20T11:18:49.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.687+0000 7fb741ffb640 10 client.4945.objecter ms_dispatch 0x7fb74c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:49.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.687+0000 7fb741ffb640 3 client.4945.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:49.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.687+0000 7fb741ffb640 3 client.4945.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:49.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.687+0000 7fb741ffb640 20 client.4945.objecter dump_active .. 0 homeless 2024-01-20T11:18:49.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.920+0000 7fb741ffb640 1 -- 172.21.15.110:0/2739456209 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb744077410 con 0x7fb74c13eca0 2024-01-20T11:18:49.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.920+0000 7fb741ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:49.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:49.920+0000 7fb741ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:50.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.092+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb74c13d060 con 0x7fb73006c860 2024-01-20T11:18:50.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.095+0000 7fb741ffb640 1 -- 172.21.15.110:0/2739456209 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64043 (secure 0 0 0) 0x7fb74c13d060 con 0x7fb73006c860 2024-01-20T11:18:50.095 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:50.096 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 10 librados: watch_flush enter 2024-01-20T11:18:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 10 librados: watch_flush exit 2024-01-20T11:18:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 20 client.4945.objecter shutdown clearing up homeless session... 2024-01-20T11:18:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 10 client.4945.objecter successfully canceled tick 2024-01-20T11:18:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb73006c860 msgr2=0x7fb73006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb73006c860 0x7fb73006ed20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb748009710 tx=0x7fb748006c90 comp rx=0 tx=0).stop 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 10 monclient: shutdown 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c13eca0 msgr2=0x7fb74c13fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c13eca0 0x7fb74c13fe00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb744009e50 tx=0x7fb744004a40 comp rx=0 tx=0).stop 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 shutdown_connections 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb73006c860 0x7fb73006ed20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb74c13eca0 0x7fb74c13fe00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c0a9ad0 0x7fb74c13d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 --2- 172.21.15.110:0/2739456209 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c0a96f0 0x7fb74c12fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 >> 172.21.15.110:0/2739456209 conn(0x7fb74c0b7bc0 msgr2=0x7fb74c1373a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 shutdown_connections 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 -- 172.21.15.110:0/2739456209 wait complete. 2024-01-20T11:18:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.097+0000 7fb75375e640 1 librados: shutdown 2024-01-20T11:18:50.108 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-01-20T11:18:48.750361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001979"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280561,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.5430000000000001},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.1859999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.2210000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.6050000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.8010000000000002},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.766999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.381},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":14.949}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.816000000000001},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":12.192}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.941000000000001},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.718}]}]},{"osd":7,"up_from":21,"seq":90194313263,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.4420000000000002},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.6120000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":5.7140000000000004},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.914}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":14.071},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.3049999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.516999999999999},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.409000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.727},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.157999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.146000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.085000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313264,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.5970000000000004},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.6420000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.4450000000000003},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.108000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":15.246},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.1460000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.708},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":12.048},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":11.372999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":16.175000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.302}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444078,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.3160000000000007},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":6.0540000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.8659999999999997},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.507}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.9849999999999994},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.571999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.9269999999999996},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.338000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.949999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.319000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":11.193}]}]},{"osd":1,"up_from":17,"seq":73014444079,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":13.891999999999999},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":12.025}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.581},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":15.843999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":8.7159999999999993},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.2640000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":12.640000000000001},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.84}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.1150000000000002},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":14.499000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.9969999999999999},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.422999999999998}]}]},{"osd":0,"up_from":17,"seq":73014444079,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":4.7480000000000002},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":9.8079999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.5359999999999996},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.2210000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.384},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.699999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.082000000000001},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.4190000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":14.773},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.102}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.739000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":7.8200000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509487,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.2649999999999997},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.9130000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.1840000000000002},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":8.9160000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.522},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.151999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.151999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.182}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.244999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":16.100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.9510000000000005},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.093}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:50.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:50.110 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:50.110 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:50.110 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:50.110 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:50.110 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:50.110 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:50.110 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:50.110 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.210+0000 7ff925b47640 10 monclient: get_monmap_and_config 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.210+0000 7ff925b47640 10 monclient: build_initial_monmap 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.210+0000 7ff925b47640 10 monclient: monmap: 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:50.210786+0000 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:50.210786+0000 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.210 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:50.211 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:50.211 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:50.211 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:50.211 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:50.211 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.210+0000 7ff925b47640 1 Processor -- start 2024-01-20T11:18:50.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.210+0000 7ff925b47640 1 -- start start 2024-01-20T11:18:50.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.210+0000 7ff925b47640 10 monclient: init 2024-01-20T11:18:50.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:50.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-01-20T11:18:50.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 0x7ff92013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient(hunting): picked mon.noname-b con 0x7ff9200a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff92013fa10 0x7ff9201423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient(hunting): picked mon.noname-a con 0x7ff92013fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff9200a9ad0 0x7ff920142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient(hunting): picked mon.noname-c con 0x7ff9200a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9200aaea0 con 0x7ff92013fa10 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9200b25c0 con 0x7ff9200a9ad0 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200ab0a0 con 0x7ff9200a96f0 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.211+0000 7ff925b47640 10 monclient(hunting): authenticate will time out at 1573.561770s 2024-01-20T11:18:50.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 0x7ff92013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff916ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff92013fa10 0x7ff9201423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff9200a9ad0 0x7ff920142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff9200a9ad0 0x7ff920142920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:60790/0 (socket says 172.21.15.110:60790) 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 1 -- 172.21.15.110:0/3932032188 learned_addr learned my addr 172.21.15.110:0/3932032188 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 10 monclient(hunting): get_auth_request con 0x7ff9200a9ad0 auth_method 0 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91f7fe640 10 monclient(hunting): get_auth_request con 0x7ff9200a96f0 auth_method 0 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff916ffd640 10 monclient(hunting): get_auth_request con 0x7ff92013fa10 auth_method 0 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff916ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff916ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff916ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.212+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.213+0000 7ff91f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.213+0000 7ff91f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.213+0000 7ff91f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.213+0000 7ff916ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.213+0000 7ff916ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.213+0000 7ff916ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 10 monclient(hunting): handle_auth_done global_id 4948 payload 274 2024-01-20T11:18:50.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 1 -- 172.21.15.110:0/3932032188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 msgr2=0x7ff92013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:50.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 1 --2- 172.21.15.110:0/3932032188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 0x7ff92013f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 1 -- 172.21.15.110:0/3932032188 >> v2:172.21.15.110:3300/0 conn(0x7ff92013fa10 msgr2=0x7ff9201423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:50.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 1 --2- 172.21.15.110:0/3932032188 >> v2:172.21.15.110:3300/0 conn(0x7ff92013fa10 0x7ff9201423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:50.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 1 monclient: found mon.noname-c 2024-01-20T11:18:50.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:50.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:50.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.214+0000 7ff91ffff640 1 -- 172.21.15.110:0/3932032188 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9200b3fb0 con 0x7ff9200a9ad0 2024-01-20T11:18:50.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.216+0000 7ff91ffff640 1 --2- 172.21.15.110:0/3932032188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 0x7ff920142920 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff9201465c0 tx=0x7ff91802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=276f8503f74e2ae8 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:50.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 1 -- 172.21.15.110:0/3932032188 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91802c9e0 con 0x7ff9200a9ad0 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 10 monclient: dump: 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:50.221 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 10 monclient: _finish_auth 0 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 1 -- 172.21.15.110:0/3932032188 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91802cb40 con 0x7ff9200a9ad0 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 1 -- 172.21.15.110:0/3932032188 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9180343f0 con 0x7ff9200a9ad0 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff925b47640 5 monclient: authenticate success, global_id 4948 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff925b47640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff925b47640 4 set_mon_vals no callback set 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff91effd640 10 monclient: dump: 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.222 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.223 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.223 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:50.223 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.223 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:50.223 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:50.223 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff925b47640 10 monclient: shutdown 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff925b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff925b47640 1 -- 172.21.15.110:0/3932032188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 msgr2=0x7ff920142920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.220+0000 7ff925b47640 1 --2- 172.21.15.110:0/3932032188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 0x7ff920142920 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff9201465c0 tx=0x7ff91802be40 comp rx=0 tx=0).stop 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.221+0000 7ff925b47640 1 -- 172.21.15.110:0/3932032188 shutdown_connections 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.221+0000 7ff925b47640 1 --2- 172.21.15.110:0/3932032188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 0x7ff920142920 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.221+0000 7ff925b47640 1 --2- 172.21.15.110:0/3932032188 >> v2:172.21.15.110:3300/0 conn(0x7ff92013fa10 0x7ff9201423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.221+0000 7ff925b47640 1 --2- 172.21.15.110:0/3932032188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 0x7ff92013f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.221+0000 7ff925b47640 1 -- 172.21.15.110:0/3932032188 >> 172.21.15.110:0/3932032188 conn(0x7ff9200b7bc0 msgr2=0x7ff9200b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.221+0000 7ff925b47640 1 -- 172.21.15.110:0/3932032188 shutdown_connections 2024-01-20T11:18:50.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.221+0000 7ff925b47640 1 -- 172.21.15.110:0/3932032188 wait complete. 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.221+0000 7ff925b47640 10 monclient: build_initial_monmap 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.221+0000 7ff925b47640 10 monclient: monmap: 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:50.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 librados: starting msgr at 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 librados: starting objecter 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 Processor -- start 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 -- start start 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 librados: setting wanted keys 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 librados: calling monclient init 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient: init 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 0x7ff9200abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient(hunting): picked mon.noname-c con 0x7ff9200a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:50.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 0x7ff9201d6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient(hunting): picked mon.noname-b con 0x7ff9200a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9201d7280 0x7ff9200b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient(hunting): picked mon.noname-a con 0x7ff9201d7280 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200ad800 con 0x7ff9201d7280 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200ada30 con 0x7ff9200a9ad0 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200adce0 con 0x7ff9200a96f0 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.222+0000 7ff925b47640 10 monclient(hunting): authenticate will time out at 1573.573095s 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9201d7280 0x7ff9200b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 0x7ff9200abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 0x7ff9201d6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 0x7ff9201d6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:60796/0 (socket says 172.21.15.110:60796) 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff916ffd640 1 -- 172.21.15.110:0/2636503478 learned_addr learned my addr 172.21.15.110:0/2636503478 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff916ffd640 10 monclient(hunting): get_auth_request con 0x7ff9200a9ad0 auth_method 0 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff916ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff916ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff916ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91f7fe640 10 monclient(hunting): get_auth_request con 0x7ff9200a96f0 auth_method 0 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91ffff640 10 monclient(hunting): get_auth_request con 0x7ff9201d7280 auth_method 0 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.223+0000 7ff91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.224+0000 7ff916ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.224+0000 7ff916ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.224+0000 7ff916ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.224+0000 7ff91f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.224+0000 7ff91f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.224+0000 7ff91f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.224+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.224+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.224+0000 7ff91ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 10 monclient(hunting): handle_auth_done global_id 4951 payload 824 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 1 -- 172.21.15.110:0/2636503478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 msgr2=0x7ff9200abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 0x7ff9200abfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 1 -- 172.21.15.110:0/2636503478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9201d7280 msgr2=0x7ff9200b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9201d7280 0x7ff9200b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:18:50.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 1 monclient: found mon.noname-b 2024-01-20T11:18:50.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:50.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:50.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.227+0000 7ff916ffd640 1 -- 172.21.15.110:0/2636503478 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff918009b30 con 0x7ff9200a9ad0 2024-01-20T11:18:50.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.229+0000 7ff916ffd640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 0x7ff9201d6d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff90800e840 tx=0x7ff90800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:50.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.229+0000 7ff91cff9640 10 client.?.objecter ms_handle_connect 0x7ff9200a9ad0 2024-01-20T11:18:50.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.229+0000 7ff91cff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:50.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 1 -- 172.21.15.110:0/2636503478 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90800c730 con 0x7ff9200a9ad0 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 10 monclient: dump: 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 10 monclient: _finish_auth 0 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 10 monclient: _check_auth_tickets 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 1 -- 172.21.15.110:0/2636503478 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90800c890 con 0x7ff9200a9ad0 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff925b47640 5 monclient: authenticate success, global_id 4951 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff925b47640 10 monclient: _renew_subs 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 1 -- 172.21.15.110:0/2636503478 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90800cc40 con 0x7ff9200a9ad0 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff925b47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9200ac840 con 0x7ff9200a9ad0 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91cff9640 10 monclient: dump: 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.236 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff91dffb640 4 set_mon_vals no callback set 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff925b47640 10 client.4951.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff925b47640 10 monclient: _renew_subs 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff925b47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9201d7700 con 0x7ff9200a9ad0 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.232+0000 7ff925b47640 1 librados: init done 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.233+0000 7ff925b47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:50.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.233+0000 7ff925b47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:50.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.233+0000 7ff925b47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:50.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.233+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff920075490 con 0x7ff9200a9ad0 2024-01-20T11:18:50.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.252+0000 7ff91cff9640 1 -- 172.21.15.110:0/2636503478 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff908013470 con 0x7ff9200a9ad0 2024-01-20T11:18:50.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.252+0000 7ff91cff9640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff90406c8e0 0x7ff90406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.253+0000 7ff91f7fe640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff90406c8e0 0x7ff90406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.253+0000 7ff91f7fe640 10 monclient: get_auth_request con 0x7ff90406c8e0 auth_method 0 2024-01-20T11:18:50.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.253+0000 7ff91f7fe640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff90406c8e0 0x7ff90406eda0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff910009ac0 tx=0x7ff910009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:50.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.253+0000 7ff91cff9640 10 client.4951.objecter ms_handle_connect 0x7ff90406c8e0 2024-01-20T11:18:50.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.254+0000 7ff91cff9640 1 -- 172.21.15.110:0/2636503478 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff9080589b0 con 0x7ff9200a9ad0 2024-01-20T11:18:50.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.254+0000 7ff91cff9640 10 client.4951.objecter ms_dispatch 0x7ff920000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:50.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.254+0000 7ff91cff9640 3 client.4951.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:50.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.254+0000 7ff91cff9640 3 client.4951.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:50.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.254+0000 7ff91cff9640 20 client.4951.objecter dump_active .. 0 homeless 2024-01-20T11:18:50.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.486+0000 7ff91cff9640 1 -- 172.21.15.110:0/2636503478 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff908054cf0 con 0x7ff9200a9ad0 2024-01-20T11:18:50.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.486+0000 7ff91cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:50.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.486+0000 7ff91cff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:50.733 INFO:tasks.ceph.osd.5.smithi134.stderr:*** Caught signal (Aborted) ** 2024-01-20T11:18:50.734 INFO:tasks.ceph.osd.5.smithi134.stderr: in thread 24e82640 thread_name:tp_osd_tp 2024-01-20T11:18:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.735+0000 7ff925b47640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.735+0000 7ff925b47640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.735+0000 7ff925b47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.735+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ff9201d7980 con 0x7ff9200a9ad0 2024-01-20T11:18:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.742+0000 7ff91cff9640 1 -- 172.21.15.110:0/2636503478 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7ff908021520 con 0x7ff9200a9ad0 2024-01-20T11:18:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.742+0000 7ff91cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.742+0000 7ff91cff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:50.742 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:50.742 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":265,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 10 librados: watch_flush enter 2024-01-20T11:18:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 10 librados: watch_flush exit 2024-01-20T11:18:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 20 client.4951.objecter shutdown clearing up homeless session... 2024-01-20T11:18:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 10 client.4951.objecter successfully canceled tick 2024-01-20T11:18:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff90406c8e0 msgr2=0x7ff90406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff90406c8e0 0x7ff90406eda0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff910009ac0 tx=0x7ff910009190 comp rx=0 tx=0).stop 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 10 monclient: shutdown 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 msgr2=0x7ff9201d6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 0x7ff9201d6d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff90800e840 tx=0x7ff90800ed00 comp rx=0 tx=0).stop 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 shutdown_connections 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff90406c8e0 0x7ff90406eda0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9201d7280 0x7ff9200b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9200a9ad0 0x7ff9201d6d40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 --2- 172.21.15.110:0/2636503478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9200a96f0 0x7ff9200abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 >> 172.21.15.110:0/2636503478 conn(0x7ff9200b7bc0 msgr2=0x7ff920145a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 shutdown_connections 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 -- 172.21.15.110:0/2636503478 wait complete. 2024-01-20T11:18:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.744+0000 7ff925b47640 1 librados: shutdown 2024-01-20T11:18:50.755 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:50.756 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:50.786 INFO:tasks.ceph.osd.5.smithi134.stderr: ceph version 19.0.0-763-ga00c9940 (a00c9940991c5e1e9ab595faa3f6e1281d8ae167) squid (dev) 2024-01-20T11:18:50.786 INFO:tasks.ceph.osd.5.smithi134.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2024-01-20T11:18:50.786 INFO:tasks.ceph.osd.5.smithi134.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2024-01-20T11:18:50.786 INFO:tasks.ceph.osd.5.smithi134.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x26f) [0xb8b6cf] 2024-01-20T11:18:50.786 INFO:tasks.ceph.osd.5.smithi134.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x72) [0xb8b7c2] 2024-01-20T11:18:50.786 INFO:tasks.ceph.osd.5.smithi134.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x297) [0xb8f347] 2024-01-20T11:18:50.786 INFO:tasks.ceph.osd.5.smithi134.stderr: 6: ceph-osd(+0xa87904) [0xb8f904] 2024-01-20T11:18:50.787 INFO:tasks.ceph.osd.5.smithi134.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-01-20T11:18:50.787 INFO:tasks.ceph.osd.5.smithi134.stderr: 8: clone() 2024-01-20T11:18:50.826 INFO:tasks.ceph.osd.5.smithi134.stderr:2024-01-20T11:18:50.824+0000 24e82640 -1 *** Caught signal (Aborted) ** 2024-01-20T11:18:50.826 INFO:tasks.ceph.osd.5.smithi134.stderr: in thread 24e82640 thread_name:tp_osd_tp 2024-01-20T11:18:50.826 INFO:tasks.ceph.osd.5.smithi134.stderr: 2024-01-20T11:18:50.826 INFO:tasks.ceph.osd.5.smithi134.stderr: ceph version 19.0.0-763-ga00c9940 (a00c9940991c5e1e9ab595faa3f6e1281d8ae167) squid (dev) 2024-01-20T11:18:50.826 INFO:tasks.ceph.osd.5.smithi134.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2024-01-20T11:18:50.826 INFO:tasks.ceph.osd.5.smithi134.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2024-01-20T11:18:50.827 INFO:tasks.ceph.osd.5.smithi134.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x26f) [0xb8b6cf] 2024-01-20T11:18:50.827 INFO:tasks.ceph.osd.5.smithi134.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x72) [0xb8b7c2] 2024-01-20T11:18:50.827 INFO:tasks.ceph.osd.5.smithi134.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x297) [0xb8f347] 2024-01-20T11:18:50.827 INFO:tasks.ceph.osd.5.smithi134.stderr: 6: ceph-osd(+0xa87904) [0xb8f904] 2024-01-20T11:18:50.827 INFO:tasks.ceph.osd.5.smithi134.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-01-20T11:18:50.827 INFO:tasks.ceph.osd.5.smithi134.stderr: 8: clone() 2024-01-20T11:18:50.827 INFO:tasks.ceph.osd.5.smithi134.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-01-20T11:18:50.827 INFO:tasks.ceph.osd.5.smithi134.stderr: 2024-01-20T11:18:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.855+0000 7f67550d4640 10 monclient: get_monmap_and_config 2024-01-20T11:18:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.855+0000 7f67550d4640 10 monclient: build_initial_monmap 2024-01-20T11:18:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.855+0000 7f67550d4640 10 monclient: monmap: 2024-01-20T11:18:50.855 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:50.856159+0000 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:50.856159+0000 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:50.856 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 1 Processor -- start 2024-01-20T11:18:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 1 -- start start 2024-01-20T11:18:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient: init 2024-01-20T11:18:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:50.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67500b0c00 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f67500accf0 0x7f6750143200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient(hunting): picked mon.noname-c con 0x7f67500accf0 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6750143740 0x7f675014a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient(hunting): picked mon.noname-a con 0x7f6750143740 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67500ad450 con 0x7f67500b0c00 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67500ad8e0 con 0x7f6750143740 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67500ad130 con 0x7f67500accf0 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.856+0000 7f67550d4640 10 monclient(hunting): authenticate will time out at 1574.207119s 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.857+0000 7f674f59e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6750143740 0x7f675014a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.857+0000 7f6747fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f67500accf0 0x7f6750143200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.857+0000 7f674ed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f67500b0c00 0x7f675013d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35834/0 (socket says 172.21.15.110:35834) 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.857+0000 7f674ed9d640 1 -- 172.21.15.110:0/25117658 learned_addr learned my addr 172.21.15.110:0/25117658 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f6747fff640 10 monclient(hunting): get_auth_request con 0x7f67500accf0 auth_method 0 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f6747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f6747fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f6747fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f674f59e640 10 monclient(hunting): get_auth_request con 0x7f6750143740 auth_method 0 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f674f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f674f59e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f674f59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f6747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f6747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.858+0000 7f6747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.859+0000 7f674f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.859+0000 7f674f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.859+0000 7f674f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.860+0000 7f6747fff640 10 monclient(hunting): handle_auth_done global_id 4954 payload 274 2024-01-20T11:18:50.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.860+0000 7f6747fff640 1 -- 172.21.15.110:0/25117658 >> v2:172.21.15.110:3300/0 conn(0x7f6750143740 msgr2=0x7f675014a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:50.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.860+0000 7f6747fff640 1 --2- 172.21.15.110:0/25117658 >> v2:172.21.15.110:3300/0 conn(0x7f6750143740 0x7f675014a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.860+0000 7f6747fff640 1 -- 172.21.15.110:0/25117658 >> v1:172.21.15.110:6789/0 conn(0x7f67500b0c00 legacy=0x7f675013d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.860+0000 7f6747fff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:50.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.860+0000 7f6747fff640 1 monclient: found mon.noname-c 2024-01-20T11:18:50.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.860+0000 7f6747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:50.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.860+0000 7f6747fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:50.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.860+0000 7f6747fff640 1 -- 172.21.15.110:0/25117658 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67500aa300 con 0x7f67500accf0 2024-01-20T11:18:50.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.862+0000 7f6747fff640 1 --2- 172.21.15.110:0/25117658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67500accf0 0x7f6750143200 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f67500b12a0 tx=0x7f673802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11c10780e8a38227 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:50.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 1 -- 172.21.15.110:0/25117658 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738031710 con 0x7f67500accf0 2024-01-20T11:18:50.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:50.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:50.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 10 monclient: dump: 2024-01-20T11:18:50.865 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:50.865 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 10 monclient: _finish_auth 0 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 10 monclient: _check_auth_tickets 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 1 -- 172.21.15.110:0/25117658 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673802f050 con 0x7f67500accf0 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 1 -- 172.21.15.110:0/25117658 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738031a90 con 0x7f67500accf0 2024-01-20T11:18:50.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f67550d4640 5 monclient: authenticate success, global_id 4954 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f67550d4640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f67550d4640 4 set_mon_vals no callback set 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.865+0000 7f674e59c640 10 monclient: dump: 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:50.867 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:50.868 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 10 monclient: shutdown 2024-01-20T11:18:50.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:50.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 1 -- 172.21.15.110:0/25117658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67500accf0 msgr2=0x7f6750143200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 1 --2- 172.21.15.110:0/25117658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67500accf0 0x7f6750143200 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f67500b12a0 tx=0x7f673802b7b0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 1 -- 172.21.15.110:0/25117658 shutdown_connections 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 1 --2- 172.21.15.110:0/25117658 >> v2:172.21.15.110:3300/0 conn(0x7f6750143740 0x7f675014a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 1 --2- 172.21.15.110:0/25117658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67500accf0 0x7f6750143200 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 1 -- 172.21.15.110:0/25117658 >> 172.21.15.110:0/25117658 conn(0x7f67500b3e60 msgr2=0x7f67500b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 1 -- 172.21.15.110:0/25117658 shutdown_connections 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.866+0000 7f67550d4640 1 -- 172.21.15.110:0/25117658 wait complete. 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.867+0000 7f67550d4640 10 monclient: build_initial_monmap 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.867+0000 7f67550d4640 10 monclient: monmap: 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.872 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.867+0000 7f67550d4640 1 librados: starting msgr at 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.867+0000 7f67550d4640 1 librados: starting objecter 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.867+0000 7f67550d4640 1 Processor -- start 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.867+0000 7f67550d4640 1 -- start start 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.867+0000 7f67550d4640 1 librados: setting wanted keys 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.867+0000 7f67550d4640 1 librados: calling monclient init 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.867+0000 7f67550d4640 10 monclient: init 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67500accf0 0x7f67501cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient(hunting): picked mon.noname-b con 0x7f67500accf0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:50.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67500b0c00 0x7f67501db2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient(hunting): picked mon.noname-c con 0x7f67500b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6750143740 0x7f6750137e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient(hunting): picked mon.noname-a con 0x7f6750143740 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67501cbd40 con 0x7f6750143740 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67501dadd0 con 0x7f67500accf0 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67501cccf0 con 0x7f67500b0c00 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f67550d4640 10 monclient(hunting): authenticate will time out at 1574.218359s 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f674f59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6750143740 0x7f6750137e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.868+0000 7f6747fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67500b0c00 0x7f67501db2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674f59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6750143740 0x7f6750137e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34382/0 (socket says 172.21.15.110:34382) 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674f59e640 1 -- 172.21.15.110:0/3338615551 learned_addr learned my addr 172.21.15.110:0/3338615551 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674f59e640 10 monclient(hunting): get_auth_request con 0x7f6750143740 auth_method 0 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674f59e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674f59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f6747fff640 10 monclient(hunting): get_auth_request con 0x7f67500b0c00 auth_method 0 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f6747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f6747fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f6747fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f6747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f6747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f6747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.869+0000 7f674ed9d640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67500accf0 0x7f67501cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.870+0000 7f674ed9d640 10 monclient(hunting): get_auth_request con 0x7f67500accf0 auth_method 0 2024-01-20T11:18:50.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.870+0000 7f674ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:50.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.870+0000 7f674ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:50.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.870+0000 7f674ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:50.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.870+0000 7f674ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:50.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.870+0000 7f674ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:50.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.870+0000 7f674ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:50.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 10 monclient(hunting): handle_auth_done global_id 4733 payload 824 2024-01-20T11:18:50.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 1 -- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67500accf0 msgr2=0x7f67501cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67500accf0 0x7f67501cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 1 -- 172.21.15.110:0/3338615551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6750143740 msgr2=0x7f6750137e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:50.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6750143740 0x7f6750137e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:50.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:50.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 1 monclient: found mon.noname-c 2024-01-20T11:18:50.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:50.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:18:50.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.872+0000 7f6747fff640 1 -- 172.21.15.110:0/3338615551 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6738009400 con 0x7f67500b0c00 2024-01-20T11:18:50.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.874+0000 7f6747fff640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67500b0c00 0x7f67501db2f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6738033000 tx=0x7f67380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:50.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.874+0000 7f67477fe640 10 client.?.objecter ms_handle_connect 0x7f67500b0c00 2024-01-20T11:18:50.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.875+0000 7f67477fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:50.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 1 -- 172.21.15.110:0/3338615551 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738033950 con 0x7f67500b0c00 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 10 monclient: dump: 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:50.880 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 10 monclient: _finish_auth 0 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 10 monclient: _check_auth_tickets 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 1 -- 172.21.15.110:0/3338615551 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738033af0 con 0x7f67500b0c00 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 1 -- 172.21.15.110:0/3338615551 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738033ea0 con 0x7f67500b0c00 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67550d4640 5 monclient: authenticate success, global_id 4733 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67550d4640 10 monclient: _renew_subs 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67550d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67501c99c0 con 0x7f67500b0c00 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f674dd9b640 4 set_mon_vals no callback set 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:50.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67477fe640 10 monclient: dump: 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67550d4640 10 client.4733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67550d4640 10 monclient: _renew_subs 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67550d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:18:50.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6750139200 con 0x7f67500b0c00 2024-01-20T11:18:50.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.879+0000 7f67550d4640 1 librados: init done 2024-01-20T11:18:50.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.880+0000 7f67550d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:50.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.880+0000 7f67550d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:50.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.880+0000 7f67550d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:18:50.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.880+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67501dac30 con 0x7f67500b0c00 2024-01-20T11:18:50.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.899+0000 7f67477fe640 1 -- 172.21.15.110:0/3338615551 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f673803a040 con 0x7f67500b0c00 2024-01-20T11:18:50.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.899+0000 7f67477fe640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f673006c8b0 0x7f673006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:50.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.899+0000 7f674ed9d640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f673006c8b0 0x7f673006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:50.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.899+0000 7f674ed9d640 10 monclient: get_auth_request con 0x7f673006c8b0 auth_method 0 2024-01-20T11:18:50.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.900+0000 7f67477fe640 10 client.4733.objecter ms_handle_connect 0x7f673006c8b0 2024-01-20T11:18:50.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:50.900+0000 7f674ed9d640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f673006c8b0 0x7f673006ed70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f6740007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:50.985 INFO:tasks.ceph.osd.5.smithi134.stderr: -5653> 2024-01-20T11:15:08.427+0000 5dc2480 -1 osd.5 0 log_to_monitors true 2024-01-20T11:18:50.993 INFO:tasks.ceph.osd.5.smithi134.stderr: -5357> 2024-01-20T11:15:11.662+0000 ac7e640 -1 osd.5 0 waiting for initial osdmap 2024-01-20T11:18:50.997 INFO:tasks.ceph.osd.5.smithi134.stderr: -5217> 2024-01-20T11:15:12.167+0000 10253640 -1 osd.5 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 0> 2024-01-20T11:18:50.824+0000 24e82640 -1 *** Caught signal (Aborted) ** 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: in thread 24e82640 thread_name:tp_osd_tp 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: ceph version 19.0.0-763-ga00c9940 (a00c9940991c5e1e9ab595faa3f6e1281d8ae167) squid (dev) 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x26f) [0xb8b6cf] 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x72) [0xb8b7c2] 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x297) [0xb8f347] 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 6: ceph-osd(+0xa87904) [0xb8f904] 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-01-20T11:18:51.140 INFO:tasks.ceph.osd.5.smithi134.stderr: 8: clone() 2024-01-20T11:18:51.141 INFO:tasks.ceph.osd.5.smithi134.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-01-20T11:18:51.141 INFO:tasks.ceph.osd.5.smithi134.stderr: 2024-01-20T11:18:51.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.141+0000 7f67477fe640 1 -- 172.21.15.110:0/3338615551 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f673807a580 con 0x7f67500b0c00 2024-01-20T11:18:51.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.141+0000 7f67477fe640 10 client.4733.objecter ms_dispatch 0x7f6750000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:51.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.141+0000 7f67477fe640 3 client.4733.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:51.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.141+0000 7f67477fe640 3 client.4733.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:51.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.141+0000 7f67477fe640 20 client.4733.objecter dump_active .. 0 homeless 2024-01-20T11:18:51.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.141+0000 7f67477fe640 1 -- 172.21.15.110:0/3338615551 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f67380768c0 con 0x7f67500b0c00 2024-01-20T11:18:51.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.141+0000 7f67477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:51.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.141+0000 7f67477fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:51.154 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:04:19.627 34419== 2024-01-20T11:18:51.154 INFO:tasks.ceph.osd.5.smithi134.stderr:==00:00:04:19.627 34419== Exit program on first error (--exit-on-first-error=yes) 2024-01-20T11:18:51.241 INFO:tasks.ceph.osd.5.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-01-20T11:18:51.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.311+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67501c9cc0 con 0x7f673006c8b0 2024-01-20T11:18:51.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.314+0000 7f67477fe640 1 -- 172.21.15.110:0/3338615551 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64043 (secure 0 0 0) 0x7f67501c9cc0 con 0x7f673006c8b0 2024-01-20T11:18:51.314 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:51.315 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 10 librados: watch_flush enter 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 10 librados: watch_flush exit 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 20 client.4733.objecter shutdown clearing up homeless session... 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 10 client.4733.objecter successfully canceled tick 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f673006c8b0 msgr2=0x7f673006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f673006c8b0 0x7f673006ed70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f6740007e80 comp rx=0 tx=0).stop 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 10 monclient: shutdown 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67500b0c00 msgr2=0x7f67501db2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:51.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67500b0c00 0x7f67501db2f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6738033000 tx=0x7f67380046c0 comp rx=0 tx=0).stop 2024-01-20T11:18:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 shutdown_connections 2024-01-20T11:18:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f673006c8b0 0x7f673006ed70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6750143740 0x7f6750137e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67500b0c00 0x7f67501db2f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 --2- 172.21.15.110:0/3338615551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67500accf0 0x7f67501cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 >> 172.21.15.110:0/3338615551 conn(0x7f67500b3e60 msgr2=0x7f6750141000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 shutdown_connections 2024-01-20T11:18:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 -- 172.21.15.110:0/3338615551 wait complete. 2024-01-20T11:18:51.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:51.316+0000 7f67550d4640 1 librados: shutdown 2024-01-20T11:18:51.327 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-01-20T11:18:50.750690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002013"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280561,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.5430000000000001},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.1859999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.2210000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.6050000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.8010000000000002},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.766999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.381},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":14.949}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.816000000000001},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":12.192}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.941000000000001},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.718}]}]},{"osd":7,"up_from":21,"seq":90194313263,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.4420000000000002},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.6120000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":5.7140000000000004},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.914}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":14.071},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.3049999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.516999999999999},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.409000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.727},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.157999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.146000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.085000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313264,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.5970000000000004},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.6420000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.4450000000000003},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.108000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":15.246},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.1460000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.708},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":12.048},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":11.372999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":16.175000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.302}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444078,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.3160000000000007},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":6.0540000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.8659999999999997},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.507}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.9849999999999994},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.571999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.9269999999999996},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.338000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.949999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.319000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":11.193}]}]},{"osd":1,"up_from":17,"seq":73014444079,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":13.891999999999999},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":12.025}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.581},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":15.843999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":8.7159999999999993},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.2640000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":12.640000000000001},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.84}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.1150000000000002},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":14.499000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.9969999999999999},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.422999999999998}]}]},{"osd":0,"up_from":17,"seq":73014444079,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":4.7480000000000002},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":9.8079999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.5359999999999996},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.2210000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.384},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.699999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.082000000000001},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.4190000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":14.773},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.102}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.739000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":7.8200000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509487,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.2649999999999997},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.9130000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.1840000000000002},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":8.9160000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.522},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.151999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.151999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.182}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.244999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":16.100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.9510000000000005},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.093}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:54.329 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:54.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.428+0000 7fba0cb73640 10 monclient: get_monmap_and_config 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.428+0000 7fba0cb73640 10 monclient: build_initial_monmap 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.428+0000 7fba0cb73640 10 monclient: monmap: 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:54.429499+0000 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:54.429499+0000 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:54.429 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:54.430 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:54.430 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.429+0000 7fba0cb73640 1 Processor -- start 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.429+0000 7fba0cb73640 1 -- start start 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.429+0000 7fba0cb73640 10 monclient: init 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba0806a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba0806ac40 0x7fba081455c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 10 monclient(hunting): picked mon.noname-a con 0x7fba0806ac40 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fba08072320 0x7fba08145b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:54.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 10 monclient(hunting): picked mon.noname-c con 0x7fba08072320 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba080af8b0 con 0x7fba0806a860 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba080b01e0 con 0x7fba0806ac40 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba080b0320 con 0x7fba08072320 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba0cb73640 10 monclient(hunting): authenticate will time out at 1577.780452s 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fb9fffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba0806ac40 0x7fba081455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.430+0000 7fba06d9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fba08072320 0x7fba08145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fb9fffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba0806ac40 0x7fba081455c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34388/0 (socket says 172.21.15.110:34388) 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fb9fffff640 1 -- 172.21.15.110:0/2120484541 learned_addr learned my addr 172.21.15.110:0/2120484541 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fb9fffff640 10 monclient(hunting): get_auth_request con 0x7fba0806ac40 auth_method 0 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fb9fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:54.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fb9fffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fb9fffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fba06d9d640 10 monclient(hunting): get_auth_request con 0x7fba08072320 auth_method 0 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fba06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fba06d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fba06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fb9fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fb9fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.431+0000 7fb9fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.432+0000 7fba06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.432+0000 7fba06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:54.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.432+0000 7fba06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:54.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.433+0000 7fb9fffff640 10 monclient(hunting): handle_auth_done global_id 4878 payload 274 2024-01-20T11:18:54.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.433+0000 7fb9fffff640 1 -- 172.21.15.110:0/2120484541 >> v2:172.21.15.134:3300/0 conn(0x7fba08072320 msgr2=0x7fba08145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:54.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.433+0000 7fb9fffff640 1 --2- 172.21.15.110:0/2120484541 >> v2:172.21.15.134:3300/0 conn(0x7fba08072320 0x7fba08145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:54.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.433+0000 7fb9fffff640 1 -- 172.21.15.110:0/2120484541 >> v1:172.21.15.134:6789/0 conn(0x7fba0806a860 legacy=0x7fba0813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:54.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.433+0000 7fb9fffff640 10 monclient: _finish_hunting 0 2024-01-20T11:18:54.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.433+0000 7fb9fffff640 1 monclient: found mon.noname-a 2024-01-20T11:18:54.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.433+0000 7fb9fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:54.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.433+0000 7fb9fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:54.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.433+0000 7fb9fffff640 1 -- 172.21.15.110:0/2120484541 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0813fb60 con 0x7fba0806ac40 2024-01-20T11:18:54.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.435+0000 7fb9fffff640 1 --2- 172.21.15.110:0/2120484541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0806ac40 0x7fba081455c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fba08071d80 tx=0x7fb9f002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df4ff74278cb19 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:54.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 1 -- 172.21.15.110:0/2120484541 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f002f850 con 0x7fba0806ac40 2024-01-20T11:18:54.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:54.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:54.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 10 monclient: dump: 2024-01-20T11:18:54.438 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 10 monclient: _finish_auth 0 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 1 -- 172.21.15.110:0/2120484541 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f002f9b0 con 0x7fba0806ac40 2024-01-20T11:18:54.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 1 -- 172.21.15.110:0/2120484541 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f002fd40 con 0x7fba0806ac40 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba05d9b640 10 monclient: dump: 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:54.440 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:54.441 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:54.441 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:54.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba0cb73640 5 monclient: authenticate success, global_id 4878 2024-01-20T11:18:54.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba0cb73640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:54.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.438+0000 7fba0cb73640 4 set_mon_vals no callback set 2024-01-20T11:18:54.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 10 monclient: shutdown 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 1 -- 172.21.15.110:0/2120484541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0806ac40 msgr2=0x7fba081455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 1 --2- 172.21.15.110:0/2120484541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0806ac40 0x7fba081455c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fba08071d80 tx=0x7fb9f002ba30 comp rx=0 tx=0).stop 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 1 -- 172.21.15.110:0/2120484541 shutdown_connections 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 1 --2- 172.21.15.110:0/2120484541 >> v2:172.21.15.134:3300/0 conn(0x7fba08072320 0x7fba08145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 1 --2- 172.21.15.110:0/2120484541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0806ac40 0x7fba081455c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 1 -- 172.21.15.110:0/2120484541 >> 172.21.15.110:0/2120484541 conn(0x7fba080b47d0 msgr2=0x7fba080b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 1 -- 172.21.15.110:0/2120484541 shutdown_connections 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.439+0000 7fba0cb73640 1 -- 172.21.15.110:0/2120484541 wait complete. 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.440+0000 7fba0cb73640 10 monclient: build_initial_monmap 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.440+0000 7fba0cb73640 10 monclient: monmap: 2024-01-20T11:18:54.445 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:54.446 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:54.446 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:54.446 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:54.446 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:54.446 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:54.446 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:54.446 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:54.446 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:54.446 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.440+0000 7fba0cb73640 1 librados: starting msgr at 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.440+0000 7fba0cb73640 1 librados: starting objecter 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.440+0000 7fba0cb73640 1 Processor -- start 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.440+0000 7fba0cb73640 1 -- start start 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.440+0000 7fba0cb73640 1 librados: setting wanted keys 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.440+0000 7fba0cb73640 1 librados: calling monclient init 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.440+0000 7fba0cb73640 10 monclient: init 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba0806a860 0x7fba081400b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient(hunting): picked mon.noname-c con 0x7fba0806a860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:54.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0806ac40 0x7fba081d7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:54.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient(hunting): picked mon.noname-b con 0x7fba0806ac40 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:54.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba08072320 0x7fba081d8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:54.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient(hunting): picked mon.noname-a con 0x7fba08072320 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:54.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:54.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba08140db0 con 0x7fba08072320 2024-01-20T11:18:54.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:54.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba08141c30 con 0x7fba0806ac40 2024-01-20T11:18:54.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:54.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba08141ea0 con 0x7fba0806a860 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0cb73640 10 monclient(hunting): authenticate will time out at 1577.791569s 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba06d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba08072320 0x7fba081d8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.441+0000 7fba0659c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba0806a860 0x7fba081400b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba06d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba08072320 0x7fba081d8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34402/0 (socket says 172.21.15.110:34402) 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba06d9d640 1 -- 172.21.15.110:0/3318025829 learned_addr learned my addr 172.21.15.110:0/3318025829 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba06d9d640 10 monclient(hunting): get_auth_request con 0x7fba08072320 auth_method 0 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba06d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fb9fffff640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0806ac40 0x7fba081d7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:54.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba0659c640 10 monclient(hunting): get_auth_request con 0x7fba0806a860 auth_method 0 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba0659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba0659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fb9fffff640 10 monclient(hunting): get_auth_request con 0x7fba0806ac40 auth_method 0 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fb9fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fb9fffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fb9fffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.442+0000 7fba0659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:54.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.443+0000 7fb9fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:54.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.443+0000 7fb9fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:54.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.443+0000 7fb9fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:54.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 10 monclient(hunting): handle_auth_done global_id 4881 payload 824 2024-01-20T11:18:54.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 1 -- 172.21.15.110:0/3318025829 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba0806a860 msgr2=0x7fba081400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:54.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba0806a860 0x7fba081400b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:54.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 1 -- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0806ac40 msgr2=0x7fba081d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:54.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0806ac40 0x7fba081d7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:54.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:18:54.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 1 monclient: found mon.noname-a 2024-01-20T11:18:54.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:54.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.445+0000 7fba06d9d640 1 -- 172.21.15.110:0/3318025829 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f00094d0 con 0x7fba08072320 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.448+0000 7fba06d9d640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba08072320 0x7fba081d8120 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fba0000efc0 tx=0x7fba0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.448+0000 7fb9feffd640 10 client.?.objecter ms_handle_connect 0x7fba08072320 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.448+0000 7fb9feffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 1 -- 172.21.15.110:0/3318025829 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00009b20 con 0x7fba08072320 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 10 monclient: dump: 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:54.455 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:54.456 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:54.456 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:54.456 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:54.456 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:54.456 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:54.456 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:54.456 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:54.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:54.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:54.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:54.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0cb73640 5 monclient: authenticate success, global_id 4881 2024-01-20T11:18:54.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 1 -- 172.21.15.110:0/3318025829 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0000f040 con 0x7fba08072320 2024-01-20T11:18:54.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0cb73640 10 monclient: _renew_subs 2024-01-20T11:18:54.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0cb73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:54.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba08142180 con 0x7fba08072320 2024-01-20T11:18:54.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:54.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 1 -- 172.21.15.110:0/3318025829 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000146c0 con 0x7fba08072320 2024-01-20T11:18:54.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:54.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:54.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fb9feffd640 10 monclient: dump: 2024-01-20T11:18:54.458 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:54.458 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:54.459 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:54.459 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:54.459 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:54.459 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:54.459 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:54.459 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:54.460 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:54.460 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:54.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0559a640 4 set_mon_vals no callback set 2024-01-20T11:18:54.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0cb73640 10 client.4881.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:54.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0cb73640 10 monclient: _renew_subs 2024-01-20T11:18:54.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0cb73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:54.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba08140940 con 0x7fba08072320 2024-01-20T11:18:54.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.452+0000 7fba0cb73640 1 librados: init done 2024-01-20T11:18:54.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.453+0000 7fba0cb73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:54.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.453+0000 7fba0cb73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:54.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.453+0000 7fba0cb73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:18:54.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.453+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba08075490 con 0x7fba08072320 2024-01-20T11:18:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.690+0000 7fb9feffd640 1 -- 172.21.15.110:0/3318025829 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fba00014820 con 0x7fba08072320 2024-01-20T11:18:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.690+0000 7fb9feffd640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb9e806c8b0 0x7fb9e806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.691+0000 7fba0659c640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb9e806c8b0 0x7fb9e806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.691+0000 7fba0659c640 10 monclient: get_auth_request con 0x7fb9e806c8b0 auth_method 0 2024-01-20T11:18:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.691+0000 7fba0659c640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb9e806c8b0 0x7fb9e806ed70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8001780 tx=0x7fb9f8009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.691+0000 7fb9feffd640 10 client.4881.objecter ms_handle_connect 0x7fb9e806c8b0 2024-01-20T11:18:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.703+0000 7fb9feffd640 1 -- 172.21.15.110:0/3318025829 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fba00059500 con 0x7fba08072320 2024-01-20T11:18:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.703+0000 7fb9feffd640 10 client.4881.objecter ms_dispatch 0x7fba08000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.703+0000 7fb9feffd640 3 client.4881.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.703+0000 7fb9feffd640 3 client.4881.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.704+0000 7fb9feffd640 20 client.4881.objecter dump_active .. 0 homeless 2024-01-20T11:18:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.704+0000 7fb9feffd640 1 -- 172.21.15.110:0/3318025829 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fba00055840 con 0x7fba08072320 2024-01-20T11:18:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.704+0000 7fb9feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.704+0000 7fb9feffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.888+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba080af1d0 con 0x7fb9e806c8b0 2024-01-20T11:18:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.891+0000 7fb9feffd640 1 -- 172.21.15.110:0/3318025829 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63955 (secure 0 0 0) 0x7fba080af1d0 con 0x7fb9e806c8b0 2024-01-20T11:18:54.892 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:54.892 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 10 librados: watch_flush enter 2024-01-20T11:18:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 10 librados: watch_flush exit 2024-01-20T11:18:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 20 client.4881.objecter shutdown clearing up homeless session... 2024-01-20T11:18:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 10 client.4881.objecter successfully canceled tick 2024-01-20T11:18:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb9e806c8b0 msgr2=0x7fb9e806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb9e806c8b0 0x7fb9e806ed70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8001780 tx=0x7fb9f8009040 comp rx=0 tx=0).stop 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 10 monclient: shutdown 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba08072320 msgr2=0x7fba081d8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba08072320 0x7fba081d8120 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fba0000efc0 tx=0x7fba0000c490 comp rx=0 tx=0).stop 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 shutdown_connections 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb9e806c8b0 0x7fb9e806ed70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba08072320 0x7fba081d8120 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0806ac40 0x7fba081d7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 --2- 172.21.15.110:0/3318025829 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba0806a860 0x7fba081400b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 >> 172.21.15.110:0/3318025829 conn(0x7fba080b47d0 msgr2=0x7fba0813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 shutdown_connections 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.893+0000 7fba0cb73640 1 -- 172.21.15.110:0/3318025829 wait complete. 2024-01-20T11:18:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:54.894+0000 7fba0cb73640 1 librados: shutdown 2024-01-20T11:18:54.905 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-01-20T11:18:54.751525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002134"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280562,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.0050000000000008},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.7480000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.2619999999999996},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.6959999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.763999999999999},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.311999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.4239999999999995},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":13.202999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.135999999999999},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":6.5250000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":12.506},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":14.343999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313264,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.6909999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.103}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.069},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.829000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.212},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.8040000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.298},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.454000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.395},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.531000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.710000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.3889999999999993}]}]},{"osd":3,"up_from":21,"seq":90194313265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.0819999999999999},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":12.364000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.9489999999999998},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.2080000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":16.574999999999999},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":7.9130000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.1210000000000004},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.102}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":15.768000000000001},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":11.153}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":10.573},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.879}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444079,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.5239999999999991},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":14.657999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.1130000000000004},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.7750000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.42},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.9630000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.869999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.786}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.521000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.128}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":7.6580000000000004},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":10.791}]}]},{"osd":1,"up_from":17,"seq":73014444080,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.763999999999999},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":8.7669999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.9860000000000007},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.198}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":13.845000000000001},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":5.8899999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":10.58},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.946}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.4870000000000001},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":12.331}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.1959999999999997},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.054}]}]},{"osd":0,"up_from":17,"seq":73014444080,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":10.122},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":11.375}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.6200000000000001},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.5099999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":11.109},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.035}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.609999999999999},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.766}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":11.956},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":13.882999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.433},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":7.6600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509488,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.2649999999999997},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.9130000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.1840000000000002},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":8.9160000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.522},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.151999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.151999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.182}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.244999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":16.100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.9510000000000005},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.093}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:54.906 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.008+0000 7f1faf428640 10 monclient: get_monmap_and_config 2024-01-20T11:18:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.008+0000 7f1faf428640 10 monclient: build_initial_monmap 2024-01-20T11:18:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.008+0000 7f1faf428640 10 monclient: monmap: 2024-01-20T11:18:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:55.009184+0000 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:55.009184+0000 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:55.009 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 1 Processor -- start 2024-01-20T11:18:55.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 1 -- start start 2024-01-20T11:18:55.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient: init 2024-01-20T11:18:55.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:55.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-01-20T11:18:55.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806bb00 0x7f1fa80b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient(hunting): picked mon.noname-b con 0x7f1fa806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1fa80af790 0x7f1fa8145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient(hunting): picked mon.noname-c con 0x7f1fa80af790 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1fa80b0900 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa806c220 con 0x7f1fa80b0900 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa806bee0 con 0x7f1fa80af790 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa80ada30 con 0x7f1fa806bb00 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.009+0000 7f1faf428640 10 monclient(hunting): authenticate will time out at 1578.360124s 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fad1c4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806bb00 0x7f1fa80b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fad9c5640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1fa80b0900 0x7f1fa8145cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35844/0 (socket says 172.21.15.110:35844) 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fad9c5640 1 -- 172.21.15.110:0/2580098158 learned_addr learned my addr 172.21.15.110:0/2580098158 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fac9c3640 1 --2- 172.21.15.110:0/2580098158 >> v2:172.21.15.134:3300/0 conn(0x7f1fa80af790 0x7f1fa8145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fad1c4640 10 monclient(hunting): get_auth_request con 0x7f1fa806bb00 auth_method 0 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fad1c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fad1c4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fad1c4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fac9c3640 10 monclient(hunting): get_auth_request con 0x7f1fa80af790 auth_method 0 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fac9c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fac9c3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.010+0000 7f1fac9c3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.011+0000 7f1fac9c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.011+0000 7f1fac9c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.011+0000 7f1fac9c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.011+0000 7f1fad1c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:55.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.011+0000 7f1fad1c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:55.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.011+0000 7f1fad1c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:55.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.013+0000 7f1fac9c3640 10 monclient(hunting): handle_auth_done global_id 4966 payload 274 2024-01-20T11:18:55.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.013+0000 7f1fac9c3640 1 -- 172.21.15.110:0/2580098158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806bb00 msgr2=0x7f1fa80b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:18:55.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.013+0000 7f1fac9c3640 1 --2- 172.21.15.110:0/2580098158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806bb00 0x7f1fa80b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.013+0000 7f1fac9c3640 1 -- 172.21.15.110:0/2580098158 >> v1:172.21.15.110:6789/0 conn(0x7f1fa80b0900 legacy=0x7f1fa8145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.013+0000 7f1fac9c3640 10 monclient: _finish_hunting 0 2024-01-20T11:18:55.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.013+0000 7f1fac9c3640 1 monclient: found mon.noname-c 2024-01-20T11:18:55.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.013+0000 7f1fac9c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:55.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.013+0000 7f1fac9c3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:55.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.013+0000 7f1fac9c3640 1 -- 172.21.15.110:0/2580098158 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa80add10 con 0x7f1fa80af790 2024-01-20T11:18:55.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.015+0000 7f1fac9c3640 1 --2- 172.21.15.110:0/2580098158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80af790 0x7f1fa8145770 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1f98009790 tx=0x7f1f9802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9255ec2844ead02 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:55.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 1 -- 172.21.15.110:0/2580098158 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f98031770 con 0x7f1fa80af790 2024-01-20T11:18:55.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 10 monclient: dump: 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 10 monclient: _finish_auth 0 2024-01-20T11:18:55.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 1 -- 172.21.15.110:0/2580098158 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f980318d0 con 0x7f1fa80af790 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 1 -- 172.21.15.110:0/2580098158 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f98031c60 con 0x7f1fa80af790 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1f9ffff640 10 monclient: dump: 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:55.020 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:55.021 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:55.021 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1faf428640 5 monclient: authenticate success, global_id 4966 2024-01-20T11:18:55.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1faf428640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:55.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.018+0000 7f1faf428640 4 set_mon_vals no callback set 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 10 monclient: shutdown 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 1 -- 172.21.15.110:0/2580098158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80af790 msgr2=0x7f1fa8145770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 1 --2- 172.21.15.110:0/2580098158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80af790 0x7f1fa8145770 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1f98009790 tx=0x7f1f9802b810 comp rx=0 tx=0).stop 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 1 -- 172.21.15.110:0/2580098158 shutdown_connections 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 1 --2- 172.21.15.110:0/2580098158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80af790 0x7f1fa8145770 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 1 --2- 172.21.15.110:0/2580098158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806bb00 0x7f1fa80b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 1 -- 172.21.15.110:0/2580098158 >> 172.21.15.110:0/2580098158 conn(0x7f1fa8066d60 msgr2=0x7f1fa80b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 1 -- 172.21.15.110:0/2580098158 shutdown_connections 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.019+0000 7f1faf428640 1 -- 172.21.15.110:0/2580098158 wait complete. 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.020+0000 7f1faf428640 10 monclient: build_initial_monmap 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.020+0000 7f1faf428640 10 monclient: monmap: 2024-01-20T11:18:55.029 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.020+0000 7f1faf428640 1 librados: starting msgr at 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.020+0000 7f1faf428640 1 librados: starting objecter 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.020+0000 7f1faf428640 1 Processor -- start 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.020+0000 7f1faf428640 1 -- start start 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.020+0000 7f1faf428640 1 librados: setting wanted keys 2024-01-20T11:18:55.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.020+0000 7f1faf428640 1 librados: calling monclient init 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.020+0000 7f1faf428640 10 monclient: init 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa806bb00 0x7f1fa8141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient(hunting): picked mon.noname-a con 0x7f1fa806bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa80af790 0x7f1fa81da6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient(hunting): picked mon.noname-c con 0x7f1fa80af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80b0900 0x7f1fa81dac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient(hunting): picked mon.noname-b con 0x7f1fa80b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa81db160 con 0x7f1fa806bb00 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa811dbe0 con 0x7f1fa80b0900 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:55.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa811efc0 con 0x7f1fa80af790 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1faf428640 10 monclient(hunting): authenticate will time out at 1578.371488s 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1fad1c4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa806bb00 0x7f1fa8141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.021+0000 7f1fac9c3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa80af790 0x7f1fa81da6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad9c5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80b0900 0x7f1fa81dac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad1c4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa806bb00 0x7f1fa8141740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34416/0 (socket says 172.21.15.110:34416) 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad1c4640 1 -- 172.21.15.110:0/3513974565 learned_addr learned my addr 172.21.15.110:0/3513974565 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad1c4640 10 monclient(hunting): get_auth_request con 0x7f1fa806bb00 auth_method 0 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad1c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad1c4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad1c4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fac9c3640 10 monclient(hunting): get_auth_request con 0x7f1fa80af790 auth_method 0 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fac9c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fac9c3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fac9c3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad9c5640 10 monclient(hunting): get_auth_request con 0x7f1fa80b0900 auth_method 0 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad9c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad9c5640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad9c5640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fac9c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fac9c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fac9c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad1c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad1c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad1c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad9c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad9c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.022+0000 7f1fad9c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 10 monclient(hunting): handle_auth_done global_id 4969 payload 824 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 1 -- 172.21.15.110:0/3513974565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa80af790 msgr2=0x7f1fa81da6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa80af790 0x7f1fa81da6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 1 -- 172.21.15.110:0/3513974565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa806bb00 msgr2=0x7f1fa8141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa806bb00 0x7f1fa8141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 10 monclient: _finish_hunting 0 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 1 monclient: found mon.noname-b 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.025+0000 7f1fad9c5640 1 -- 172.21.15.110:0/3513974565 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f980093a0 con 0x7f1fa80b0900 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.027+0000 7f1fad9c5640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80b0900 0x7f1fa81dac20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1fa000cad0 tx=0x7f1fa000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.027+0000 7f1f9dffb640 10 client.?.objecter ms_handle_connect 0x7f1fa80b0900 2024-01-20T11:18:55.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.027+0000 7f1f9dffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 1 -- 172.21.15.110:0/3513974565 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa000ab70 con 0x7f1fa80b0900 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 10 monclient: dump: 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:55.036 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 10 monclient: _finish_auth 0 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 10 monclient: _check_auth_tickets 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 1 -- 172.21.15.110:0/3513974565 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa000acd0 con 0x7f1fa80b0900 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 1 -- 172.21.15.110:0/3513974565 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0012600 con 0x7f1fa80b0900 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1faf428640 5 monclient: authenticate success, global_id 4969 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1faf428640 10 monclient: _renew_subs 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1faf428640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa811e0d0 con 0x7f1fa80b0900 2024-01-20T11:18:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9f7fe640 4 set_mon_vals no callback set 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1f9dffb640 10 monclient: dump: 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1faf428640 10 client.4969.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1faf428640 10 monclient: _renew_subs 2024-01-20T11:18:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1faf428640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa811d780 con 0x7f1fa80b0900 2024-01-20T11:18:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.031+0000 7f1faf428640 1 librados: init done 2024-01-20T11:18:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.032+0000 7f1faf428640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.032+0000 7f1faf428640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.032+0000 7f1faf428640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.032+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa811dd70 con 0x7f1fa80b0900 2024-01-20T11:18:55.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.271+0000 7f1f9dffb640 1 -- 172.21.15.110:0/3513974565 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1fa0007070 con 0x7f1fa80b0900 2024-01-20T11:18:55.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.271+0000 7f1f9dffb640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f8c06c860 0x7f1f8c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.272+0000 7f1fad1c4640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f8c06c860 0x7f1f8c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.272+0000 7f1fad1c4640 10 monclient: get_auth_request con 0x7f1f8c06c860 auth_method 0 2024-01-20T11:18:55.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.272+0000 7f1fad1c4640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f8c06c860 0x7f1f8c06ed20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009c00 tx=0x7f1fa4009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:55.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.272+0000 7f1f9dffb640 10 client.4969.objecter ms_handle_connect 0x7f1f8c06c860 2024-01-20T11:18:55.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.275+0000 7f1f9dffb640 1 -- 172.21.15.110:0/3513974565 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1fa000ae40 con 0x7f1fa80b0900 2024-01-20T11:18:55.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.275+0000 7f1f9dffb640 10 client.4969.objecter ms_dispatch 0x7f1fa8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:55.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.275+0000 7f1f9dffb640 3 client.4969.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:55.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.275+0000 7f1f9dffb640 3 client.4969.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:55.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.275+0000 7f1f9dffb640 20 client.4969.objecter dump_active .. 0 homeless 2024-01-20T11:18:55.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.285+0000 7f1f9dffb640 1 -- 172.21.15.110:0/3513974565 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1fa0054750 con 0x7f1fa80b0900 2024-01-20T11:18:55.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.285+0000 7f1f9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:55.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.285+0000 7f1f9dffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:55.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.451+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1fa806c910 con 0x7f1f8c06c860 2024-01-20T11:18:55.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.454+0000 7f1f9dffb640 1 -- 172.21.15.110:0/3513974565 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63955 (secure 0 0 0) 0x7f1fa4011b20 con 0x7f1f8c06c860 2024-01-20T11:18:55.454 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:55.455 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 10 librados: watch_flush enter 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 10 librados: watch_flush exit 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 20 client.4969.objecter shutdown clearing up homeless session... 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 10 client.4969.objecter successfully canceled tick 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f8c06c860 msgr2=0x7f1f8c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f8c06c860 0x7f1f8c06ed20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009c00 tx=0x7f1fa4009190 comp rx=0 tx=0).stop 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 10 monclient: shutdown 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80b0900 msgr2=0x7f1fa81dac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80b0900 0x7f1fa81dac20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1fa000cad0 tx=0x7f1fa000cf90 comp rx=0 tx=0).stop 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 shutdown_connections 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1f8c06c860 0x7f1f8c06ed20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa80b0900 0x7f1fa81dac20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa80af790 0x7f1fa81da6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 --2- 172.21.15.110:0/3513974565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa806bb00 0x7f1fa8141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 >> 172.21.15.110:0/3513974565 conn(0x7f1fa8066d60 msgr2=0x7f1fa8142bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 shutdown_connections 2024-01-20T11:18:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 -- 172.21.15.110:0/3513974565 wait complete. 2024-01-20T11:18:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.456+0000 7f1faf428640 1 librados: shutdown 2024-01-20T11:18:55.467 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-01-20T11:18:54.751525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002134"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280562,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.0050000000000008},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.7480000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.2619999999999996},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.6959999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.763999999999999},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.311999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.4239999999999995},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":13.202999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.135999999999999},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":6.5250000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":12.506},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":14.343999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313264,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.6909999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.103}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.069},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.829000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.212},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.8040000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.298},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.454000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.395},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.531000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.710000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.3889999999999993}]}]},{"osd":3,"up_from":21,"seq":90194313265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.0819999999999999},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":12.364000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.9489999999999998},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.2080000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":16.574999999999999},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":7.9130000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.1210000000000004},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.102}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":15.768000000000001},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":11.153}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":10.573},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.879}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444079,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.5239999999999991},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":14.657999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.1130000000000004},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.7750000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.42},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.9630000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.869999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.786}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.521000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.128}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":7.6580000000000004},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":10.791}]}]},{"osd":1,"up_from":17,"seq":73014444080,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.763999999999999},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":8.7669999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.9860000000000007},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.198}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":13.845000000000001},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":5.8899999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":10.58},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.946}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.4870000000000001},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":12.331}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.1959999999999997},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.054}]}]},{"osd":0,"up_from":17,"seq":73014444080,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":10.122},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":11.375}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.6200000000000001},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.5099999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":11.109},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.035}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.609999999999999},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.766}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":11.956},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":13.882999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.433},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":7.6600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509488,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.2649999999999997},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.9130000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.1840000000000002},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":8.9160000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.522},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.151999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.151999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.182}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.244999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":16.100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.9510000000000005},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.093}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:55.468 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:18:55.468 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.984927+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.984927+0000', 'last_peered': '2024-01-20T11:16:52.984927+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.984927+0000', 'last_undegraded': '2024-01-20T11:16:52.984927+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:18:55.468 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:18:55.469 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:18:55.469 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:18:55.469 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:18:55.469 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:18:55.469 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:18:55.469 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:18:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.567+0000 7f4138725640 10 monclient: get_monmap_and_config 2024-01-20T11:18:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.567+0000 7f4138725640 10 monclient: build_initial_monmap 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.567+0000 7f4138725640 10 monclient: monmap: 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:55.568062+0000 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:55.568062+0000 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:55.568 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 1 Processor -- start 2024-01-20T11:18:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 1 -- start start 2024-01-20T11:18:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 10 monclient: init 2024-01-20T11:18:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-01-20T11:18:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f413006bb00 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41300af790 0x7f41300ae410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 10 monclient(hunting): picked mon.noname-a con 0x7f41300af790 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41300ae950 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413006bee0 con 0x7f41300ae950 2024-01-20T11:18:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41300ada30 con 0x7f413006bb00 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41300add10 con 0x7f41300af790 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.568+0000 7f4138725640 10 monclient(hunting): authenticate will time out at 1578.919031s 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.569+0000 7f4135cc0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41300af790 0x7f41300ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.569+0000 7f41364c1640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f413006bb00 0x7f41300b03c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:46792/0 (socket says 172.21.15.110:46792) 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.569+0000 7f41364c1640 1 -- 172.21.15.110:0/942764572 learned_addr learned my addr 172.21.15.110:0/942764572 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.570+0000 7f4135cc0640 10 monclient(hunting): get_auth_request con 0x7f41300af790 auth_method 0 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.570+0000 7f4135cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.570+0000 7f4135cc0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.570+0000 7f4135cc0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.571+0000 7f4135cc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.571+0000 7f4135cc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.571+0000 7f4135cc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.572+0000 7f41354bf640 1 -- 172.21.15.110:0/942764572 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448391920 0 0) 0x7f41300ada30 con 0x7f413006bb00 2024-01-20T11:18:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.572+0000 7f41354bf640 10 monclient(hunting): _negotiate 2024-01-20T11:18:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.572+0000 7f41354bf640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.572+0000 7f41354bf640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.572+0000 7f41354bf640 10 monclient(hunting): my global_id is 4972 2024-01-20T11:18:55.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.572+0000 7f41354bf640 1 -- 172.21.15.110:0/942764572 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c003520 con 0x7f413006bb00 2024-01-20T11:18:55.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.574+0000 7f4135cc0640 10 monclient(hunting): handle_auth_done global_id 4890 payload 274 2024-01-20T11:18:55.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.574+0000 7f4135cc0640 1 -- 172.21.15.110:0/942764572 >> v1:172.21.15.134:6789/0 conn(0x7f413006bb00 legacy=0x7f41300b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.574+0000 7f4135cc0640 1 -- 172.21.15.110:0/942764572 >> v1:172.21.15.110:6789/0 conn(0x7f41300ae950 legacy=0x7f41300aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.574+0000 7f4135cc0640 10 monclient: _finish_hunting 0 2024-01-20T11:18:55.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.574+0000 7f4135cc0640 1 monclient: found mon.noname-a 2024-01-20T11:18:55.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.574+0000 7f4135cc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:55.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.574+0000 7f4135cc0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:18:55.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.574+0000 7f4135cc0640 1 -- 172.21.15.110:0/942764572 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41300adff0 con 0x7f41300af790 2024-01-20T11:18:55.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.577+0000 7f4135cc0640 1 --2- 172.21.15.110:0/942764572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41300af790 0x7f41300ae410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4120009790 tx=0x7f412002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d63393664f725b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:55.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 1 -- 172.21.15.110:0/942764572 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120031830 con 0x7f41300af790 2024-01-20T11:18:55.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:55.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:18:55.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 10 monclient: dump: 2024-01-20T11:18:55.582 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:55.582 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 10 monclient: _finish_auth 0 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 10 monclient: _check_auth_tickets 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 1 -- 172.21.15.110:0/942764572 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4120031990 con 0x7f41300af790 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:55.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 1 -- 172.21.15.110:0/942764572 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120031d20 con 0x7f41300af790 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f4138725640 5 monclient: authenticate success, global_id 4890 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f4138725640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f4138725640 4 set_mon_vals no callback set 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.581+0000 7f41354bf640 10 monclient: dump: 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:55.584 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.582+0000 7f4138725640 10 monclient: shutdown 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.582+0000 7f4138725640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.582+0000 7f4138725640 1 -- 172.21.15.110:0/942764572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41300af790 msgr2=0x7f41300ae410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.582+0000 7f4138725640 1 --2- 172.21.15.110:0/942764572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41300af790 0x7f41300ae410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4120009790 tx=0x7f412002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.582+0000 7f4138725640 1 -- 172.21.15.110:0/942764572 shutdown_connections 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.582+0000 7f4138725640 1 --2- 172.21.15.110:0/942764572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41300af790 0x7f41300ae410 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.582+0000 7f4138725640 1 -- 172.21.15.110:0/942764572 >> 172.21.15.110:0/942764572 conn(0x7f4130066d60 msgr2=0x7f41300b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.582+0000 7f4138725640 1 -- 172.21.15.110:0/942764572 shutdown_connections 2024-01-20T11:18:55.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.582+0000 7f4138725640 1 -- 172.21.15.110:0/942764572 wait complete. 2024-01-20T11:18:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.583+0000 7f4138725640 10 monclient: build_initial_monmap 2024-01-20T11:18:55.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.583+0000 7f4138725640 10 monclient: monmap: 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.583+0000 7f4138725640 1 librados: starting msgr at 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.583+0000 7f4138725640 1 librados: starting objecter 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.583+0000 7f4138725640 1 Processor -- start 2024-01-20T11:18:55.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.583+0000 7f4138725640 1 -- start start 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.583+0000 7f4138725640 1 librados: setting wanted keys 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.583+0000 7f4138725640 1 librados: calling monclient init 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.583+0000 7f4138725640 10 monclient: init 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f413006bb00 0x7f413011ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient(hunting): picked mon.noname-b con 0x7f413006bb00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41300ae950 0x7f413011f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient(hunting): picked mon.noname-c con 0x7f41300ae950 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41300af790 0x7f41301d6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient(hunting): picked mon.noname-a con 0x7f41300af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413011f730 con 0x7f41300af790 2024-01-20T11:18:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4130121130 con 0x7f413006bb00 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413011f950 con 0x7f41300ae950 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4138725640 10 monclient(hunting): authenticate will time out at 1578.934530s 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4135cc0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41300ae950 0x7f413011f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.584+0000 7f4136cc2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41300af790 0x7f41301d6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f41364c1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f413006bb00 0x7f413011ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4135cc0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41300ae950 0x7f413011f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33456/0 (socket says 172.21.15.110:33456) 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4135cc0640 1 -- 172.21.15.110:0/815345214 learned_addr learned my addr 172.21.15.110:0/815345214 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4135cc0640 10 monclient(hunting): get_auth_request con 0x7f41300ae950 auth_method 0 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4135cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4135cc0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4135cc0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4136cc2640 10 monclient(hunting): get_auth_request con 0x7f41300af790 auth_method 0 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4136cc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4136cc2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4136cc2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f41364c1640 10 monclient(hunting): get_auth_request con 0x7f413006bb00 auth_method 0 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f41364c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f41364c1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f41364c1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f41364c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f41364c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f41364c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4136cc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4136cc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4136cc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4135cc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4135cc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.585+0000 7f4135cc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 10 monclient(hunting): handle_auth_done global_id 4975 payload 824 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 1 -- 172.21.15.110:0/815345214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41300ae950 msgr2=0x7f413011f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41300ae950 0x7f413011f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 1 -- 172.21.15.110:0/815345214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41300af790 msgr2=0x7f41301d6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41300af790 0x7f41301d6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 10 monclient: _finish_hunting 0 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 1 monclient: found mon.noname-b 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.588+0000 7f41364c1640 1 -- 172.21.15.110:0/815345214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41200093a0 con 0x7f413006bb00 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.590+0000 7f41364c1640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f413006bb00 0x7f413011ec40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f412800cb10 tx=0x7f412800a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.590+0000 7f4126ffd640 10 client.?.objecter ms_handle_connect 0x7f413006bb00 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.590+0000 7f4126ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 1 -- 172.21.15.110:0/815345214 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412800ac70 con 0x7f413006bb00 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 10 monclient: dump: 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 10 monclient: _finish_auth 0 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 1 -- 172.21.15.110:0/815345214 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412800add0 con 0x7f413006bb00 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 1 -- 172.21.15.110:0/815345214 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128012620 con 0x7f413006bb00 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4138725640 5 monclient: authenticate success, global_id 4975 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4138725640 10 monclient: _renew_subs 2024-01-20T11:18:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4138725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413011fe50 con 0x7f413006bb00 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4126ffd640 10 monclient: dump: 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:55.597 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4134cbe640 4 set_mon_vals no callback set 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4138725640 10 client.4975.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4138725640 10 monclient: _renew_subs 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4138725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41300b4f30 con 0x7f413006bb00 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.594+0000 7f4138725640 1 librados: init done 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.595+0000 7f4138725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.595+0000 7f4138725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.595+0000 7f4138725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:55.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.595+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413011fb10 con 0x7f413006bb00 2024-01-20T11:18:55.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.833+0000 7f4126ffd640 1 -- 172.21.15.110:0/815345214 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4128007070 con 0x7f413006bb00 2024-01-20T11:18:55.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.833+0000 7f4126ffd640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f411406c8b0 0x7f411406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:55.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.833+0000 7f4135cc0640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f411406c8b0 0x7f411406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:55.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.833+0000 7f4135cc0640 10 monclient: get_auth_request con 0x7f411406c8b0 auth_method 0 2024-01-20T11:18:55.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.834+0000 7f4135cc0640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f411406c8b0 0x7f411406ed70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f412002b910 tx=0x7f4120008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:55.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.834+0000 7f4126ffd640 10 client.4975.objecter ms_handle_connect 0x7f411406c8b0 2024-01-20T11:18:55.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.835+0000 7f4126ffd640 1 -- 172.21.15.110:0/815345214 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4128058600 con 0x7f413006bb00 2024-01-20T11:18:55.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.835+0000 7f4126ffd640 10 client.4975.objecter ms_dispatch 0x7f4130000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:55.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.835+0000 7f4126ffd640 3 client.4975.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:55.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.835+0000 7f4126ffd640 3 client.4975.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:55.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.835+0000 7f4126ffd640 20 client.4975.objecter dump_active .. 0 homeless 2024-01-20T11:18:55.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.846+0000 7f4126ffd640 1 -- 172.21.15.110:0/815345214 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4128054940 con 0x7f413006bb00 2024-01-20T11:18:55.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.846+0000 7f4126ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:55.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:55.846+0000 7f4126ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.107+0000 7f4138725640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:18:56.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.107+0000 7f4138725640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:56.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.107+0000 7f4138725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:56.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.107+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f4130075490 con 0x7f413006bb00 2024-01-20T11:18:56.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.114+0000 7f4126ffd640 1 -- 172.21.15.110:0/815345214 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f4128017080 con 0x7f413006bb00 2024-01-20T11:18:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.114+0000 7f4126ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:18:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.114+0000 7f4126ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:18:56.114 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:56.114 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":270,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:18:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.115+0000 7f4138725640 10 librados: watch_flush enter 2024-01-20T11:18:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 10 librados: watch_flush exit 2024-01-20T11:18:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 20 client.4975.objecter shutdown clearing up homeless session... 2024-01-20T11:18:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 10 client.4975.objecter successfully canceled tick 2024-01-20T11:18:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f411406c8b0 msgr2=0x7f411406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f411406c8b0 0x7f411406ed70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f412002b910 tx=0x7f4120008030 comp rx=0 tx=0).stop 2024-01-20T11:18:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 10 monclient: shutdown 2024-01-20T11:18:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f413006bb00 msgr2=0x7f413011ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f413006bb00 0x7f413011ec40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f412800cb10 tx=0x7f412800a060 comp rx=0 tx=0).stop 2024-01-20T11:18:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 shutdown_connections 2024-01-20T11:18:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f411406c8b0 0x7f411406ed70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41300af790 0x7f41301d6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41300ae950 0x7f413011f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 --2- 172.21.15.110:0/815345214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f413006bb00 0x7f413011ec40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 >> 172.21.15.110:0/815345214 conn(0x7f4130066d60 msgr2=0x7f41300b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 shutdown_connections 2024-01-20T11:18:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 -- 172.21.15.110:0/815345214 wait complete. 2024-01-20T11:18:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.116+0000 7f4138725640 1 librados: shutdown 2024-01-20T11:18:56.127 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:18:56.128 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.227+0000 7fb25c27c640 10 monclient: get_monmap_and_config 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.227+0000 7fb25c27c640 10 monclient: build_initial_monmap 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.227+0000 7fb25c27c640 10 monclient: monmap: 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:56.228332+0000 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:56.228332+0000 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:56.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:56.229 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.228+0000 7fb25c27c640 1 Processor -- start 2024-01-20T11:18:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.228+0000 7fb25c27c640 1 -- start start 2024-01-20T11:18:56.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.228+0000 7fb25c27c640 10 monclient: init 2024-01-20T11:18:56.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:56.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540af3a0 0x7fb25413b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 10 monclient(hunting): picked mon.noname-b con 0x7fb2540af3a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2540ab6d0 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb2540adb30 0x7fb25413bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 10 monclient(hunting): picked mon.noname-a con 0x7fb2540adb30 addr v2:172.21.15.110:3300/0 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb254141930 con 0x7fb2540ab6d0 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb254142920 con 0x7fb2540adb30 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2540af780 con 0x7fb2540af3a0 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25c27c640 10 monclient(hunting): authenticate will time out at 1579.579374s 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25a018640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540af3a0 0x7fb25413b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:56.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb259817640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fb2540ab6d0 0x7fb2540ad420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:46798/0 (socket says 172.21.15.110:46798) 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb259817640 1 -- 172.21.15.110:0/2330331648 learned_addr learned my addr 172.21.15.110:0/2330331648 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.229+0000 7fb25a819640 1 --2- 172.21.15.110:0/2330331648 >> v2:172.21.15.110:3300/0 conn(0x7fb2540adb30 0x7fb25413bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a819640 10 monclient(hunting): get_auth_request con 0x7fb2540adb30 auth_method 0 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a819640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a819640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a018640 10 monclient(hunting): get_auth_request con 0x7fb2540af3a0 auth_method 0 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a018640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a018640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a018640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:56.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a819640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.230+0000 7fb25a819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.231+0000 7fb25a819640 1 -- 172.21.15.110:0/2330331648 >> v2:172.21.15.110:3300/0 conn(0x7fb2540adb30 msgr2=0x7fb25413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.231+0000 7fb25a819640 1 -- 172.21.15.110:0/2330331648 >> v2:172.21.15.110:3300/0 conn(0x7fb2540adb30 msgr2=0x7fb25413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.231+0000 7fb25a819640 1 --2- 172.21.15.110:0/2330331648 >> v2:172.21.15.110:3300/0 conn(0x7fb2540adb30 0x7fb25413bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.231+0000 7fb25a819640 1 --2- 172.21.15.110:0/2330331648 >> v2:172.21.15.110:3300/0 conn(0x7fb2540adb30 0x7fb25413bc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb259016640 1 -- 172.21.15.110:0/2330331648 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144305830 0 0) 0x7fb254141930 con 0x7fb2540ab6d0 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb259016640 10 monclient(hunting): _negotiate 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb259016640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb259016640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb259016640 10 monclient(hunting): my global_id is 4978 2024-01-20T11:18:56.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb259016640 1 -- 172.21.15.110:0/2330331648 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb240003520 con 0x7fb2540ab6d0 2024-01-20T11:18:56.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb25a018640 10 monclient(hunting): handle_auth_done global_id 4748 payload 274 2024-01-20T11:18:56.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb25a018640 1 -- 172.21.15.110:0/2330331648 >> v2:172.21.15.110:3300/0 conn(0x7fb2540adb30 msgr2=0x7fb25413bc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:18:56.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb25a018640 1 --2- 172.21.15.110:0/2330331648 >> v2:172.21.15.110:3300/0 conn(0x7fb2540adb30 0x7fb25413bc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb25a018640 1 -- 172.21.15.110:0/2330331648 >> v1:172.21.15.134:6789/0 conn(0x7fb2540ab6d0 legacy=0x7fb2540ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:56.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb25a018640 10 monclient: _finish_hunting 0 2024-01-20T11:18:56.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb25a018640 1 monclient: found mon.noname-b 2024-01-20T11:18:56.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb25a018640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:56.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb25a018640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:18:56.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.232+0000 7fb25a018640 1 -- 172.21.15.110:0/2330331648 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2540ab2b0 con 0x7fb2540af3a0 2024-01-20T11:18:56.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.234+0000 7fb25a018640 1 --2- 172.21.15.110:0/2330331648 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540af3a0 0x7fb25413b730 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00aea0 tx=0x7fb24c02cfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e073f7f23975c48 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:56.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 1 -- 172.21.15.110:0/2330331648 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c02db40 con 0x7fb2540af3a0 2024-01-20T11:18:56.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 10 monclient: dump: 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 10 monclient: _finish_auth 0 2024-01-20T11:18:56.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 10 monclient: _check_auth_tickets 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 1 -- 172.21.15.110:0/2330331648 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24c02dca0 con 0x7fb2540af3a0 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb25c27c640 5 monclient: authenticate success, global_id 4748 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb25c27c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb25c27c640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 1 -- 172.21.15.110:0/2330331648 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c008660 con 0x7fb2540af3a0 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb25c27c640 4 set_mon_vals no callback set 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.237+0000 7fb259016640 10 monclient: dump: 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:56.239 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:56.240 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:56.240 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:56.240 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:56.240 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:56.240 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:56.240 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:56.240 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:56.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 10 monclient: shutdown 2024-01-20T11:18:56.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:56.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 1 -- 172.21.15.110:0/2330331648 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540af3a0 msgr2=0x7fb25413b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:56.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 1 --2- 172.21.15.110:0/2330331648 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540af3a0 0x7fb25413b730 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00aea0 tx=0x7fb24c02cfa0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 1 -- 172.21.15.110:0/2330331648 shutdown_connections 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 1 --2- 172.21.15.110:0/2330331648 >> v2:172.21.15.110:3300/0 conn(0x7fb2540adb30 0x7fb25413bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 1 --2- 172.21.15.110:0/2330331648 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540af3a0 0x7fb25413b730 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 1 -- 172.21.15.110:0/2330331648 >> 172.21.15.110:0/2330331648 conn(0x7fb2540b3e60 msgr2=0x7fb2540b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 1 -- 172.21.15.110:0/2330331648 shutdown_connections 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.238+0000 7fb25c27c640 1 -- 172.21.15.110:0/2330331648 wait complete. 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.239+0000 7fb25c27c640 10 monclient: build_initial_monmap 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.239+0000 7fb25c27c640 10 monclient: monmap: 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.239+0000 7fb25c27c640 1 librados: starting msgr at 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.239+0000 7fb25c27c640 1 librados: starting objecter 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.239+0000 7fb25c27c640 1 Processor -- start 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.239+0000 7fb25c27c640 1 -- start start 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.239+0000 7fb25c27c640 1 librados: setting wanted keys 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.239+0000 7fb25c27c640 1 librados: calling monclient init 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.239+0000 7fb25c27c640 10 monclient: init 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2540ab6d0 0x7fb25413d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:56.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient(hunting): picked mon.noname-a con 0x7fb2540ab6d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540adb30 0x7fb2541d4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient(hunting): picked mon.noname-c con 0x7fb2540adb30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2540af3a0 0x7fb2541d8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient(hunting): picked mon.noname-b con 0x7fb2540af3a0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25413e020 con 0x7fb2540ab6d0 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25413dcb0 con 0x7fb2540af3a0 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25413d9c0 con 0x7fb2540adb30 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb25c27c640 10 monclient(hunting): authenticate will time out at 1579.590563s 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.240+0000 7fb259817640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540adb30 0x7fb2541d4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a018640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2540ab6d0 0x7fb25413d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:56.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a819640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2540af3a0 0x7fb2541d8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb259817640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540adb30 0x7fb2541d4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33480/0 (socket says 172.21.15.110:33480) 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb259817640 1 -- 172.21.15.110:0/3047462924 learned_addr learned my addr 172.21.15.110:0/3047462924 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb259817640 10 monclient(hunting): get_auth_request con 0x7fb2540adb30 auth_method 0 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb259817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb259817640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb259817640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a018640 10 monclient(hunting): get_auth_request con 0x7fb2540ab6d0 auth_method 0 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a018640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a018640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a819640 10 monclient(hunting): get_auth_request con 0x7fb2540af3a0 auth_method 0 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a819640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a819640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:56.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb259817640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:56.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb259817640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:56.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb259817640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:56.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a819640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:56.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:56.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:56.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a018640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:56.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:56.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.241+0000 7fb25a018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:56.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 10 monclient(hunting): handle_auth_done global_id 4981 payload 824 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 1 -- 172.21.15.110:0/3047462924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540adb30 msgr2=0x7fb2541d4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540adb30 0x7fb2541d4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 1 -- 172.21.15.110:0/3047462924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2540ab6d0 msgr2=0x7fb25413d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2540ab6d0 0x7fb25413d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 10 monclient: _finish_hunting 0 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 1 monclient: found mon.noname-b 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.244+0000 7fb25a819640 1 -- 172.21.15.110:0/3047462924 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c00ab80 con 0x7fb2540af3a0 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.246+0000 7fb25a819640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2540af3a0 0x7fb2541d8310 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb2500094a0 tx=0x7fb25000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.246+0000 7fb24affd640 10 client.?.objecter ms_handle_connect 0x7fb2540af3a0 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.246+0000 7fb24affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.249+0000 7fb24affd640 1 -- 172.21.15.110:0/3047462924 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25000c730 con 0x7fb2540af3a0 2024-01-20T11:18:56.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 10 monclient: dump: 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 10 monclient: _finish_auth 0 2024-01-20T11:18:56.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 10 monclient: _check_auth_tickets 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 1 -- 172.21.15.110:0/3047462924 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25000c8b0 con 0x7fb2540af3a0 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 5 monclient: authenticate success, global_id 4981 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 10 monclient: _renew_subs 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb254072220 con 0x7fb2540af3a0 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 1 -- 172.21.15.110:0/3047462924 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25000cc60 con 0x7fb2540af3a0 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb24affd640 10 monclient: dump: 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:56.255 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 10 client.4981.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 10 monclient: _renew_subs 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb258815640 4 set_mon_vals no callback set 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25413c110 con 0x7fb2540af3a0 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 1 librados: init done 2024-01-20T11:18:56.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:56.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:56.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:56.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.250+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25413e190 con 0x7fb2540af3a0 2024-01-20T11:18:56.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.492+0000 7fb24affd640 1 -- 172.21.15.110:0/3047462924 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb250013470 con 0x7fb2540af3a0 2024-01-20T11:18:56.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.492+0000 7fb24affd640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb23806c8b0 0x7fb23806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:56.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.493+0000 7fb25a018640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb23806c8b0 0x7fb23806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:56.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.493+0000 7fb25a018640 10 monclient: get_auth_request con 0x7fb23806c8b0 auth_method 0 2024-01-20T11:18:56.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.493+0000 7fb25a018640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb23806c8b0 0x7fb23806ed70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb24c036000 tx=0x7fb24c031000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:56.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.493+0000 7fb24affd640 10 client.4981.objecter ms_handle_connect 0x7fb23806c8b0 2024-01-20T11:18:56.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.495+0000 7fb24affd640 1 -- 172.21.15.110:0/3047462924 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb2500589d0 con 0x7fb2540af3a0 2024-01-20T11:18:56.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.495+0000 7fb24affd640 10 client.4981.objecter ms_dispatch 0x7fb254000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:18:56.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.495+0000 7fb24affd640 3 client.4981.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:18:56.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.495+0000 7fb24affd640 3 client.4981.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:18:56.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.495+0000 7fb24affd640 20 client.4981.objecter dump_active .. 0 homeless 2024-01-20T11:18:56.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.505+0000 7fb24affd640 1 -- 172.21.15.110:0/3047462924 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb250054d10 con 0x7fb2540af3a0 2024-01-20T11:18:56.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.505+0000 7fb24affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:56.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.505+0000 7fb24affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:18:56.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.669+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2540acd20 con 0x7fb23806c8b0 2024-01-20T11:18:56.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.672+0000 7fb24affd640 1 -- 172.21.15.110:0/3047462924 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63955 (secure 0 0 0) 0x7fb24c02dad0 con 0x7fb23806c8b0 2024-01-20T11:18:56.672 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:18:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 10 librados: watch_flush enter 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 10 librados: watch_flush exit 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 20 client.4981.objecter shutdown clearing up homeless session... 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 10 client.4981.objecter successfully canceled tick 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb23806c8b0 msgr2=0x7fb23806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb23806c8b0 0x7fb23806ed70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb24c036000 tx=0x7fb24c031000 comp rx=0 tx=0).stop 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 10 monclient: shutdown 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2540af3a0 msgr2=0x7fb2541d8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2540af3a0 0x7fb2541d8310 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb2500094a0 tx=0x7fb25000ec50 comp rx=0 tx=0).stop 2024-01-20T11:18:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 shutdown_connections 2024-01-20T11:18:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb23806c8b0 0x7fb23806ed70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2540af3a0 0x7fb2541d8310 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2540adb30 0x7fb2541d4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 --2- 172.21.15.110:0/3047462924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2540ab6d0 0x7fb25413d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 >> 172.21.15.110:0/3047462924 conn(0x7fb2540b3e60 msgr2=0x7fb2540a9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 shutdown_connections 2024-01-20T11:18:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 -- 172.21.15.110:0/3047462924 wait complete. 2024-01-20T11:18:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:56.674+0000 7fb25c27c640 1 librados: shutdown 2024-01-20T11:18:56.684 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-01-20T11:18:54.751525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002134"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.984927+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.984927+0000","last_peered":"2024-01-20T11:16:52.984927+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.984927+0000","last_undegraded":"2024-01-20T11:16:52.984927+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280562,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.0050000000000008},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.7480000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.2619999999999996},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":9.6959999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.763999999999999},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.311999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.4239999999999995},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":13.202999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.135999999999999},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":6.5250000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":12.506},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":14.343999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313264,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.6909999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.103}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.069},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.829000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.212},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.8040000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.298},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.454000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.395},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.531000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.710000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.3889999999999993}]}]},{"osd":3,"up_from":21,"seq":90194313265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.0819999999999999},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":12.364000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.9489999999999998},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.2080000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":16.574999999999999},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":7.9130000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.1210000000000004},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.102}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":15.768000000000001},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":11.153}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":10.573},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.879}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444079,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.5239999999999991},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":14.657999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.1130000000000004},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.7750000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.42},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.9630000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.869999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.786}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.521000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.128}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":7.6580000000000004},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":10.791}]}]},{"osd":1,"up_from":17,"seq":73014444080,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.763999999999999},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":8.7669999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.9860000000000007},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":13.198}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":13.845000000000001},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":5.8899999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":10.58},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.946}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.4870000000000001},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":12.331}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.1959999999999997},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.054}]}]},{"osd":0,"up_from":17,"seq":73014444080,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":10.122},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":11.375}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.6200000000000001},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.5099999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":11.109},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.035}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.609999999999999},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.766}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":11.956},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":13.882999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.433},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":7.6600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509488,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.2649999999999997},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.9130000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.1840000000000002},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":8.9160000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.522},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.151999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.151999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.182}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.244999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":16.100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.9510000000000005},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.093}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:18:58.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-01-20T11:18:58.490 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:49.515 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:58.538 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:04:49.544 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:58.539 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:04:49.545 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:58.540 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:49.565 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:58.563 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:04:49.570 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:58.566 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:49.590 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:58.593 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:49.617 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:58.595 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:49.620 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:58.616 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:49.640 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:18:59.687 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:18:59.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.788+0000 7fc5a1f08640 10 monclient: get_monmap_and_config 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.788+0000 7fc5a1f08640 10 monclient: build_initial_monmap 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.788+0000 7fc5a1f08640 10 monclient: monmap: 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:18:59.790551+0000 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:18:59.790551+0000 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:18:59.790 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.789+0000 7fc5a1f08640 1 Processor -- start 2024-01-20T11:18:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.789+0000 7fc5a1f08640 1 -- start start 2024-01-20T11:18:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.789+0000 7fc5a1f08640 10 monclient: init 2024-01-20T11:18:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-01-20T11:18:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc59c06a860 0x7fc59c13f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 10 monclient(hunting): picked mon.noname-c con 0x7fc59c06a860 addr v2:172.21.15.134:3300/0 2024-01-20T11:18:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc59c0af730 addr v1:172.21.15.110:6789/0 2024-01-20T11:18:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc59c13f990 addr v1:172.21.15.134:6789/0 2024-01-20T11:18:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59c0afc50 con 0x7fc59c0af730 2024-01-20T11:18:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59c0b0170 con 0x7fc59c13f990 2024-01-20T11:18:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59c06e450 con 0x7fc59c06a860 2024-01-20T11:18:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc5a1f08640 10 monclient(hunting): authenticate will time out at 1583.141625s 2024-01-20T11:18:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.790+0000 7fc59affd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc59c0af730 0x7fc59c1455b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36290/0 (socket says 172.21.15.110:36290) 2024-01-20T11:18:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.791+0000 7fc59affd640 1 -- 172.21.15.110:0/781382312 learned_addr learned my addr 172.21.15.110:0/781382312 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:18:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.791+0000 7fc59b7fe640 1 --2- 172.21.15.110:0/781382312 >> v2:172.21.15.134:3300/0 conn(0x7fc59c06a860 0x7fc59c13f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.792+0000 7fc59b7fe640 10 monclient(hunting): get_auth_request con 0x7fc59c06a860 auth_method 0 2024-01-20T11:18:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.792+0000 7fc59b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.792+0000 7fc59b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.792+0000 7fc59b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.792+0000 7fc59b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.792+0000 7fc59b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.793+0000 7fc59b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:59.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.795+0000 7fc59b7fe640 10 monclient(hunting): handle_auth_done global_id 4984 payload 274 2024-01-20T11:18:59.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.795+0000 7fc59b7fe640 1 -- 172.21.15.110:0/781382312 >> v1:172.21.15.134:6789/0 conn(0x7fc59c13f990 legacy=0x7fc59c14c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:59.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.795+0000 7fc59b7fe640 1 -- 172.21.15.110:0/781382312 >> v1:172.21.15.110:6789/0 conn(0x7fc59c0af730 legacy=0x7fc59c1455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:59.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.795+0000 7fc59b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:18:59.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.795+0000 7fc59b7fe640 1 monclient: found mon.noname-c 2024-01-20T11:18:59.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.795+0000 7fc59b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:59.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.795+0000 7fc59b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:18:59.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.795+0000 7fc59b7fe640 1 -- 172.21.15.110:0/781382312 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c06ac40 con 0x7fc59c06a860 2024-01-20T11:18:59.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.799+0000 7fc59b7fe640 1 --2- 172.21.15.110:0/781382312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c06a860 0x7fc59c13f450 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc5840097a0 tx=0x7fc58402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e8c32bdd27faef9 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:59.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 1 -- 172.21.15.110:0/781382312 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58402e070 con 0x7fc59c06a860 2024-01-20T11:18:59.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:59.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:18:59.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 10 monclient: dump: 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 1 -- 172.21.15.110:0/781382312 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5840315a0 con 0x7fc59c06a860 2024-01-20T11:18:59.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc5a1f08640 5 monclient: authenticate success, global_id 4984 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc5a1f08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 1 -- 172.21.15.110:0/781382312 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584031930 con 0x7fc59c06a860 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc59a7fc640 10 monclient: dump: 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:59.806 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:59.807 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:59.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc5a1f08640 10 monclient: get_monmap_and_config success 2024-01-20T11:18:59.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.803+0000 7fc5a1f08640 4 set_mon_vals no callback set 2024-01-20T11:18:59.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.804+0000 7fc5a1f08640 10 monclient: shutdown 2024-01-20T11:18:59.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.804+0000 7fc5a1f08640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:18:59.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.804+0000 7fc5a1f08640 1 -- 172.21.15.110:0/781382312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c06a860 msgr2=0x7fc59c13f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:59.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.804+0000 7fc5a1f08640 1 --2- 172.21.15.110:0/781382312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c06a860 0x7fc59c13f450 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc5840097a0 tx=0x7fc58402b840 comp rx=0 tx=0).stop 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.804+0000 7fc5a1f08640 1 -- 172.21.15.110:0/781382312 shutdown_connections 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.804+0000 7fc5a1f08640 1 --2- 172.21.15.110:0/781382312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c06a860 0x7fc59c13f450 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.804+0000 7fc5a1f08640 1 -- 172.21.15.110:0/781382312 >> 172.21.15.110:0/781382312 conn(0x7fc59c0b47d0 msgr2=0x7fc59c0b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.804+0000 7fc5a1f08640 1 -- 172.21.15.110:0/781382312 shutdown_connections 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.804+0000 7fc5a1f08640 1 -- 172.21.15.110:0/781382312 wait complete. 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.805+0000 7fc5a1f08640 10 monclient: build_initial_monmap 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.805+0000 7fc5a1f08640 10 monclient: monmap: 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:18:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.805+0000 7fc5a1f08640 1 librados: starting msgr at 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.805+0000 7fc5a1f08640 1 librados: starting objecter 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.805+0000 7fc5a1f08640 1 Processor -- start 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.805+0000 7fc5a1f08640 1 -- start start 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.805+0000 7fc5a1f08640 1 librados: setting wanted keys 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.805+0000 7fc5a1f08640 1 librados: calling monclient init 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.805+0000 7fc5a1f08640 10 monclient: init 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient: _reopen_session rank -1 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc59c06a860 0x7fc59c13b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient(hunting): picked mon.noname-c con 0x7fc59c06a860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:18:59.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c0af730 0x7fc59c1d7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient(hunting): picked mon.noname-b con 0x7fc59c0af730 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc59c13f990 0x7fc59c1d7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient(hunting): picked mon.noname-a con 0x7fc59c13f990 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1d7590 con 0x7fc59c13f990 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1d84a0 con 0x7fc59c0af730 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient(hunting): start opening mon connection 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c06edf0 con 0x7fc59c06a860 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient(hunting): _renew_subs 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc5a1f08640 10 monclient(hunting): authenticate will time out at 1583.157572s 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc59affd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c0af730 0x7fc59c1d7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.806+0000 7fc59b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc59c06a860 0x7fc59c13b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc59c13f990 0x7fc59c1d7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:18:59.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc59c06a860 0x7fc59c13b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47082/0 (socket says 172.21.15.110:47082) 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59affd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c0af730 0x7fc59c1d7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:42974/0 (socket says 172.21.15.110:42974) 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59b7fe640 1 -- 172.21.15.110:0/3899426751 learned_addr learned my addr 172.21.15.110:0/3899426751 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59b7fe640 10 monclient(hunting): get_auth_request con 0x7fc59c06a860 auth_method 0 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59affd640 10 monclient(hunting): get_auth_request con 0x7fc59c0af730 auth_method 0 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59bfff640 10 monclient(hunting): get_auth_request con 0x7fc59c13f990 auth_method 0 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:18:59.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:59.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:59.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:59.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:59.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:59.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:59.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:18:59.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:18:59.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.807+0000 7fc59bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:18:59.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 10 monclient(hunting): handle_auth_done global_id 4990 payload 824 2024-01-20T11:18:59.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 1 -- 172.21.15.110:0/3899426751 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc59c06a860 msgr2=0x7fc59c13b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:59.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc59c06a860 0x7fc59c13b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:59.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 1 -- 172.21.15.110:0/3899426751 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc59c13f990 msgr2=0x7fc59c1d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:18:59.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc59c13f990 0x7fc59c1d7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:18:59.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 10 monclient: _finish_hunting 0 2024-01-20T11:18:59.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 1 monclient: found mon.noname-b 2024-01-20T11:18:59.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:18:59.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:18:59.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.810+0000 7fc59affd640 1 -- 172.21.15.110:0/3899426751 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc584009400 con 0x7fc59c0af730 2024-01-20T11:18:59.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.813+0000 7fc59affd640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c0af730 0x7fc59c1d7a20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc58c005880 tx=0x7fc58c00c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:18:59.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.813+0000 7fc593fff640 10 client.?.objecter ms_handle_connect 0x7fc59c0af730 2024-01-20T11:18:59.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.813+0000 7fc593fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 1 -- 172.21.15.110:0/3899426751 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c005ce0 con 0x7fc59c0af730 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 10 monclient: dump: 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:59.818 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 10 monclient: _finish_auth 0 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 10 monclient: _check_auth_tickets 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 1 -- 172.21.15.110:0/3899426751 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c005e60 con 0x7fc59c0af730 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 1 -- 172.21.15.110:0/3899426751 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c007310 con 0x7fc59c0af730 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc593fff640 10 monclient: dump: 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:18:59.819 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc5a1f08640 5 monclient: authenticate success, global_id 4990 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc5a1f08640 10 monclient: _renew_subs 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc5a1f08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c13aac0 con 0x7fc59c0af730 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.816+0000 7fc599ffb640 4 set_mon_vals no callback set 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.817+0000 7fc5a1f08640 10 client.4990.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.817+0000 7fc5a1f08640 10 monclient: _renew_subs 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.817+0000 7fc5a1f08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.817+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c136850 con 0x7fc59c0af730 2024-01-20T11:18:59.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.817+0000 7fc5a1f08640 1 librados: init done 2024-01-20T11:18:59.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.817+0000 7fc5a1f08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:59.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.817+0000 7fc5a1f08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:18:59.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.817+0000 7fc5a1f08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:18:59.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:18:59.817+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c075490 con 0x7fc59c0af730 2024-01-20T11:19:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.061+0000 7fc593fff640 1 -- 172.21.15.110:0/3899426751 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc58c007470 con 0x7fc59c0af730 2024-01-20T11:19:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.062+0000 7fc593fff640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc57c06c8b0 0x7fc57c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.062+0000 7fc59b7fe640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc57c06c8b0 0x7fc57c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.062+0000 7fc59b7fe640 10 monclient: get_auth_request con 0x7fc57c06c8b0 auth_method 0 2024-01-20T11:19:00.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.063+0000 7fc593fff640 10 client.4990.objecter ms_handle_connect 0x7fc57c06c8b0 2024-01-20T11:19:00.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.063+0000 7fc59b7fe640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc57c06c8b0 0x7fc57c06ed70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc584000c00 tx=0x7fc58403d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.066+0000 7fc593fff640 1 -- 172.21.15.110:0/3899426751 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc58c001080 con 0x7fc59c0af730 2024-01-20T11:19:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.066+0000 7fc593fff640 10 client.4990.objecter ms_dispatch 0x7fc59c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.066+0000 7fc593fff640 3 client.4990.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.066+0000 7fc593fff640 3 client.4990.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.066+0000 7fc593fff640 20 client.4990.objecter dump_active .. 0 homeless 2024-01-20T11:19:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.077+0000 7fc593fff640 1 -- 172.21.15.110:0/3899426751 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc58c053e70 con 0x7fc59c0af730 2024-01-20T11:19:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.077+0000 7fc593fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.077+0000 7fc593fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.246+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc59c06ef60 con 0x7fc57c06c8b0 2024-01-20T11:19:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.249+0000 7fc593fff640 1 -- 172.21.15.110:0/3899426751 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63929 (secure 0 0 0) 0x7fc59c06ef60 con 0x7fc57c06c8b0 2024-01-20T11:19:00.250 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.250+0000 7fc5a1f08640 10 librados: watch_flush enter 2024-01-20T11:19:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 10 librados: watch_flush exit 2024-01-20T11:19:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 20 client.4990.objecter shutdown clearing up homeless session... 2024-01-20T11:19:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 10 client.4990.objecter successfully canceled tick 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc57c06c8b0 msgr2=0x7fc57c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc57c06c8b0 0x7fc57c06ed70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc584000c00 tx=0x7fc58403d040 comp rx=0 tx=0).stop 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 10 monclient: shutdown 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c0af730 msgr2=0x7fc59c1d7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c0af730 0x7fc59c1d7a20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc58c005880 tx=0x7fc58c00c820 comp rx=0 tx=0).stop 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 shutdown_connections 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc57c06c8b0 0x7fc57c06ed70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc59c13f990 0x7fc59c1d7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc59c0af730 0x7fc59c1d7a20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 --2- 172.21.15.110:0/3899426751 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc59c06a860 0x7fc59c13b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 >> 172.21.15.110:0/3899426751 conn(0x7fc59c0b47d0 msgr2=0x7fc59c0adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 shutdown_connections 2024-01-20T11:19:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 -- 172.21.15.110:0/3899426751 wait complete. 2024-01-20T11:19:00.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.251+0000 7fc5a1f08640 1 librados: shutdown 2024-01-20T11:19:00.262 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-01-20T11:18:58.752241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002225"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.6639999999999997},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":12.459}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.8179999999999996},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.0739999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":8.4849999999999994},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.227}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.390000000000001},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":13.096}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":9.8360000000000003},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":9.2240000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.175999999999998},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.154}]}]},{"osd":7,"up_from":21,"seq":90194313265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.6909999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.103}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.069},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.829000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.212},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.8040000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.298},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.454000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.395},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.531000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.710000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.3889999999999993}]}]},{"osd":3,"up_from":21,"seq":90194313266,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.133},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.7960000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":11.412000000000001},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.079000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":9.3699999999999992},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.113}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.380000000000001},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":7.8470000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":8.4719999999999995},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":14.656000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.476000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.708}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444080,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.1929999999999996},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":9.3290000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":13.417999999999999},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.218}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.1010000000000009},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.281000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":8.7650000000000006},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.965999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":10.023},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.760999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.542999999999999},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":12.683}]}]},{"osd":1,"up_from":17,"seq":73014444081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.9930000000000003},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":16.992999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.932},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.364000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":14.079000000000001},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.874000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":13.336},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.856}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.209},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":18.094999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.813000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":19.370000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444081,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":12.417},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":7.298}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.1669999999999998},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.042999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.384},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.9829999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.23},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":5.1680000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":15.134},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":11.859999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.081},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":9.2050000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":7.4130000000000003},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.25}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.3620000000000001},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":12.396000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":16.323}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.004},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.5589999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":10.218999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":17.282}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.753},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.597}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:00.264 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.362+0000 7f5da3aa2640 10 monclient: get_monmap_and_config 2024-01-20T11:19:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.362+0000 7f5da3aa2640 10 monclient: build_initial_monmap 2024-01-20T11:19:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.362+0000 7f5da3aa2640 10 monclient: monmap: 2024-01-20T11:19:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:00.363848+0000 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:00.363848+0000 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.362+0000 7f5da3aa2640 1 Processor -- start 2024-01-20T11:19:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 1 -- start start 2024-01-20T11:19:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 10 monclient: init 2024-01-20T11:19:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-01-20T11:19:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d9c0a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d9c0a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c142af0 0x7f5d9c14d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f5d9c142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d9c0b25c0 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d9c142f50 con 0x7f5d9c0a9ad0 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c0aaea0 con 0x7f5d9c142af0 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.363+0000 7f5da3aa2640 10 monclient(hunting): authenticate will time out at 1583.714830s 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.364+0000 7f5da203f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c142af0 0x7f5d9c14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.364+0000 7f5da183e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5d9c0a96f0 0x7f5d9c13f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36294/0 (socket says 172.21.15.110:36294) 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.364+0000 7f5da183e640 1 -- 172.21.15.110:0/115358202 learned_addr learned my addr 172.21.15.110:0/115358202 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.364+0000 7f5da203f640 10 monclient(hunting): get_auth_request con 0x7f5d9c142af0 auth_method 0 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.364+0000 7f5da203f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.364+0000 7f5da203f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.364+0000 7f5da203f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.365+0000 7f5da203f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:00.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.365+0000 7f5da203f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:00.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.365+0000 7f5da203f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da083c640 1 -- 172.21.15.110:0/115358202 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157777078 0 0) 0x7f5d9c0b25c0 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da083c640 10 monclient(hunting): _negotiate 2024-01-20T11:19:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da083c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da083c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da083c640 10 monclient(hunting): my global_id is 4911 2024-01-20T11:19:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da083c640 1 -- 172.21.15.110:0/115358202 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d88003520 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da203f640 10 monclient(hunting): handle_auth_done global_id 4757 payload 274 2024-01-20T11:19:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da203f640 1 -- 172.21.15.110:0/115358202 >> v1:172.21.15.134:6789/0 conn(0x7f5d9c0a9ad0 legacy=0x7f5d9c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da203f640 1 -- 172.21.15.110:0/115358202 >> v1:172.21.15.110:6789/0 conn(0x7f5d9c0a96f0 legacy=0x7f5d9c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da203f640 10 monclient: _finish_hunting 0 2024-01-20T11:19:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da203f640 1 monclient: found mon.noname-b 2024-01-20T11:19:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da203f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da203f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.366+0000 7f5da203f640 1 -- 172.21.15.110:0/115358202 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9c13fbe0 con 0x7f5d9c142af0 2024-01-20T11:19:00.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.368+0000 7f5da203f640 1 --2- 172.21.15.110:0/115358202 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c142af0 0x7f5d9c14d860 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009dc0 tx=0x7f5d9402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c30c7611c21ebd9 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 1 -- 172.21.15.110:0/115358202 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9402cc40 con 0x7f5d9c142af0 2024-01-20T11:19:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 10 monclient: dump: 2024-01-20T11:19:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.377 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 10 monclient: _finish_auth 0 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 10 monclient: _check_auth_tickets 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 1 -- 172.21.15.110:0/115358202 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9402cda0 con 0x7f5d9c142af0 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da3aa2640 5 monclient: authenticate success, global_id 4757 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da3aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:00.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da3aa2640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da3aa2640 4 set_mon_vals no callback set 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 1 -- 172.21.15.110:0/115358202 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94034660 con 0x7f5d9c142af0 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da083c640 10 monclient: dump: 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da3aa2640 10 monclient: shutdown 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da3aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.372+0000 7f5da3aa2640 1 -- 172.21.15.110:0/115358202 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c142af0 msgr2=0x7f5d9c14d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.373+0000 7f5da3aa2640 1 --2- 172.21.15.110:0/115358202 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c142af0 0x7f5d9c14d860 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009dc0 tx=0x7f5d9402be90 comp rx=0 tx=0).stop 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.373+0000 7f5da3aa2640 1 -- 172.21.15.110:0/115358202 shutdown_connections 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.373+0000 7f5da3aa2640 1 --2- 172.21.15.110:0/115358202 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c142af0 0x7f5d9c14d860 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.373+0000 7f5da3aa2640 1 -- 172.21.15.110:0/115358202 >> 172.21.15.110:0/115358202 conn(0x7f5d9c0b7bc0 msgr2=0x7f5d9c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.373+0000 7f5da3aa2640 1 -- 172.21.15.110:0/115358202 shutdown_connections 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.373+0000 7f5da3aa2640 1 -- 172.21.15.110:0/115358202 wait complete. 2024-01-20T11:19:00.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.373+0000 7f5da3aa2640 10 monclient: build_initial_monmap 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.373+0000 7f5da3aa2640 10 monclient: monmap: 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 librados: starting msgr at 2024-01-20T11:19:00.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 librados: starting objecter 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 Processor -- start 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 -- start start 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 librados: setting wanted keys 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 librados: calling monclient init 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient: init 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d9c0a96f0 0x7f5d9c135670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f5d9c0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c0a9ad0 0x7f5d9c136b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f5d9c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:00.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d9c142af0 0x7f5d9c138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f5d9c142af0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c13a010 con 0x7f5d9c142af0 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c13a2f0 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c13a5d0 con 0x7f5d9c0a9ad0 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.374+0000 7f5da3aa2640 10 monclient(hunting): authenticate will time out at 1583.726097s 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da203f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d9c142af0 0x7f5d9c138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da183e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d9c0a96f0 0x7f5d9c135670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da103d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c0a9ad0 0x7f5d9c136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da103d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c0a9ad0 0x7f5d9c136b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47090/0 (socket says 172.21.15.110:47090) 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da103d640 1 -- 172.21.15.110:0/3986626868 learned_addr learned my addr 172.21.15.110:0/3986626868 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da103d640 10 monclient(hunting): get_auth_request con 0x7f5d9c0a9ad0 auth_method 0 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da103d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da103d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da103d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da203f640 10 monclient(hunting): get_auth_request con 0x7f5d9c142af0 auth_method 0 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da203f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da203f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da203f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da183e640 10 monclient(hunting): get_auth_request con 0x7f5d9c0a96f0 auth_method 0 2024-01-20T11:19:00.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da183e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da183e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.375+0000 7f5da183e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.376+0000 7f5da103d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.376+0000 7f5da103d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.376+0000 7f5da103d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.376+0000 7f5da183e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.376+0000 7f5da183e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.376+0000 7f5da183e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.376+0000 7f5da203f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.376+0000 7f5da203f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:00.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.376+0000 7f5da203f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 10 monclient(hunting): handle_auth_done global_id 4996 payload 824 2024-01-20T11:19:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 1 -- 172.21.15.110:0/3986626868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c0a9ad0 msgr2=0x7f5d9c136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c0a9ad0 0x7f5d9c136b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 1 -- 172.21.15.110:0/3986626868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d9c142af0 msgr2=0x7f5d9c138dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d9c142af0 0x7f5d9c138dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 10 monclient: _finish_hunting 0 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 1 monclient: found mon.noname-b 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.379+0000 7f5da183e640 1 -- 172.21.15.110:0/3986626868 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d94009a70 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.381+0000 7f5da183e640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d9c0a96f0 0x7f5d9c135670 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00c430 tx=0x7f5d8c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.381+0000 7f5d927fc640 10 client.?.objecter ms_handle_connect 0x7f5d9c0a96f0 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.381+0000 7f5d927fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 1 -- 172.21.15.110:0/3986626868 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c005ce0 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:00.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 10 monclient: dump: 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:00.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 1 -- 172.21.15.110:0/3986626868 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c005e40 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 1 -- 172.21.15.110:0/3986626868 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c0072d0 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5da3aa2640 5 monclient: authenticate success, global_id 4996 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5da3aa2640 10 monclient: _renew_subs 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5da3aa2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9c1388b0 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d93fff640 4 set_mon_vals no callback set 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5d927fc640 10 monclient: dump: 2024-01-20T11:19:00.390 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5da3aa2640 10 client.4996.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5da3aa2640 10 monclient: _renew_subs 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5da3aa2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:00.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d9c1380e0 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.385+0000 7f5da3aa2640 1 librados: init done 2024-01-20T11:19:00.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.386+0000 7f5da3aa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:00.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.386+0000 7f5da3aa2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:00.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.386+0000 7f5da3aa2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:00.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.386+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d9c14dd20 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.625+0000 7f5d927fc640 1 -- 172.21.15.110:0/3986626868 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5d8c007430 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.626+0000 7f5d927fc640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5d8006c8b0 0x7f5d8006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.626+0000 7f5da103d640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5d8006c8b0 0x7f5d8006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.626+0000 7f5da103d640 10 monclient: get_auth_request con 0x7f5d8006c8b0 auth_method 0 2024-01-20T11:19:00.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.627+0000 7f5da103d640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5d8006c8b0 0x7f5d8006ed70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0b3750 tx=0x7f5d98009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:00.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.627+0000 7f5d927fc640 10 client.4996.objecter ms_handle_connect 0x7f5d8006c8b0 2024-01-20T11:19:00.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.628+0000 7f5d927fc640 1 -- 172.21.15.110:0/3986626868 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5d8c057d10 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.628+0000 7f5d927fc640 10 client.4996.objecter ms_dispatch 0x7f5d9c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:00.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.628+0000 7f5d927fc640 3 client.4996.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:00.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.628+0000 7f5d927fc640 3 client.4996.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:00.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.628+0000 7f5d927fc640 20 client.4996.objecter dump_active .. 0 homeless 2024-01-20T11:19:00.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.638+0000 7f5d927fc640 1 -- 172.21.15.110:0/3986626868 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5d8c054050 con 0x7f5d9c0a96f0 2024-01-20T11:19:00.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.638+0000 7f5d927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:00.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.638+0000 7f5d927fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:00.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.834+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d9c0b4400 con 0x7f5d8006c8b0 2024-01-20T11:19:00.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.837+0000 7f5d927fc640 1 -- 172.21.15.110:0/3986626868 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63929 (secure 0 0 0) 0x7f5d9800f820 con 0x7f5d8006c8b0 2024-01-20T11:19:00.838 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:00.839 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:00.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 10 librados: watch_flush enter 2024-01-20T11:19:00.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 10 librados: watch_flush exit 2024-01-20T11:19:00.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 20 client.4996.objecter shutdown clearing up homeless session... 2024-01-20T11:19:00.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 10 client.4996.objecter successfully canceled tick 2024-01-20T11:19:00.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5d8006c8b0 msgr2=0x7f5d8006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5d8006c8b0 0x7f5d8006ed70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0b3750 tx=0x7f5d98009040 comp rx=0 tx=0).stop 2024-01-20T11:19:00.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 10 monclient: shutdown 2024-01-20T11:19:00.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:00.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d9c0a96f0 msgr2=0x7f5d9c135670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d9c0a96f0 0x7f5d9c135670 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00c430 tx=0x7f5d8c00c8f0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 shutdown_connections 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5d8006c8b0 0x7f5d8006ed70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d9c142af0 0x7f5d9c138dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d9c0a9ad0 0x7f5d9c136b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 --2- 172.21.15.110:0/3986626868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d9c0a96f0 0x7f5d9c135670 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 >> 172.21.15.110:0/3986626868 conn(0x7f5d9c0b7bc0 msgr2=0x7f5d9c1423e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 shutdown_connections 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 -- 172.21.15.110:0/3986626868 wait complete. 2024-01-20T11:19:00.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.839+0000 7f5da3aa2640 1 librados: shutdown 2024-01-20T11:19:00.850 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-01-20T11:19:00.752600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002238"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.6639999999999997},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":12.459}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.8179999999999996},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.0739999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":8.4849999999999994},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.227}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.390000000000001},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":13.096}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":9.8360000000000003},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":9.2240000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.175999999999998},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.154}]}]},{"osd":7,"up_from":21,"seq":90194313265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.6909999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.103}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.069},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.829000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.212},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.8040000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.298},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.454000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.395},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.531000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.710000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.3889999999999993}]}]},{"osd":3,"up_from":21,"seq":90194313266,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.133},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.7960000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":11.412000000000001},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.079000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":9.3699999999999992},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.113}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.380000000000001},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":7.8470000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":8.4719999999999995},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":14.656000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.476000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.708}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444080,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.1929999999999996},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":9.3290000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":13.417999999999999},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.218}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.1010000000000009},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.281000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":8.7650000000000006},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.965999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":10.023},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.760999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.542999999999999},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":12.683}]}]},{"osd":1,"up_from":17,"seq":73014444081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.9930000000000003},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":16.992999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.932},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.364000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":14.079000000000001},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.874000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":13.336},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.856}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.209},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":18.094999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.813000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":19.370000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444081,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":12.417},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":7.298}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.1669999999999998},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.042999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.384},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.9829999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.23},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":5.1680000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":15.134},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":11.859999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.081},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":9.2050000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":7.4130000000000003},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.25}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.3620000000000001},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":12.396000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":16.323}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.004},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.5589999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":10.218999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":17.282}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.753},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.597}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:00.852 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:00.852 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:00.852 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:00.852 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:00.852 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:00.852 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:00.852 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:00.852 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:00.853 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.952+0000 7fcb72d0f640 10 monclient: get_monmap_and_config 2024-01-20T11:19:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.952+0000 7fcb72d0f640 10 monclient: build_initial_monmap 2024-01-20T11:19:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.952+0000 7fcb72d0f640 10 monclient: monmap: 2024-01-20T11:19:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:00.954126+0000 2024-01-20T11:19:00.954 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:00.954126+0000 2024-01-20T11:19:00.954 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:00.954 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.954 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:00.954 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:00.954 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:00.954 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:00.954 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:00.954 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 1 Processor -- start 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 1 -- start start 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient: init 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c06bb00 0x7fcb6c0b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient(hunting): picked mon.noname-b con 0x7fcb6c06bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fcb6c0af790 0x7fcb6c145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6c0af790 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb6c0b0900 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c06c220 con 0x7fcb6c0b0900 2024-01-20T11:19:00.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6c06bee0 con 0x7fcb6c0af790 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0ada30 con 0x7fcb6c06bb00 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.953+0000 7fcb72d0f640 10 monclient(hunting): authenticate will time out at 1584.305067s 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb70aab640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c06bb00 0x7fcb6c0b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb6bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fcb6c0af790 0x7fcb6c145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb6bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fcb6c0af790 0x7fcb6c145770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:42988/0 (socket says 172.21.15.110:42988) 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb6bfff640 1 -- 172.21.15.110:0/3759295542 learned_addr learned my addr 172.21.15.110:0/3759295542 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb6bfff640 10 monclient(hunting): get_auth_request con 0x7fcb6c0af790 auth_method 0 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb6bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb70aab640 10 monclient(hunting): get_auth_request con 0x7fcb6c06bb00 auth_method 0 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb70aab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:00.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb70aab640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.954+0000 7fcb70aab640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.955+0000 7fcb70aab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:00.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.955+0000 7fcb70aab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:00.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.955+0000 7fcb70aab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:00.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.955+0000 7fcb6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:00.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.955+0000 7fcb6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:00.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.955+0000 7fcb6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:00.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.957+0000 7fcb70aab640 10 monclient(hunting): handle_auth_done global_id 4763 payload 274 2024-01-20T11:19:00.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.957+0000 7fcb70aab640 1 -- 172.21.15.110:0/3759295542 >> v2:172.21.15.134:3300/0 conn(0x7fcb6c0af790 msgr2=0x7fcb6c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:00.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.957+0000 7fcb70aab640 1 --2- 172.21.15.110:0/3759295542 >> v2:172.21.15.134:3300/0 conn(0x7fcb6c0af790 0x7fcb6c145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.957+0000 7fcb70aab640 1 -- 172.21.15.110:0/3759295542 >> v1:172.21.15.134:6789/0 conn(0x7fcb6c0b0900 legacy=0x7fcb6c145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.957+0000 7fcb70aab640 10 monclient: _finish_hunting 0 2024-01-20T11:19:00.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.957+0000 7fcb70aab640 1 monclient: found mon.noname-b 2024-01-20T11:19:00.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.957+0000 7fcb70aab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:00.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.957+0000 7fcb70aab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:00.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.957+0000 7fcb70aab640 1 -- 172.21.15.110:0/3759295542 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c0add10 con 0x7fcb6c06bb00 2024-01-20T11:19:00.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.959+0000 7fcb70aab640 1 --2- 172.21.15.110:0/3759295542 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c06bb00 0x7fcb6c0b03c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009d90 tx=0x7fcb6402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3e755aec1bc977 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:00.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 1 -- 172.21.15.110:0/3759295542 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6402cbc0 con 0x7fcb6c06bb00 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 10 monclient: dump: 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:00.964 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 1 -- 172.21.15.110:0/3759295542 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6402cd20 con 0x7fcb6c06bb00 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb72d0f640 5 monclient: authenticate success, global_id 4763 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb72d0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb72d0f640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb72d0f640 4 set_mon_vals no callback set 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 1 -- 172.21.15.110:0/3759295542 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb64034660 con 0x7fcb6c06bb00 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.962+0000 7fcb6b7fe640 10 monclient: dump: 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:00.965 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:00.966 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:00.966 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 10 monclient: shutdown 2024-01-20T11:19:00.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:00.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 1 -- 172.21.15.110:0/3759295542 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c06bb00 msgr2=0x7fcb6c0b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 1 --2- 172.21.15.110:0/3759295542 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c06bb00 0x7fcb6c0b03c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009d90 tx=0x7fcb6402be10 comp rx=0 tx=0).stop 2024-01-20T11:19:00.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 1 -- 172.21.15.110:0/3759295542 shutdown_connections 2024-01-20T11:19:00.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 1 --2- 172.21.15.110:0/3759295542 >> v2:172.21.15.134:3300/0 conn(0x7fcb6c0af790 0x7fcb6c145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 1 --2- 172.21.15.110:0/3759295542 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c06bb00 0x7fcb6c0b03c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 1 -- 172.21.15.110:0/3759295542 >> 172.21.15.110:0/3759295542 conn(0x7fcb6c066d60 msgr2=0x7fcb6c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 1 -- 172.21.15.110:0/3759295542 shutdown_connections 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.963+0000 7fcb72d0f640 1 -- 172.21.15.110:0/3759295542 wait complete. 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.964+0000 7fcb72d0f640 10 monclient: build_initial_monmap 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.964+0000 7fcb72d0f640 10 monclient: monmap: 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:00.971 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:00.972 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.964+0000 7fcb72d0f640 1 librados: starting msgr at 2024-01-20T11:19:00.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.964+0000 7fcb72d0f640 1 librados: starting objecter 2024-01-20T11:19:00.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 Processor -- start 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 -- start start 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 librados: setting wanted keys 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 librados: calling monclient init 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient: init 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb6c06bb00 0x7fcb6c1207a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient(hunting): picked mon.noname-b con 0x7fcb6c06bb00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0af790 0x7fcb6c1daab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6c0af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c0b0900 0x7fcb6c1daff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6c0b0900 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:00.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c121d50 con 0x7fcb6c0b0900 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c1da620 con 0x7fcb6c06bb00 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c121f50 con 0x7fcb6c0af790 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.965+0000 7fcb72d0f640 10 monclient(hunting): authenticate will time out at 1584.317103s 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb6bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0af790 0x7fcb6c1daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb70aab640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb6c06bb00 0x7fcb6c1207a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb712ac640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c0b0900 0x7fcb6c1daff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb70aab640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb6c06bb00 0x7fcb6c1207a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:42994/0 (socket says 172.21.15.110:42994) 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb70aab640 1 -- 172.21.15.110:0/1943864048 learned_addr learned my addr 172.21.15.110:0/1943864048 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb70aab640 10 monclient(hunting): get_auth_request con 0x7fcb6c06bb00 auth_method 0 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb70aab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb70aab640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb70aab640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb6bfff640 10 monclient(hunting): get_auth_request con 0x7fcb6c0af790 auth_method 0 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb6bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.966+0000 7fcb6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb712ac640 10 monclient(hunting): get_auth_request con 0x7fcb6c0b0900 auth_method 0 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb712ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb712ac640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb712ac640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb70aab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb70aab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb70aab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:00.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb712ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb712ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.967+0000 7fcb712ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 10 monclient(hunting): handle_auth_done global_id 5005 payload 824 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 1 -- 172.21.15.110:0/1943864048 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0af790 msgr2=0x7fcb6c1daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0af790 0x7fcb6c1daab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 1 -- 172.21.15.110:0/1943864048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c0b0900 msgr2=0x7fcb6c1daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c0b0900 0x7fcb6c1daff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 10 monclient: _finish_hunting 0 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 1 monclient: found mon.noname-b 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.969+0000 7fcb70aab640 1 -- 172.21.15.110:0/1943864048 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb64009a70 con 0x7fcb6c06bb00 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.971+0000 7fcb70aab640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb6c06bb00 0x7fcb6c1207a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009d90 tx=0x7fcb64004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.971+0000 7fcb697fa640 10 client.?.objecter ms_handle_connect 0x7fcb6c06bb00 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.971+0000 7fcb697fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb697fa640 1 -- 172.21.15.110:0/1943864048 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6403e610 con 0x7fcb6c06bb00 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb697fa640 10 monclient: dump: 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb697fa640 10 monclient: _finish_auth 0 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb697fa640 10 monclient: _check_auth_tickets 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb697fa640 1 -- 172.21.15.110:0/1943864048 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6402ce90 con 0x7fcb6c06bb00 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb72d0f640 5 monclient: authenticate success, global_id 5005 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb72d0f640 10 monclient: _renew_subs 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb72d0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.976+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c11f880 con 0x7fcb6c06bb00 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb697fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:00.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb697fa640 1 -- 172.21.15.110:0/1943864048 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6403e9d0 con 0x7fcb6c06bb00 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb697fa640 10 monclient: dump: 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb6affd640 4 set_mon_vals no callback set 2024-01-20T11:19:00.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb72d0f640 10 client.5005.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb72d0f640 10 monclient: _renew_subs 2024-01-20T11:19:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb72d0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c120ce0 con 0x7fcb6c06bb00 2024-01-20T11:19:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb72d0f640 1 librados: init done 2024-01-20T11:19:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb72d0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb72d0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb72d0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.977+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c075490 con 0x7fcb6c06bb00 2024-01-20T11:19:00.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.995+0000 7fcb697fa640 1 -- 172.21.15.110:0/1943864048 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fcb6403b050 con 0x7fcb6c06bb00 2024-01-20T11:19:00.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.995+0000 7fcb697fa640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb4c06c8b0 0x7fcb4c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.996+0000 7fcb6bfff640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb4c06c8b0 0x7fcb4c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.996+0000 7fcb6bfff640 10 monclient: get_auth_request con 0x7fcb4c06c8b0 auth_method 0 2024-01-20T11:19:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.996+0000 7fcb6bfff640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb4c06c8b0 0x7fcb4c06ed70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c121560 tx=0x7fcb54009210 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.996+0000 7fcb697fa640 10 client.5005.objecter ms_handle_connect 0x7fcb4c06c8b0 2024-01-20T11:19:00.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.998+0000 7fcb697fa640 1 -- 172.21.15.110:0/1943864048 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcb6407b000 con 0x7fcb6c06bb00 2024-01-20T11:19:00.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.998+0000 7fcb697fa640 10 client.5005.objecter ms_dispatch 0x7fcb6c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:00.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.998+0000 7fcb697fa640 3 client.5005.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:00.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.998+0000 7fcb697fa640 3 client.5005.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:00.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:00.998+0000 7fcb697fa640 20 client.5005.objecter dump_active .. 0 homeless 2024-01-20T11:19:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.233+0000 7fcb697fa640 1 -- 172.21.15.110:0/1943864048 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fcb64077340 con 0x7fcb6c06bb00 2024-01-20T11:19:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.233+0000 7fcb697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.233+0000 7fcb697fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:01.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.472+0000 7fcb72d0f640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:01.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.472+0000 7fcb72d0f640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:01.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.472+0000 7fcb72d0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:01.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.472+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fcb6c120ed0 con 0x7fcb6c06bb00 2024-01-20T11:19:01.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.478+0000 7fcb697fa640 1 -- 172.21.15.110:0/1943864048 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fcb6404e520 con 0x7fcb6c06bb00 2024-01-20T11:19:01.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.478+0000 7fcb697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.478+0000 7fcb697fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:01.479 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:01.479 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":276,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.479+0000 7fcb72d0f640 10 librados: watch_flush enter 2024-01-20T11:19:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.479+0000 7fcb72d0f640 10 librados: watch_flush exit 2024-01-20T11:19:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.479+0000 7fcb72d0f640 20 client.5005.objecter shutdown clearing up homeless session... 2024-01-20T11:19:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.479+0000 7fcb72d0f640 10 client.5005.objecter successfully canceled tick 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb4c06c8b0 msgr2=0x7fcb4c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb4c06c8b0 0x7fcb4c06ed70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c121560 tx=0x7fcb54009210 comp rx=0 tx=0).stop 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 10 monclient: shutdown 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb6c06bb00 msgr2=0x7fcb6c1207a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb6c06bb00 0x7fcb6c1207a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009d90 tx=0x7fcb64004a40 comp rx=0 tx=0).stop 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 shutdown_connections 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb4c06c8b0 0x7fcb4c06ed70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c0b0900 0x7fcb6c1daff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0af790 0x7fcb6c1daab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:01.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 --2- 172.21.15.110:0/1943864048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb6c06bb00 0x7fcb6c1207a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:01.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 >> 172.21.15.110:0/1943864048 conn(0x7fcb6c066d60 msgr2=0x7fcb6c144dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:01.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 shutdown_connections 2024-01-20T11:19:01.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 -- 172.21.15.110:0/1943864048 wait complete. 2024-01-20T11:19:01.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.480+0000 7fcb72d0f640 1 librados: shutdown 2024-01-20T11:19:01.491 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:01.492 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.589+0000 7f0ee1d33640 10 monclient: get_monmap_and_config 2024-01-20T11:19:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.589+0000 7f0ee1d33640 10 monclient: build_initial_monmap 2024-01-20T11:19:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.589+0000 7f0ee1d33640 10 monclient: monmap: 2024-01-20T11:19:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:01.591155+0000 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:01.591155+0000 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:01.591 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 1 Processor -- start 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 1 -- start start 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 10 monclient: init 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0edc0b0c00 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0edc0accf0 0x7f0edc143200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 10 monclient(hunting): picked mon.noname-c con 0x7f0edc0accf0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0edc143740 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:01.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0edc0ad450 con 0x7f0edc0b0c00 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0edc0ad8e0 con 0x7f0edc143740 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0edc0ad130 con 0x7f0edc0accf0 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.590+0000 7f0ee1d33640 10 monclient(hunting): authenticate will time out at 1584.942114s 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.591+0000 7f0edaffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0edc0accf0 0x7f0edc143200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.591+0000 7f0edb7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0edc0b0c00 0x7f0edc13d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36304/0 (socket says 172.21.15.110:36304) 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.591+0000 7f0edb7fe640 1 -- 172.21.15.110:0/4258195115 learned_addr learned my addr 172.21.15.110:0/4258195115 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.591+0000 7f0edaffd640 10 monclient(hunting): get_auth_request con 0x7f0edc0accf0 auth_method 0 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.591+0000 7f0edaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.591+0000 7f0edaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:01.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.591+0000 7f0edaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:01.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.592+0000 7f0edaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:01.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.592+0000 7f0edaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:01.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.592+0000 7f0edaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.593+0000 7f0eda7fc640 1 -- 172.21.15.110:0/4258195115 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234261193 0 0) 0x7f0edc0ad450 con 0x7f0edc0b0c00 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0eda7fc640 10 monclient(hunting): _negotiate 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0eda7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0eda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0eda7fc640 10 monclient(hunting): my global_id is 4920 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0eda7fc640 1 -- 172.21.15.110:0/4258195115 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec4003520 con 0x7f0edc0b0c00 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0edaffd640 10 monclient(hunting): handle_auth_done global_id 5008 payload 274 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0edaffd640 1 -- 172.21.15.110:0/4258195115 >> v1:172.21.15.134:6789/0 conn(0x7f0edc143740 legacy=0x7f0edc14a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0edaffd640 1 -- 172.21.15.110:0/4258195115 >> v1:172.21.15.110:6789/0 conn(0x7f0edc0b0c00 legacy=0x7f0edc13d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0edaffd640 10 monclient: _finish_hunting 0 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0edaffd640 1 monclient: found mon.noname-c 2024-01-20T11:19:01.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0edaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:01.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0edaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:19:01.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.594+0000 7f0edaffd640 1 -- 172.21.15.110:0/4258195115 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0edc0aa300 con 0x7f0edc0accf0 2024-01-20T11:19:01.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.597+0000 7f0edaffd640 1 --2- 172.21.15.110:0/4258195115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 0x7f0edc143200 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00097a0 tx=0x7f0ed002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b953a753e0d12592 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 1 -- 172.21.15.110:0/4258195115 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed002f050 con 0x7f0edc0accf0 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 10 monclient: dump: 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:01.602 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 1 -- 172.21.15.110:0/4258195115 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed0031700 con 0x7f0edc0accf0 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 1 -- 172.21.15.110:0/4258195115 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0031a90 con 0x7f0edc0accf0 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 5 monclient: authenticate success, global_id 5008 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 4 set_mon_vals no callback set 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:01.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0eda7fc640 10 monclient: dump: 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:01.604 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 10 monclient: shutdown 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 1 -- 172.21.15.110:0/4258195115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 msgr2=0x7f0edc143200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 1 --2- 172.21.15.110:0/4258195115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 0x7f0edc143200 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00097a0 tx=0x7f0ed002b7a0 comp rx=0 tx=0).stop 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 1 -- 172.21.15.110:0/4258195115 shutdown_connections 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 1 --2- 172.21.15.110:0/4258195115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 0x7f0edc143200 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 1 -- 172.21.15.110:0/4258195115 >> 172.21.15.110:0/4258195115 conn(0x7f0edc0b3e60 msgr2=0x7f0edc0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 1 -- 172.21.15.110:0/4258195115 shutdown_connections 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.601+0000 7f0ee1d33640 1 -- 172.21.15.110:0/4258195115 wait complete. 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.602+0000 7f0ee1d33640 10 monclient: build_initial_monmap 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.602+0000 7f0ee1d33640 10 monclient: monmap: 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:01.608 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.602+0000 7f0ee1d33640 1 librados: starting msgr at 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.602+0000 7f0ee1d33640 1 librados: starting objecter 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.602+0000 7f0ee1d33640 1 Processor -- start 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 1 -- start start 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 1 librados: setting wanted keys 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 1 librados: calling monclient init 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient: init 2024-01-20T11:19:01.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 0x7f0edc0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient(hunting): picked mon.noname-b con 0x7f0edc0accf0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0edc0b0c00 0x7f0edc1387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient(hunting): picked mon.noname-c con 0x7f0edc0b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0edc143740 0x7f0edc1d4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient(hunting): picked mon.noname-a con 0x7f0edc143740 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc11b530 con 0x7f0edc143740 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc11b780 con 0x7f0edc0accf0 2024-01-20T11:19:01.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc11ba60 con 0x7f0edc0b0c00 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.603+0000 7f0ee1d33640 10 monclient(hunting): authenticate will time out at 1584.954959s 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0edc143740 0x7f0edc1d4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edaffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0edc0b0c00 0x7f0edc1387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edb7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 0x7f0edc0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0edc143740 0x7f0edc1d4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:46550/0 (socket says 172.21.15.110:46550) 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edbfff640 1 -- 172.21.15.110:0/1113269092 learned_addr learned my addr 172.21.15.110:0/1113269092 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edbfff640 10 monclient(hunting): get_auth_request con 0x7f0edc143740 auth_method 0 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edb7fe640 10 monclient(hunting): get_auth_request con 0x7f0edc0accf0 auth_method 0 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:01.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edaffd640 10 monclient(hunting): get_auth_request con 0x7f0edc0b0c00 auth_method 0 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.604+0000 7f0edaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.605+0000 7f0edb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.605+0000 7f0edb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.605+0000 7f0edb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.605+0000 7f0edbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.605+0000 7f0edbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.605+0000 7f0edbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.605+0000 7f0edaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:01.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.605+0000 7f0edaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:01.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.605+0000 7f0edaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:01.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 10 monclient(hunting): handle_auth_done global_id 5011 payload 824 2024-01-20T11:19:01.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 1 -- 172.21.15.110:0/1113269092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0edc0b0c00 msgr2=0x7f0edc1387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:01.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0edc0b0c00 0x7f0edc1387c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 1 -- 172.21.15.110:0/1113269092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0edc143740 msgr2=0x7f0edc1d4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0edc143740 0x7f0edc1d4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 1 monclient: found mon.noname-b 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.607+0000 7f0edb7fe640 1 -- 172.21.15.110:0/1113269092 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed0009400 con 0x7f0edc0accf0 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.609+0000 7f0edb7fe640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 0x7f0edc0709a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00c400 tx=0x7f0ecc00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.610+0000 7f0ec3fff640 10 client.?.objecter ms_handle_connect 0x7f0edc0accf0 2024-01-20T11:19:01.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.610+0000 7f0ec3fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:01.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 1 -- 172.21.15.110:0/1113269092 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc005ce0 con 0x7f0edc0accf0 2024-01-20T11:19:01.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 10 monclient: dump: 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:01.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 10 monclient: _finish_auth 0 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 1 -- 172.21.15.110:0/1113269092 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ecc005e40 con 0x7f0edc0accf0 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 1 -- 172.21.15.110:0/1113269092 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0072f0 con 0x7f0edc0accf0 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ec3fff640 10 monclient: dump: 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:01.617 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ee1d33640 5 monclient: authenticate success, global_id 5011 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ee1d33640 10 monclient: _renew_subs 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ed97fa640 4 set_mon_vals no callback set 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ee1d33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0edc071b50 con 0x7f0edc0accf0 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ee1d33640 10 client.5011.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ee1d33640 10 monclient: _renew_subs 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ee1d33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0edc071740 con 0x7f0edc0accf0 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.613+0000 7f0ee1d33640 1 librados: init done 2024-01-20T11:19:01.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.614+0000 7f0ee1d33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:01.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.614+0000 7f0ee1d33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:01.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.614+0000 7f0ee1d33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:01.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.614+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0edc071990 con 0x7f0edc0accf0 2024-01-20T11:19:01.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.633+0000 7f0ec3fff640 1 -- 172.21.15.110:0/1113269092 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f0ecc007450 con 0x7f0edc0accf0 2024-01-20T11:19:01.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.633+0000 7f0ec3fff640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0ec806c8b0 0x7f0ec806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:01.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.633+0000 7f0edaffd640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0ec806c8b0 0x7f0ec806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:01.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.634+0000 7f0edaffd640 10 monclient: get_auth_request con 0x7f0ec806c8b0 auth_method 0 2024-01-20T11:19:01.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.634+0000 7f0edaffd640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0ec806c8b0 0x7f0ec806ed70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0035000 tx=0x7f0ed0008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:01.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.634+0000 7f0ec3fff640 10 client.5011.objecter ms_handle_connect 0x7f0ec806c8b0 2024-01-20T11:19:01.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.854+0000 7f0ec3fff640 1 -- 172.21.15.110:0/1113269092 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0ecc057f20 con 0x7f0edc0accf0 2024-01-20T11:19:01.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.854+0000 7f0ec3fff640 10 client.5011.objecter ms_dispatch 0x7f0edc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:01.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.854+0000 7f0ec3fff640 3 client.5011.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:01.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.854+0000 7f0ec3fff640 3 client.5011.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:01.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.854+0000 7f0ec3fff640 20 client.5011.objecter dump_active .. 0 homeless 2024-01-20T11:19:01.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.864+0000 7f0ec3fff640 1 -- 172.21.15.110:0/1113269092 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0ecc054260 con 0x7f0edc0accf0 2024-01-20T11:19:01.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.864+0000 7f0ec3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:01.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:01.864+0000 7f0ec3fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:02.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.044+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0edc143ba0 con 0x7f0ec806c8b0 2024-01-20T11:19:02.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.046+0000 7f0ec3fff640 1 -- 172.21.15.110:0/1113269092 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63929 (secure 0 0 0) 0x7f0ed0041070 con 0x7f0ec806c8b0 2024-01-20T11:19:02.048 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:02.048 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:02.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 10 librados: watch_flush enter 2024-01-20T11:19:02.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 10 librados: watch_flush exit 2024-01-20T11:19:02.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 20 client.5011.objecter shutdown clearing up homeless session... 2024-01-20T11:19:02.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 10 client.5011.objecter successfully canceled tick 2024-01-20T11:19:02.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0ec806c8b0 msgr2=0x7f0ec806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:02.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0ec806c8b0 0x7f0ec806ed70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0035000 tx=0x7f0ed0008030 comp rx=0 tx=0).stop 2024-01-20T11:19:02.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 10 monclient: shutdown 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 msgr2=0x7f0edc0709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 0x7f0edc0709a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00c400 tx=0x7f0ecc00c8c0 comp rx=0 tx=0).stop 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 shutdown_connections 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0ec806c8b0 0x7f0ec806ed70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0edc143740 0x7f0edc1d4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0edc0b0c00 0x7f0edc1387c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 --2- 172.21.15.110:0/1113269092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0edc0accf0 0x7f0edc0709a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.048+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 >> 172.21.15.110:0/1113269092 conn(0x7f0edc0b3e60 msgr2=0x7f0edc0ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.049+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 shutdown_connections 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.049+0000 7f0ee1d33640 1 -- 172.21.15.110:0/1113269092 wait complete. 2024-01-20T11:19:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:02.049+0000 7f0ee1d33640 1 librados: shutdown 2024-01-20T11:19:02.060 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-01-20T11:19:00.752600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002238"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.6639999999999997},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":12.459}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.8179999999999996},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.0739999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":8.4849999999999994},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.227}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.390000000000001},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":13.096}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":9.8360000000000003},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":9.2240000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.175999999999998},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.154}]}]},{"osd":7,"up_from":21,"seq":90194313265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.6909999999999998},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.103}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.069},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.829000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.212},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.8040000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.298},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.454000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.395},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.531000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.710000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":8.3889999999999993}]}]},{"osd":3,"up_from":21,"seq":90194313266,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.133},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.7960000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":11.412000000000001},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.079000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":9.3699999999999992},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.113}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.380000000000001},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":7.8470000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":8.4719999999999995},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":14.656000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.476000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.708}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444080,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.1929999999999996},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":9.3290000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":13.417999999999999},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.218}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.1010000000000009},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.281000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":8.7650000000000006},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.965999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":10.023},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.760999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.542999999999999},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":12.683}]}]},{"osd":1,"up_from":17,"seq":73014444081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.9930000000000003},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":16.992999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.932},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.364000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":14.079000000000001},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.874000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":13.336},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.856}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.209},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":18.094999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.813000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":19.370000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444081,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":12.417},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":7.298}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.1669999999999998},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.042999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.384},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.9829999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.23},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":5.1680000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":15.134},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":11.859999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.081},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":9.2050000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":7.4130000000000003},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.25}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.3620000000000001},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":12.396000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":16.323}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.004},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.5589999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":10.218999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":17.282}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.753},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.597}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:04.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~6s 2024-01-20T11:19:05.063 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:05.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.166+0000 7f67d7415640 10 monclient: get_monmap_and_config 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.166+0000 7f67d7415640 10 monclient: build_initial_monmap 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.166+0000 7f67d7415640 10 monclient: monmap: 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:05.168020+0000 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:05.168020+0000 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:05.168 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:05.169 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:05.169 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.166+0000 7f67d7415640 1 Processor -- start 2024-01-20T11:19:05.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 1 -- start start 2024-01-20T11:19:05.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient: init 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f67d00a96f0 0x7f67d013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient(hunting): picked mon.noname-a con 0x7f67d00a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67d013fa10 0x7f67d01423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient(hunting): picked mon.noname-b con 0x7f67d013fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f67d00a9ad0 0x7f67d0142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient(hunting): picked mon.noname-c con 0x7f67d00a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d00aaea0 con 0x7f67d00a96f0 2024-01-20T11:19:05.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d00b25c0 con 0x7f67d00a9ad0 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d00ab0a0 con 0x7f67d013fa10 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.167+0000 7f67d7415640 10 monclient(hunting): authenticate will time out at 1588.518968s 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d51b1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f67d00a96f0 0x7f67d013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d59b2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f67d00a9ad0 0x7f67d0142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d51b1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f67d00a96f0 0x7f67d013f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:46558/0 (socket says 172.21.15.110:46558) 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d51b1640 1 -- 172.21.15.110:0/118355896 learned_addr learned my addr 172.21.15.110:0/118355896 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d51b1640 10 monclient(hunting): get_auth_request con 0x7f67d00a96f0 auth_method 0 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d51b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d51b1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d51b1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d59b2640 10 monclient(hunting): get_auth_request con 0x7f67d00a9ad0 auth_method 0 2024-01-20T11:19:05.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d59b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:05.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d59b2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.168+0000 7f67d59b2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.169+0000 7f67d51b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:05.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.169+0000 7f67d51b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:05.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.169+0000 7f67d51b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:05.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.169+0000 7f67d59b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:05.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.169+0000 7f67d59b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:05.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.169+0000 7f67d59b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 10 monclient(hunting): handle_auth_done global_id 4926 payload 274 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 1 -- 172.21.15.110:0/118355896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67d013fa10 msgr2=0x7f67d01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 1 --2- 172.21.15.110:0/118355896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67d013fa10 0x7f67d01423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 1 -- 172.21.15.110:0/118355896 >> v2:172.21.15.134:3300/0 conn(0x7f67d00a9ad0 msgr2=0x7f67d0142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 1 --2- 172.21.15.110:0/118355896 >> v2:172.21.15.134:3300/0 conn(0x7f67d00a9ad0 0x7f67d0142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 10 monclient: _finish_hunting 0 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 1 monclient: found mon.noname-a 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.170+0000 7f67d51b1640 1 -- 172.21.15.110:0/118355896 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d00b3fb0 con 0x7f67d00a96f0 2024-01-20T11:19:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.172+0000 7f67d51b1640 1 --2- 172.21.15.110:0/118355896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a96f0 0x7f67d013f4d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f67d00749e0 tx=0x7f67cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32291e110a461aaa server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 1 -- 172.21.15.110:0/118355896 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc0317e0 con 0x7f67d00a96f0 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 10 monclient: dump: 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 10 monclient: _finish_auth 0 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 1 -- 172.21.15.110:0/118355896 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67cc031940 con 0x7f67d00a96f0 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 1 -- 172.21.15.110:0/118355896 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc031cd0 con 0x7f67d00a96f0 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67d7415640 5 monclient: authenticate success, global_id 4926 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67d7415640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67d7415640 4 set_mon_vals no callback set 2024-01-20T11:19:05.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67c7fff640 10 monclient: dump: 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67d7415640 10 monclient: shutdown 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67d7415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67d7415640 1 -- 172.21.15.110:0/118355896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a96f0 msgr2=0x7f67d013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.177+0000 7f67d7415640 1 --2- 172.21.15.110:0/118355896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a96f0 0x7f67d013f4d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f67d00749e0 tx=0x7f67cc02b880 comp rx=0 tx=0).stop 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.178+0000 7f67d7415640 1 -- 172.21.15.110:0/118355896 shutdown_connections 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.178+0000 7f67d7415640 1 --2- 172.21.15.110:0/118355896 >> v2:172.21.15.134:3300/0 conn(0x7f67d00a9ad0 0x7f67d0142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.178+0000 7f67d7415640 1 --2- 172.21.15.110:0/118355896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67d013fa10 0x7f67d01423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.178+0000 7f67d7415640 1 --2- 172.21.15.110:0/118355896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a96f0 0x7f67d013f4d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.178+0000 7f67d7415640 1 -- 172.21.15.110:0/118355896 >> 172.21.15.110:0/118355896 conn(0x7f67d00b7bc0 msgr2=0x7f67d00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.178+0000 7f67d7415640 1 -- 172.21.15.110:0/118355896 shutdown_connections 2024-01-20T11:19:05.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.178+0000 7f67d7415640 1 -- 172.21.15.110:0/118355896 wait complete. 2024-01-20T11:19:05.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.178+0000 7f67d7415640 10 monclient: build_initial_monmap 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 10 monclient: monmap: 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 1 librados: starting msgr at 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 1 librados: starting objecter 2024-01-20T11:19:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 1 Processor -- start 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 1 -- start start 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 1 librados: setting wanted keys 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 1 librados: calling monclient init 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 10 monclient: init 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67d00a96f0 0x7f67d01cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 10 monclient(hunting): picked mon.noname-c con 0x7f67d00a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a9ad0 0x7f67d00b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.179+0000 7f67d7415640 10 monclient(hunting): picked mon.noname-a con 0x7f67d00a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67d013e890 0x7f67d01cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 10 monclient(hunting): picked mon.noname-b con 0x7f67d013e890 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d00ab620 con 0x7f67d00a9ad0 2024-01-20T11:19:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d01cc660 con 0x7f67d013e890 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d01da560 con 0x7f67d00a96f0 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d7415640 10 monclient(hunting): authenticate will time out at 1588.531209s 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d49b0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a9ad0 0x7f67d00b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d59b2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67d013e890 0x7f67d01cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d49b0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a9ad0 0x7f67d00b0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:46568/0 (socket says 172.21.15.110:46568) 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d49b0640 1 -- 172.21.15.110:0/2758701926 learned_addr learned my addr 172.21.15.110:0/2758701926 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d49b0640 10 monclient(hunting): get_auth_request con 0x7f67d00a9ad0 auth_method 0 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d49b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d49b0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d49b0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d59b2640 10 monclient(hunting): get_auth_request con 0x7f67d013e890 auth_method 0 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d59b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:05.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d59b2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.180+0000 7f67d59b2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.181+0000 7f67d49b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.181+0000 7f67d49b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.181+0000 7f67d49b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.181+0000 7f67d59b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.181+0000 7f67d59b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.181+0000 7f67d59b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.183+0000 7f67d49b0640 10 monclient(hunting): handle_auth_done global_id 4929 payload 824 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.183+0000 7f67d49b0640 1 -- 172.21.15.110:0/2758701926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67d00a96f0 msgr2=0x7f67d01cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.183+0000 7f67d49b0640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67d00a96f0 0x7f67d01cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.184+0000 7f67d49b0640 1 -- 172.21.15.110:0/2758701926 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67d013e890 msgr2=0x7f67d01cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.184+0000 7f67d49b0640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67d013e890 0x7f67d01cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.184+0000 7f67d49b0640 10 monclient: _finish_hunting 0 2024-01-20T11:19:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.184+0000 7f67d49b0640 1 monclient: found mon.noname-a 2024-01-20T11:19:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.184+0000 7f67d49b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.184+0000 7f67d49b0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.184+0000 7f67d49b0640 1 -- 172.21.15.110:0/2758701926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc0094d0 con 0x7f67d00a9ad0 2024-01-20T11:19:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.186+0000 7f67d49b0640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a9ad0 0x7f67d00b0be0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f67c000b390 tx=0x7f67c000b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.186+0000 7f67c5ffb640 10 client.?.objecter ms_handle_connect 0x7f67d00a9ad0 2024-01-20T11:19:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.186+0000 7f67c5ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 1 -- 172.21.15.110:0/2758701926 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c0011830 con 0x7f67d00a9ad0 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 10 monclient: dump: 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 10 monclient: _finish_auth 0 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67d7415640 5 monclient: authenticate success, global_id 4929 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 1 -- 172.21.15.110:0/2758701926 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c0011990 con 0x7f67d00a9ad0 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 1 -- 172.21.15.110:0/2758701926 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c0011d20 con 0x7f67d00a9ad0 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c5ffb640 10 monclient: dump: 2024-01-20T11:19:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67c77fe640 4 set_mon_vals no callback set 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67d7415640 10 monclient: _renew_subs 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67d7415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:05.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d01cb980 con 0x7f67d00a9ad0 2024-01-20T11:19:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67d7415640 10 client.4929.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67d7415640 10 monclient: _renew_subs 2024-01-20T11:19:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67d7415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d01cbe10 con 0x7f67d00a9ad0 2024-01-20T11:19:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.189+0000 7f67d7415640 1 librados: init done 2024-01-20T11:19:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.190+0000 7f67d7415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.190+0000 7f67d7415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.190+0000 7f67d7415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:05.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.190+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d0075490 con 0x7f67d00a9ad0 2024-01-20T11:19:05.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.210+0000 7f67c5ffb640 1 -- 172.21.15.110:0/2758701926 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f67c001a400 con 0x7f67d00a9ad0 2024-01-20T11:19:05.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.210+0000 7f67c5ffb640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67b406c8b0 0x7f67b406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.211+0000 7f67d51b1640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67b406c8b0 0x7f67b406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.211+0000 7f67d51b1640 10 monclient: get_auth_request con 0x7f67b406c8b0 auth_method 0 2024-01-20T11:19:05.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.211+0000 7f67c5ffb640 10 client.4929.objecter ms_handle_connect 0x7f67b406c8b0 2024-01-20T11:19:05.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.211+0000 7f67d51b1640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67b406c8b0 0x7f67b406ed70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f67cc036000 tx=0x7f67cc030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:05.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.213+0000 7f67c5ffb640 1 -- 172.21.15.110:0/2758701926 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f67c0058ec0 con 0x7f67d00a9ad0 2024-01-20T11:19:05.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.213+0000 7f67c5ffb640 10 client.4929.objecter ms_dispatch 0x7f67d0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:05.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.213+0000 7f67c5ffb640 3 client.4929.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:05.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.213+0000 7f67c5ffb640 3 client.4929.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:05.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.213+0000 7f67c5ffb640 20 client.4929.objecter dump_active .. 0 homeless 2024-01-20T11:19:05.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.444+0000 7f67c5ffb640 1 -- 172.21.15.110:0/2758701926 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f67c0054b00 con 0x7f67d00a9ad0 2024-01-20T11:19:05.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.444+0000 7f67c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:05.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.444+0000 7f67c5ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:05.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.615+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67d00b4400 con 0x7f67b406c8b0 2024-01-20T11:19:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.619+0000 7f67c5ffb640 1 -- 172.21.15.110:0/2758701926 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63954 (secure 0 0 0) 0x7f67cc031ab0 con 0x7f67b406c8b0 2024-01-20T11:19:05.620 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 10 librados: watch_flush enter 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 10 librados: watch_flush exit 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 20 client.4929.objecter shutdown clearing up homeless session... 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 10 client.4929.objecter successfully canceled tick 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67b406c8b0 msgr2=0x7f67b406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67b406c8b0 0x7f67b406ed70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f67cc036000 tx=0x7f67cc030000 comp rx=0 tx=0).stop 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 10 monclient: shutdown 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a9ad0 msgr2=0x7f67d00b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:05.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a9ad0 0x7f67d00b0be0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f67c000b390 tx=0x7f67c000b850 comp rx=0 tx=0).stop 2024-01-20T11:19:05.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 shutdown_connections 2024-01-20T11:19:05.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67b406c8b0 0x7f67b406ed70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67d013e890 0x7f67d01cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67d00a9ad0 0x7f67d00b0be0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 --2- 172.21.15.110:0/2758701926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67d00a96f0 0x7f67d01cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 >> 172.21.15.110:0/2758701926 conn(0x7f67d00b7bc0 msgr2=0x7f67d01423e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:05.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 shutdown_connections 2024-01-20T11:19:05.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.620+0000 7f67d7415640 1 -- 172.21.15.110:0/2758701926 wait complete. 2024-01-20T11:19:05.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.621+0000 7f67d7415640 1 librados: shutdown 2024-01-20T11:19:05.632 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-01-20T11:19:04.753469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002365"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.9160000000000004},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":9.4309999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":10.742000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.617}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.58},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":7.968}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.048},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":12.791}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.089},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":8.8030000000000008}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.952999999999999},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":14.31}]}]},{"osd":7,"up_from":21,"seq":90194313266,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.3700000000000001},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.403}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.0540000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.401999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.644},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.3490000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.0180000000000007},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.789999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.6470000000000002},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":11.999000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.632},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.7360000000000007}]}]},{"osd":3,"up_from":21,"seq":90194313267,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.9409999999999998},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.3339999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7550000000000008},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.199000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":8.5809999999999995},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.962}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.606999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.101000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":10.682},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":13.548999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.484999999999999},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.863}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.6400000000000006},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":14.542999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.9740000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.3030000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.263},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.8609999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.846},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.994999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.771000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":8.0510000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.483000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":13.18}]}]},{"osd":1,"up_from":17,"seq":73014444082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.682},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":5.8959999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":7.5199999999999996},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.875999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":14.744},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":12.214}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":8.1920000000000002},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.519}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.8209999999999997},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":9.5709999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.507},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":8.7940000000000005}]}]},{"osd":0,"up_from":17,"seq":73014444082,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":7.524},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":12.279999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.7350000000000003},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.1969999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.436},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.8719999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.628},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.4340000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":10.156000000000001},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.914}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.867000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":9.4600000000000009}]}]},{"osd":6,"up_from":15,"seq":64424509490,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":4.7380000000000004},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":10.125}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.5289999999999999},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":8.6620000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.324999999999999},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.093}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.736000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.875}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.77},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.641999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.944000000000001},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":9.3379999999999992}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:05.633 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:05.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.731+0000 7fb9668f2640 10 monclient: get_monmap_and_config 2024-01-20T11:19:05.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.731+0000 7fb9668f2640 10 monclient: build_initial_monmap 2024-01-20T11:19:05.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.731+0000 7fb9668f2640 10 monclient: monmap: 2024-01-20T11:19:05.732 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:05.733271+0000 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:05.733271+0000 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:05.733 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.732+0000 7fb9668f2640 1 Processor -- start 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.732+0000 7fb9668f2640 1 -- start start 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.732+0000 7fb9668f2640 10 monclient: init 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb9600b0c00 0x7fb9600accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 10 monclient(hunting): picked mon.noname-c con 0x7fb9600b0c00 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb96013d0e0 0x7fb96013d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 10 monclient(hunting): picked mon.noname-b con 0x7fb96013d0e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb960143200 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96006dcc0 con 0x7fb960143200 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9600ad450 con 0x7fb9600b0c00 2024-01-20T11:19:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600ad8e0 con 0x7fb96013d0e0 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb9668f2640 10 monclient(hunting): authenticate will time out at 1589.084317s 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb95f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb96013d0e0 0x7fb96013d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb964e8f640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb960143200 0x7fb960143690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36320/0 (socket says 172.21.15.110:36320) 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb964e8f640 1 -- 172.21.15.110:0/969485734 learned_addr learned my addr 172.21.15.110:0/969485734 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.733+0000 7fb95ffff640 1 --2- 172.21.15.110:0/969485734 >> v2:172.21.15.134:3300/0 conn(0x7fb9600b0c00 0x7fb9600accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95f7fe640 10 monclient(hunting): get_auth_request con 0x7fb96013d0e0 auth_method 0 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95ffff640 10 monclient(hunting): get_auth_request con 0x7fb9600b0c00 auth_method 0 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:05.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:05.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:05.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:05.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:05.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.734+0000 7fb95ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:05.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95effd640 1 -- 172.21.15.110:0/969485734 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709191833 0 0) 0x7fb96006dcc0 con 0x7fb960143200 2024-01-20T11:19:05.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95effd640 10 monclient(hunting): _negotiate 2024-01-20T11:19:05.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95effd640 10 monclient(hunting): my global_id is 4932 2024-01-20T11:19:05.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95effd640 1 -- 172.21.15.110:0/969485734 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb948003520 con 0x7fb960143200 2024-01-20T11:19:05.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95f7fe640 10 monclient(hunting): handle_auth_done global_id 4772 payload 274 2024-01-20T11:19:05.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95f7fe640 1 -- 172.21.15.110:0/969485734 >> v2:172.21.15.134:3300/0 conn(0x7fb9600b0c00 msgr2=0x7fb9600accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95f7fe640 1 --2- 172.21.15.110:0/969485734 >> v2:172.21.15.134:3300/0 conn(0x7fb9600b0c00 0x7fb9600accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95f7fe640 1 -- 172.21.15.110:0/969485734 >> v1:172.21.15.110:6789/0 conn(0x7fb960143200 legacy=0x7fb960143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95f7fe640 1 monclient: found mon.noname-b 2024-01-20T11:19:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.736+0000 7fb95f7fe640 1 -- 172.21.15.110:0/969485734 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb960143da0 con 0x7fb96013d0e0 2024-01-20T11:19:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.737+0000 7fb95f7fe640 1 --2- 172.21.15.110:0/969485734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb96013d0e0 0x7fb96013d520 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb9540097a0 tx=0x7fb95402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89db48588f6a8c08 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 1 -- 172.21.15.110:0/969485734 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95402e070 con 0x7fb96013d0e0 2024-01-20T11:19:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 10 monclient: dump: 2024-01-20T11:19:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 10 monclient: _finish_auth 0 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb9668f2640 5 monclient: authenticate success, global_id 4772 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 1 -- 172.21.15.110:0/969485734 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9540315a0 con 0x7fb96013d0e0 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb9668f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb9668f2640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb9668f2640 4 set_mon_vals no callback set 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 1 -- 172.21.15.110:0/969485734 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb954031930 con 0x7fb96013d0e0 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.741+0000 7fb95effd640 10 monclient: dump: 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:05.744 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 10 monclient: shutdown 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 1 -- 172.21.15.110:0/969485734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb96013d0e0 msgr2=0x7fb96013d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 1 --2- 172.21.15.110:0/969485734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb96013d0e0 0x7fb96013d520 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb9540097a0 tx=0x7fb95402b840 comp rx=0 tx=0).stop 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 1 -- 172.21.15.110:0/969485734 shutdown_connections 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 1 --2- 172.21.15.110:0/969485734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb96013d0e0 0x7fb96013d520 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 1 --2- 172.21.15.110:0/969485734 >> v2:172.21.15.134:3300/0 conn(0x7fb9600b0c00 0x7fb9600accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 1 -- 172.21.15.110:0/969485734 >> 172.21.15.110:0/969485734 conn(0x7fb9600b3e60 msgr2=0x7fb9600b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 1 -- 172.21.15.110:0/969485734 shutdown_connections 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 1 -- 172.21.15.110:0/969485734 wait complete. 2024-01-20T11:19:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 10 monclient: build_initial_monmap 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.742+0000 7fb9668f2640 10 monclient: monmap: 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.743+0000 7fb9668f2640 1 librados: starting msgr at 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.743+0000 7fb9668f2640 1 librados: starting objecter 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.743+0000 7fb9668f2640 1 Processor -- start 2024-01-20T11:19:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.743+0000 7fb9668f2640 1 -- start start 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.743+0000 7fb9668f2640 1 librados: setting wanted keys 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.743+0000 7fb9668f2640 1 librados: calling monclient init 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.743+0000 7fb9668f2640 10 monclient: init 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9600b0c00 0x7fb960137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient(hunting): picked mon.noname-b con 0x7fb9600b0c00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96013d0e0 0x7fb96013faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient(hunting): picked mon.noname-a con 0x7fb96013d0e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb960143200 0x7fb960131790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient(hunting): picked mon.noname-c con 0x7fb960143200 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9601404d0 con 0x7fb96013d0e0 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96013f2b0 con 0x7fb9600b0c00 2024-01-20T11:19:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96013ffe0 con 0x7fb960143200 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb9668f2640 10 monclient(hunting): authenticate will time out at 1589.095336s 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb964e8f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb960143200 0x7fb960131790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb95f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96013d0e0 0x7fb96013faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.744+0000 7fb95ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9600b0c00 0x7fb960137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb964e8f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb960143200 0x7fb960131790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47146/0 (socket says 172.21.15.110:47146) 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb964e8f640 1 -- 172.21.15.110:0/4013488777 learned_addr learned my addr 172.21.15.110:0/4013488777 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb964e8f640 10 monclient(hunting): get_auth_request con 0x7fb960143200 auth_method 0 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb964e8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb964e8f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb964e8f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95f7fe640 10 monclient(hunting): get_auth_request con 0x7fb96013d0e0 auth_method 0 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95ffff640 10 monclient(hunting): get_auth_request con 0x7fb9600b0c00 auth_method 0 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb964e8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb964e8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb964e8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.745+0000 7fb95ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 10 monclient(hunting): handle_auth_done global_id 4775 payload 824 2024-01-20T11:19:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 1 -- 172.21.15.110:0/4013488777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9600b0c00 msgr2=0x7fb960137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9600b0c00 0x7fb960137890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 1 -- 172.21.15.110:0/4013488777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96013d0e0 msgr2=0x7fb96013faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96013d0e0 0x7fb96013faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 10 monclient: _finish_hunting 0 2024-01-20T11:19:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 1 monclient: found mon.noname-c 2024-01-20T11:19:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb964e8f640 1 -- 172.21.15.110:0/4013488777 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb954009400 con 0x7fb960143200 2024-01-20T11:19:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.748+0000 7fb95f7fe640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96013d0e0 0x7fb96013faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.750+0000 7fb964e8f640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb960143200 0x7fb960131790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb95800caa0 tx=0x7fb95800cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.750+0000 7fb95cff9640 10 client.?.objecter ms_handle_connect 0x7fb960143200 2024-01-20T11:19:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.750+0000 7fb95cff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 1 -- 172.21.15.110:0/4013488777 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95800ac00 con 0x7fb960143200 2024-01-20T11:19:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 10 monclient: dump: 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 10 monclient: _finish_auth 0 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 10 monclient: _check_auth_tickets 2024-01-20T11:19:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb9668f2640 5 monclient: authenticate success, global_id 4775 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 1 -- 172.21.15.110:0/4013488777 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95800ad60 con 0x7fb960143200 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb9668f2640 10 monclient: _renew_subs 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb9668f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb960140260 con 0x7fb960143200 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 1 -- 172.21.15.110:0/4013488777 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb958012620 con 0x7fb960143200 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95cff9640 10 monclient: dump: 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:05.757 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb95e7fc640 4 set_mon_vals no callback set 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb9668f2640 10 client.4775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb9668f2640 10 monclient: _renew_subs 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb9668f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb960131eb0 con 0x7fb960143200 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.754+0000 7fb9668f2640 1 librados: init done 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.755+0000 7fb9668f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.755+0000 7fb9668f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.755+0000 7fb9668f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.755+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb960075490 con 0x7fb960143200 2024-01-20T11:19:05.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.774+0000 7fb95cff9640 1 -- 172.21.15.110:0/4013488777 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb958007070 con 0x7fb960143200 2024-01-20T11:19:05.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.775+0000 7fb95cff9640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb94c06c860 0x7fb94c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.775+0000 7fb95ffff640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb94c06c860 0x7fb94c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.775+0000 7fb95ffff640 10 monclient: get_auth_request con 0x7fb94c06c860 auth_method 0 2024-01-20T11:19:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.776+0000 7fb95ffff640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb94c06c860 0x7fb94c06ed20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb950009b40 tx=0x7fb950009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.776+0000 7fb95cff9640 10 client.4775.objecter ms_handle_connect 0x7fb94c06c860 2024-01-20T11:19:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.777+0000 7fb95cff9640 1 -- 172.21.15.110:0/4013488777 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb958058600 con 0x7fb960143200 2024-01-20T11:19:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.777+0000 7fb95cff9640 10 client.4775.objecter ms_dispatch 0x7fb960000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.777+0000 7fb95cff9640 3 client.4775.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.777+0000 7fb95cff9640 3 client.4775.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:05.777+0000 7fb95cff9640 20 client.4775.objecter dump_active .. 0 homeless 2024-01-20T11:19:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.002+0000 7fb95cff9640 1 -- 172.21.15.110:0/4013488777 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb958054940 con 0x7fb960143200 2024-01-20T11:19:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.002+0000 7fb95cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.002+0000 7fb95cff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:06.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.180+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9601d4ee0 con 0x7fb94c06c860 2024-01-20T11:19:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.183+0000 7fb95cff9640 1 -- 172.21.15.110:0/4013488777 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63954 (secure 0 0 0) 0x7fb9601d4ee0 con 0x7fb94c06c860 2024-01-20T11:19:06.184 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 10 librados: watch_flush enter 2024-01-20T11:19:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 10 librados: watch_flush exit 2024-01-20T11:19:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 20 client.4775.objecter shutdown clearing up homeless session... 2024-01-20T11:19:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 10 client.4775.objecter successfully canceled tick 2024-01-20T11:19:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb94c06c860 msgr2=0x7fb94c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb94c06c860 0x7fb94c06ed20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb950009b40 tx=0x7fb950009190 comp rx=0 tx=0).stop 2024-01-20T11:19:06.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 10 monclient: shutdown 2024-01-20T11:19:06.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:06.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb960143200 msgr2=0x7fb960131790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb960143200 0x7fb960131790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb95800caa0 tx=0x7fb95800cf60 comp rx=0 tx=0).stop 2024-01-20T11:19:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 shutdown_connections 2024-01-20T11:19:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.184+0000 7fb9668f2640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb94c06c860 0x7fb94c06ed20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.185+0000 7fb9668f2640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb960143200 0x7fb960131790 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.185+0000 7fb9668f2640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96013d0e0 0x7fb96013faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.185+0000 7fb9668f2640 1 --2- 172.21.15.110:0/4013488777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9600b0c00 0x7fb960137890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.185+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 >> 172.21.15.110:0/4013488777 conn(0x7fb9600b3e60 msgr2=0x7fb96013c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.185+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 shutdown_connections 2024-01-20T11:19:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.185+0000 7fb9668f2640 1 -- 172.21.15.110:0/4013488777 wait complete. 2024-01-20T11:19:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.185+0000 7fb9668f2640 1 librados: shutdown 2024-01-20T11:19:06.196 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-01-20T11:19:04.753469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002365"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.9160000000000004},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":9.4309999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":10.742000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.617}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.58},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":7.968}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.048},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":12.791}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.089},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":8.8030000000000008}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.952999999999999},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":14.31}]}]},{"osd":7,"up_from":21,"seq":90194313266,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.3700000000000001},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.403}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.0540000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.401999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.644},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.3490000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.0180000000000007},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.789999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.6470000000000002},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":11.999000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.632},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.7360000000000007}]}]},{"osd":3,"up_from":21,"seq":90194313267,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.9409999999999998},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.3339999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7550000000000008},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.199000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":8.5809999999999995},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.962}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.606999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.101000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":10.682},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":13.548999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.484999999999999},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.863}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.6400000000000006},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":14.542999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.9740000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.3030000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.263},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.8609999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.846},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.994999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.771000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":8.0510000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.483000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":13.18}]}]},{"osd":1,"up_from":17,"seq":73014444082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.682},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":5.8959999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":7.5199999999999996},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.875999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":14.744},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":12.214}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":8.1920000000000002},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.519}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.8209999999999997},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":9.5709999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.507},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":8.7940000000000005}]}]},{"osd":0,"up_from":17,"seq":73014444082,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":7.524},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":12.279999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.7350000000000003},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.1969999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.436},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.8719999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.628},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.4340000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":10.156000000000001},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.914}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.867000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":9.4600000000000009}]}]},{"osd":6,"up_from":15,"seq":64424509490,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":4.7380000000000004},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":10.125}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.5289999999999999},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":8.6620000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.324999999999999},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.093}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.736000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.875}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.77},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.641999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":13.944000000000001},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":9.3379999999999992}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:06.198 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:06.198 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:06.198 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:06.198 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:06.198 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:06.198 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:06.199 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:06.199 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:06.199 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.296+0000 7f4e5f705640 10 monclient: get_monmap_and_config 2024-01-20T11:19:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.296+0000 7f4e5f705640 10 monclient: build_initial_monmap 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.296+0000 7f4e5f705640 10 monclient: monmap: 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:06.298233+0000 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:06.298233+0000 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:06.298 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 1 Processor -- start 2024-01-20T11:19:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 1 -- start start 2024-01-20T11:19:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 10 monclient: init 2024-01-20T11:19:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-01-20T11:19:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e5813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 10 monclient(hunting): picked mon.noname-b con 0x7f4e580a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e5813fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e580a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e580b25c0 con 0x7f4e5813fa10 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e580ab0a0 con 0x7f4e580a9ad0 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580aaea0 con 0x7f4e580a96f0 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.297+0000 7f4e5f705640 10 monclient(hunting): authenticate will time out at 1589.649178s 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.298+0000 7f4e5d4a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e5813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.298+0000 7f4e5cca0640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4e5813fa10 0x7f4e581423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36336/0 (socket says 172.21.15.110:36336) 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.298+0000 7f4e5dca2640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4e580a9ad0 0x7f4e5814c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:41224/0 (socket says 172.21.15.110:41224) 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.298+0000 7f4e5cca0640 1 -- 172.21.15.110:0/3746452909 learned_addr learned my addr 172.21.15.110:0/3746452909 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.298+0000 7f4e5d4a1640 10 monclient(hunting): get_auth_request con 0x7f4e580a96f0 auth_method 0 2024-01-20T11:19:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.298+0000 7f4e5d4a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.298+0000 7f4e5d4a1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.298+0000 7f4e5d4a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.299+0000 7f4e5d4a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.299+0000 7f4e5d4a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.299+0000 7f4e5d4a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e5d4a1640 10 monclient(hunting): handle_auth_done global_id 4778 payload 274 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e5d4a1640 1 -- 172.21.15.110:0/3746452909 >> v1:172.21.15.134:6789/0 conn(0x7f4e580a9ad0 legacy=0x7f4e5814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e5d4a1640 1 -- 172.21.15.110:0/3746452909 >> v1:172.21.15.110:6789/0 conn(0x7f4e5813fa10 legacy=0x7f4e581423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e5d4a1640 10 monclient: _finish_hunting 0 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e5d4a1640 1 monclient: found mon.noname-b 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e4ffff640 1 -- 172.21.15.110:0/3746452909 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649786471 0 0) 0x7f4e580ab0a0 con 0x7f4e580a9ad0 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e5d4a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e5d4a1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e5d4a1640 1 -- 172.21.15.110:0/3746452909 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e580b3fb0 con 0x7f4e580a96f0 2024-01-20T11:19:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.301+0000 7f4e4ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:19:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.302+0000 7f4e5d4a1640 1 --2- 172.21.15.110:0/3746452909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e5813f4d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4e540097d0 tx=0x7f4e5402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=465eadc6ca4bdc1 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.304+0000 7f4e4ffff640 1 -- 172.21.15.110:0/3746452909 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e54031800 con 0x7f4e580a96f0 2024-01-20T11:19:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.304+0000 7f4e4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.304+0000 7f4e4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.304+0000 7f4e4ffff640 10 monclient: dump: 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.304+0000 7f4e4ffff640 10 monclient: _finish_auth 0 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.304+0000 7f4e4ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.304+0000 7f4e4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:06.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.304+0000 7f4e5f705640 5 monclient: authenticate success, global_id 4778 2024-01-20T11:19:06.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.304+0000 7f4e5f705640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.306+0000 7f4e4ffff640 1 -- 172.21.15.110:0/3746452909 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e54031960 con 0x7f4e580a96f0 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.306+0000 7f4e4ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.306+0000 7f4e4ffff640 1 -- 172.21.15.110:0/3746452909 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e54031cf0 con 0x7f4e580a96f0 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.306+0000 7f4e4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.306+0000 7f4e4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.306+0000 7f4e4ffff640 10 monclient: dump: 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.306+0000 7f4e5f705640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.306+0000 7f4e5f705640 4 set_mon_vals no callback set 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 10 monclient: shutdown 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 1 -- 172.21.15.110:0/3746452909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 msgr2=0x7f4e5813f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 1 --2- 172.21.15.110:0/3746452909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e5813f4d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4e540097d0 tx=0x7f4e5402b8a0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 1 -- 172.21.15.110:0/3746452909 shutdown_connections 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 1 --2- 172.21.15.110:0/3746452909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e5813f4d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 1 -- 172.21.15.110:0/3746452909 >> 172.21.15.110:0/3746452909 conn(0x7f4e580b7bc0 msgr2=0x7f4e580b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 1 -- 172.21.15.110:0/3746452909 shutdown_connections 2024-01-20T11:19:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 1 -- 172.21.15.110:0/3746452909 wait complete. 2024-01-20T11:19:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 10 monclient: build_initial_monmap 2024-01-20T11:19:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.307+0000 7f4e5f705640 10 monclient: monmap: 2024-01-20T11:19:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 librados: starting msgr at 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 librados: starting objecter 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 Processor -- start 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 -- start start 2024-01-20T11:19:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 librados: setting wanted keys 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 librados: calling monclient init 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient: init 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e581385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient(hunting): picked mon.noname-c con 0x7f4e580a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e580a9ad0 0x7f4e581dab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient(hunting): picked mon.noname-a con 0x7f4e580a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e581390b0 0x7f4e581db090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient(hunting): picked mon.noname-b con 0x7f4e581390b0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581db5d0 con 0x7f4e580a9ad0 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:06.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e58138b30 con 0x7f4e581390b0 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e58138ca0 con 0x7f4e580a96f0 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.308+0000 7f4e5f705640 10 monclient(hunting): authenticate will time out at 1589.660153s 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5d4a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e581385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5dca2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e581390b0 0x7f4e581db090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5cca0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e580a9ad0 0x7f4e581dab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5d4a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e581385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47164/0 (socket says 172.21.15.110:47164) 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5d4a1640 1 -- 172.21.15.110:0/2419459315 learned_addr learned my addr 172.21.15.110:0/2419459315 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5d4a1640 10 monclient(hunting): get_auth_request con 0x7f4e580a96f0 auth_method 0 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5d4a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5d4a1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5d4a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5dca2640 10 monclient(hunting): get_auth_request con 0x7f4e581390b0 auth_method 0 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5dca2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5dca2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5dca2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.309+0000 7f4e5cca0640 10 monclient(hunting): get_auth_request con 0x7f4e580a9ad0 auth_method 0 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5cca0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5cca0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5cca0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5d4a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5d4a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5d4a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5dca2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5dca2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5dca2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:06.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5cca0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:06.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5cca0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:06.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.310+0000 7f4e5cca0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:06.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 10 monclient(hunting): handle_auth_done global_id 4781 payload 824 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 1 -- 172.21.15.110:0/2419459315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e581390b0 msgr2=0x7f4e581db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e581390b0 0x7f4e581db090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 1 -- 172.21.15.110:0/2419459315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e580a9ad0 msgr2=0x7f4e581dab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e580a9ad0 0x7f4e581dab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 10 monclient: _finish_hunting 0 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 1 monclient: found mon.noname-c 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.313+0000 7f4e5d4a1640 1 -- 172.21.15.110:0/2419459315 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e54009400 con 0x7f4e580a96f0 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.315+0000 7f4e5d4a1640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e581385f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e540097d0 tx=0x7f4e540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.315+0000 7f4e4dffb640 10 client.?.objecter ms_handle_connect 0x7f4e580a96f0 2024-01-20T11:19:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.315+0000 7f4e4dffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.317+0000 7f4e4dffb640 1 -- 172.21.15.110:0/2419459315 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5403bd80 con 0x7f4e580a96f0 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.317+0000 7f4e4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.317+0000 7f4e4dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.317+0000 7f4e4dffb640 10 monclient: dump: 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.317+0000 7f4e4dffb640 10 monclient: _finish_auth 0 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.317+0000 7f4e4dffb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.317+0000 7f4e4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.318+0000 7f4e5f705640 5 monclient: authenticate success, global_id 4781 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.318+0000 7f4e5f705640 10 monclient: _renew_subs 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.318+0000 7f4e5f705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.318+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e58138e70 con 0x7f4e580a96f0 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.318+0000 7f4e5f705640 10 client.4781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.318+0000 7f4e5f705640 10 monclient: _renew_subs 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.318+0000 7f4e5f705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.318+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e580afb20 con 0x7f4e580a96f0 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.318+0000 7f4e5f705640 1 librados: init done 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.319+0000 7f4e5f705640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.319+0000 7f4e5f705640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.319+0000 7f4e5f705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.319+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e581db790 con 0x7f4e580a96f0 2024-01-20T11:19:06.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.524+0000 7f4e4dffb640 1 -- 172.21.15.110:0/2419459315 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5404e450 con 0x7f4e580a96f0 2024-01-20T11:19:06.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.524+0000 7f4e4dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:06.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.524+0000 7f4e4f7fe640 4 set_mon_vals no callback set 2024-01-20T11:19:06.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.562+0000 7f4e4dffb640 1 -- 172.21.15.110:0/2419459315 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5404e7e0 con 0x7f4e580a96f0 2024-01-20T11:19:06.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.562+0000 7f4e4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:06.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.562+0000 7f4e4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.562+0000 7f4e4dffb640 10 monclient: dump: 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:06.564 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.563+0000 7f4e4dffb640 1 -- 172.21.15.110:0/2419459315 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4e5404e940 con 0x7f4e580a96f0 2024-01-20T11:19:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.563+0000 7f4e4dffb640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4e3c06c890 0x7f4e3c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.564+0000 7f4e5cca0640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4e3c06c890 0x7f4e3c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.564+0000 7f4e5cca0640 10 monclient: get_auth_request con 0x7f4e3c06c890 auth_method 0 2024-01-20T11:19:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.564+0000 7f4e5cca0640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4e3c06c890 0x7f4e3c06ed50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e580b1a90 tx=0x7f4e48009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.564+0000 7f4e4dffb640 10 client.4781.objecter ms_handle_connect 0x7f4e3c06c890 2024-01-20T11:19:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.575+0000 7f4e4dffb640 1 -- 172.21.15.110:0/2419459315 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4e5402e070 con 0x7f4e580a96f0 2024-01-20T11:19:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.575+0000 7f4e4dffb640 10 client.4781.objecter ms_dispatch 0x7f4e58000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.575+0000 7f4e4dffb640 3 client.4781.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.575+0000 7f4e4dffb640 3 client.4781.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.575+0000 7f4e4dffb640 20 client.4781.objecter dump_active .. 0 homeless 2024-01-20T11:19:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.575+0000 7f4e4dffb640 1 -- 172.21.15.110:0/2419459315 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4e54076e20 con 0x7f4e580a96f0 2024-01-20T11:19:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.575+0000 7f4e4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.575+0000 7f4e4dffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:06.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.825+0000 7f4e5f705640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:06.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.825+0000 7f4e5f705640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:06.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.825+0000 7f4e5f705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.825+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f4e58075490 con 0x7f4e580a96f0 2024-01-20T11:19:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.853+0000 7f4e4dffb640 1 -- 172.21.15.110:0/2419459315 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f4e5407a440 con 0x7f4e580a96f0 2024-01-20T11:19:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.853+0000 7f4e4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.853+0000 7f4e4dffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:06.855 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:06.855 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":281,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 10 librados: watch_flush enter 2024-01-20T11:19:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 10 librados: watch_flush exit 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 20 client.4781.objecter shutdown clearing up homeless session... 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 10 client.4781.objecter successfully canceled tick 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4e3c06c890 msgr2=0x7f4e3c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4e3c06c890 0x7f4e3c06ed50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e580b1a90 tx=0x7f4e48009040 comp rx=0 tx=0).stop 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 10 monclient: shutdown 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 msgr2=0x7f4e581385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.855+0000 7f4e5f705640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e581385f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e540097d0 tx=0x7f4e540046c0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.856+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 shutdown_connections 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.856+0000 7f4e5f705640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4e3c06c890 0x7f4e3c06ed50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.856+0000 7f4e5f705640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e581390b0 0x7f4e581db090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.856+0000 7f4e5f705640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e580a9ad0 0x7f4e581dab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.856+0000 7f4e5f705640 1 --2- 172.21.15.110:0/2419459315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e580a96f0 0x7f4e581385f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.856+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 >> 172.21.15.110:0/2419459315 conn(0x7f4e580b7bc0 msgr2=0x7f4e580b4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.856+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 shutdown_connections 2024-01-20T11:19:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.856+0000 7f4e5f705640 1 -- 172.21.15.110:0/2419459315 wait complete. 2024-01-20T11:19:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.856+0000 7f4e5f705640 1 librados: shutdown 2024-01-20T11:19:06.868 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:06.869 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:06.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.966+0000 7fd7f73ce640 10 monclient: get_monmap_and_config 2024-01-20T11:19:06.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.966+0000 7fd7f73ce640 10 monclient: build_initial_monmap 2024-01-20T11:19:06.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.966+0000 7fd7f73ce640 10 monclient: monmap: 2024-01-20T11:19:06.967 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:06.967 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.967 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:06.967918+0000 2024-01-20T11:19:06.967 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:06.967918+0000 2024-01-20T11:19:06.967 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:06.968 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.968 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:06.968 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:06.968 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:06.968 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:06.968 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:06.968 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.966+0000 7fd7f73ce640 1 Processor -- start 2024-01-20T11:19:06.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 1 -- start start 2024-01-20T11:19:06.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient: init 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7f00af3a0 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00ab6d0 0x7fd7f00ad420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient(hunting): picked mon.noname-b con 0x7fd7f00ab6d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd7f00ad960 0x7fd7f0148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient(hunting): picked mon.noname-c con 0x7fd7f00ad960 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f0141930 con 0x7fd7f00af3a0 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f0142920 con 0x7fd7f00ad960 2024-01-20T11:19:06.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f00af780 con 0x7fd7f00ab6d0 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.967+0000 7fd7f73ce640 10 monclient(hunting): authenticate will time out at 1590.318887s 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f4969640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00ab6d0 0x7fd7f00ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f516a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd7f00af3a0 0x7fd7f013b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36348/0 (socket says 172.21.15.110:36348) 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f516a640 1 -- 172.21.15.110:0/112293613 learned_addr learned my addr 172.21.15.110:0/112293613 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f596b640 1 --2- 172.21.15.110:0/112293613 >> v2:172.21.15.134:3300/0 conn(0x7fd7f00ad960 0x7fd7f0148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f4969640 10 monclient(hunting): get_auth_request con 0x7fd7f00ab6d0 auth_method 0 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f4969640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f4969640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f4969640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f596b640 10 monclient(hunting): get_auth_request con 0x7fd7f00ad960 auth_method 0 2024-01-20T11:19:06.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f596b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:06.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f596b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.968+0000 7fd7f596b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.969+0000 7fd7f4969640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:06.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.969+0000 7fd7f4969640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:06.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.969+0000 7fd7f4969640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:06.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.969+0000 7fd7f596b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:06.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.969+0000 7fd7f596b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:06.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.969+0000 7fd7f596b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:06.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.970+0000 7fd7e7fff640 1 -- 172.21.15.110:0/112293613 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578146153 0 0) 0x7fd7f0141930 con 0x7fd7f00af3a0 2024-01-20T11:19:06.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.970+0000 7fd7e7fff640 10 monclient(hunting): _negotiate 2024-01-20T11:19:06.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.970+0000 7fd7e7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.970+0000 7fd7e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.970+0000 7fd7e7fff640 10 monclient(hunting): my global_id is 4944 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.970+0000 7fd7e7fff640 1 -- 172.21.15.110:0/112293613 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc005730 con 0x7fd7f00af3a0 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.970+0000 7fd7f4969640 10 monclient(hunting): handle_auth_done global_id 4784 payload 274 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.971+0000 7fd7f4969640 1 -- 172.21.15.110:0/112293613 >> v2:172.21.15.134:3300/0 conn(0x7fd7f00ad960 msgr2=0x7fd7f0148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.971+0000 7fd7f4969640 1 --2- 172.21.15.110:0/112293613 >> v2:172.21.15.134:3300/0 conn(0x7fd7f00ad960 0x7fd7f0148cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.971+0000 7fd7f4969640 1 -- 172.21.15.110:0/112293613 >> v1:172.21.15.110:6789/0 conn(0x7fd7f00af3a0 legacy=0x7fd7f013b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.971+0000 7fd7f4969640 10 monclient: _finish_hunting 0 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.971+0000 7fd7f4969640 1 monclient: found mon.noname-b 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.971+0000 7fd7f4969640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.971+0000 7fd7f4969640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.971+0000 7fd7f4969640 1 -- 172.21.15.110:0/112293613 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f00ab2b0 con 0x7fd7f00ab6d0 2024-01-20T11:19:06.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.972+0000 7fd7f4969640 1 --2- 172.21.15.110:0/112293613 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00ab6d0 0x7fd7f00ad420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0097a0 tx=0x7fd7ec02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28ce66ce6a461059 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:06.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 1 -- 172.21.15.110:0/112293613 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec031830 con 0x7fd7f00ab6d0 2024-01-20T11:19:06.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:06.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 10 monclient: dump: 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 10 monclient: _finish_auth 0 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:06.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 1 -- 172.21.15.110:0/112293613 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ec031990 con 0x7fd7f00ab6d0 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7f73ce640 5 monclient: authenticate success, global_id 4784 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7f73ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7f73ce640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7f73ce640 4 set_mon_vals no callback set 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 1 -- 172.21.15.110:0/112293613 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec031d20 con 0x7fd7f00ab6d0 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.976+0000 7fd7e7fff640 10 monclient: dump: 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:06.979 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.980 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:06.980 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:06.980 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:06.980 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 10 monclient: shutdown 2024-01-20T11:19:06.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:06.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 1 -- 172.21.15.110:0/112293613 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00ab6d0 msgr2=0x7fd7f00ad420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 1 --2- 172.21.15.110:0/112293613 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00ab6d0 0x7fd7f00ad420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0097a0 tx=0x7fd7ec02b8d0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 1 -- 172.21.15.110:0/112293613 shutdown_connections 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 1 --2- 172.21.15.110:0/112293613 >> v2:172.21.15.134:3300/0 conn(0x7fd7f00ad960 0x7fd7f0148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 1 --2- 172.21.15.110:0/112293613 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00ab6d0 0x7fd7f00ad420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 1 -- 172.21.15.110:0/112293613 >> 172.21.15.110:0/112293613 conn(0x7fd7f00b3e60 msgr2=0x7fd7f00b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 1 -- 172.21.15.110:0/112293613 shutdown_connections 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.977+0000 7fd7f73ce640 1 -- 172.21.15.110:0/112293613 wait complete. 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.978+0000 7fd7f73ce640 10 monclient: build_initial_monmap 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.978+0000 7fd7f73ce640 10 monclient: monmap: 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:06.984 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.978+0000 7fd7f73ce640 1 librados: starting msgr at 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.978+0000 7fd7f73ce640 1 librados: starting objecter 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.978+0000 7fd7f73ce640 1 Processor -- start 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.978+0000 7fd7f73ce640 1 -- start start 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.978+0000 7fd7f73ce640 1 librados: setting wanted keys 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.978+0000 7fd7f73ce640 1 librados: calling monclient init 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.978+0000 7fd7f73ce640 10 monclient: init 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:06.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7f00ab6d0 0x7fd7f006de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f00ab6d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f00ad960 0x7fd7f0070670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient(hunting): picked mon.noname-b con 0x7fd7f00ad960 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00af3a0 0x7fd7f0070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient(hunting): picked mon.noname-c con 0x7fd7f00af3a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f013b5c0 con 0x7fd7f00ab6d0 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f006eb00 con 0x7fd7f00ad960 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f006fc90 con 0x7fd7f00af3a0 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f73ce640 10 monclient(hunting): authenticate will time out at 1590.330651s 2024-01-20T11:19:06.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f596b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00af3a0 0x7fd7f0070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.979+0000 7fd7f516a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7f00ab6d0 0x7fd7f006de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f4969640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f00ad960 0x7fd7f0070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f596b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00af3a0 0x7fd7f0070bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47184/0 (socket says 172.21.15.110:47184) 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f596b640 1 -- 172.21.15.110:0/1832410786 learned_addr learned my addr 172.21.15.110:0/1832410786 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f596b640 10 monclient(hunting): get_auth_request con 0x7fd7f00af3a0 auth_method 0 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f596b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f596b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f596b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f516a640 10 monclient(hunting): get_auth_request con 0x7fd7f00ab6d0 auth_method 0 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f516a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f516a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f516a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f4969640 10 monclient(hunting): get_auth_request con 0x7fd7f00ad960 auth_method 0 2024-01-20T11:19:06.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f4969640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f4969640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f4969640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f596b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f596b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f596b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f516a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f516a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.980+0000 7fd7f516a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.981+0000 7fd7f4969640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.981+0000 7fd7f4969640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:06.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.981+0000 7fd7f4969640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 10 monclient(hunting): handle_auth_done global_id 4787 payload 824 2024-01-20T11:19:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 1 -- 172.21.15.110:0/1832410786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f00ad960 msgr2=0x7fd7f0070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f00ad960 0x7fd7f0070670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 1 -- 172.21.15.110:0/1832410786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7f00ab6d0 msgr2=0x7fd7f006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7f00ab6d0 0x7fd7f006de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 10 monclient: _finish_hunting 0 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 1 monclient: found mon.noname-c 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.983+0000 7fd7f596b640 1 -- 172.21.15.110:0/1832410786 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec009400 con 0x7fd7f00af3a0 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.985+0000 7fd7f596b640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00af3a0 0x7fd7f0070bb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800efc0 tx=0x7fd7e800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.985+0000 7fd7e5ffb640 10 client.?.objecter ms_handle_connect 0x7fd7f00af3a0 2024-01-20T11:19:06.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.985+0000 7fd7e5ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 1 -- 172.21.15.110:0/1832410786 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8014480 con 0x7fd7f00af3a0 2024-01-20T11:19:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 10 monclient: dump: 2024-01-20T11:19:06.992 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:06.994 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 10 monclient: _finish_auth 0 2024-01-20T11:19:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 1 -- 172.21.15.110:0/1832410786 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e800f040 con 0x7fd7f00af3a0 2024-01-20T11:19:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7f73ce640 5 monclient: authenticate success, global_id 4787 2024-01-20T11:19:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 1 -- 172.21.15.110:0/1832410786 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8014820 con 0x7fd7f00af3a0 2024-01-20T11:19:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e5ffb640 10 monclient: dump: 2024-01-20T11:19:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:06.996 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:06.996 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:06.996 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:06.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7e77fe640 4 set_mon_vals no callback set 2024-01-20T11:19:06.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7f73ce640 10 monclient: _renew_subs 2024-01-20T11:19:06.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7f73ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f006ff70 con 0x7fd7f00af3a0 2024-01-20T11:19:06.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7f73ce640 10 client.4787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7f73ce640 10 monclient: _renew_subs 2024-01-20T11:19:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7f73ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f006e3a0 con 0x7fd7f00af3a0 2024-01-20T11:19:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.989+0000 7fd7f73ce640 1 librados: init done 2024-01-20T11:19:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.990+0000 7fd7f73ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.990+0000 7fd7f73ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.990+0000 7fd7f73ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:06.990+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f0075490 con 0x7fd7f00af3a0 2024-01-20T11:19:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.237+0000 7fd7e5ffb640 1 -- 172.21.15.110:0/1832410786 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd7e8014980 con 0x7fd7f00af3a0 2024-01-20T11:19:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.237+0000 7fd7e5ffb640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7d406c8b0 0x7fd7d406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.237+0000 7fd7f516a640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7d406c8b0 0x7fd7d406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.237+0000 7fd7f516a640 10 monclient: get_auth_request con 0x7fd7d406c8b0 auth_method 0 2024-01-20T11:19:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.238+0000 7fd7f516a640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7d406c8b0 0x7fd7d406ed70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0005880 tx=0x7fd7e0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.238+0000 7fd7e5ffb640 10 client.4787.objecter ms_handle_connect 0x7fd7d406c8b0 2024-01-20T11:19:07.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.250+0000 7fd7e5ffb640 1 -- 172.21.15.110:0/1832410786 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd7e8059500 con 0x7fd7f00af3a0 2024-01-20T11:19:07.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.250+0000 7fd7e5ffb640 10 client.4787.objecter ms_dispatch 0x7fd7f0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:07.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.250+0000 7fd7e5ffb640 3 client.4787.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:07.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.250+0000 7fd7e5ffb640 3 client.4787.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:07.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.250+0000 7fd7e5ffb640 20 client.4787.objecter dump_active .. 0 homeless 2024-01-20T11:19:07.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.250+0000 7fd7e5ffb640 1 -- 172.21.15.110:0/1832410786 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd7e8055840 con 0x7fd7f00af3a0 2024-01-20T11:19:07.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.250+0000 7fd7e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:07.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.250+0000 7fd7e5ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.450+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7f00aaf60 con 0x7fd7d406c8b0 2024-01-20T11:19:07.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.453+0000 7fd7e5ffb640 1 -- 172.21.15.110:0/1832410786 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63930 (secure 0 0 0) 0x7fd7e000f820 con 0x7fd7d406c8b0 2024-01-20T11:19:07.455 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:07.455 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 10 librados: watch_flush enter 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 10 librados: watch_flush exit 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 20 client.4787.objecter shutdown clearing up homeless session... 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 10 client.4787.objecter successfully canceled tick 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7d406c8b0 msgr2=0x7fd7d406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7d406c8b0 0x7fd7d406ed70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0005880 tx=0x7fd7e0009040 comp rx=0 tx=0).stop 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 10 monclient: shutdown 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00af3a0 msgr2=0x7fd7f0070bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00af3a0 0x7fd7f0070bb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800efc0 tx=0x7fd7e800c490 comp rx=0 tx=0).stop 2024-01-20T11:19:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 shutdown_connections 2024-01-20T11:19:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd7d406c8b0 0x7fd7d406ed70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7f00af3a0 0x7fd7f0070bb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f00ad960 0x7fd7f0070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 --2- 172.21.15.110:0/1832410786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7f00ab6d0 0x7fd7f006de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.455+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 >> 172.21.15.110:0/1832410786 conn(0x7fd7f00b3e60 msgr2=0x7fd7f00ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.456+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 shutdown_connections 2024-01-20T11:19:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.456+0000 7fd7f73ce640 1 -- 172.21.15.110:0/1832410786 wait complete. 2024-01-20T11:19:07.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:07.456+0000 7fd7f73ce640 1 librados: shutdown 2024-01-20T11:19:07.467 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-01-20T11:19:06.753915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002389"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.9160000000000004},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":9.4309999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":10.742000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.617}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.58},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":7.968}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.048},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":12.791}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.089},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":8.8030000000000008}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.952999999999999},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":14.31}]}]},{"osd":7,"up_from":21,"seq":90194313267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.0819999999999999},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.032}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.6959999999999997},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.228}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.215},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.285}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":16.388000000000002},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.567}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":17.327999999999999},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.932}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.217000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":18.315000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313268,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.9409999999999998},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.3339999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7550000000000008},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.199000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":8.5809999999999995},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.962}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.606999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.101000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":10.682},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":13.548999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.484999999999999},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.863}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444081,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.6400000000000006},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":14.542999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.9740000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.3030000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.263},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.8609999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.846},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.994999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.771000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":8.0510000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.483000000000001},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":13.18}]}]},{"osd":1,"up_from":17,"seq":73014444082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.682},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":5.8959999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":7.5199999999999996},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.875999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":14.744},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":12.214}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":8.1920000000000002},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.519}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.8209999999999997},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":9.5709999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.507},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":8.7940000000000005}]}]},{"osd":0,"up_from":17,"seq":73014444082,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":7.524},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":12.279999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.7350000000000003},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.1969999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":13.436},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.8719999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.628},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.4340000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":10.156000000000001},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.914}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.867000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":9.4600000000000009}]}]},{"osd":6,"up_from":15,"seq":64424509491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":11.664},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.8250000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.5499999999999998},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":14.401}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.752000000000001},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.521000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.768000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.4000000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.0790000000000006},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.234999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.109},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":17.707999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:10.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~12s 2024-01-20T11:19:10.470 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.571+0000 7f633d508640 10 monclient: get_monmap_and_config 2024-01-20T11:19:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.571+0000 7f633d508640 10 monclient: build_initial_monmap 2024-01-20T11:19:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.571+0000 7f633d508640 10 monclient: monmap: 2024-01-20T11:19:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:10.573065+0000 2024-01-20T11:19:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:10.573065+0000 2024-01-20T11:19:10.573 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:10.573 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:10.573 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:10.573 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:10.573 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:10.573 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:10.573 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:10.573 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:10.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 1 Processor -- start 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 1 -- start start 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient: init 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f63380b4d10 0x7f63380b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient(hunting): picked mon.noname-a con 0x7f63380b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63380b17c0 0x7f63380b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient(hunting): picked mon.noname-b con 0x7f63380b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6338145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:10.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63380b2120 con 0x7f6338145980 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63380b50f0 con 0x7f63380b4d10 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63380aade0 con 0x7f63380b17c0 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.572+0000 7f633d508640 10 monclient(hunting): authenticate will time out at 1593.924006s 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.573+0000 7f632ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63380b17c0 0x7f63380b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.573+0000 7f6336ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f63380b4d10 0x7f63380b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.573+0000 7f632ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63380b17c0 0x7f63380b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47716/0 (socket says 172.21.15.110:47716) 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.573+0000 7f632ffff640 1 -- 172.21.15.110:0/1017049706 learned_addr learned my addr 172.21.15.110:0/1017049706 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.573+0000 7f632ffff640 10 monclient(hunting): get_auth_request con 0x7f63380b17c0 auth_method 0 2024-01-20T11:19:10.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.573+0000 7f632ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.573+0000 7f632ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.573+0000 7f632ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f6336ffd640 10 monclient(hunting): get_auth_request con 0x7f63380b4d10 auth_method 0 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f6336ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f6336ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f6336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f632ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f632ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f632ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f6336ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f6336ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:10.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.574+0000 7f6336ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.576+0000 7f632ffff640 10 monclient(hunting): handle_auth_done global_id 4790 payload 274 2024-01-20T11:19:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.576+0000 7f632ffff640 1 -- 172.21.15.110:0/1017049706 >> v2:172.21.15.110:3300/0 conn(0x7f63380b4d10 msgr2=0x7f63380b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.576+0000 7f632ffff640 1 --2- 172.21.15.110:0/1017049706 >> v2:172.21.15.110:3300/0 conn(0x7f63380b4d10 0x7f63380b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.576+0000 7f632ffff640 1 -- 172.21.15.110:0/1017049706 >> v1:172.21.15.110:6789/0 conn(0x7f6338145980 legacy=0x7f6338145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.576+0000 7f632ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.576+0000 7f632ffff640 1 monclient: found mon.noname-b 2024-01-20T11:19:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.576+0000 7f632ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.576+0000 7f632ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.576+0000 7f632ffff640 1 -- 172.21.15.110:0/1017049706 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6338145460 con 0x7f63380b17c0 2024-01-20T11:19:10.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.578+0000 7f632ffff640 1 --2- 172.21.15.110:0/1017049706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63380b17c0 0x7f63380b1be0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6320009870 tx=0x7f632002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c609cb8fc202a19b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 1 -- 172.21.15.110:0/1017049706 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320031640 con 0x7f63380b17c0 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 10 monclient: dump: 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 1 -- 172.21.15.110:0/1017049706 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63200317a0 con 0x7f63380b17c0 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f633d508640 5 monclient: authenticate success, global_id 4790 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f633d508640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f633d508640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 1 -- 172.21.15.110:0/1017049706 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320031b30 con 0x7f63380b17c0 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f633d508640 4 set_mon_vals no callback set 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:10.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.581+0000 7f63367fc640 10 monclient: dump: 2024-01-20T11:19:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:10.588 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:10.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 10 monclient: shutdown 2024-01-20T11:19:10.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:10.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 1 -- 172.21.15.110:0/1017049706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63380b17c0 msgr2=0x7f63380b1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:10.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 1 --2- 172.21.15.110:0/1017049706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63380b17c0 0x7f63380b1be0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6320009870 tx=0x7f632002b8f0 comp rx=0 tx=0).stop 2024-01-20T11:19:10.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 1 -- 172.21.15.110:0/1017049706 shutdown_connections 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 1 --2- 172.21.15.110:0/1017049706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63380b17c0 0x7f63380b1be0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 1 --2- 172.21.15.110:0/1017049706 >> v2:172.21.15.110:3300/0 conn(0x7f63380b4d10 0x7f63380b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 1 -- 172.21.15.110:0/1017049706 >> 172.21.15.110:0/1017049706 conn(0x7f6338066590 msgr2=0x7f63380669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 1 -- 172.21.15.110:0/1017049706 shutdown_connections 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.582+0000 7f633d508640 1 -- 172.21.15.110:0/1017049706 wait complete. 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.583+0000 7f633d508640 10 monclient: build_initial_monmap 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.583+0000 7f633d508640 10 monclient: monmap: 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:10.593 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.583+0000 7f633d508640 1 librados: starting msgr at 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.583+0000 7f633d508640 1 librados: starting objecter 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.583+0000 7f633d508640 1 Processor -- start 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.583+0000 7f633d508640 1 -- start start 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.583+0000 7f633d508640 1 librados: setting wanted keys 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.583+0000 7f633d508640 1 librados: calling monclient init 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.583+0000 7f633d508640 10 monclient: init 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63380b17c0 0x7f633813eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient(hunting): picked mon.noname-b con 0x7f63380b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63380b4d10 0x7f63381370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:10.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient(hunting): picked mon.noname-a con 0x7f63380b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6338145980 0x7f63381375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient(hunting): picked mon.noname-c con 0x7f6338145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63380ac7b0 con 0x7f63380b4d10 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6338137b20 con 0x7f63380b17c0 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63380aca00 con 0x7f6338145980 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f633d508640 10 monclient(hunting): authenticate will time out at 1593.935534s 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f63377fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6338145980 0x7f63381375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.584+0000 7f632ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63380b4d10 0x7f63381370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f63377fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6338145980 0x7f63381375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47726/0 (socket says 172.21.15.110:47726) 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f63377fe640 1 -- 172.21.15.110:0/551606707 learned_addr learned my addr 172.21.15.110:0/551606707 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f63377fe640 10 monclient(hunting): get_auth_request con 0x7f6338145980 auth_method 0 2024-01-20T11:19:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f63377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f63377fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f63377fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f6336ffd640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63380b17c0 0x7f633813eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f632ffff640 10 monclient(hunting): get_auth_request con 0x7f63380b4d10 auth_method 0 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f632ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f632ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f632ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f63377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f63377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f63377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f6336ffd640 10 monclient(hunting): get_auth_request con 0x7f63380b17c0 auth_method 0 2024-01-20T11:19:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f6336ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f6336ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f6336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f632ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f632ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.585+0000 7f632ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.586+0000 7f6336ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.586+0000 7f6336ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.586+0000 7f6336ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 10 monclient(hunting): handle_auth_done global_id 4793 payload 824 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 1 -- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63380b17c0 msgr2=0x7f633813eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63380b17c0 0x7f633813eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 1 -- 172.21.15.110:0/551606707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63380b4d10 msgr2=0x7f63381370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63380b4d10 0x7f63381370a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:10.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 1 monclient: found mon.noname-c 2024-01-20T11:19:10.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:10.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:10.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.588+0000 7f63377fe640 1 -- 172.21.15.110:0/551606707 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63200094d0 con 0x7f6338145980 2024-01-20T11:19:10.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.590+0000 7f63377fe640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6338145980 0x7f63381375e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f633000c730 tx=0x7f633000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:10.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.590+0000 7f632f7fe640 10 client.?.objecter ms_handle_connect 0x7f6338145980 2024-01-20T11:19:10.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.590+0000 7f632f7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 1 -- 172.21.15.110:0/551606707 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6330006d50 con 0x7f6338145980 2024-01-20T11:19:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 10 monclient: dump: 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 1 -- 172.21.15.110:0/551606707 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6330003970 con 0x7f6338145980 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 1 -- 172.21.15.110:0/551606707 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6330003d20 con 0x7f6338145980 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f632f7fe640 10 monclient: dump: 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:10.601 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f633d508640 5 monclient: authenticate success, global_id 4793 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f633d508640 10 monclient: _renew_subs 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f6335ffb640 4 set_mon_vals no callback set 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f633d508640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63380acce0 con 0x7f6338145980 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f633d508640 10 client.4793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f633d508640 10 monclient: _renew_subs 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f633d508640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63380ab780 con 0x7f6338145980 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.594+0000 7f633d508640 1 librados: init done 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.595+0000 7f633d508640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.595+0000 7f633d508640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:10.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.595+0000 7f633d508640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.595+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63380ab970 con 0x7f6338145980 2024-01-20T11:19:10.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.835+0000 7f632f7fe640 1 -- 172.21.15.110:0/551606707 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f63300050d0 con 0x7f6338145980 2024-01-20T11:19:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.835+0000 7f632f7fe640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f631806c8b0 0x7f631806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.836+0000 7f6336ffd640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f631806c8b0 0x7f631806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.836+0000 7f6336ffd640 10 monclient: get_auth_request con 0x7f631806c8b0 auth_method 0 2024-01-20T11:19:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.836+0000 7f6336ffd640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f631806c8b0 0x7f631806ed70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6328009b40 tx=0x7f6328009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.836+0000 7f632f7fe640 10 client.4793.objecter ms_handle_connect 0x7f631806c8b0 2024-01-20T11:19:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.847+0000 7f632f7fe640 1 -- 172.21.15.110:0/551606707 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6330058a20 con 0x7f6338145980 2024-01-20T11:19:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.847+0000 7f632f7fe640 10 client.4793.objecter ms_dispatch 0x7f6338000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.847+0000 7f632f7fe640 3 client.4793.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.847+0000 7f632f7fe640 3 client.4793.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.847+0000 7f632f7fe640 20 client.4793.objecter dump_active .. 0 homeless 2024-01-20T11:19:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.847+0000 7f632f7fe640 1 -- 172.21.15.110:0/551606707 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6330054d60 con 0x7f6338145980 2024-01-20T11:19:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.847+0000 7f632f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:10.847+0000 7f632f7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:11.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.021+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f633814c7c0 con 0x7f631806c8b0 2024-01-20T11:19:11.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.024+0000 7f632f7fe640 1 -- 172.21.15.110:0/551606707 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63970 (secure 0 0 0) 0x7f633814c7c0 con 0x7f631806c8b0 2024-01-20T11:19:11.026 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:11.026 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 10 librados: watch_flush enter 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 10 librados: watch_flush exit 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 20 client.4793.objecter shutdown clearing up homeless session... 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 10 client.4793.objecter successfully canceled tick 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f631806c8b0 msgr2=0x7f631806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f631806c8b0 0x7f631806ed70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6328009b40 tx=0x7f6328009190 comp rx=0 tx=0).stop 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 10 monclient: shutdown 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6338145980 msgr2=0x7f63381375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6338145980 0x7f63381375e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f633000c730 tx=0x7f633000cbf0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 shutdown_connections 2024-01-20T11:19:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f631806c8b0 0x7f631806ed70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6338145980 0x7f63381375e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63380b4d10 0x7f63381370a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 --2- 172.21.15.110:0/551606707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63380b17c0 0x7f633813eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 >> 172.21.15.110:0/551606707 conn(0x7f6338066590 msgr2=0x7f6338073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.026+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 shutdown_connections 2024-01-20T11:19:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.027+0000 7f633d508640 1 -- 172.21.15.110:0/551606707 wait complete. 2024-01-20T11:19:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.027+0000 7f633d508640 1 librados: shutdown 2024-01-20T11:19:11.038 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-01-20T11:19:10.754893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002652"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.3250000000000002},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.1689999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.4220000000000006},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.412000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.0899999999999999},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.071}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":14.734},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":10.180999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.064},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":10.795999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.690999999999999},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.477}]}]},{"osd":7,"up_from":21,"seq":90194313267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.0819999999999999},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.032}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.6959999999999997},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.228}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.215},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.285}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":16.388000000000002},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.567}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":17.327999999999999},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.932}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.217000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":18.315000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313268,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.9409999999999998},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.3339999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7550000000000008},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.199000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":8.5809999999999995},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.962}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.606999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.101000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":10.682},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":13.548999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.484999999999999},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.863}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.0649999999999995},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":6.9820000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":4.7720000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.358000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.597},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.420999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.542},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.987}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.672000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.9729999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.7420000000000009},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":16.292999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.9939999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":7.484}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.109999999999999},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.9459999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":9.3610000000000007},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.4390000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":14.061},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.944000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.617000000000001},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":15.782}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.210000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.867000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444083,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":4.3250000000000002},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":8.0459999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.359},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.3410000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":9.6620000000000008},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.8300000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.972},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":13.065}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":10.265000000000001},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.746}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.554},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":12.163}]}]},{"osd":6,"up_from":15,"seq":64424509491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":11.664},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.8250000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.5499999999999998},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":14.401}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.752000000000001},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.521000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.768000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.4000000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.0790000000000006},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.234999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.109},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":17.707999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:11.040 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:11.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.138+0000 7f85a0461640 10 monclient: get_monmap_and_config 2024-01-20T11:19:11.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.138+0000 7f85a0461640 10 monclient: build_initial_monmap 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.138+0000 7f85a0461640 10 monclient: monmap: 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:11.140407+0000 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:11.140407+0000 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:11.141 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.139+0000 7f85a0461640 1 Processor -- start 2024-01-20T11:19:11.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.139+0000 7f85a0461640 1 -- start start 2024-01-20T11:19:11.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.139+0000 7f85a0461640 10 monclient: init 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85980b4d10 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85980b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 10 monclient(hunting): picked mon.noname-b con 0x7f85980b1ed0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8598145980 0x7f8598145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 10 monclient(hunting): picked mon.noname-a con 0x7f8598145980 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8598146370 con 0x7f85980b4d10 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85980b50f0 con 0x7f8598145980 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980aade0 con 0x7f85980b1ed0 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f85a0461640 10 monclient(hunting): authenticate will time out at 1594.491595s 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f859d9fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85980b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.140+0000 7f859e9fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8598145980 0x7f8598145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859e1fd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f85980b4d10 0x7f85980b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:48762/0 (socket says 172.21.15.110:48762) 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859e1fd640 1 -- 172.21.15.110:0/334801183 learned_addr learned my addr 172.21.15.110:0/334801183 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859d9fc640 10 monclient(hunting): get_auth_request con 0x7f85980b1ed0 auth_method 0 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859d9fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859d9fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859d9fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859e9fe640 10 monclient(hunting): get_auth_request con 0x7f8598145980 auth_method 0 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859e9fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859e9fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859e9fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859d9fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859d9fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.141+0000 7f859d9fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.142+0000 7f859e9fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.142+0000 7f859e9fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.142+0000 7f859e9fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.143+0000 7f859d9fc640 10 monclient(hunting): handle_auth_done global_id 4796 payload 274 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.143+0000 7f859d9fc640 1 -- 172.21.15.110:0/334801183 >> v2:172.21.15.110:3300/0 conn(0x7f8598145980 msgr2=0x7f8598145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.143+0000 7f859d9fc640 1 --2- 172.21.15.110:0/334801183 >> v2:172.21.15.110:3300/0 conn(0x7f8598145980 0x7f8598145e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.143+0000 7f859d9fc640 1 -- 172.21.15.110:0/334801183 >> v1:172.21.15.134:6789/0 conn(0x7f85980b4d10 legacy=0x7f85980b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.143+0000 7f859d9fc640 10 monclient: _finish_hunting 0 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.143+0000 7f859d9fc640 1 monclient: found mon.noname-b 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.143+0000 7f859d9fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.143+0000 7f859d9fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.143+0000 7f859d9fc640 1 -- 172.21.15.110:0/334801183 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8598145460 con 0x7f85980b1ed0 2024-01-20T11:19:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.145+0000 7f859d9fc640 1 --2- 172.21.15.110:0/334801183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85980b3da0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f85880097a0 tx=0x7f858802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d36adb76cb58a59 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 1 -- 172.21.15.110:0/334801183 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858802e070 con 0x7f85980b1ed0 2024-01-20T11:19:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 10 monclient: dump: 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 10 monclient: _finish_auth 0 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 1 -- 172.21.15.110:0/334801183 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85880315a0 con 0x7f85980b1ed0 2024-01-20T11:19:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f85a0461640 5 monclient: authenticate success, global_id 4796 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f85a0461640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f85a0461640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f85a0461640 4 set_mon_vals no callback set 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 1 -- 172.21.15.110:0/334801183 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588031930 con 0x7f85980b1ed0 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.148+0000 7f859d1fb640 10 monclient: dump: 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:11.152 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 10 monclient: shutdown 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 1 -- 172.21.15.110:0/334801183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 msgr2=0x7f85980b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 1 --2- 172.21.15.110:0/334801183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85980b3da0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f85880097a0 tx=0x7f858802b840 comp rx=0 tx=0).stop 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 1 -- 172.21.15.110:0/334801183 shutdown_connections 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 1 --2- 172.21.15.110:0/334801183 >> v2:172.21.15.110:3300/0 conn(0x7f8598145980 0x7f8598145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 1 --2- 172.21.15.110:0/334801183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85980b3da0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 1 -- 172.21.15.110:0/334801183 >> 172.21.15.110:0/334801183 conn(0x7f8598066590 msgr2=0x7f85980669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 1 -- 172.21.15.110:0/334801183 shutdown_connections 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.149+0000 7f85a0461640 1 -- 172.21.15.110:0/334801183 wait complete. 2024-01-20T11:19:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.150+0000 7f85a0461640 10 monclient: build_initial_monmap 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.150+0000 7f85a0461640 10 monclient: monmap: 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.150+0000 7f85a0461640 1 librados: starting msgr at 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.150+0000 7f85a0461640 1 librados: starting objecter 2024-01-20T11:19:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.150+0000 7f85a0461640 1 Processor -- start 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 1 -- start start 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 1 librados: setting wanted keys 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 1 librados: calling monclient init 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient: init 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85981dae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient(hunting): picked mon.noname-c con 0x7f85980b1ed0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85980b4d10 0x7f8598135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient(hunting): picked mon.noname-b con 0x7f85980b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8598145980 0x7f8598136430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient(hunting): picked mon.noname-a con 0x7f8598145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85981411d0 con 0x7f8598145980 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859813d6a0 con 0x7f85980b4d10 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859813d940 con 0x7f85980b1ed0 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.151+0000 7f85a0461640 10 monclient(hunting): authenticate will time out at 1594.502760s 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85981dae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e9fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8598145980 0x7f8598136430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859d9fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85980b4d10 0x7f8598135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85981dae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47744/0 (socket says 172.21.15.110:47744) 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 1 -- 172.21.15.110:0/1912323001 learned_addr learned my addr 172.21.15.110:0/1912323001 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 10 monclient(hunting): get_auth_request con 0x7f85980b1ed0 auth_method 0 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e9fe640 10 monclient(hunting): get_auth_request con 0x7f8598145980 auth_method 0 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e9fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e9fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e9fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859d9fc640 10 monclient(hunting): get_auth_request con 0x7f85980b4d10 auth_method 0 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859d9fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859d9fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859d9fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.152+0000 7f859e1fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.153+0000 7f859e9fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.153+0000 7f859e9fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.153+0000 7f859e9fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.153+0000 7f859d9fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.153+0000 7f859d9fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.153+0000 7f859d9fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 10 monclient(hunting): handle_auth_done global_id 4799 payload 824 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 1 -- 172.21.15.110:0/1912323001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85980b4d10 msgr2=0x7f8598135ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85980b4d10 0x7f8598135ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 1 -- 172.21.15.110:0/1912323001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8598145980 msgr2=0x7f8598136430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8598145980 0x7f8598136430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 10 monclient: _finish_hunting 0 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 1 monclient: found mon.noname-c 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.155+0000 7f859e1fd640 1 -- 172.21.15.110:0/1912323001 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588009400 con 0x7f85980b1ed0 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.158+0000 7f859e1fd640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85981dae70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f859400c4d0 tx=0x7f859400c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.158+0000 7f858effd640 10 client.?.objecter ms_handle_connect 0x7f85980b1ed0 2024-01-20T11:19:11.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.158+0000 7f858effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:11.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 1 -- 172.21.15.110:0/1912323001 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594005ce0 con 0x7f85980b1ed0 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 10 monclient: dump: 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 10 monclient: _finish_auth 0 2024-01-20T11:19:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 1 -- 172.21.15.110:0/1912323001 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8594005e40 con 0x7f85980b1ed0 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f85a0461640 5 monclient: authenticate success, global_id 4799 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f85a0461640 10 monclient: _renew_subs 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f85a0461640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8598141ef0 con 0x7f85980b1ed0 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 1 -- 172.21.15.110:0/1912323001 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594007270 con 0x7f85980b1ed0 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858effd640 10 monclient: dump: 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:11.164 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f858ffff640 4 set_mon_vals no callback set 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f85a0461640 10 client.4799.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f85a0461640 10 monclient: _renew_subs 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f85a0461640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859813fee0 con 0x7f85980b1ed0 2024-01-20T11:19:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.161+0000 7f85a0461640 1 librados: init done 2024-01-20T11:19:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.162+0000 7f85a0461640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.162+0000 7f85a0461640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.162+0000 7f85a0461640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.162+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8598075490 con 0x7f85980b1ed0 2024-01-20T11:19:11.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.405+0000 7f858effd640 1 -- 172.21.15.110:0/1912323001 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f85940073d0 con 0x7f85980b1ed0 2024-01-20T11:19:11.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.405+0000 7f858effd640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f857c06c860 0x7f857c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.406+0000 7f859d9fc640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f857c06c860 0x7f857c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.406+0000 7f859d9fc640 10 monclient: get_auth_request con 0x7f857c06c860 auth_method 0 2024-01-20T11:19:11.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.406+0000 7f858effd640 10 client.4799.objecter ms_handle_connect 0x7f857c06c860 2024-01-20T11:19:11.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.406+0000 7f859d9fc640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f857c06c860 0x7f857c06ed20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8588036000 tx=0x7f858803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:11.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.418+0000 7f858effd640 1 -- 172.21.15.110:0/1912323001 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8594057b90 con 0x7f85980b1ed0 2024-01-20T11:19:11.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.418+0000 7f858effd640 10 client.4799.objecter ms_dispatch 0x7f8598000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:11.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.418+0000 7f858effd640 3 client.4799.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:11.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.418+0000 7f858effd640 3 client.4799.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:11.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.418+0000 7f858effd640 20 client.4799.objecter dump_active .. 0 homeless 2024-01-20T11:19:11.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.418+0000 7f858effd640 1 -- 172.21.15.110:0/1912323001 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8594053ed0 con 0x7f85980b1ed0 2024-01-20T11:19:11.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.418+0000 7f858effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:11.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.418+0000 7f858effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:11.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.592+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f859814c7f0 con 0x7f857c06c860 2024-01-20T11:19:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.595+0000 7f858effd640 1 -- 172.21.15.110:0/1912323001 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63970 (secure 0 0 0) 0x7f8588031990 con 0x7f857c06c860 2024-01-20T11:19:11.596 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 10 librados: watch_flush enter 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 10 librados: watch_flush exit 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 20 client.4799.objecter shutdown clearing up homeless session... 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 10 client.4799.objecter successfully canceled tick 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f857c06c860 msgr2=0x7f857c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f857c06c860 0x7f857c06ed20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8588036000 tx=0x7f858803d040 comp rx=0 tx=0).stop 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 10 monclient: shutdown 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 msgr2=0x7f85981dae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85981dae70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f859400c4d0 tx=0x7f859400c990 comp rx=0 tx=0).stop 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 shutdown_connections 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f857c06c860 0x7f857c06ed20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8598145980 0x7f8598136430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85980b4d10 0x7f8598135ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 --2- 172.21.15.110:0/1912323001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980b1ed0 0x7f85981dae70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 >> 172.21.15.110:0/1912323001 conn(0x7f8598066590 msgr2=0x7f8598144a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:11.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 shutdown_connections 2024-01-20T11:19:11.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 -- 172.21.15.110:0/1912323001 wait complete. 2024-01-20T11:19:11.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.597+0000 7f85a0461640 1 librados: shutdown 2024-01-20T11:19:11.609 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-01-20T11:19:10.754893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002652"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.3250000000000002},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.1689999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.4220000000000006},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.412000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.0899999999999999},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.071}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":14.734},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":10.180999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.064},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":10.795999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.690999999999999},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.477}]}]},{"osd":7,"up_from":21,"seq":90194313267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.0819999999999999},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.032}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.6959999999999997},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.228}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.215},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.285}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":16.388000000000002},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.567}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":17.327999999999999},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.932}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.217000000000001},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":18.315000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313268,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.9409999999999998},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.3339999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7550000000000008},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.199000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":8.5809999999999995},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.962}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.606999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.101000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":10.682},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":13.548999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":15.484999999999999},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.863}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.0649999999999995},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":6.9820000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":4.7720000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.358000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.597},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.420999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.542},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.987}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.672000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.9729999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.7420000000000009},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":16.292999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.9939999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":7.484}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.109999999999999},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.9459999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":9.3610000000000007},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.4390000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":14.061},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.944000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.617000000000001},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":15.782}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.210000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.867000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444083,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":4.3250000000000002},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":8.0459999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.359},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.3410000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":9.6620000000000008},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.8300000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.972},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":13.065}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":10.265000000000001},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.746}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.554},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":12.163}]}]},{"osd":6,"up_from":15,"seq":64424509491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":11.664},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.8250000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.5499999999999998},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":14.401}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.752000000000001},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.521000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.768000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.4000000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.0790000000000006},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.234999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.109},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":17.707999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:11.610 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:11.610 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:11.610 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:11.610 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:11.610 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:11.611 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:11.611 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:11.611 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:11.611 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.710+0000 7f1fdc75e640 10 monclient: get_monmap_and_config 2024-01-20T11:19:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.710+0000 7f1fdc75e640 10 monclient: build_initial_monmap 2024-01-20T11:19:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.710+0000 7f1fdc75e640 10 monclient: monmap: 2024-01-20T11:19:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:11.711644+0000 2024-01-20T11:19:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:11.711644+0000 2024-01-20T11:19:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:11.751 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.710+0000 7f1fdc75e640 1 Processor -- start 2024-01-20T11:19:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.710+0000 7f1fdc75e640 1 -- start start 2024-01-20T11:19:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.710+0000 7f1fdc75e640 10 monclient: init 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1fd40a96f0 0x7f1fd413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd40a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fd413fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd40a9ad0 0x7f1fd4142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd40a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd40b25c0 con 0x7f1fd413fa10 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd40aaea0 con 0x7f1fd40a96f0 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd40ab0a0 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.711+0000 7f1fdc75e640 10 monclient(hunting): authenticate will time out at 1595.062595s 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd40a9ad0 0x7f1fd4142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fda4fa640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f1fd40a96f0 0x7f1fd413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd40a9ad0 0x7f1fd4142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47748/0 (socket says 172.21.15.110:47748) 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 1 -- 172.21.15.110:0/355574683 learned_addr learned my addr 172.21.15.110:0/355574683 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 10 monclient(hunting): get_auth_request con 0x7f1fd40a9ad0 auth_method 0 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fda4fa640 10 monclient(hunting): get_auth_request con 0x7f1fd40a96f0 auth_method 0 2024-01-20T11:19:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fda4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fda4fa640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fda4fa640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.712+0000 7f1fdacfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.713+0000 7f1fda4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.713+0000 7f1fda4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.713+0000 7f1fda4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.714+0000 7f1fdacfb640 10 monclient(hunting): handle_auth_done global_id 4802 payload 274 2024-01-20T11:19:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.714+0000 7f1fdacfb640 1 -- 172.21.15.110:0/355574683 >> v2:172.21.15.134:3300/0 conn(0x7f1fd40a96f0 msgr2=0x7f1fd413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.714+0000 7f1fdacfb640 1 --2- 172.21.15.110:0/355574683 >> v2:172.21.15.134:3300/0 conn(0x7f1fd40a96f0 0x7f1fd413f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.714+0000 7f1fdacfb640 1 -- 172.21.15.110:0/355574683 >> v1:172.21.15.134:6789/0 conn(0x7f1fd413fa10 legacy=0x7f1fd41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.714+0000 7f1fdacfb640 10 monclient: _finish_hunting 0 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.714+0000 7f1fdacfb640 1 monclient: found mon.noname-b 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.714+0000 7f1fdacfb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.714+0000 7f1fdacfb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.714+0000 7f1fdacfb640 1 -- 172.21.15.110:0/355574683 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd40b3fb0 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.716+0000 7f1fdacfb640 1 --2- 172.21.15.110:0/355574683 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd40a9ad0 0x7f1fd4142af0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009870 tx=0x7f1fd002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b776c0bbccf6b3b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 1 -- 172.21.15.110:0/355574683 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0031640 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 10 monclient: dump: 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 10 monclient: _finish_auth 0 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 10 monclient: _check_auth_tickets 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 1 -- 172.21.15.110:0/355574683 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd00317a0 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 1 -- 172.21.15.110:0/355574683 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0031b30 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fdc75e640 5 monclient: authenticate success, global_id 4802 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fdc75e640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fdc75e640 4 set_mon_vals no callback set 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.719+0000 7f1fd94f8640 10 monclient: dump: 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 10 monclient: shutdown 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 1 -- 172.21.15.110:0/355574683 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd40a9ad0 msgr2=0x7f1fd4142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 1 --2- 172.21.15.110:0/355574683 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd40a9ad0 0x7f1fd4142af0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009870 tx=0x7f1fd002b8f0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 1 -- 172.21.15.110:0/355574683 shutdown_connections 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 1 --2- 172.21.15.110:0/355574683 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd40a9ad0 0x7f1fd4142af0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 1 --2- 172.21.15.110:0/355574683 >> v2:172.21.15.134:3300/0 conn(0x7f1fd40a96f0 0x7f1fd413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 1 -- 172.21.15.110:0/355574683 >> 172.21.15.110:0/355574683 conn(0x7f1fd40b7bc0 msgr2=0x7f1fd40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 1 -- 172.21.15.110:0/355574683 shutdown_connections 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.720+0000 7f1fdc75e640 1 -- 172.21.15.110:0/355574683 wait complete. 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.721+0000 7f1fdc75e640 10 monclient: build_initial_monmap 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.721+0000 7f1fdc75e640 10 monclient: monmap: 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.721+0000 7f1fdc75e640 1 librados: starting msgr at 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.721+0000 7f1fdc75e640 1 librados: starting objecter 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.721+0000 7f1fdc75e640 1 Processor -- start 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.721+0000 7f1fdc75e640 1 -- start start 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.721+0000 7f1fdc75e640 1 librados: setting wanted keys 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.721+0000 7f1fdc75e640 1 librados: calling monclient init 2024-01-20T11:19:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.721+0000 7f1fdc75e640 10 monclient: init 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fd40a96f0 0x7f1fd413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient(hunting): picked mon.noname-a con 0x7f1fd40a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fd40a9ad0 0x7f1fd4140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd40a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd4133df0 0x7f1fd4132320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd4133df0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd41334b0 con 0x7f1fd40a96f0 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd41336f0 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd413e5e0 con 0x7f1fd4133df0 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.722+0000 7f1fdc75e640 10 monclient(hunting): authenticate will time out at 1595.073744s 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fdacfb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd4133df0 0x7f1fd4132320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fda4fa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fd40a96f0 0x7f1fd413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fd9cf9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fd40a9ad0 0x7f1fd4140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fdacfb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd4133df0 0x7f1fd4132320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47760/0 (socket says 172.21.15.110:47760) 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fdacfb640 1 -- 172.21.15.110:0/2918461175 learned_addr learned my addr 172.21.15.110:0/2918461175 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fdacfb640 10 monclient(hunting): get_auth_request con 0x7f1fd4133df0 auth_method 0 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fdacfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fdacfb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fdacfb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fd9cf9640 10 monclient(hunting): get_auth_request con 0x7f1fd40a9ad0 auth_method 0 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fd9cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fd9cf9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fd9cf9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fda4fa640 10 monclient(hunting): get_auth_request con 0x7f1fd40a96f0 auth_method 0 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fda4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fda4fa640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.723+0000 7f1fda4fa640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.724+0000 7f1fd9cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.724+0000 7f1fd9cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.724+0000 7f1fd9cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.724+0000 7f1fdacfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.724+0000 7f1fdacfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.724+0000 7f1fdacfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.724+0000 7f1fda4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.724+0000 7f1fda4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.724+0000 7f1fda4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 10 monclient(hunting): handle_auth_done global_id 5053 payload 824 2024-01-20T11:19:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 1 -- 172.21.15.110:0/2918461175 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd4133df0 msgr2=0x7f1fd4132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd4133df0 0x7f1fd4132320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 1 -- 172.21.15.110:0/2918461175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fd40a96f0 msgr2=0x7f1fd413ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fd40a96f0 0x7f1fd413ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 10 monclient: _finish_hunting 0 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 1 monclient: found mon.noname-b 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.726+0000 7f1fd9cf9640 1 -- 172.21.15.110:0/2918461175 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd00094d0 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.729+0000 7f1fd9cf9640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fd40a9ad0 0x7f1fd4140b30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1fd40aac70 tx=0x7f1fc800cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.729+0000 7f1fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f1fd40a9ad0 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.729+0000 7f1fc6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 1 -- 172.21.15.110:0/2918461175 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8006d50 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:11.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 10 monclient: dump: 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 1 -- 172.21.15.110:0/2918461175 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc8003970 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 1 -- 172.21.15.110:0/2918461175 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8003d60 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fc6ffd640 10 monclient: dump: 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:11.772 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fdc75e640 5 monclient: authenticate success, global_id 5053 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fdc75e640 10 monclient: _renew_subs 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fdc75e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd413e2d0 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.732+0000 7f1fd8cf7640 4 set_mon_vals no callback set 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.733+0000 7f1fdc75e640 10 client.5053.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.733+0000 7f1fdc75e640 10 monclient: _renew_subs 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.733+0000 7f1fdc75e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.733+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd4134270 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.733+0000 7f1fdc75e640 1 librados: init done 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.734+0000 7f1fdc75e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.734+0000 7f1fdc75e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.734+0000 7f1fdc75e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.734+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd413e800 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.751+0000 7f1fc6ffd640 1 -- 172.21.15.110:0/2918461175 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1fc80050d0 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.752+0000 7f1fc6ffd640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1fb806c860 0x7f1fb806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.752+0000 7f1fda4fa640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1fb806c860 0x7f1fb806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.752+0000 7f1fda4fa640 10 monclient: get_auth_request con 0x7f1fb806c860 auth_method 0 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.752+0000 7f1fda4fa640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1fb806c860 0x7f1fb806ed20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009b40 tx=0x7f1fcc009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.752+0000 7f1fc6ffd640 10 client.5053.objecter ms_handle_connect 0x7f1fb806c860 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.756+0000 7f1fc6ffd640 1 -- 172.21.15.110:0/2918461175 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1fc80589f0 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.756+0000 7f1fc6ffd640 10 client.5053.objecter ms_dispatch 0x7f1fd4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.756+0000 7f1fc6ffd640 3 client.5053.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.756+0000 7f1fc6ffd640 3 client.5053.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.756+0000 7f1fc6ffd640 20 client.5053.objecter dump_active .. 0 homeless 2024-01-20T11:19:11.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.985+0000 7f1fc6ffd640 1 -- 172.21.15.110:0/2918461175 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1fc8054d30 con 0x7f1fd40a9ad0 2024-01-20T11:19:11.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.985+0000 7f1fc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:11.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:11.985+0000 7f1fc6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:12.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.228+0000 7f1fdc75e640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:12.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.228+0000 7f1fdc75e640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:12.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.228+0000 7f1fdc75e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:12.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.228+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1fd4075490 con 0x7f1fd40a9ad0 2024-01-20T11:19:12.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.235+0000 7f1fc6ffd640 1 -- 172.21.15.110:0/2918461175 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f1fc8058350 con 0x7f1fd40a9ad0 2024-01-20T11:19:12.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.235+0000 7f1fc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:12.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.235+0000 7f1fc6ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:12.236 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:12.236 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":286,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.236+0000 7f1fdc75e640 10 librados: watch_flush enter 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.236+0000 7f1fdc75e640 10 librados: watch_flush exit 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.236+0000 7f1fdc75e640 20 client.5053.objecter shutdown clearing up homeless session... 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.236+0000 7f1fdc75e640 10 client.5053.objecter successfully canceled tick 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.236+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1fb806c860 msgr2=0x7f1fb806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.236+0000 7f1fdc75e640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1fb806c860 0x7f1fb806ed20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009b40 tx=0x7f1fcc009190 comp rx=0 tx=0).stop 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.236+0000 7f1fdc75e640 10 monclient: shutdown 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.236+0000 7f1fdc75e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.236+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fd40a9ad0 msgr2=0x7f1fd4140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fd40a9ad0 0x7f1fd4140b30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1fd40aac70 tx=0x7f1fc800cb90 comp rx=0 tx=0).stop 2024-01-20T11:19:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 shutdown_connections 2024-01-20T11:19:12.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1fb806c860 0x7f1fb806ed20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fd4133df0 0x7f1fd4132320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fd40a9ad0 0x7f1fd4140b30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 --2- 172.21.15.110:0/2918461175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fd40a96f0 0x7f1fd413ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 >> 172.21.15.110:0/2918461175 conn(0x7f1fd40b7bc0 msgr2=0x7f1fd41373a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:12.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 shutdown_connections 2024-01-20T11:19:12.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 -- 172.21.15.110:0/2918461175 wait complete. 2024-01-20T11:19:12.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.237+0000 7f1fdc75e640 1 librados: shutdown 2024-01-20T11:19:12.248 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:12.249 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.347+0000 7f23a944f640 10 monclient: get_monmap_and_config 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.347+0000 7f23a944f640 10 monclient: build_initial_monmap 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.347+0000 7f23a944f640 10 monclient: monmap: 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:12.348749+0000 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:12.348749+0000 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:12.349 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:12.349 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:12.349 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.347+0000 7f23a944f640 1 Processor -- start 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.347+0000 7f23a944f640 1 -- start start 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.347+0000 7f23a944f640 10 monclient: init 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f23a40a96f0 0x7f23a413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 10 monclient(hunting): picked mon.noname-c con 0x7f23a40a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23a413fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23a40a9ad0 0x7f23a4142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 10 monclient(hunting): picked mon.noname-a con 0x7f23a40a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a40b25c0 con 0x7f23a413fa10 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a40aaea0 con 0x7f23a40a9ad0 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a40ab0a0 con 0x7f23a40a96f0 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.348+0000 7f23a944f640 10 monclient(hunting): authenticate will time out at 1595.699728s 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a2ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f23a40a96f0 0x7f23a413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a37fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23a40a9ad0 0x7f23a4142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a2ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f23a40a96f0 0x7f23a413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:40114/0 (socket says 172.21.15.110:40114) 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a2ffd640 1 -- 172.21.15.110:0/2486372870 learned_addr learned my addr 172.21.15.110:0/2486372870 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a2ffd640 10 monclient(hunting): get_auth_request con 0x7f23a40a96f0 auth_method 0 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a37fe640 10 monclient(hunting): get_auth_request con 0x7f23a40a9ad0 auth_method 0 2024-01-20T11:19:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.349+0000 7f23a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.350+0000 7f23a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.350+0000 7f23a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.350+0000 7f23a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.350+0000 7f23a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.350+0000 7f23a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.350+0000 7f23a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.351+0000 7f23a37fe640 10 monclient(hunting): handle_auth_done global_id 4968 payload 274 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.351+0000 7f23a37fe640 1 -- 172.21.15.110:0/2486372870 >> v2:172.21.15.134:3300/0 conn(0x7f23a40a96f0 msgr2=0x7f23a413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.351+0000 7f23a37fe640 1 --2- 172.21.15.110:0/2486372870 >> v2:172.21.15.134:3300/0 conn(0x7f23a40a96f0 0x7f23a413f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.351+0000 7f23a37fe640 1 -- 172.21.15.110:0/2486372870 >> v1:172.21.15.134:6789/0 conn(0x7f23a413fa10 legacy=0x7f23a41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.351+0000 7f23a37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.351+0000 7f23a37fe640 1 monclient: found mon.noname-a 2024-01-20T11:19:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.351+0000 7f23a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.351+0000 7f23a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.351+0000 7f23a37fe640 1 -- 172.21.15.110:0/2486372870 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40b3fb0 con 0x7f23a40a9ad0 2024-01-20T11:19:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.353+0000 7f23a37fe640 1 --2- 172.21.15.110:0/2486372870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a40a9ad0 0x7f23a4142af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23a40b5720 tx=0x7f239c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7ec20363ed4603 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 1 -- 172.21.15.110:0/2486372870 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c02cbf0 con 0x7f23a40a9ad0 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 10 monclient: dump: 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 1 -- 172.21.15.110:0/2486372870 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c02cd50 con 0x7f23a40a9ad0 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 5 monclient: authenticate success, global_id 4968 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 4 set_mon_vals no callback set 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 1 -- 172.21.15.110:0/2486372870 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c034660 con 0x7f23a40a9ad0 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a27fc640 10 monclient: dump: 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:12.360 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 10 monclient: shutdown 2024-01-20T11:19:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 1 -- 172.21.15.110:0/2486372870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a40a9ad0 msgr2=0x7f23a4142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 1 --2- 172.21.15.110:0/2486372870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a40a9ad0 0x7f23a4142af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23a40b5720 tx=0x7f239c02be40 comp rx=0 tx=0).stop 2024-01-20T11:19:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 1 -- 172.21.15.110:0/2486372870 shutdown_connections 2024-01-20T11:19:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 1 --2- 172.21.15.110:0/2486372870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a40a9ad0 0x7f23a4142af0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.357+0000 7f23a944f640 1 --2- 172.21.15.110:0/2486372870 >> v2:172.21.15.134:3300/0 conn(0x7f23a40a96f0 0x7f23a413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.358+0000 7f23a944f640 1 -- 172.21.15.110:0/2486372870 >> 172.21.15.110:0/2486372870 conn(0x7f23a40b7bc0 msgr2=0x7f23a40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.358+0000 7f23a944f640 1 -- 172.21.15.110:0/2486372870 shutdown_connections 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.358+0000 7f23a944f640 1 -- 172.21.15.110:0/2486372870 wait complete. 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.358+0000 7f23a944f640 10 monclient: build_initial_monmap 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.358+0000 7f23a944f640 10 monclient: monmap: 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.359+0000 7f23a944f640 1 librados: starting msgr at 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.359+0000 7f23a944f640 1 librados: starting objecter 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.359+0000 7f23a944f640 1 Processor -- start 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.359+0000 7f23a944f640 1 -- start start 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.359+0000 7f23a944f640 1 librados: setting wanted keys 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.359+0000 7f23a944f640 1 librados: calling monclient init 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.359+0000 7f23a944f640 10 monclient: init 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23a40a96f0 0x7f23a4137f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient(hunting): picked mon.noname-b con 0x7f23a40a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a40a9ad0 0x7f23a4133ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient(hunting): picked mon.noname-c con 0x7f23a40a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a4134530 0x7f23a41dac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient(hunting): picked mon.noname-a con 0x7f23a4134530 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a4139b10 con 0x7f23a4134530 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a4138530 con 0x7f23a40a96f0 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a4139d10 con 0x7f23a40a9ad0 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a944f640 10 monclient(hunting): authenticate will time out at 1595.711268s 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a4134530 0x7f23a41dac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f239bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a40a9ad0 0x7f23a4133ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a2ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23a40a96f0 0x7f23a4137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a4134530 0x7f23a41dac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43858/0 (socket says 172.21.15.110:43858) 2024-01-20T11:19:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a37fe640 1 -- 172.21.15.110:0/524205516 learned_addr learned my addr 172.21.15.110:0/524205516 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a37fe640 10 monclient(hunting): get_auth_request con 0x7f23a4134530 auth_method 0 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.360+0000 7f23a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f239bfff640 10 monclient(hunting): get_auth_request con 0x7f23a40a9ad0 auth_method 0 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f239bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f239bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f239bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a2ffd640 10 monclient(hunting): get_auth_request con 0x7f23a40a96f0 auth_method 0 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:12.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f239bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:12.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f239bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:12.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f239bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:12.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:12.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:12.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.361+0000 7f23a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 10 monclient(hunting): handle_auth_done global_id 4808 payload 824 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 1 -- 172.21.15.110:0/524205516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23a40a96f0 msgr2=0x7f23a4137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23a40a96f0 0x7f23a4137f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 1 -- 172.21.15.110:0/524205516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a4134530 msgr2=0x7f23a41dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a4134530 0x7f23a41dac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 1 monclient: found mon.noname-c 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f239bfff640 1 -- 172.21.15.110:0/524205516 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c009a70 con 0x7f23a40a9ad0 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.364+0000 7f23a37fe640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a4134530 0x7f23a41dac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.366+0000 7f239bfff640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a40a9ad0 0x7f23a4133ff0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f23a4067980 tx=0x7f238c00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.366+0000 7f239b7fe640 10 client.?.objecter ms_handle_connect 0x7f23a40a9ad0 2024-01-20T11:19:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.366+0000 7f239b7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 1 -- 172.21.15.110:0/524205516 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c0052d0 con 0x7f23a40a9ad0 2024-01-20T11:19:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 10 monclient: dump: 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 1 -- 172.21.15.110:0/524205516 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c005450 con 0x7f23a40a9ad0 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 1 -- 172.21.15.110:0/524205516 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c007260 con 0x7f23a40a9ad0 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f239b7fe640 10 monclient: dump: 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a944f640 5 monclient: authenticate success, global_id 4808 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a944f640 10 monclient: _renew_subs 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a944f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a1ffb640 4 set_mon_vals no callback set 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a413a050 con 0x7f23a40a9ad0 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a944f640 10 client.4808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a944f640 10 monclient: _renew_subs 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a944f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a41365e0 con 0x7f23a40a9ad0 2024-01-20T11:19:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.370+0000 7f23a944f640 1 librados: init done 2024-01-20T11:19:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.371+0000 7f23a944f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.371+0000 7f23a944f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.371+0000 7f23a944f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.371+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a41386f0 con 0x7f23a40a9ad0 2024-01-20T11:19:12.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.615+0000 7f239b7fe640 1 -- 172.21.15.110:0/524205516 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f238c0073c0 con 0x7f23a40a9ad0 2024-01-20T11:19:12.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.615+0000 7f239b7fe640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f238406c8b0 0x7f238406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:12.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.615+0000 7f23a2ffd640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f238406c8b0 0x7f238406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:12.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.615+0000 7f23a2ffd640 10 monclient: get_auth_request con 0x7f238406c8b0 auth_method 0 2024-01-20T11:19:12.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.616+0000 7f239b7fe640 10 client.4808.objecter ms_handle_connect 0x7f238406c8b0 2024-01-20T11:19:12.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.616+0000 7f23a2ffd640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f238406c8b0 0x7f238406ed70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2394009bd0 tx=0x7f2394009240 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:12.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.632+0000 7f239b7fe640 1 -- 172.21.15.110:0/524205516 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f238c057c60 con 0x7f23a40a9ad0 2024-01-20T11:19:12.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.632+0000 7f239b7fe640 10 client.4808.objecter ms_dispatch 0x7f23a4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:12.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.632+0000 7f239b7fe640 3 client.4808.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:12.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.632+0000 7f239b7fe640 3 client.4808.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:12.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.632+0000 7f239b7fe640 20 client.4808.objecter dump_active .. 0 homeless 2024-01-20T11:19:12.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.633+0000 7f239b7fe640 1 -- 172.21.15.110:0/524205516 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f238c053fa0 con 0x7f23a40a9ad0 2024-01-20T11:19:12.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.633+0000 7f239b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:12.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.633+0000 7f239b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.802+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23a4075490 con 0x7f238406c8b0 2024-01-20T11:19:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.805+0000 7f239b7fe640 1 -- 172.21.15.110:0/524205516 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63919 (secure 0 0 0) 0x7f23a4075490 con 0x7f238406c8b0 2024-01-20T11:19:12.806 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.806+0000 7f23a944f640 10 librados: watch_flush enter 2024-01-20T11:19:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.806+0000 7f23a944f640 10 librados: watch_flush exit 2024-01-20T11:19:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.806+0000 7f23a944f640 20 client.4808.objecter shutdown clearing up homeless session... 2024-01-20T11:19:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.806+0000 7f23a944f640 10 client.4808.objecter successfully canceled tick 2024-01-20T11:19:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f238406c8b0 msgr2=0x7f238406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f238406c8b0 0x7f238406ed70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2394009bd0 tx=0x7f2394009240 comp rx=0 tx=0).stop 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 10 monclient: shutdown 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a40a9ad0 msgr2=0x7f23a4133ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a40a9ad0 0x7f23a4133ff0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f23a4067980 tx=0x7f238c00c820 comp rx=0 tx=0).stop 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 shutdown_connections 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f238406c8b0 0x7f238406ed70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a4134530 0x7f23a41dac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a40a9ad0 0x7f23a4133ff0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 --2- 172.21.15.110:0/524205516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23a40a96f0 0x7f23a4137f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 >> 172.21.15.110:0/524205516 conn(0x7f23a40b7bc0 msgr2=0x7f23a4137210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 shutdown_connections 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 -- 172.21.15.110:0/524205516 wait complete. 2024-01-20T11:19:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:12.807+0000 7f23a944f640 1 librados: shutdown 2024-01-20T11:19:12.818 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-01-20T11:19:12.755375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002775"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.3250000000000002},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.1689999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.4220000000000006},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.412000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.0899999999999999},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.071}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":14.734},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":10.180999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.064},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":10.795999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":16.690999999999999},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":15.477}]}]},{"osd":7,"up_from":21,"seq":90194313268,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.5990000000000002},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":4.8140000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.508},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.5530000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.44},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.872}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4730000000000008},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.547000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.526},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":11.269}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.935},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.263}]}]},{"osd":3,"up_from":21,"seq":90194313269,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":11.135999999999999},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.8419999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.1999999999999993},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":6.7560000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":10.307},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.132}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.930999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.164999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":12.151},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":15.551}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":17.405999999999999},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.686}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.0649999999999995},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":6.9820000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":4.7720000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.358000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.597},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.420999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.542},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.987}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.672000000000001},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.9729999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.7420000000000009},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":16.292999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.9939999999999998},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":7.484}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.109999999999999},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.9459999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":9.3610000000000007},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.4390000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":14.061},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.944000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.617000000000001},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":15.782}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.210000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.867000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444083,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":4.3250000000000002},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":8.0459999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.359},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.3410000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":9.6620000000000008},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.8300000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.972},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":13.065}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":10.265000000000001},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":14.746}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.554},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":12.163}]}]},{"osd":6,"up_from":15,"seq":64424509492,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":7.79},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.4790000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.706},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":9.0779999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.805999999999999},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.673}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.877000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":16.809000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.672999999999998},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.260999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.368},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":9.8889999999999993}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:15.820 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:15.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.920+0000 7f4e203e9640 10 monclient: get_monmap_and_config 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.920+0000 7f4e203e9640 10 monclient: build_initial_monmap 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.920+0000 7f4e203e9640 10 monclient: monmap: 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:15.922142+0000 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:15.922142+0000 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:15.926 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 1 Processor -- start 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 1 -- start start 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient: init 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e1806bb00 0x7f4e180b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient(hunting): picked mon.noname-b con 0x7f4e1806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4e180af790 0x7f4e18145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient(hunting): picked mon.noname-a con 0x7f4e180af790 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e180b0900 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:15.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1806c220 con 0x7f4e180b0900 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e1806bee0 con 0x7f4e180af790 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e180ada30 con 0x7f4e1806bb00 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.921+0000 7f4e203e9640 10 monclient(hunting): authenticate will time out at 1599.273074s 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.922+0000 7f4e1d984640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4e180af790 0x7f4e18145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.922+0000 7f4e1e185640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e1806bb00 0x7f4e180b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.922+0000 7f4e1e185640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e1806bb00 0x7f4e180b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47784/0 (socket says 172.21.15.110:47784) 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.922+0000 7f4e1e185640 1 -- 172.21.15.110:0/2930115065 learned_addr learned my addr 172.21.15.110:0/2930115065 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.922+0000 7f4e1e185640 10 monclient(hunting): get_auth_request con 0x7f4e1806bb00 auth_method 0 2024-01-20T11:19:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.922+0000 7f4e1e185640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.922+0000 7f4e1e185640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.922+0000 7f4e1e185640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.923+0000 7f4e1d984640 10 monclient(hunting): get_auth_request con 0x7f4e180af790 auth_method 0 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.923+0000 7f4e1d984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.923+0000 7f4e1d984640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.923+0000 7f4e1d984640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.923+0000 7f4e1e185640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.923+0000 7f4e1e185640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.923+0000 7f4e1e185640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.924+0000 7f4e1d984640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.924+0000 7f4e1d984640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.924+0000 7f4e1d984640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:15.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.925+0000 7f4e1e185640 10 monclient(hunting): handle_auth_done global_id 4811 payload 274 2024-01-20T11:19:15.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.925+0000 7f4e1e185640 1 -- 172.21.15.110:0/2930115065 >> v2:172.21.15.110:3300/0 conn(0x7f4e180af790 msgr2=0x7f4e18145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.925+0000 7f4e1e185640 1 --2- 172.21.15.110:0/2930115065 >> v2:172.21.15.110:3300/0 conn(0x7f4e180af790 0x7f4e18145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.925+0000 7f4e1e185640 1 -- 172.21.15.110:0/2930115065 >> v1:172.21.15.110:6789/0 conn(0x7f4e180b0900 legacy=0x7f4e18145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.925+0000 7f4e1e185640 10 monclient: _finish_hunting 0 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.925+0000 7f4e1e185640 1 monclient: found mon.noname-b 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.925+0000 7f4e1e185640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.925+0000 7f4e1e185640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.925+0000 7f4e1e185640 1 -- 172.21.15.110:0/2930115065 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e180add10 con 0x7f4e1806bb00 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.927+0000 7f4e1e185640 1 --2- 172.21.15.110:0/2930115065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e1806bb00 0x7f4e180b03c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009870 tx=0x7f4e1402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62bd04a8c4073fbb server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.930+0000 7f4e1d183640 1 -- 172.21.15.110:0/2930115065 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14031640 con 0x7f4e1806bb00 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.930+0000 7f4e1d183640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.930+0000 7f4e1d183640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.930+0000 7f4e1d183640 10 monclient: dump: 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:15.931 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.930+0000 7f4e1d183640 10 monclient: _finish_auth 0 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.930+0000 7f4e1d183640 10 monclient: _check_auth_tickets 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.930+0000 7f4e1d183640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.930+0000 7f4e203e9640 5 monclient: authenticate success, global_id 4811 2024-01-20T11:19:15.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.930+0000 7f4e203e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:15.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.931+0000 7f4e1d183640 1 -- 172.21.15.110:0/2930115065 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e140317a0 con 0x7f4e1806bb00 2024-01-20T11:19:15.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.931+0000 7f4e1d183640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:15.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.931+0000 7f4e1d183640 1 -- 172.21.15.110:0/2930115065 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14031b30 con 0x7f4e1806bb00 2024-01-20T11:19:15.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.931+0000 7f4e1d183640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:15.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.931+0000 7f4e1d183640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:15.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.931+0000 7f4e1d183640 10 monclient: dump: 2024-01-20T11:19:15.934 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:15.934 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.931+0000 7f4e203e9640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.931+0000 7f4e203e9640 4 set_mon_vals no callback set 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 10 monclient: shutdown 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 1 -- 172.21.15.110:0/2930115065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e1806bb00 msgr2=0x7f4e180b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:15.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 1 --2- 172.21.15.110:0/2930115065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e1806bb00 0x7f4e180b03c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009870 tx=0x7f4e1402b8f0 comp rx=0 tx=0).stop 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 1 -- 172.21.15.110:0/2930115065 shutdown_connections 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 1 --2- 172.21.15.110:0/2930115065 >> v2:172.21.15.110:3300/0 conn(0x7f4e180af790 0x7f4e18145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 1 --2- 172.21.15.110:0/2930115065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e1806bb00 0x7f4e180b03c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 1 -- 172.21.15.110:0/2930115065 >> 172.21.15.110:0/2930115065 conn(0x7f4e18066d60 msgr2=0x7f4e180b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 1 -- 172.21.15.110:0/2930115065 shutdown_connections 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.932+0000 7f4e203e9640 1 -- 172.21.15.110:0/2930115065 wait complete. 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.933+0000 7f4e203e9640 10 monclient: build_initial_monmap 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.933+0000 7f4e203e9640 10 monclient: monmap: 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:15.936 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.933+0000 7f4e203e9640 1 librados: starting msgr at 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.933+0000 7f4e203e9640 1 librados: starting objecter 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.933+0000 7f4e203e9640 1 Processor -- start 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.933+0000 7f4e203e9640 1 -- start start 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.933+0000 7f4e203e9640 1 librados: setting wanted keys 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.933+0000 7f4e203e9640 1 librados: calling monclient init 2024-01-20T11:19:15.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.933+0000 7f4e203e9640 10 monclient: init 2024-01-20T11:19:15.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:15.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:19:15.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1806bb00 0x7f4e18120040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:15.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient(hunting): picked mon.noname-a con 0x7f4e1806bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e180af790 0x7f4e18120c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient(hunting): picked mon.noname-c con 0x7f4e180af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e180b0900 0x7f4e181d7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient(hunting): picked mon.noname-b con 0x7f4e180b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e18121180 con 0x7f4e1806bb00 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e18121ea0 con 0x7f4e180b0900 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e181d7470 con 0x7f4e180af790 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e203e9640 10 monclient(hunting): authenticate will time out at 1599.285608s 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e1d984640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e180af790 0x7f4e18120c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:15.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.934+0000 7f4e1e185640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1806bb00 0x7f4e18120040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e185640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1806bb00 0x7f4e18120040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43888/0 (socket says 172.21.15.110:43888) 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1d984640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e180af790 0x7f4e18120c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47788/0 (socket says 172.21.15.110:47788) 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e185640 1 -- 172.21.15.110:0/910030339 learned_addr learned my addr 172.21.15.110:0/910030339 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e185640 10 monclient(hunting): get_auth_request con 0x7f4e1806bb00 auth_method 0 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e185640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e185640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e185640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1d984640 10 monclient(hunting): get_auth_request con 0x7f4e180af790 auth_method 0 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1d984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1d984640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1d984640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e986640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e180b0900 0x7f4e181d7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e986640 10 monclient(hunting): get_auth_request con 0x7f4e180b0900 auth_method 0 2024-01-20T11:19:15.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e986640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e986640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1d984640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1d984640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1d984640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e185640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e185640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.935+0000 7f4e1e185640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.936+0000 7f4e1e986640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.936+0000 7f4e1e986640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:15.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.936+0000 7f4e1e986640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:15.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 10 monclient(hunting): handle_auth_done global_id 5065 payload 824 2024-01-20T11:19:15.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 1 -- 172.21.15.110:0/910030339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e180af790 msgr2=0x7f4e18120c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:15.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e180af790 0x7f4e18120c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:15.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 1 -- 172.21.15.110:0/910030339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1806bb00 msgr2=0x7f4e18120040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1806bb00 0x7f4e18120040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 10 monclient: _finish_hunting 0 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 1 monclient: found mon.noname-b 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1e986640 1 -- 172.21.15.110:0/910030339 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e140094d0 con 0x7f4e180b0900 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.939+0000 7f4e1d984640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e180af790 0x7f4e18120c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.942+0000 7f4e1e986640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e180b0900 0x7f4e181d7990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000c700 tx=0x7f4e1000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.942+0000 7f4e0effd640 10 client.?.objecter ms_handle_connect 0x7f4e180b0900 2024-01-20T11:19:15.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.942+0000 7f4e0effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 1 -- 172.21.15.110:0/910030339 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e10003b70 con 0x7f4e180b0900 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 10 monclient: dump: 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:15.948 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 10 monclient: _finish_auth 0 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 1 -- 172.21.15.110:0/910030339 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e10003cd0 con 0x7f4e180b0900 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 1 -- 172.21.15.110:0/910030339 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e10005310 con 0x7f4e180b0900 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e203e9640 5 monclient: authenticate success, global_id 5065 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e203e9640 10 monclient: _renew_subs 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e203e9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1811d5a0 con 0x7f4e180b0900 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:15.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e0effd640 10 monclient: dump: 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.946+0000 7f4e1c982640 4 set_mon_vals no callback set 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.947+0000 7f4e203e9640 10 client.5065.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.947+0000 7f4e203e9640 10 monclient: _renew_subs 2024-01-20T11:19:15.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.947+0000 7f4e203e9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:15.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.947+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e181d7ed0 con 0x7f4e180b0900 2024-01-20T11:19:15.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.947+0000 7f4e203e9640 1 librados: init done 2024-01-20T11:19:15.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.947+0000 7f4e203e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:15.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.947+0000 7f4e203e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:15.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.947+0000 7f4e203e9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:15.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:15.947+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e181d7690 con 0x7f4e180b0900 2024-01-20T11:19:16.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.191+0000 7f4e0effd640 1 -- 172.21.15.110:0/910030339 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4e10005470 con 0x7f4e180b0900 2024-01-20T11:19:16.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.191+0000 7f4e0effd640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dfc06c8b0 0x7f4dfc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:16.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.191+0000 7f4e1e185640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dfc06c8b0 0x7f4dfc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:16.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.191+0000 7f4e1e185640 10 monclient: get_auth_request con 0x7f4dfc06c8b0 auth_method 0 2024-01-20T11:19:16.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.192+0000 7f4e0effd640 10 client.5065.objecter ms_handle_connect 0x7f4dfc06c8b0 2024-01-20T11:19:16.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.192+0000 7f4e1e185640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dfc06c8b0 0x7f4dfc06ed70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009870 tx=0x7f4e1403c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.194+0000 7f4e0effd640 1 -- 172.21.15.110:0/910030339 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4e100588b0 con 0x7f4e180b0900 2024-01-20T11:19:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.194+0000 7f4e0effd640 10 client.5065.objecter ms_dispatch 0x7f4e18000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.194+0000 7f4e0effd640 3 client.5065.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.194+0000 7f4e0effd640 3 client.5065.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.194+0000 7f4e0effd640 20 client.5065.objecter dump_active .. 0 homeless 2024-01-20T11:19:16.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.205+0000 7f4e0effd640 1 -- 172.21.15.110:0/910030339 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4e10054bf0 con 0x7f4e180b0900 2024-01-20T11:19:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.205+0000 7f4e0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.205+0000 7f4e0effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:16.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.400+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e1813fda0 con 0x7f4dfc06c8b0 2024-01-20T11:19:16.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.403+0000 7f4e0effd640 1 -- 172.21.15.110:0/910030339 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63955 (secure 0 0 0) 0x7f4e1813fda0 con 0x7f4dfc06c8b0 2024-01-20T11:19:16.404 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:16.405 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 10 librados: watch_flush enter 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 10 librados: watch_flush exit 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 20 client.5065.objecter shutdown clearing up homeless session... 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 10 client.5065.objecter successfully canceled tick 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dfc06c8b0 msgr2=0x7f4dfc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dfc06c8b0 0x7f4dfc06ed70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009870 tx=0x7f4e1403c040 comp rx=0 tx=0).stop 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 10 monclient: shutdown 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e180b0900 msgr2=0x7f4e181d7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e180b0900 0x7f4e181d7990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000c700 tx=0x7f4e1000cbc0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 shutdown_connections 2024-01-20T11:19:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dfc06c8b0 0x7f4dfc06ed70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e180b0900 0x7f4e181d7990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e180af790 0x7f4e18120c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 --2- 172.21.15.110:0/910030339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1806bb00 0x7f4e18120040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 >> 172.21.15.110:0/910030339 conn(0x7f4e18066d60 msgr2=0x7f4e181429e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 shutdown_connections 2024-01-20T11:19:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 -- 172.21.15.110:0/910030339 wait complete. 2024-01-20T11:19:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.405+0000 7f4e203e9640 1 librados: shutdown 2024-01-20T11:19:16.416 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-01-20T11:19:14.755935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002922"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1429999999999998},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":13.545999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.255000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.324}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.753},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.968999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.9770000000000003},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":10.936999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.904},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":14.885999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7029999999999994},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":16.803000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313268,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.5990000000000002},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":4.8140000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.508},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.5530000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.44},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":13.872}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4730000000000008},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":14.547000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.526},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":11.269}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.935},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.263}]}]},{"osd":3,"up_from":21,"seq":90194313269,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":11.135999999999999},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.8419999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.1999999999999993},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":6.7560000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":10.307},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.132}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.930999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.164999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":12.151},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":15.551}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":17.405999999999999},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.686}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.4889999999999999},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":11.525}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7720000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.9009999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.523},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.202}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.930999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.183}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.811},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7220000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.0749999999999993},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":14.885999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444084,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.7809999999999997},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":9.0030000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.35},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.9400000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":11.436999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":14.593999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":13.945},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.510999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.8630000000000004},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":16.497}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.191000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.382999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444084,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":7.7750000000000004},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":5.9500000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.6630000000000003},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.031000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":11.698},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.4039999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":9.6189999999999998},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":15.127000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":13.635999999999999},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":10.308}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.335000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":9.0540000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509492,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":7.79},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.4790000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.706},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":9.0779999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":15.805999999999999},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.673}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.877000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":16.809000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.672999999999998},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.260999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.368},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":9.8889999999999993}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:16.418 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:16.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.518+0000 7fbb51d41640 10 monclient: get_monmap_and_config 2024-01-20T11:19:16.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.518+0000 7fbb51d41640 10 monclient: build_initial_monmap 2024-01-20T11:19:16.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.518+0000 7fbb51d41640 10 monclient: monmap: 2024-01-20T11:19:16.519 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:16.519 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:16.519 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:16.519864+0000 2024-01-20T11:19:16.519 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:16.519864+0000 2024-01-20T11:19:16.519 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:16.519 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:16.520 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:16.520 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:16.520 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:16.520 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:16.520 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:16.520 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:16.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 1 Processor -- start 2024-01-20T11:19:16.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 1 -- start start 2024-01-20T11:19:16.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 10 monclient: init 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb4c0a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb4c0a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb4c142af0 0x7fbb4c14d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 10 monclient(hunting): picked mon.noname-a con 0x7fbb4c142af0 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4c0b25c0 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4c142f50 con 0x7fbb4c0a9ad0 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4c0aaea0 con 0x7fbb4c142af0 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.519+0000 7fbb51d41640 10 monclient(hunting): authenticate will time out at 1599.870946s 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.520+0000 7fbb4bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb4c142af0 0x7fbb4c14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:16.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.520+0000 7fbb42ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbb4c0a9ad0 0x7fbb4c1423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:48788/0 (socket says 172.21.15.110:48788) 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.520+0000 7fbb42ffd640 1 -- 172.21.15.110:0/961224983 learned_addr learned my addr 172.21.15.110:0/961224983 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.520+0000 7fbb4bfff640 10 monclient(hunting): get_auth_request con 0x7fbb4c142af0 auth_method 0 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.520+0000 7fbb4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.520+0000 7fbb4bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.520+0000 7fbb4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.521+0000 7fbb4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.521+0000 7fbb4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.521+0000 7fbb4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.522+0000 7fbb4affd640 1 -- 172.21.15.110:0/961224983 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695781012 0 0) 0x7fbb34001110 con 0x7fbb4c0a9ad0 2024-01-20T11:19:16.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.522+0000 7fbb4affd640 10 monclient(hunting): _negotiate 2024-01-20T11:19:16.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.522+0000 7fbb4affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:16.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.522+0000 7fbb4affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:16.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.522+0000 7fbb4affd640 10 monclient(hunting): my global_id is 5068 2024-01-20T11:19:16.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.522+0000 7fbb4affd640 1 -- 172.21.15.110:0/961224983 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb38003520 con 0x7fbb4c0a9ad0 2024-01-20T11:19:16.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.523+0000 7fbb4bfff640 10 monclient(hunting): handle_auth_done global_id 4983 payload 274 2024-01-20T11:19:16.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.523+0000 7fbb4bfff640 1 -- 172.21.15.110:0/961224983 >> v1:172.21.15.134:6789/0 conn(0x7fbb4c0a9ad0 legacy=0x7fbb4c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:16.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.523+0000 7fbb4bfff640 1 -- 172.21.15.110:0/961224983 >> v1:172.21.15.110:6789/0 conn(0x7fbb4c0a96f0 legacy=0x7fbb4c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:16.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.523+0000 7fbb4bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:16.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.523+0000 7fbb4bfff640 1 monclient: found mon.noname-a 2024-01-20T11:19:16.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.523+0000 7fbb4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:16.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.523+0000 7fbb4bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:16.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.523+0000 7fbb4bfff640 1 -- 172.21.15.110:0/961224983 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c13fbe0 con 0x7fbb4c142af0 2024-01-20T11:19:16.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.527+0000 7fbb4bfff640 1 --2- 172.21.15.110:0/961224983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 0x7fbb4c14d860 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009d90 tx=0x7fbb4402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555c90240a857490 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:16.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 1 -- 172.21.15.110:0/961224983 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4402cc60 con 0x7fbb4c142af0 2024-01-20T11:19:16.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:16.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:16.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 10 monclient: dump: 2024-01-20T11:19:16.535 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:16.535 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 10 monclient: _finish_auth 0 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 1 -- 172.21.15.110:0/961224983 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4402cdc0 con 0x7fbb4c142af0 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:16.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 1 -- 172.21.15.110:0/961224983 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44034660 con 0x7fbb4c142af0 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 5 monclient: authenticate success, global_id 4983 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 4 set_mon_vals no callback set 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb4affd640 10 monclient: dump: 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:16.537 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:16.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 10 monclient: shutdown 2024-01-20T11:19:16.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:16.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 1 -- 172.21.15.110:0/961224983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 msgr2=0x7fbb4c14d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:16.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 1 --2- 172.21.15.110:0/961224983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 0x7fbb4c14d860 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009d90 tx=0x7fbb4402beb0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 1 -- 172.21.15.110:0/961224983 shutdown_connections 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 1 --2- 172.21.15.110:0/961224983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 0x7fbb4c14d860 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 1 -- 172.21.15.110:0/961224983 >> 172.21.15.110:0/961224983 conn(0x7fbb4c0b7bc0 msgr2=0x7fbb4c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 1 -- 172.21.15.110:0/961224983 shutdown_connections 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.532+0000 7fbb51d41640 1 -- 172.21.15.110:0/961224983 wait complete. 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.533+0000 7fbb51d41640 10 monclient: build_initial_monmap 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.533+0000 7fbb51d41640 10 monclient: monmap: 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:16.542 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.533+0000 7fbb51d41640 1 librados: starting msgr at 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.533+0000 7fbb51d41640 1 librados: starting objecter 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.533+0000 7fbb51d41640 1 Processor -- start 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 1 -- start start 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 1 librados: setting wanted keys 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 1 librados: calling monclient init 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient: init 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4c0a96f0 0x7fbb4c134ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient(hunting): picked mon.noname-c con 0x7fbb4c0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4c0a9ad0 0x7fbb4c1386c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:16.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient(hunting): picked mon.noname-b con 0x7fbb4c0a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 0x7fbb4c132cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient(hunting): picked mon.noname-a con 0x7fbb4c142af0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c13a3f0 con 0x7fbb4c142af0 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c138fb0 con 0x7fbb4c0a9ad0 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c139440 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.534+0000 7fbb51d41640 10 monclient(hunting): authenticate will time out at 1599.885892s 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 0x7fbb4c132cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4c0a96f0 0x7fbb4c134ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 0x7fbb4c132cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43898/0 (socket says 172.21.15.110:43898) 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4bfff640 1 -- 172.21.15.110:0/389086331 learned_addr learned my addr 172.21.15.110:0/389086331 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:16.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4bfff640 10 monclient(hunting): get_auth_request con 0x7fbb4c142af0 auth_method 0 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb42ffd640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4c0a9ad0 0x7fbb4c1386c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4b7fe640 10 monclient(hunting): get_auth_request con 0x7fbb4c0a96f0 auth_method 0 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb42ffd640 10 monclient(hunting): get_auth_request con 0x7fbb4c0a9ad0 auth_method 0 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb42ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.535+0000 7fbb42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.536+0000 7fbb4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:16.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.536+0000 7fbb4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.536+0000 7fbb4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.536+0000 7fbb4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.536+0000 7fbb4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.536+0000 7fbb4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.536+0000 7fbb42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.536+0000 7fbb42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.536+0000 7fbb42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 10 monclient(hunting): handle_auth_done global_id 4817 payload 824 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 1 -- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4c0a9ad0 msgr2=0x7fbb4c1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4c0a9ad0 0x7fbb4c1386c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 1 -- 172.21.15.110:0/389086331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 msgr2=0x7fbb4c132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 0x7fbb4c132cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 1 monclient: found mon.noname-c 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:16.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:16.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.538+0000 7fbb4b7fe640 1 -- 172.21.15.110:0/389086331 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb44009a70 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.541+0000 7fbb4b7fe640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4c0a96f0 0x7fbb4c134ca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c370 tx=0x7fbb3c00c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:16.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.541+0000 7fbb48ff9640 10 client.?.objecter ms_handle_connect 0x7fbb4c0a96f0 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.541+0000 7fbb48ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 1 -- 172.21.15.110:0/389086331 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c0052d0 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 10 monclient: dump: 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:16.549 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 10 monclient: _finish_auth 0 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 1 -- 172.21.15.110:0/389086331 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c005450 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb51d41640 5 monclient: authenticate success, global_id 4817 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb51d41640 10 monclient: _renew_subs 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb51d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4c138c00 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 1 -- 172.21.15.110:0/389086331 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c007260 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb4a7fc640 4 set_mon_vals no callback set 2024-01-20T11:19:16.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb48ff9640 10 monclient: dump: 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb51d41640 10 client.4817.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb51d41640 10 monclient: _renew_subs 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb51d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:16.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4c1331f0 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.544+0000 7fbb51d41640 1 librados: init done 2024-01-20T11:19:16.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.545+0000 7fbb51d41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:16.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.545+0000 7fbb51d41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:16.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.545+0000 7fbb51d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:16.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.545+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4c074ae0 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.564+0000 7fbb48ff9640 1 -- 172.21.15.110:0/389086331 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fbb3c0073c0 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.565+0000 7fbb48ff9640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c8e0 0x7fbb2c06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:16.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.565+0000 7fbb42ffd640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c8e0 0x7fbb2c06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:16.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.565+0000 7fbb42ffd640 10 monclient: get_auth_request con 0x7fbb2c06c8e0 auth_method 0 2024-01-20T11:19:16.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.566+0000 7fbb42ffd640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c8e0 0x7fbb2c06eda0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c138120 tx=0x7fbb34007bb0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:16.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.566+0000 7fbb48ff9640 10 client.4817.objecter ms_handle_connect 0x7fbb2c06c8e0 2024-01-20T11:19:16.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~19s 2024-01-20T11:19:16.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.797+0000 7fbb48ff9640 1 -- 172.21.15.110:0/389086331 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbb3c057a10 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.797+0000 7fbb48ff9640 10 client.4817.objecter ms_dispatch 0x7fbb4c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:16.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.797+0000 7fbb48ff9640 3 client.4817.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:16.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.797+0000 7fbb48ff9640 3 client.4817.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:16.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.797+0000 7fbb48ff9640 20 client.4817.objecter dump_active .. 0 homeless 2024-01-20T11:19:16.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.797+0000 7fbb48ff9640 1 -- 172.21.15.110:0/389086331 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fbb3c053d50 con 0x7fbb4c0a96f0 2024-01-20T11:19:16.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.797+0000 7fbb48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:16.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.797+0000 7fbb48ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:16.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.968+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb4c0b4400 con 0x7fbb2c06c8e0 2024-01-20T11:19:16.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.971+0000 7fbb48ff9640 1 -- 172.21.15.110:0/389086331 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63946 (secure 0 0 0) 0x7fbb34005850 con 0x7fbb2c06c8e0 2024-01-20T11:19:16.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:16.973 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 10 librados: watch_flush enter 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 10 librados: watch_flush exit 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 20 client.4817.objecter shutdown clearing up homeless session... 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 10 client.4817.objecter successfully canceled tick 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c8e0 msgr2=0x7fbb2c06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c8e0 0x7fbb2c06eda0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c138120 tx=0x7fbb34007bb0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 10 monclient: shutdown 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4c0a96f0 msgr2=0x7fbb4c134ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4c0a96f0 0x7fbb4c134ca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c370 tx=0x7fbb3c00c830 comp rx=0 tx=0).stop 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 shutdown_connections 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb2c06c8e0 0x7fbb2c06eda0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb4c142af0 0x7fbb4c132cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb4c0a9ad0 0x7fbb4c1386c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 --2- 172.21.15.110:0/389086331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb4c0a96f0 0x7fbb4c134ca0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:16.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 >> 172.21.15.110:0/389086331 conn(0x7fbb4c0b7bc0 msgr2=0x7fbb4c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:16.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 shutdown_connections 2024-01-20T11:19:16.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 -- 172.21.15.110:0/389086331 wait complete. 2024-01-20T11:19:16.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:16.973+0000 7fbb51d41640 1 librados: shutdown 2024-01-20T11:19:16.985 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-01-20T11:19:16.756307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002837"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1429999999999998},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":13.545999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.255000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.324}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.753},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.968999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.9770000000000003},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":10.936999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.904},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":14.885999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7029999999999994},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":16.803000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313269,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.2569999999999997},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":15.512}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7590000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.551}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.896000000000001},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":16.513000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.051},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.625}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.256},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":18.236999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":17.183},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.287000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313270,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":18.129000000000001},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.0660000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.7390000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":15.366},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.016}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.901999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":18.693000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":15.975},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":12.997}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":13.942},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.183}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.4889999999999999},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":11.525}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7720000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.9009999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.523},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.202}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.930999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.183}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.811},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7220000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.0749999999999993},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":14.885999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444084,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.7809999999999997},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":9.0030000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.35},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.9400000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":11.436999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":14.593999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":13.945},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.510999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.8630000000000004},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":16.497}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.191000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.382999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444084,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":7.7750000000000004},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":5.9500000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.6630000000000003},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.031000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":11.698},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.4039999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":9.6189999999999998},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":15.127000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":13.635999999999999},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":10.308}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.335000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":9.0540000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509493,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.3789999999999996},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.0980000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.0629999999999997},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":6.9409999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.286},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.618}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.561999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.888}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.8960000000000008},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.84},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8.9250000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:16.986 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:16.986 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:16.986 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:16.986 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:16.987 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:16.987 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:16.987 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:16.987 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 128, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:16:52.902829+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:16:52.902829+0000', 'last_peered': '2024-01-20T11:16:52.902829+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:16:52.902829+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:16.987 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:17.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.083+0000 7f1493cda640 10 monclient: get_monmap_and_config 2024-01-20T11:19:17.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.083+0000 7f1493cda640 10 monclient: build_initial_monmap 2024-01-20T11:19:17.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.083+0000 7f1493cda640 10 monclient: monmap: 2024-01-20T11:19:17.084 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:17.084 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.084 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:17.084767+0000 2024-01-20T11:19:17.084 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:17.084767+0000 2024-01-20T11:19:17.084 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:17.084 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.085 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:17.085 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:17.085 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:17.085 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:17.085 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:17.085 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.083+0000 7f1493cda640 1 Processor -- start 2024-01-20T11:19:17.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.083+0000 7f1493cda640 1 -- start start 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.083+0000 7f1493cda640 10 monclient: init 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f148c0b4d10 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f148c0b1ed0 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c146090 0x7f148c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 10 monclient(hunting): picked mon.noname-b con 0x7f148c146090 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148c0aade0 con 0x7f148c0b1ed0 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148c145460 con 0x7f148c0b4d10 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c0b50f0 con 0x7f148c146090 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.084+0000 7f1493cda640 10 monclient(hunting): authenticate will time out at 1600.435742s 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.085+0000 7f1492277640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c146090 0x7f148c0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.085+0000 7f1491275640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f148c0b1ed0 0x7f148c145980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51134/0 (socket says 172.21.15.110:51134) 2024-01-20T11:19:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.085+0000 7f1491275640 1 -- 172.21.15.110:0/1017243024 learned_addr learned my addr 172.21.15.110:0/1017243024 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.085+0000 7f1492277640 10 monclient(hunting): get_auth_request con 0x7f148c146090 auth_method 0 2024-01-20T11:19:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.085+0000 7f1492277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.085+0000 7f1492277640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.085+0000 7f1492277640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.086+0000 7f1492277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.086+0000 7f1492277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.086+0000 7f1492277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:17.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.087+0000 7f1490a74640 1 -- 172.21.15.110:0/1017243024 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202302574 0 0) 0x7f148c145460 con 0x7f148c0b4d10 2024-01-20T11:19:17.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.087+0000 7f1490a74640 10 monclient(hunting): _negotiate 2024-01-20T11:19:17.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.087+0000 7f1490a74640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.087+0000 7f1490a74640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.087+0000 7f1490a74640 10 monclient(hunting): my global_id is 5074 2024-01-20T11:19:17.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.087+0000 7f1490a74640 1 -- 172.21.15.110:0/1017243024 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1478003520 con 0x7f148c0b4d10 2024-01-20T11:19:17.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.087+0000 7f1492277640 10 monclient(hunting): handle_auth_done global_id 4820 payload 274 2024-01-20T11:19:17.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.087+0000 7f1492277640 1 -- 172.21.15.110:0/1017243024 >> v1:172.21.15.134:6789/0 conn(0x7f148c0b4d10 legacy=0x7f148c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.088+0000 7f1492277640 1 -- 172.21.15.110:0/1017243024 >> v1:172.21.15.110:6789/0 conn(0x7f148c0b1ed0 legacy=0x7f148c145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.088+0000 7f1492277640 10 monclient: _finish_hunting 0 2024-01-20T11:19:17.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.088+0000 7f1492277640 1 monclient: found mon.noname-b 2024-01-20T11:19:17.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.088+0000 7f1492277640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:17.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.088+0000 7f1492277640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:17.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.088+0000 7f1492277640 1 -- 172.21.15.110:0/1017243024 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c0b2660 con 0x7f148c146090 2024-01-20T11:19:17.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.092+0000 7f1492277640 1 --2- 172.21.15.110:0/1017243024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c146090 0x7f148c0b3da0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1484009df0 tx=0x7f148402be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc7aadb5425dec15 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:17.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 1 -- 172.21.15.110:0/1017243024 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148402cc20 con 0x7f148c146090 2024-01-20T11:19:17.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:17.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:17.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 10 monclient: dump: 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 10 monclient: _finish_auth 0 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 10 monclient: _check_auth_tickets 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:17.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 1 -- 172.21.15.110:0/1017243024 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148402cd80 con 0x7f148c146090 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 1 -- 172.21.15.110:0/1017243024 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484034660 con 0x7f148c146090 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.095+0000 7f1490a74640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1490a74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1490a74640 10 monclient: dump: 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:17.099 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 5 monclient: authenticate success, global_id 4820 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 4 set_mon_vals no callback set 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 10 monclient: shutdown 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 1 -- 172.21.15.110:0/1017243024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c146090 msgr2=0x7f148c0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 1 --2- 172.21.15.110:0/1017243024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c146090 0x7f148c0b3da0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1484009df0 tx=0x7f148402be70 comp rx=0 tx=0).stop 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 1 -- 172.21.15.110:0/1017243024 shutdown_connections 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 1 --2- 172.21.15.110:0/1017243024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c146090 0x7f148c0b3da0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 1 -- 172.21.15.110:0/1017243024 >> 172.21.15.110:0/1017243024 conn(0x7f148c066590 msgr2=0x7f148c0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 1 -- 172.21.15.110:0/1017243024 shutdown_connections 2024-01-20T11:19:17.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.096+0000 7f1493cda640 1 -- 172.21.15.110:0/1017243024 wait complete. 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.097+0000 7f1493cda640 10 monclient: build_initial_monmap 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.097+0000 7f1493cda640 10 monclient: monmap: 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.097+0000 7f1493cda640 1 librados: starting msgr at 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.097+0000 7f1493cda640 1 librados: starting objecter 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.097+0000 7f1493cda640 1 Processor -- start 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 1 -- start start 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 1 librados: setting wanted keys 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 1 librados: calling monclient init 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient: init 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f148c0b1ed0 0x7f148c13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient(hunting): picked mon.noname-b con 0x7f148c0b1ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c0b4d10 0x7f148c0ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient(hunting): picked mon.noname-c con 0x7f148c0b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148c146090 0x7f148c13e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient(hunting): picked mon.noname-a con 0x7f148c146090 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:17.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c0abee0 con 0x7f148c146090 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c13dff0 con 0x7f148c0b1ed0 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c13dcf0 con 0x7f148c0b4d10 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.098+0000 7f1493cda640 10 monclient(hunting): authenticate will time out at 1600.449730s 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c0b4d10 0x7f148c0ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1492277640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148c146090 0x7f148c13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491a76640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f148c0b1ed0 0x7f148c13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c0b4d10 0x7f148c0ac300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47812/0 (socket says 172.21.15.110:47812) 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 1 -- 172.21.15.110:0/166002054 learned_addr learned my addr 172.21.15.110:0/166002054 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 10 monclient(hunting): get_auth_request con 0x7f148c0b4d10 auth_method 0 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1492277640 10 monclient(hunting): get_auth_request con 0x7f148c146090 auth_method 0 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1492277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1492277640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1492277640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491a76640 10 monclient(hunting): get_auth_request con 0x7f148c0b1ed0 auth_method 0 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491a76640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491a76640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.099+0000 7f1491275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.100+0000 7f1492277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.100+0000 7f1492277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.100+0000 7f1492277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:17.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.100+0000 7f1491a76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.100+0000 7f1491a76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.100+0000 7f1491a76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 10 monclient(hunting): handle_auth_done global_id 4823 payload 824 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 1 -- 172.21.15.110:0/166002054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f148c0b1ed0 msgr2=0x7f148c13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f148c0b1ed0 0x7f148c13d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 1 -- 172.21.15.110:0/166002054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148c146090 msgr2=0x7f148c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148c146090 0x7f148c13e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 10 monclient: _finish_hunting 0 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 1 monclient: found mon.noname-c 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:17.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.102+0000 7f1491275640 1 -- 172.21.15.110:0/166002054 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484009a70 con 0x7f148c0b4d10 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.105+0000 7f1491275640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c0b4d10 0x7f148c0ac300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f148800c500 tx=0x7f148800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.105+0000 7f14827fc640 10 client.?.objecter ms_handle_connect 0x7f148c0b4d10 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.105+0000 7f14827fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 1 -- 172.21.15.110:0/166002054 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488005ce0 con 0x7f148c0b4d10 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 10 monclient: dump: 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 1 -- 172.21.15.110:0/166002054 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1488005e40 con 0x7f148c0b4d10 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 1 -- 172.21.15.110:0/166002054 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14880072f0 con 0x7f148c0b4d10 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1493cda640 5 monclient: authenticate success, global_id 4823 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1493cda640 10 monclient: _renew_subs 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1493cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148c13c410 con 0x7f148c0b4d10 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1483fff640 4 set_mon_vals no callback set 2024-01-20T11:19:17.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f14827fc640 10 monclient: dump: 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1493cda640 10 client.4823.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1493cda640 10 monclient: _renew_subs 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1493cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:17.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148c13ea70 con 0x7f148c0b4d10 2024-01-20T11:19:17.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.108+0000 7f1493cda640 1 librados: init done 2024-01-20T11:19:17.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.109+0000 7f1493cda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:17.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.109+0000 7f1493cda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:17.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.109+0000 7f1493cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:17.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.109+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148c13e200 con 0x7f148c0b4d10 2024-01-20T11:19:17.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.128+0000 7f14827fc640 1 -- 172.21.15.110:0/166002054 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1488007510 con 0x7f148c0b4d10 2024-01-20T11:19:17.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.129+0000 7f14827fc640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f147006c8b0 0x7f147006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.129+0000 7f1491a76640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f147006c8b0 0x7f147006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.129+0000 7f1491a76640 10 monclient: get_auth_request con 0x7f147006c8b0 auth_method 0 2024-01-20T11:19:17.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.130+0000 7f1491a76640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f147006c8b0 0x7f147006ed70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f147c007fd0 tx=0x7f147c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:17.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.130+0000 7f14827fc640 10 client.4823.objecter ms_handle_connect 0x7f147006c8b0 2024-01-20T11:19:17.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.362+0000 7f14827fc640 1 -- 172.21.15.110:0/166002054 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1488057fb0 con 0x7f148c0b4d10 2024-01-20T11:19:17.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.362+0000 7f14827fc640 10 client.4823.objecter ms_dispatch 0x7f148c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:17.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.362+0000 7f14827fc640 3 client.4823.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:17.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.362+0000 7f14827fc640 3 client.4823.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:17.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.363+0000 7f14827fc640 20 client.4823.objecter dump_active .. 0 homeless 2024-01-20T11:19:17.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.363+0000 7f14827fc640 1 -- 172.21.15.110:0/166002054 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f14880542f0 con 0x7f148c0b4d10 2024-01-20T11:19:17.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.363+0000 7f14827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:17.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.363+0000 7f14827fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.615+0000 7f1493cda640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.615+0000 7f1493cda640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.615+0000 7f1493cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.615+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f148c13de30 con 0x7f148c0b4d10 2024-01-20T11:19:17.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.621+0000 7f14827fc640 1 -- 172.21.15.110:0/166002054 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f1488057910 con 0x7f148c0b4d10 2024-01-20T11:19:17.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.621+0000 7f14827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:17.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.621+0000 7f14827fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:17.622 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:17.622 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":292,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:17.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.622+0000 7f1493cda640 10 librados: watch_flush enter 2024-01-20T11:19:17.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.622+0000 7f1493cda640 10 librados: watch_flush exit 2024-01-20T11:19:17.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 20 client.4823.objecter shutdown clearing up homeless session... 2024-01-20T11:19:17.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 10 client.4823.objecter successfully canceled tick 2024-01-20T11:19:17.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f147006c8b0 msgr2=0x7f147006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f147006c8b0 0x7f147006ed70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f147c007fd0 tx=0x7f147c009040 comp rx=0 tx=0).stop 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 10 monclient: shutdown 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c0b4d10 msgr2=0x7f148c0ac300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c0b4d10 0x7f148c0ac300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f148800c500 tx=0x7f148800c9c0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 shutdown_connections 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f147006c8b0 0x7f147006ed70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148c146090 0x7f148c13e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f148c0b4d10 0x7f148c0ac300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 --2- 172.21.15.110:0/166002054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f148c0b1ed0 0x7f148c13d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 >> 172.21.15.110:0/166002054 conn(0x7f148c066590 msgr2=0x7f148c13f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 shutdown_connections 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 -- 172.21.15.110:0/166002054 wait complete. 2024-01-20T11:19:17.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.623+0000 7f1493cda640 1 librados: shutdown 2024-01-20T11:19:17.635 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:17.635 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:17.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.733+0000 7f61a94f7640 10 monclient: get_monmap_and_config 2024-01-20T11:19:17.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.733+0000 7f61a94f7640 10 monclient: build_initial_monmap 2024-01-20T11:19:17.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.733+0000 7f61a94f7640 10 monclient: monmap: 2024-01-20T11:19:17.734 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:17.734 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.734 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:17.734956+0000 2024-01-20T11:19:17.735 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:17.734956+0000 2024-01-20T11:19:17.735 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:17.735 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.735 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:17.735 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:17.735 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:17.735 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:17.735 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:17.735 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 1 Processor -- start 2024-01-20T11:19:17.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 1 -- start start 2024-01-20T11:19:17.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient: init 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b4d10 0x7f61a40b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient(hunting): picked mon.noname-b con 0x7f61a40b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f61a40b17c0 0x7f61a40b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient(hunting): picked mon.noname-a con 0x7f61a40b17c0 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61a4145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a40b2120 con 0x7f61a4145980 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a40b50f0 con 0x7f61a40b17c0 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40aade0 con 0x7f61a40b4d10 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:17.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.734+0000 7f61a94f7640 10 monclient(hunting): authenticate will time out at 1601.085974s 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f61a2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b4d10 0x7f61a40b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f619bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f61a40b17c0 0x7f61a40b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f61a2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b4d10 0x7f61a40b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47820/0 (socket says 172.21.15.110:47820) 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f61a2ffd640 1 -- 172.21.15.110:0/1092288855 learned_addr learned my addr 172.21.15.110:0/1092288855 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f61a2ffd640 10 monclient(hunting): get_auth_request con 0x7f61a40b4d10 auth_method 0 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f61a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f61a2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f61a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f619bfff640 10 monclient(hunting): get_auth_request con 0x7f61a40b17c0 auth_method 0 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f619bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f619bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.735+0000 7f619bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.736+0000 7f61a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:17.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.736+0000 7f61a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:17.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.736+0000 7f61a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.737+0000 7f619bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.737+0000 7f619bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.737+0000 7f619bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.738+0000 7f61a2ffd640 10 monclient(hunting): handle_auth_done global_id 4826 payload 274 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.738+0000 7f61a2ffd640 1 -- 172.21.15.110:0/1092288855 >> v2:172.21.15.110:3300/0 conn(0x7f61a40b17c0 msgr2=0x7f61a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.738+0000 7f61a2ffd640 1 --2- 172.21.15.110:0/1092288855 >> v2:172.21.15.110:3300/0 conn(0x7f61a40b17c0 0x7f61a40b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.738+0000 7f61a2ffd640 1 -- 172.21.15.110:0/1092288855 >> v1:172.21.15.110:6789/0 conn(0x7f61a4145980 legacy=0x7f61a4145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.738+0000 7f61a2ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.738+0000 7f61a2ffd640 1 monclient: found mon.noname-b 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.738+0000 7f61a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.738+0000 7f61a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:17.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.738+0000 7f61a2ffd640 1 -- 172.21.15.110:0/1092288855 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a4145460 con 0x7f61a40b4d10 2024-01-20T11:19:17.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.740+0000 7f61a2ffd640 1 --2- 172.21.15.110:0/1092288855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b4d10 0x7f61a40b3da0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6194009870 tx=0x7f619402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d0e21af4152fb19 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:17.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 1 -- 172.21.15.110:0/1092288855 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194031640 con 0x7f61a40b4d10 2024-01-20T11:19:17.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:17.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:17.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 10 monclient: dump: 2024-01-20T11:19:17.745 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 1 -- 172.21.15.110:0/1092288855 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61940317a0 con 0x7f61a40b4d10 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:17.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 1 -- 172.21.15.110:0/1092288855 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194031b30 con 0x7f61a40b4d10 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a94f7640 5 monclient: authenticate success, global_id 4826 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a94f7640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a94f7640 4 set_mon_vals no callback set 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.744+0000 7f61a27fc640 10 monclient: dump: 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:17.747 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 10 monclient: shutdown 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 1 -- 172.21.15.110:0/1092288855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b4d10 msgr2=0x7f61a40b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 1 --2- 172.21.15.110:0/1092288855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b4d10 0x7f61a40b3da0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6194009870 tx=0x7f619402b8f0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 1 -- 172.21.15.110:0/1092288855 shutdown_connections 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 1 --2- 172.21.15.110:0/1092288855 >> v2:172.21.15.110:3300/0 conn(0x7f61a40b17c0 0x7f61a40b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 1 --2- 172.21.15.110:0/1092288855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b4d10 0x7f61a40b3da0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 1 -- 172.21.15.110:0/1092288855 >> 172.21.15.110:0/1092288855 conn(0x7f61a4066590 msgr2=0x7f61a40669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 1 -- 172.21.15.110:0/1092288855 shutdown_connections 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.745+0000 7f61a94f7640 1 -- 172.21.15.110:0/1092288855 wait complete. 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.746+0000 7f61a94f7640 10 monclient: build_initial_monmap 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.746+0000 7f61a94f7640 10 monclient: monmap: 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:17.751 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.746+0000 7f61a94f7640 1 librados: starting msgr at 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.746+0000 7f61a94f7640 1 librados: starting objecter 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.746+0000 7f61a94f7640 1 Processor -- start 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.746+0000 7f61a94f7640 1 -- start start 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.746+0000 7f61a94f7640 1 librados: setting wanted keys 2024-01-20T11:19:17.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.746+0000 7f61a94f7640 1 librados: calling monclient init 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.746+0000 7f61a94f7640 10 monclient: init 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b17c0 0x7f61a41467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient(hunting): picked mon.noname-c con 0x7f61a40b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61a40b4d10 0x7f61a41406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient(hunting): picked mon.noname-b con 0x7f61a40b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61a4145980 0x7f61a4140c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient(hunting): picked mon.noname-a con 0x7f61a4145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a4148070 con 0x7f61a4145980 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a4143060 con 0x7f61a40b4d10 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:17.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a4141170 con 0x7f61a40b17c0 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a94f7640 10 monclient(hunting): authenticate will time out at 1601.098437s 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b17c0 0x7f61a41467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.747+0000 7f61a37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61a4145980 0x7f61a4140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f619bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61a40b4d10 0x7f61a41406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b17c0 0x7f61a41467c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47824/0 (socket says 172.21.15.110:47824) 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a2ffd640 1 -- 172.21.15.110:0/1457469230 learned_addr learned my addr 172.21.15.110:0/1457469230 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a2ffd640 10 monclient(hunting): get_auth_request con 0x7f61a40b17c0 auth_method 0 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a37fe640 10 monclient(hunting): get_auth_request con 0x7f61a4145980 auth_method 0 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f619bfff640 10 monclient(hunting): get_auth_request con 0x7f61a40b4d10 auth_method 0 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f619bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f619bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f619bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f619bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f619bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f619bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:17.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.748+0000 7f61a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:17.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 10 monclient(hunting): handle_auth_done global_id 5080 payload 824 2024-01-20T11:19:17.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 1 -- 172.21.15.110:0/1457469230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b17c0 msgr2=0x7f61a41467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b17c0 0x7f61a41467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 1 -- 172.21.15.110:0/1457469230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61a4145980 msgr2=0x7f61a4140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61a4145980 0x7f61a4140c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 1 monclient: found mon.noname-b 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.751+0000 7f619bfff640 1 -- 172.21.15.110:0/1457469230 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61940094d0 con 0x7f61a40b4d10 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.753+0000 7f619bfff640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61a40b4d10 0x7f61a41406f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f618c00e850 tx=0x7f618c00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.753+0000 7f619b7fe640 10 client.?.objecter ms_handle_connect 0x7f61a40b4d10 2024-01-20T11:19:17.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.753+0000 7f619b7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:17.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 1 -- 172.21.15.110:0/1457469230 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c0095b0 con 0x7f61a40b4d10 2024-01-20T11:19:17.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:17.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:17.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 10 monclient: dump: 2024-01-20T11:19:17.758 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:17.758 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 1 -- 172.21.15.110:0/1457469230 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c00c940 con 0x7f61a40b4d10 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 1 -- 172.21.15.110:0/1457469230 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c00ccd0 con 0x7f61a40b4d10 2024-01-20T11:19:17.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f619b7fe640 10 monclient: dump: 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:17.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a94f7640 5 monclient: authenticate success, global_id 5080 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a94f7640 10 monclient: _renew_subs 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a94f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a41434f0 con 0x7f61a40b4d10 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a1ffb640 4 set_mon_vals no callback set 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a94f7640 10 client.5080.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a94f7640 10 monclient: _renew_subs 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a94f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a4147340 con 0x7f61a40b4d10 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.757+0000 7f61a94f7640 1 librados: init done 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.758+0000 7f61a94f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:17.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.758+0000 7f61a94f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:17.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.758+0000 7f61a94f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:17.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.758+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a4075490 con 0x7f61a40b4d10 2024-01-20T11:19:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.776+0000 7f619b7fe640 1 -- 172.21.15.110:0/1457469230 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f618c013470 con 0x7f61a40b4d10 2024-01-20T11:19:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.776+0000 7f619b7fe640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f618406c890 0x7f618406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.776+0000 7f61a2ffd640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f618406c890 0x7f618406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.777+0000 7f61a2ffd640 10 monclient: get_auth_request con 0x7f618406c890 auth_method 0 2024-01-20T11:19:17.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.777+0000 7f61a2ffd640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f618406c890 0x7f618406ed50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6194009870 tx=0x7f619403c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:17.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.777+0000 7f619b7fe640 10 client.5080.objecter ms_handle_connect 0x7f618406c890 2024-01-20T11:19:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.778+0000 7f619b7fe640 1 -- 172.21.15.110:0/1457469230 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f618c00ce30 con 0x7f61a40b4d10 2024-01-20T11:19:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.778+0000 7f619b7fe640 10 client.5080.objecter ms_dispatch 0x7f61a4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.778+0000 7f619b7fe640 3 client.5080.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.778+0000 7f619b7fe640 3 client.5080.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:17.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:17.778+0000 7f619b7fe640 20 client.5080.objecter dump_active .. 0 homeless 2024-01-20T11:19:18.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.009+0000 7f619b7fe640 1 -- 172.21.15.110:0/1457469230 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f618c054ea0 con 0x7f61a40b4d10 2024-01-20T11:19:18.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.009+0000 7f619b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:18.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.009+0000 7f619b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:18.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.186+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61a414c7f0 con 0x7f618406c890 2024-01-20T11:19:18.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.189+0000 7f619b7fe640 1 -- 172.21.15.110:0/1457469230 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63946 (secure 0 0 0) 0x7f6194044400 con 0x7f618406c890 2024-01-20T11:19:18.190 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:18.191 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:18.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 10 librados: watch_flush enter 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 10 librados: watch_flush exit 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 20 client.5080.objecter shutdown clearing up homeless session... 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 10 client.5080.objecter successfully canceled tick 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f618406c890 msgr2=0x7f618406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f618406c890 0x7f618406ed50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6194009870 tx=0x7f619403c040 comp rx=0 tx=0).stop 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 10 monclient: shutdown 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61a40b4d10 msgr2=0x7f61a41406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61a40b4d10 0x7f61a41406f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f618c00e850 tx=0x7f618c00ed10 comp rx=0 tx=0).stop 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 shutdown_connections 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f618406c890 0x7f618406ed50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61a4145980 0x7f61a4140c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61a40b4d10 0x7f61a41406f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:18.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 --2- 172.21.15.110:0/1457469230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61a40b17c0 0x7f61a41467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:18.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 >> 172.21.15.110:0/1457469230 conn(0x7f61a4066590 msgr2=0x7f61a40b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:18.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 shutdown_connections 2024-01-20T11:19:18.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 -- 172.21.15.110:0/1457469230 wait complete. 2024-01-20T11:19:18.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:18.191+0000 7f61a94f7640 1 librados: shutdown 2024-01-20T11:19:18.203 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-01-20T11:19:16.756307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002837"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":128,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:16:52.902829+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:16:52.902829+0000","last_peered":"2024-01-20T11:16:52.902829+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:16:52.902829+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1429999999999998},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":13.545999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.255000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.324}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.753},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.968999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.9770000000000003},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":10.936999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.904},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":14.885999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7029999999999994},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":16.803000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313269,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.2569999999999997},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":15.512}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7590000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.551}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.896000000000001},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":16.513000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.051},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.625}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.256},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":18.236999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":17.183},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.287000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313270,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":18.129000000000001},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.0660000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.7390000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":15.366},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.016}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.901999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":18.693000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":15.975},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":12.997}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":13.942},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.183}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.4889999999999999},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":11.525}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7720000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.9009999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.523},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.202}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.930999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.183}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.811},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7220000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.0749999999999993},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":14.885999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444084,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":9.2669999999999995,"15min":9.5809999999999995},"min":{"1min":4.657,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.374000000000001,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.7809999999999997},{"interface":"front","average":{"1min":8.2289999999999992,"5min":9.8260000000000005,"15min":10.339},"min":{"1min":5.6760000000000002,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":11.693,"5min":30.439,"15min":30.439},"last":9.0030000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.714,"5min":12.791,"15min":13.225},"min":{"1min":6.351,"5min":6.351,"15min":6.351},"max":{"1min":14.565,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.35},{"interface":"front","average":{"1min":9.6359999999999992,"5min":12.599,"15min":13.265000000000001},"min":{"1min":6.6180000000000003,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":14.939,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.9400000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":11.436999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":14.593999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":13.945},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.510999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:18:16 2024","interfaces":[{"interface":"back","average":{"1min":10.361000000000001,"5min":11.929,"15min":12.164999999999999},"min":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"max":{"1min":16.533999999999999,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.8630000000000004},{"interface":"front","average":{"1min":13.428000000000001,"5min":15.054,"15min":15.538},"min":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":16.329999999999998,"5min":31.634,"15min":31.634},"last":16.497}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.191000000000001},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.382999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444084,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":7.7750000000000004},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":5.9500000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.6630000000000003},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.031000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":11.698},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.4039999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":9.6189999999999998},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":15.127000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":13.635999999999999},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":10.308}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.335000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":9.0540000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509493,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.3789999999999996},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.0980000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.0629999999999997},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":6.9409999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.286},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.618}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.561999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.888}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.8960000000000008},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.84},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8.9250000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:18.616 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:09.640 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:18.618 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:09.642 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:18.632 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:09.656 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:18.660 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:05:09.666 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:18.662 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:05:09.669 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:18.667 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:09.691 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:18.668 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:09.693 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:18.677 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:05:09.683 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:18.682 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:09.707 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:21.205 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:21.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.304+0000 7f8cf8fb6640 10 monclient: get_monmap_and_config 2024-01-20T11:19:21.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.304+0000 7f8cf8fb6640 10 monclient: build_initial_monmap 2024-01-20T11:19:21.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.304+0000 7f8cf8fb6640 10 monclient: monmap: 2024-01-20T11:19:21.306 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:21.306 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.307 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:21.306171+0000 2024-01-20T11:19:21.307 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:21.306171+0000 2024-01-20T11:19:21.307 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:21.307 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.307 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:21.307 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:21.308 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:21.308 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:21.308 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:21.308 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 1 Processor -- start 2024-01-20T11:19:21.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 1 -- start start 2024-01-20T11:19:21.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 10 monclient: init 2024-01-20T11:19:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-01-20T11:19:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf413bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 10 monclient(hunting): picked mon.noname-b con 0x7f8cf4066620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8cf406df70 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8cf4145b50 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf40adf80 con 0x7f8cf406df70 2024-01-20T11:19:21.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf406f650 con 0x7f8cf4145b50 2024-01-20T11:19:21.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:21.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf406e690 con 0x7f8cf4066620 2024-01-20T11:19:21.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:21.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.305+0000 7f8cf8fb6640 10 monclient(hunting): authenticate will time out at 1604.657135s 2024-01-20T11:19:21.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.306+0000 7f8cf259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf413bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.306+0000 7f8cebfff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8cf406df70 0x7f8cf4145440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33908/0 (socket says 172.21.15.110:33908) 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.306+0000 7f8cebfff640 1 -- 172.21.15.110:0/1831252273 learned_addr learned my addr 172.21.15.110:0/1831252273 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.306+0000 7f8cf259c640 10 monclient(hunting): get_auth_request con 0x7f8cf4066620 auth_method 0 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.306+0000 7f8cf259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.306+0000 7f8cf259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.306+0000 7f8cf259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.307+0000 7f8cf259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.307+0000 7f8cf259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.307+0000 7f8cf259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf1d9b640 1 -- 172.21.15.110:0/1831252273 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043221510 0 0) 0x7f8cf40adf80 con 0x7f8cf406df70 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf1d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf1d9b640 10 monclient(hunting): my global_id is 5007 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf1d9b640 1 -- 172.21.15.110:0/1831252273 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce0003520 con 0x7f8cf406df70 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf259c640 10 monclient(hunting): handle_auth_done global_id 4832 payload 274 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf259c640 1 -- 172.21.15.110:0/1831252273 >> v1:172.21.15.134:6789/0 conn(0x7f8cf4145b50 legacy=0x7f8cf414c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf259c640 1 -- 172.21.15.110:0/1831252273 >> v1:172.21.15.110:6789/0 conn(0x7f8cf406df70 legacy=0x7f8cf4145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf259c640 10 monclient: _finish_hunting 0 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf259c640 1 monclient: found mon.noname-b 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.309+0000 7f8cf259c640 1 -- 172.21.15.110:0/1831252273 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf4072be0 con 0x7f8cf4066620 2024-01-20T11:19:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.311+0000 7f8cf259c640 1 --2- 172.21.15.110:0/1831252273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf413bcc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40097d0 tx=0x7f8ce402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69fa55a13c2fd87d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:21.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 1 -- 172.21.15.110:0/1831252273 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4031800 con 0x7f8cf4066620 2024-01-20T11:19:21.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:21.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:21.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 10 monclient: dump: 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 10 monclient: _finish_auth 0 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:19:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 1 -- 172.21.15.110:0/1831252273 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce4031960 con 0x7f8cf4066620 2024-01-20T11:19:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf8fb6640 5 monclient: authenticate success, global_id 4832 2024-01-20T11:19:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf8fb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf8fb6640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 1 -- 172.21.15.110:0/1831252273 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4031cf0 con 0x7f8cf4066620 2024-01-20T11:19:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf8fb6640 4 set_mon_vals no callback set 2024-01-20T11:19:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.314+0000 7f8cf1d9b640 10 monclient: dump: 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.315+0000 7f8cf8fb6640 10 monclient: shutdown 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.315+0000 7f8cf8fb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.315+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1831252273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 msgr2=0x7f8cf413bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.315+0000 7f8cf8fb6640 1 --2- 172.21.15.110:0/1831252273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf413bcc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40097d0 tx=0x7f8ce402b8a0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.315+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1831252273 shutdown_connections 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.315+0000 7f8cf8fb6640 1 --2- 172.21.15.110:0/1831252273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf413bcc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.315+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1831252273 >> 172.21.15.110:0/1831252273 conn(0x7f8cf40b09e0 msgr2=0x7f8cf40b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.315+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1831252273 shutdown_connections 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.315+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1831252273 wait complete. 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.316+0000 7f8cf8fb6640 10 monclient: build_initial_monmap 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.316+0000 7f8cf8fb6640 10 monclient: monmap: 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:21.326 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.316+0000 7f8cf8fb6640 1 librados: starting msgr at 2024-01-20T11:19:21.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.316+0000 7f8cf8fb6640 1 librados: starting objecter 2024-01-20T11:19:21.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.316+0000 7f8cf8fb6640 1 Processor -- start 2024-01-20T11:19:21.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.316+0000 7f8cf8fb6640 1 -- start start 2024-01-20T11:19:21.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.316+0000 7f8cf8fb6640 1 librados: setting wanted keys 2024-01-20T11:19:21.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.316+0000 7f8cf8fb6640 1 librados: calling monclient init 2024-01-20T11:19:21.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.316+0000 7f8cf8fb6640 10 monclient: init 2024-01-20T11:19:21.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:21.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:19:21.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf41cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient(hunting): picked mon.noname-c con 0x7f8cf4066620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:21.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf406df70 0x7f8cf41cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient(hunting): picked mon.noname-b con 0x7f8cf406df70 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cf4145b50 0x7f8cf41ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient(hunting): picked mon.noname-a con 0x7f8cf4145b50 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf41cd550 con 0x7f8cf4145b50 2024-01-20T11:19:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf41cd7c0 con 0x7f8cf406df70 2024-01-20T11:19:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf41cbf50 con 0x7f8cf4066620 2024-01-20T11:19:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.317+0000 7f8cf8fb6640 10 monclient(hunting): authenticate will time out at 1604.668679s 2024-01-20T11:19:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf2d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cf4145b50 0x7f8cf41ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf41cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cebfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf406df70 0x7f8cf41cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf41cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50612/0 (socket says 172.21.15.110:50612) 2024-01-20T11:19:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 1 -- 172.21.15.110:0/1012140607 learned_addr learned my addr 172.21.15.110:0/1012140607 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 10 monclient(hunting): get_auth_request con 0x7f8cf4066620 auth_method 0 2024-01-20T11:19:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:21.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf2d9d640 10 monclient(hunting): get_auth_request con 0x7f8cf4145b50 auth_method 0 2024-01-20T11:19:21.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:21.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:21.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:21.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cebfff640 10 monclient(hunting): get_auth_request con 0x7f8cf406df70 auth_method 0 2024-01-20T11:19:21.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cebfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:21.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:21.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.318+0000 7f8cf2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:21.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.319+0000 7f8cebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:21.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.319+0000 7f8cebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:21.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.319+0000 7f8cebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:21.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 10 monclient(hunting): handle_auth_done global_id 4835 payload 824 2024-01-20T11:19:21.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 1 -- 172.21.15.110:0/1012140607 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf406df70 msgr2=0x7f8cf41cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf406df70 0x7f8cf41cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 1 -- 172.21.15.110:0/1012140607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cf4145b50 msgr2=0x7f8cf41ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cf4145b50 0x7f8cf41ce120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 10 monclient: _finish_hunting 0 2024-01-20T11:19:21.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 1 monclient: found mon.noname-c 2024-01-20T11:19:21.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:21.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:21.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.321+0000 7f8cf259c640 1 -- 172.21.15.110:0/1012140607 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce4009400 con 0x7f8cf4066620 2024-01-20T11:19:21.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.323+0000 7f8cf259c640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf41cb3d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8ce402beb0 tx=0x7f8ce40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:21.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.323+0000 7f8ceaffd640 10 client.?.objecter ms_handle_connect 0x7f8cf4066620 2024-01-20T11:19:21.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.323+0000 7f8ceaffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:21.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 1 -- 172.21.15.110:0/1012140607 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce403bc30 con 0x7f8cf4066620 2024-01-20T11:19:21.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:21.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:21.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 10 monclient: dump: 2024-01-20T11:19:21.339 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:21.339 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.339 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.339 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.340 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:21.340 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.340 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:21.340 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:21.340 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:21.340 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 10 monclient: _finish_auth 0 2024-01-20T11:19:21.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:21.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:21.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 1 -- 172.21.15.110:0/1012140607 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce403bdb0 con 0x7f8cf4066620 2024-01-20T11:19:21.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:21.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 1 -- 172.21.15.110:0/1012140607 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4045630 con 0x7f8cf4066620 2024-01-20T11:19:21.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:21.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:21.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8ceaffd640 10 monclient: dump: 2024-01-20T11:19:21.342 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:21.342 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.342 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.342 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.343 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:21.343 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.343 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:21.343 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:21.343 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:21.343 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf8fb6640 5 monclient: authenticate success, global_id 4835 2024-01-20T11:19:21.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf8fb6640 10 monclient: _renew_subs 2024-01-20T11:19:21.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf8fb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:21.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf413c380 con 0x7f8cf4066620 2024-01-20T11:19:21.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf159a640 4 set_mon_vals no callback set 2024-01-20T11:19:21.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf8fb6640 10 client.4835.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf8fb6640 10 monclient: _renew_subs 2024-01-20T11:19:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf8fb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf41cbce0 con 0x7f8cf4066620 2024-01-20T11:19:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.327+0000 7f8cf8fb6640 1 librados: init done 2024-01-20T11:19:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.328+0000 7f8cf8fb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.328+0000 7f8cf8fb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.328+0000 7f8cf8fb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.328+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf4075490 con 0x7f8cf4066620 2024-01-20T11:19:21.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.346+0000 7f8ceaffd640 1 -- 172.21.15.110:0/1012140607 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8ce404e450 con 0x7f8cf4066620 2024-01-20T11:19:21.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.347+0000 7f8ceaffd640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8cd406c8b0 0x7f8cd406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.347+0000 7f8cebfff640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8cd406c8b0 0x7f8cd406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.347+0000 7f8cebfff640 10 monclient: get_auth_request con 0x7f8cd406c8b0 auth_method 0 2024-01-20T11:19:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.347+0000 7f8ceaffd640 10 client.4835.objecter ms_handle_connect 0x7f8cd406c8b0 2024-01-20T11:19:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.347+0000 7f8cebfff640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8cd406c8b0 0x7f8cd406ed70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8cf413b910 tx=0x7f8cdc009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.580+0000 7f8ceaffd640 1 -- 172.21.15.110:0/1012140607 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8ce402e070 con 0x7f8cf4066620 2024-01-20T11:19:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.580+0000 7f8ceaffd640 10 client.4835.objecter ms_dispatch 0x7f8cf4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.580+0000 7f8ceaffd640 3 client.4835.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.580+0000 7f8ceaffd640 3 client.4835.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.580+0000 7f8ceaffd640 20 client.4835.objecter dump_active .. 0 homeless 2024-01-20T11:19:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.581+0000 7f8ceaffd640 1 -- 172.21.15.110:0/1012140607 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8ce4076c30 con 0x7f8cf4066620 2024-01-20T11:19:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.581+0000 7f8ceaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.581+0000 7f8ceaffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:21.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.748+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8cf413c200 con 0x7f8cd406c8b0 2024-01-20T11:19:21.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.751+0000 7f8ceaffd640 1 -- 172.21.15.110:0/1012140607 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63960 (secure 0 0 0) 0x7f8cdc001580 con 0x7f8cd406c8b0 2024-01-20T11:19:21.753 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:21.753 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:21.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 10 librados: watch_flush enter 2024-01-20T11:19:21.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 10 librados: watch_flush exit 2024-01-20T11:19:21.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 20 client.4835.objecter shutdown clearing up homeless session... 2024-01-20T11:19:21.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 10 client.4835.objecter successfully canceled tick 2024-01-20T11:19:21.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8cd406c8b0 msgr2=0x7f8cd406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8cd406c8b0 0x7f8cd406ed70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8cf413b910 tx=0x7f8cdc009040 comp rx=0 tx=0).stop 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 10 monclient: shutdown 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 msgr2=0x7f8cf41cb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf41cb3d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8ce402beb0 tx=0x7f8ce40046c0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 shutdown_connections 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8cd406c8b0 0x7f8cd406ed70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cf4145b50 0x7f8cf41ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf406df70 0x7f8cf41cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 --2- 172.21.15.110:0/1012140607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf4066620 0x7f8cf41cb3d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 >> 172.21.15.110:0/1012140607 conn(0x7f8cf40b09e0 msgr2=0x7f8cf40afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 shutdown_connections 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 -- 172.21.15.110:0/1012140607 wait complete. 2024-01-20T11:19:21.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.753+0000 7f8cf8fb6640 1 librados: shutdown 2024-01-20T11:19:21.765 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-01-20T11:19:20.757116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002600"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1429999999999998},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":13.545999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.255000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.324}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.753},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.968999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.9770000000000003},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":10.936999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.904},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":14.885999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7029999999999994},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":16.803000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313269,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.2569999999999997},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":15.512}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7590000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.551}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.896000000000001},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":16.513000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.051},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.625}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.256},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":18.236999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":17.183},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.287000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313270,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":18.129000000000001},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.0660000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.7390000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":15.366},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.016}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.901999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":18.693000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":15.975},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":12.997}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":13.942},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.183}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444084,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.4889999999999999},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":11.525}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7720000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.9009999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.523},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.202}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.930999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.183}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.811},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7220000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.0749999999999993},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":14.885999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444085,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.6870000000000003},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.0990000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.4559999999999995},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.916}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.077999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.032999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":10.456},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.704000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.0149999999999997},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":14.522}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.1219999999999999},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.339}]}]},{"osd":0,"up_from":17,"seq":73014444085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":6.6369999999999996},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":4.3040000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":13.349},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.5659999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":9.2829999999999995},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.976000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.614000000000001},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.1869999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":8.7390000000000008},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":12.638}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.912000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":14.662000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509493,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.3789999999999996},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.0980000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.0629999999999997},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":6.9409999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.286},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.618}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.561999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.888}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.8960000000000008},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.84},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8.9250000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:21.766 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:21.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.868+0000 7fccd0105640 10 monclient: get_monmap_and_config 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.868+0000 7fccd0105640 10 monclient: build_initial_monmap 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.868+0000 7fccd0105640 10 monclient: monmap: 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:21.870425+0000 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:21.870425+0000 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:21.870 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:21.871 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.869+0000 7fccd0105640 1 Processor -- start 2024-01-20T11:19:21.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.869+0000 7fccd0105640 1 -- start start 2024-01-20T11:19:21.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.869+0000 7fccd0105640 10 monclient: init 2024-01-20T11:19:21.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fccc80b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccc80b1ed0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fccc8146090 0x7fccc80b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 10 monclient(hunting): picked mon.noname-a con 0x7fccc8146090 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc80aade0 con 0x7fccc80b4d10 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc8145460 con 0x7fccc80b1ed0 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc80b50f0 con 0x7fccc8146090 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.870+0000 7fccd0105640 10 monclient(hunting): authenticate will time out at 1605.221860s 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.871+0000 7fcccd6a0640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fccc80b1ed0 0x7fccc8145980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:40626/0 (socket says 172.21.15.110:40626) 2024-01-20T11:19:21.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.871+0000 7fcccd6a0640 1 -- 172.21.15.110:0/892675828 learned_addr learned my addr 172.21.15.110:0/892675828 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:21.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.883+0000 7fccce6a2640 1 --2- 172.21.15.110:0/892675828 >> v2:172.21.15.110:3300/0 conn(0x7fccc8146090 0x7fccc80b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.884+0000 7fccce6a2640 10 monclient(hunting): get_auth_request con 0x7fccc8146090 auth_method 0 2024-01-20T11:19:21.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.884+0000 7fccce6a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:21.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.884+0000 7fccce6a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:21.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.884+0000 7fccce6a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:21.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.885+0000 7fccce6a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:21.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.885+0000 7fccce6a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:21.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.885+0000 7fccce6a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:21.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.887+0000 7fccce6a2640 10 monclient(hunting): handle_auth_done global_id 5013 payload 274 2024-01-20T11:19:21.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.887+0000 7fccce6a2640 1 -- 172.21.15.110:0/892675828 >> v1:172.21.15.134:6789/0 conn(0x7fccc80b1ed0 legacy=0x7fccc8145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.887+0000 7fccce6a2640 1 -- 172.21.15.110:0/892675828 >> v1:172.21.15.110:6789/0 conn(0x7fccc80b4d10 legacy=0x7fccc80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.887+0000 7fccce6a2640 10 monclient: _finish_hunting 0 2024-01-20T11:19:21.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.887+0000 7fccce6a2640 1 monclient: found mon.noname-a 2024-01-20T11:19:21.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.887+0000 7fccce6a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:21.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.887+0000 7fccce6a2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:21.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.887+0000 7fccce6a2640 1 -- 172.21.15.110:0/892675828 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc80b2660 con 0x7fccc8146090 2024-01-20T11:19:21.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.890+0000 7fccce6a2640 1 --2- 172.21.15.110:0/892675828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc8146090 0x7fccc80b3da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009dc0 tx=0x7fccc002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a541021208dc85e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:21.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 1 -- 172.21.15.110:0/892675828 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc002cc40 con 0x7fccc8146090 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 10 monclient: dump: 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 10 monclient: _finish_auth 0 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 10 monclient: _check_auth_tickets 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 1 -- 172.21.15.110:0/892675828 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc002cda0 con 0x7fccc8146090 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fccd0105640 5 monclient: authenticate success, global_id 5013 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fccd0105640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 1 -- 172.21.15.110:0/892675828 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0034660 con 0x7fccc8146090 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fcccce9f640 10 monclient: dump: 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.901 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:21.902 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.902 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:21.902 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:21.902 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:21.902 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fccd0105640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:21.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.898+0000 7fccd0105640 4 set_mon_vals no callback set 2024-01-20T11:19:21.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.899+0000 7fccd0105640 10 monclient: shutdown 2024-01-20T11:19:21.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.899+0000 7fccd0105640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:21.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.899+0000 7fccd0105640 1 -- 172.21.15.110:0/892675828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc8146090 msgr2=0x7fccc80b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.899+0000 7fccd0105640 1 --2- 172.21.15.110:0/892675828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc8146090 0x7fccc80b3da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009dc0 tx=0x7fccc002be90 comp rx=0 tx=0).stop 2024-01-20T11:19:21.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.899+0000 7fccd0105640 1 -- 172.21.15.110:0/892675828 shutdown_connections 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.899+0000 7fccd0105640 1 --2- 172.21.15.110:0/892675828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc8146090 0x7fccc80b3da0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.899+0000 7fccd0105640 1 -- 172.21.15.110:0/892675828 >> 172.21.15.110:0/892675828 conn(0x7fccc8066590 msgr2=0x7fccc80669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.899+0000 7fccd0105640 1 -- 172.21.15.110:0/892675828 shutdown_connections 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.899+0000 7fccd0105640 1 -- 172.21.15.110:0/892675828 wait complete. 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.900+0000 7fccd0105640 10 monclient: build_initial_monmap 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.900+0000 7fccd0105640 10 monclient: monmap: 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:21.906 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 librados: starting msgr at 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 librados: starting objecter 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 Processor -- start 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 -- start start 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 librados: setting wanted keys 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 librados: calling monclient init 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient: init 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccc80b1ed0 0x7fccc813e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient(hunting): picked mon.noname-b con 0x7fccc80b1ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc80b4d10 0x7fccc813c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient(hunting): picked mon.noname-a con 0x7fccc80b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccc8146090 0x7fccc80ac180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient(hunting): picked mon.noname-c con 0x7fccc8146090 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc8136d00 con 0x7fccc80b4d10 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc813c630 con 0x7fccc80b1ed0 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80ab7a0 con 0x7fccc8146090 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.901+0000 7fccd0105640 10 monclient(hunting): authenticate will time out at 1605.252935s 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fccce6a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccc8146090 0x7fccc80ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccd6a0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc80b4d10 0x7fccc813c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccdea1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccc80b1ed0 0x7fccc813e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fccce6a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccc8146090 0x7fccc80ac180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50628/0 (socket says 172.21.15.110:50628) 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccd6a0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc80b4d10 0x7fccc813c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51410/0 (socket says 172.21.15.110:51410) 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fccce6a2640 1 -- 172.21.15.110:0/2357993166 learned_addr learned my addr 172.21.15.110:0/2357993166 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fccce6a2640 10 monclient(hunting): get_auth_request con 0x7fccc8146090 auth_method 0 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fccce6a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fccce6a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fccce6a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccd6a0640 10 monclient(hunting): get_auth_request con 0x7fccc80b4d10 auth_method 0 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccd6a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccd6a0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccd6a0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccdea1640 10 monclient(hunting): get_auth_request con 0x7fccc80b1ed0 auth_method 0 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccdea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccdea1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:21.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.902+0000 7fcccdea1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:21.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.903+0000 7fcccd6a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:21.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.903+0000 7fcccd6a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:21.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.903+0000 7fcccd6a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:21.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.903+0000 7fccce6a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:21.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.903+0000 7fccce6a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:21.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.903+0000 7fccce6a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:21.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.903+0000 7fcccdea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:21.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.903+0000 7fcccdea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:21.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.903+0000 7fcccdea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:21.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 10 monclient(hunting): handle_auth_done global_id 4838 payload 824 2024-01-20T11:19:21.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 1 -- 172.21.15.110:0/2357993166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccc80b1ed0 msgr2=0x7fccc813e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccc80b1ed0 0x7fccc813e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 1 -- 172.21.15.110:0/2357993166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc80b4d10 msgr2=0x7fccc813c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:21.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc80b4d10 0x7fccc813c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:21.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 10 monclient: _finish_hunting 0 2024-01-20T11:19:21.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 1 monclient: found mon.noname-c 2024-01-20T11:19:21.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:21.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:21.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.905+0000 7fccce6a2640 1 -- 172.21.15.110:0/2357993166 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc0009a70 con 0x7fccc8146090 2024-01-20T11:19:21.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.907+0000 7fccce6a2640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccc8146090 0x7fccc80ac180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fccc002c4a0 tx=0x7fccc0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:21.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.907+0000 7fccbe7fc640 10 client.?.objecter ms_handle_connect 0x7fccc8146090 2024-01-20T11:19:21.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.907+0000 7fccbe7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:21.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 1 -- 172.21.15.110:0/2357993166 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc003e610 con 0x7fccc8146090 2024-01-20T11:19:21.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:21.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:21.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 10 monclient: dump: 2024-01-20T11:19:21.913 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:21.913 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 1 -- 172.21.15.110:0/2357993166 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc003e770 con 0x7fccc8146090 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 1 -- 172.21.15.110:0/2357993166 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc003eb00 con 0x7fccc8146090 2024-01-20T11:19:21.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccd0105640 5 monclient: authenticate success, global_id 4838 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccd0105640 10 monclient: _renew_subs 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccd0105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc80aba80 con 0x7fccc8146090 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbffff640 4 set_mon_vals no callback set 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccbe7fc640 10 monclient: dump: 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:21.915 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccd0105640 10 client.4838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccd0105640 10 monclient: _renew_subs 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccd0105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc813da60 con 0x7fccc8146090 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.911+0000 7fccd0105640 1 librados: init done 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.912+0000 7fccd0105640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.912+0000 7fccd0105640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.912+0000 7fccd0105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:21.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.912+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc813dc50 con 0x7fccc8146090 2024-01-20T11:19:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.930+0000 7fccbe7fc640 1 -- 172.21.15.110:0/2357993166 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fccc003b050 con 0x7fccc8146090 2024-01-20T11:19:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.930+0000 7fccbe7fc640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fccac06c860 0x7fccac06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.930+0000 7fcccdea1640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fccac06c860 0x7fccac06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.930+0000 7fcccdea1640 10 monclient: get_auth_request con 0x7fccac06c860 auth_method 0 2024-01-20T11:19:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.931+0000 7fcccdea1640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fccac06c860 0x7fccac06ed20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fccc4001780 tx=0x7fccc4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.931+0000 7fccbe7fc640 10 client.4838.objecter ms_handle_connect 0x7fccac06c860 2024-01-20T11:19:21.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.933+0000 7fccbe7fc640 1 -- 172.21.15.110:0/2357993166 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fccc007aff0 con 0x7fccc8146090 2024-01-20T11:19:21.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.933+0000 7fccbe7fc640 10 client.4838.objecter ms_dispatch 0x7fccc8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:21.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.933+0000 7fccbe7fc640 3 client.4838.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:21.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.933+0000 7fccbe7fc640 3 client.4838.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:21.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:21.933+0000 7fccbe7fc640 20 client.4838.objecter dump_active .. 0 homeless 2024-01-20T11:19:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.162+0000 7fccbe7fc640 1 -- 172.21.15.110:0/2357993166 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fccc0077330 con 0x7fccc8146090 2024-01-20T11:19:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.162+0000 7fccbe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.162+0000 7fccbe7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:22.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.331+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fccc8075490 con 0x7fccac06c860 2024-01-20T11:19:22.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.334+0000 7fccbe7fc640 1 -- 172.21.15.110:0/2357993166 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63960 (secure 0 0 0) 0x7fccc8075490 con 0x7fccac06c860 2024-01-20T11:19:22.335 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:22.335 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 10 librados: watch_flush enter 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 10 librados: watch_flush exit 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 20 client.4838.objecter shutdown clearing up homeless session... 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 10 client.4838.objecter successfully canceled tick 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fccac06c860 msgr2=0x7fccac06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fccac06c860 0x7fccac06ed20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fccc4001780 tx=0x7fccc4009040 comp rx=0 tx=0).stop 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 10 monclient: shutdown 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccc8146090 msgr2=0x7fccc80ac180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccc8146090 0x7fccc80ac180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fccc002c4a0 tx=0x7fccc0004a40 comp rx=0 tx=0).stop 2024-01-20T11:19:22.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 shutdown_connections 2024-01-20T11:19:22.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fccac06c860 0x7fccac06ed20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccc8146090 0x7fccc80ac180 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccc80b4d10 0x7fccc813c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 --2- 172.21.15.110:0/2357993166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccc80b1ed0 0x7fccc813e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 >> 172.21.15.110:0/2357993166 conn(0x7fccc8066590 msgr2=0x7fccc8139be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:22.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 shutdown_connections 2024-01-20T11:19:22.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 -- 172.21.15.110:0/2357993166 wait complete. 2024-01-20T11:19:22.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.336+0000 7fccd0105640 1 librados: shutdown 2024-01-20T11:19:22.347 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-01-20T11:19:20.757116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002600"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1429999999999998},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":13.545999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.255000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.324}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.753},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.968999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.9770000000000003},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":10.936999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.904},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":14.885999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7029999999999994},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":16.803000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313269,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.2569999999999997},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":15.512}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.7590000000000003},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.551}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.896000000000001},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":16.513000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.051},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.625}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.256},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":18.236999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":17.183},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.287000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313270,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":18.129000000000001},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.0660000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.7390000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":15.366},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.016}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.901999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":18.693000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":15.975},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":12.997}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":13.942},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.183}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444084,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.4889999999999999},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":11.525}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7720000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.9009999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.523},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.202}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.930999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.183}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.811},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7220000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.0749999999999993},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":14.885999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444085,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.6870000000000003},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.0990000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.4559999999999995},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.916}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.077999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.032999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":10.456},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.704000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.0149999999999997},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":14.522}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.1219999999999999},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.339}]}]},{"osd":0,"up_from":17,"seq":73014444085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":6.6369999999999996},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":4.3040000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":13.349},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.5659999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":9.2829999999999995},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.976000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.614000000000001},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.1869999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":8.7390000000000008},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":12.638}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.912000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":14.662000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509493,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":7.4080000000000004,"5min":10.327999999999999,"15min":11.295999999999999},"min":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"max":{"1min":11.795999999999999,"5min":106.059,"15min":106.059},"last":6.3789999999999996},{"interface":"front","average":{"1min":8.6549999999999994,"5min":12.192,"15min":14.045},"min":{"1min":4.2450000000000001,"5min":3.96,"15min":3.96},"max":{"1min":13.644,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.0980000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.881,"5min":22.417000000000002,"15min":27.847000000000001},"min":{"1min":6.9379999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.265999999999998,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.0629999999999997},{"interface":"front","average":{"1min":11.308,"5min":23.657,"15min":29.041},"min":{"1min":6.7039999999999997,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":16.733000000000001,"5min":190.304,"15min":190.304},"last":6.9409999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:18:14 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":13.129,"15min":14.474},"min":{"1min":6.5819999999999999,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.843,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.286},{"interface":"front","average":{"1min":11.49,"5min":14.509,"15min":15.615},"min":{"1min":6.8159999999999998,"5min":6.8159999999999998,"15min":6.8159999999999998},"max":{"1min":17.050000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.618}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.561999999999999},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.888}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.8960000000000008},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.84},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8.9250000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:22.349 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:22.349 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:22.349 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:22.349 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:22.350 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:22.350 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:22.350 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:22.350 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:22.350 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.448+0000 7f4884de9640 10 monclient: get_monmap_and_config 2024-01-20T11:19:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.448+0000 7f4884de9640 10 monclient: build_initial_monmap 2024-01-20T11:19:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.448+0000 7f4884de9640 10 monclient: monmap: 2024-01-20T11:19:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:22.450098+0000 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:22.450098+0000 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:22.450 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 1 Processor -- start 2024-01-20T11:19:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 1 -- start start 2024-01-20T11:19:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient: init 2024-01-20T11:19:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-01-20T11:19:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f488006bb00 0x7f48800b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient(hunting): picked mon.noname-c con 0x7f488006bb00 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48800af790 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48800b0900 0x7f4880145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient(hunting): picked mon.noname-b con 0x7f48800b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488006c220 con 0x7f48800af790 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488006bee0 con 0x7f488006bb00 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48800ada30 con 0x7f48800b0900 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.449+0000 7f4884de9640 10 monclient(hunting): authenticate will time out at 1605.801070s 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.450+0000 7f487ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48800b0900 0x7f4880145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.450+0000 7f487e59c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f488006bb00 0x7f48800b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.450+0000 7f487ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48800b0900 0x7f4880145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50642/0 (socket says 172.21.15.110:50642) 2024-01-20T11:19:22.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.450+0000 7f487ed9d640 1 -- 172.21.15.110:0/3830712705 learned_addr learned my addr 172.21.15.110:0/3830712705 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:22.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.450+0000 7f487ed9d640 10 monclient(hunting): get_auth_request con 0x7f48800b0900 auth_method 0 2024-01-20T11:19:22.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.450+0000 7f487ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:22.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.450+0000 7f487ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:22.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.450+0000 7f487ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:22.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.451+0000 7f487e59c640 10 monclient(hunting): get_auth_request con 0x7f488006bb00 auth_method 0 2024-01-20T11:19:22.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.451+0000 7f487e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:22.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.451+0000 7f487e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:22.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.451+0000 7f487e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:22.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.451+0000 7f487ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:22.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.451+0000 7f487ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:22.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.451+0000 7f487ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.452+0000 7f487e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.452+0000 7f487e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.452+0000 7f487e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.453+0000 7f487ed9d640 10 monclient(hunting): handle_auth_done global_id 4841 payload 274 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.453+0000 7f487ed9d640 1 -- 172.21.15.110:0/3830712705 >> v2:172.21.15.134:3300/0 conn(0x7f488006bb00 msgr2=0x7f48800b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.453+0000 7f487ed9d640 1 --2- 172.21.15.110:0/3830712705 >> v2:172.21.15.134:3300/0 conn(0x7f488006bb00 0x7f48800b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.453+0000 7f487ed9d640 1 -- 172.21.15.110:0/3830712705 >> v1:172.21.15.134:6789/0 conn(0x7f48800af790 legacy=0x7f4880145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.453+0000 7f487ed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.453+0000 7f487ed9d640 1 monclient: found mon.noname-b 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.453+0000 7f487ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.453+0000 7f487ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.453+0000 7f487ed9d640 1 -- 172.21.15.110:0/3830712705 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48800add10 con 0x7f48800b0900 2024-01-20T11:19:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.454+0000 7f487ed9d640 1 --2- 172.21.15.110:0/3830712705 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48800b0900 0x7f4880145e80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4878009e50 tx=0x7f487802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbf86897735796e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 1 -- 172.21.15.110:0/3830712705 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487802ca70 con 0x7f48800b0900 2024-01-20T11:19:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 10 monclient: dump: 2024-01-20T11:19:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 10 monclient: _finish_auth 0 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 10 monclient: _check_auth_tickets 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 1 -- 172.21.15.110:0/3830712705 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487802cbd0 con 0x7f48800b0900 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 1 -- 172.21.15.110:0/3830712705 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48780343f0 con 0x7f48800b0900 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f487d59a640 10 monclient: dump: 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f4884de9640 5 monclient: authenticate success, global_id 4841 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f4884de9640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.458+0000 7f4884de9640 4 set_mon_vals no callback set 2024-01-20T11:19:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 10 monclient: shutdown 2024-01-20T11:19:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 1 -- 172.21.15.110:0/3830712705 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48800b0900 msgr2=0x7f4880145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 1 --2- 172.21.15.110:0/3830712705 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48800b0900 0x7f4880145e80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4878009e50 tx=0x7f487802bed0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 1 -- 172.21.15.110:0/3830712705 shutdown_connections 2024-01-20T11:19:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 1 --2- 172.21.15.110:0/3830712705 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48800b0900 0x7f4880145e80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 1 --2- 172.21.15.110:0/3830712705 >> v2:172.21.15.134:3300/0 conn(0x7f488006bb00 0x7f48800b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 1 -- 172.21.15.110:0/3830712705 >> 172.21.15.110:0/3830712705 conn(0x7f4880066d60 msgr2=0x7f48800b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 1 -- 172.21.15.110:0/3830712705 shutdown_connections 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 1 -- 172.21.15.110:0/3830712705 wait complete. 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 10 monclient: build_initial_monmap 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.459+0000 7f4884de9640 10 monclient: monmap: 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 librados: starting msgr at 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 librados: starting objecter 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 Processor -- start 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 -- start start 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 librados: setting wanted keys 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 librados: calling monclient init 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient: init 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f488006bb00 0x7f488013c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient(hunting): picked mon.noname-c con 0x7f488006bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48800af790 0x7f488011de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient(hunting): picked mon.noname-b con 0x7f48800af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48800b0900 0x7f4880070ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient(hunting): picked mon.noname-a con 0x7f48800b0900 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48800b5540 con 0x7f48800b0900 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488011d880 con 0x7f48800af790 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48800b5820 con 0x7f488006bb00 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.460+0000 7f4884de9640 10 monclient(hunting): authenticate will time out at 1605.812104s 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f488006bb00 0x7f488013c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48800b0900 0x7f4880070ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f488006bb00 0x7f488013c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50656/0 (socket says 172.21.15.110:50656) 2024-01-20T11:19:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487e59c640 1 -- 172.21.15.110:0/3550412589 learned_addr learned my addr 172.21.15.110:0/3550412589 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487e59c640 10 monclient(hunting): get_auth_request con 0x7f488006bb00 auth_method 0 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487ed9d640 10 monclient(hunting): get_auth_request con 0x7f48800b0900 auth_method 0 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.461+0000 7f487ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487dd9b640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48800af790 0x7f488011de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487dd9b640 10 monclient(hunting): get_auth_request con 0x7f48800af790 auth_method 0 2024-01-20T11:19:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:22.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:22.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:22.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:22.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:22.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:22.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:22.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.462+0000 7f487dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:22.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.463+0000 7f487dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:22.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 10 monclient(hunting): handle_auth_done global_id 4844 payload 824 2024-01-20T11:19:22.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 1 -- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48800af790 msgr2=0x7f488011de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48800af790 0x7f488011de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 1 -- 172.21.15.110:0/3550412589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48800b0900 msgr2=0x7f4880070ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48800b0900 0x7f4880070ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 10 monclient: _finish_hunting 0 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 1 monclient: found mon.noname-c 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.465+0000 7f487e59c640 1 -- 172.21.15.110:0/3550412589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4878009b30 con 0x7f488006bb00 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.467+0000 7f487e59c640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f488006bb00 0x7f488013c3a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f487400e9b0 tx=0x7f487400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.467+0000 7f486effd640 10 client.?.objecter ms_handle_connect 0x7f488006bb00 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.467+0000 7f486effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 1 -- 172.21.15.110:0/3550412589 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874009670 con 0x7f488006bb00 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:22.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 10 monclient: dump: 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 10 monclient: _finish_auth 0 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:22.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 1 -- 172.21.15.110:0/3550412589 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487400c9a0 con 0x7f488006bb00 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 1 -- 172.21.15.110:0/3550412589 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487400cd50 con 0x7f488006bb00 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f486effd640 10 monclient: dump: 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:22.475 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f4884de9640 5 monclient: authenticate success, global_id 4844 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f4884de9640 10 monclient: _renew_subs 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f4884de9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48801dae00 con 0x7f488006bb00 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f487cd99640 4 set_mon_vals no callback set 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f4884de9640 10 client.4844.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f4884de9640 10 monclient: _renew_subs 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f4884de9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488011e3b0 con 0x7f488006bb00 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.470+0000 7f4884de9640 1 librados: init done 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.471+0000 7f4884de9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.471+0000 7f4884de9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:22.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.471+0000 7f4884de9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:22.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.471+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4880075490 con 0x7f488006bb00 2024-01-20T11:19:22.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.489+0000 7f486effd640 1 -- 172.21.15.110:0/3550412589 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4874013470 con 0x7f488006bb00 2024-01-20T11:19:22.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.489+0000 7f486effd640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f486006c890 0x7f486006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:22.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.490+0000 7f487dd9b640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f486006c890 0x7f486006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:22.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.490+0000 7f487dd9b640 10 monclient: get_auth_request con 0x7f486006c890 auth_method 0 2024-01-20T11:19:22.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.490+0000 7f487dd9b640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f486006c890 0x7f486006ed50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f48800718d0 tx=0x7f4870009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:22.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.490+0000 7f486effd640 10 client.4844.objecter ms_handle_connect 0x7f486006c890 2024-01-20T11:19:22.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.493+0000 7f486effd640 1 -- 172.21.15.110:0/3550412589 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4874058be0 con 0x7f488006bb00 2024-01-20T11:19:22.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.493+0000 7f486effd640 10 client.4844.objecter ms_dispatch 0x7f4880000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:22.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.493+0000 7f486effd640 3 client.4844.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:22.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.493+0000 7f486effd640 3 client.4844.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:22.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.493+0000 7f486effd640 20 client.4844.objecter dump_active .. 0 homeless 2024-01-20T11:19:22.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.729+0000 7f486effd640 1 -- 172.21.15.110:0/3550412589 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4874054f20 con 0x7f488006bb00 2024-01-20T11:19:22.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.729+0000 7f486effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:22.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.729+0000 7f486effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:22.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~25s 2024-01-20T11:19:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.992+0000 7f4884de9640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.992+0000 7f4884de9640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.992+0000 7f4884de9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:22.992+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f48801461a0 con 0x7f488006bb00 2024-01-20T11:19:23.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.030+0000 7f486effd640 1 -- 172.21.15.110:0/3550412589 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f4874021520 con 0x7f488006bb00 2024-01-20T11:19:23.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.030+0000 7f486effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:23.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.030+0000 7f486effd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:23.032 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:23.032 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":297,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 10 librados: watch_flush enter 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 10 librados: watch_flush exit 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 20 client.4844.objecter shutdown clearing up homeless session... 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 10 client.4844.objecter successfully canceled tick 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f486006c890 msgr2=0x7f486006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f486006c890 0x7f486006ed50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f48800718d0 tx=0x7f4870009040 comp rx=0 tx=0).stop 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 10 monclient: shutdown 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f488006bb00 msgr2=0x7f488013c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f488006bb00 0x7f488013c3a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f487400e9b0 tx=0x7f487400ee70 comp rx=0 tx=0).stop 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 shutdown_connections 2024-01-20T11:19:23.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f486006c890 0x7f486006ed50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48800b0900 0x7f4880070ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48800af790 0x7f488011de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 --2- 172.21.15.110:0/3550412589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f488006bb00 0x7f488013c3a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 >> 172.21.15.110:0/3550412589 conn(0x7f4880066d60 msgr2=0x7f4880142970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:23.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 shutdown_connections 2024-01-20T11:19:23.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 -- 172.21.15.110:0/3550412589 wait complete. 2024-01-20T11:19:23.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.032+0000 7f4884de9640 1 librados: shutdown 2024-01-20T11:19:23.044 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:23.044 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.143+0000 7fc4cdd9e640 10 monclient: get_monmap_and_config 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.143+0000 7fc4cdd9e640 10 monclient: build_initial_monmap 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.143+0000 7fc4cdd9e640 10 monclient: monmap: 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:23.144706+0000 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:23.144706+0000 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:23.144 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:23.145 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:23.145 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:23.145 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:23.145 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.143+0000 7fc4cdd9e640 1 Processor -- start 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.143+0000 7fc4cdd9e640 1 -- start start 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.143+0000 7fc4cdd9e640 10 monclient: init 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc4c80b4d10 0x7fc4c80b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c80b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4c80b17c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c8145980 0x7fc4c8145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c8145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:23.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c8146370 con 0x7fc4c80b17c0 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c80b50f0 con 0x7fc4c80b4d10 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80aade0 con 0x7fc4c8145980 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.144+0000 7fc4cdd9e640 10 monclient(hunting): authenticate will time out at 1606.495642s 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c8145980 0x7fc4c8145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c6ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc4c80b17c0 0x7fc4c80b1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33932/0 (socket says 172.21.15.110:33932) 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c6ffd640 1 -- 172.21.15.110:0/1992741264 learned_addr learned my addr 172.21.15.110:0/1992741264 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c77fe640 1 --2- 172.21.15.110:0/1992741264 >> v2:172.21.15.134:3300/0 conn(0x7fc4c80b4d10 0x7fc4c80b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c7fff640 10 monclient(hunting): get_auth_request con 0x7fc4c8145980 auth_method 0 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:23.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c77fe640 10 monclient(hunting): get_auth_request con 0x7fc4c80b4d10 auth_method 0 2024-01-20T11:19:23.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:23.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:23.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.145+0000 7fc4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:23.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.146+0000 7fc4c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:23.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.146+0000 7fc4c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:23.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.146+0000 7fc4c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:23.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.146+0000 7fc4c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:23.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.146+0000 7fc4c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:23.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.146+0000 7fc4c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:23.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.147+0000 7fc4c7fff640 10 monclient(hunting): handle_auth_done global_id 4847 payload 274 2024-01-20T11:19:23.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.147+0000 7fc4c7fff640 1 -- 172.21.15.110:0/1992741264 >> v2:172.21.15.134:3300/0 conn(0x7fc4c80b4d10 msgr2=0x7fc4c80b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:23.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.147+0000 7fc4c7fff640 1 --2- 172.21.15.110:0/1992741264 >> v2:172.21.15.134:3300/0 conn(0x7fc4c80b4d10 0x7fc4c80b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.147+0000 7fc4c7fff640 1 -- 172.21.15.110:0/1992741264 >> v1:172.21.15.110:6789/0 conn(0x7fc4c80b17c0 legacy=0x7fc4c80b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:23.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.147+0000 7fc4c7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:23.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.147+0000 7fc4c7fff640 1 monclient: found mon.noname-b 2024-01-20T11:19:23.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.147+0000 7fc4c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:23.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.147+0000 7fc4c7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:23.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.147+0000 7fc4c7fff640 1 -- 172.21.15.110:0/1992741264 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c8145460 con 0x7fc4c8145980 2024-01-20T11:19:23.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.150+0000 7fc4c7fff640 1 --2- 172.21.15.110:0/1992741264 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c8145980 0x7fc4c8145e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80097a0 tx=0x7fc4b802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff86cdf0585d7143 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:23.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 1 -- 172.21.15.110:0/1992741264 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b802e070 con 0x7fc4c8145980 2024-01-20T11:19:23.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:23.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:23.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 10 monclient: dump: 2024-01-20T11:19:23.154 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:23.154 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:23.154 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 1 -- 172.21.15.110:0/1992741264 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b80315a0 con 0x7fc4c8145980 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 1 -- 172.21.15.110:0/1992741264 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8031930 con 0x7fc4c8145980 2024-01-20T11:19:23.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4c67fc640 10 monclient: dump: 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4cdd9e640 5 monclient: authenticate success, global_id 4847 2024-01-20T11:19:23.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4cdd9e640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:23.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.153+0000 7fc4cdd9e640 4 set_mon_vals no callback set 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 10 monclient: shutdown 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1992741264 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c8145980 msgr2=0x7fc4c8145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 1 --2- 172.21.15.110:0/1992741264 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c8145980 0x7fc4c8145e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80097a0 tx=0x7fc4b802b840 comp rx=0 tx=0).stop 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1992741264 shutdown_connections 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 1 --2- 172.21.15.110:0/1992741264 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c8145980 0x7fc4c8145e30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 1 --2- 172.21.15.110:0/1992741264 >> v2:172.21.15.134:3300/0 conn(0x7fc4c80b4d10 0x7fc4c80b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1992741264 >> 172.21.15.110:0/1992741264 conn(0x7fc4c8066590 msgr2=0x7fc4c80669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1992741264 shutdown_connections 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1992741264 wait complete. 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 10 monclient: build_initial_monmap 2024-01-20T11:19:23.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.154+0000 7fc4cdd9e640 10 monclient: monmap: 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.155+0000 7fc4cdd9e640 1 librados: starting msgr at 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.155+0000 7fc4cdd9e640 1 librados: starting objecter 2024-01-20T11:19:23.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.155+0000 7fc4cdd9e640 1 Processor -- start 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.155+0000 7fc4cdd9e640 1 -- start start 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.155+0000 7fc4cdd9e640 1 librados: setting wanted keys 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.155+0000 7fc4cdd9e640 1 librados: calling monclient init 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.155+0000 7fc4cdd9e640 10 monclient: init 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4c80b17c0 0x7fc4c8146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c80b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c80b4d10 0x7fc4c8137f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c80b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4c8145980 0x7fc4c8138450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c8145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c8147090 con 0x7fc4c8145980 2024-01-20T11:19:23.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c8148170 con 0x7fc4c80b17c0 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c8146d40 con 0x7fc4c80b4d10 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4cdd9e640 10 monclient(hunting): authenticate will time out at 1606.507293s 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4c6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c80b4d10 0x7fc4c8137f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4c77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4c80b17c0 0x7fc4c8146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4c7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4c8145980 0x7fc4c8138450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4c6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c80b4d10 0x7fc4c8137f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50678/0 (socket says 172.21.15.110:50678) 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4c6ffd640 1 -- 172.21.15.110:0/1077989413 learned_addr learned my addr 172.21.15.110:0/1077989413 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4c6ffd640 10 monclient(hunting): get_auth_request con 0x7fc4c80b4d10 auth_method 0 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4c6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.156+0000 7fc4c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:23.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c77fe640 10 monclient(hunting): get_auth_request con 0x7fc4c80b17c0 auth_method 0 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c7fff640 10 monclient(hunting): get_auth_request con 0x7fc4c8145980 auth_method 0 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:23.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.157+0000 7fc4c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 10 monclient(hunting): handle_auth_done global_id 4850 payload 824 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 1 -- 172.21.15.110:0/1077989413 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4c80b17c0 msgr2=0x7fc4c8146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4c80b17c0 0x7fc4c8146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 1 -- 172.21.15.110:0/1077989413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4c8145980 msgr2=0x7fc4c8138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4c8145980 0x7fc4c8138450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 1 monclient: found mon.noname-c 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:23.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.160+0000 7fc4c6ffd640 1 -- 172.21.15.110:0/1077989413 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b8009400 con 0x7fc4c80b4d10 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.162+0000 7fc4c6ffd640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c80b4d10 0x7fc4c8137f10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000c530 tx=0x7fc4b000c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.162+0000 7fc4bffff640 10 client.?.objecter ms_handle_connect 0x7fc4c80b4d10 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.162+0000 7fc4bffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 1 -- 172.21.15.110:0/1077989413 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0005ce0 con 0x7fc4c80b4d10 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 10 monclient: dump: 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:23.168 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 10 monclient: _finish_auth 0 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 1 -- 172.21.15.110:0/1077989413 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b0005e40 con 0x7fc4c80b4d10 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4cdd9e640 5 monclient: authenticate success, global_id 4850 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4cdd9e640 10 monclient: _renew_subs 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4cdd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:23.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c80b13b0 con 0x7fc4c80b4d10 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 1 -- 172.21.15.110:0/1077989413 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0007270 con 0x7fc4c80b4d10 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4bffff640 10 monclient: dump: 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:23.170 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4c5ffb640 4 set_mon_vals no callback set 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4cdd9e640 10 client.4850.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4cdd9e640 10 monclient: _renew_subs 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4cdd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c813ee50 con 0x7fc4c80b4d10 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.165+0000 7fc4cdd9e640 1 librados: init done 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.166+0000 7fc4cdd9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.166+0000 7fc4cdd9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.166+0000 7fc4cdd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:23.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.166+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c813ca00 con 0x7fc4c80b4d10 2024-01-20T11:19:23.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.183+0000 7fc4bffff640 1 -- 172.21.15.110:0/1077989413 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc4b00073d0 con 0x7fc4c80b4d10 2024-01-20T11:19:23.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.184+0000 7fc4bffff640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4a806c8b0 0x7fc4a806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:23.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.184+0000 7fc4c77fe640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4a806c8b0 0x7fc4a806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:23.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.184+0000 7fc4c77fe640 10 monclient: get_auth_request con 0x7fc4a806c8b0 auth_method 0 2024-01-20T11:19:23.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.185+0000 7fc4bffff640 10 client.4850.objecter ms_handle_connect 0x7fc4a806c8b0 2024-01-20T11:19:23.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.185+0000 7fc4c77fe640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4a806c8b0 0x7fc4a806ed70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00113e0 tx=0x7fc4c000b040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:23.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.186+0000 7fc4bffff640 1 -- 172.21.15.110:0/1077989413 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc4b0057d50 con 0x7fc4c80b4d10 2024-01-20T11:19:23.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.186+0000 7fc4bffff640 10 client.4850.objecter ms_dispatch 0x7fc4c8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:23.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.186+0000 7fc4bffff640 3 client.4850.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:23.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.186+0000 7fc4bffff640 3 client.4850.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:23.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.186+0000 7fc4bffff640 20 client.4850.objecter dump_active .. 0 homeless 2024-01-20T11:19:23.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.418+0000 7fc4bffff640 1 -- 172.21.15.110:0/1077989413 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc4b0054090 con 0x7fc4c80b4d10 2024-01-20T11:19:23.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.418+0000 7fc4bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:23.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.418+0000 7fc4bffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:23.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.593+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4c814c7f0 con 0x7fc4a806c8b0 2024-01-20T11:19:23.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.596+0000 7fc4bffff640 1 -- 172.21.15.110:0/1077989413 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63983 (secure 0 0 0) 0x7fc4c000f040 con 0x7fc4a806c8b0 2024-01-20T11:19:23.597 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:23.598 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 10 librados: watch_flush enter 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 10 librados: watch_flush exit 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 20 client.4850.objecter shutdown clearing up homeless session... 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 10 client.4850.objecter successfully canceled tick 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4a806c8b0 msgr2=0x7fc4a806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4a806c8b0 0x7fc4a806ed70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00113e0 tx=0x7fc4c000b040 comp rx=0 tx=0).stop 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 10 monclient: shutdown 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c80b4d10 msgr2=0x7fc4c8137f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c80b4d10 0x7fc4c8137f10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000c530 tx=0x7fc4b000c9f0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 shutdown_connections 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc4a806c8b0 0x7fc4a806ed70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4c8145980 0x7fc4c8138450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4c80b4d10 0x7fc4c8137f10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 --2- 172.21.15.110:0/1077989413 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4c80b17c0 0x7fc4c8146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:23.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 >> 172.21.15.110:0/1077989413 conn(0x7fc4c8066590 msgr2=0x7fc4c8144bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:23.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 shutdown_connections 2024-01-20T11:19:23.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 -- 172.21.15.110:0/1077989413 wait complete. 2024-01-20T11:19:23.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:23.598+0000 7fc4cdd9e640 1 librados: shutdown 2024-01-20T11:19:23.610 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-01-20T11:19:22.757602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002708"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.1429999999999998},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":13.545999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":14.255000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.324}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.753},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.968999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.9770000000000003},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":10.936999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.904},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":14.885999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7029999999999994},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":16.803000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313270,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.0469999999999997},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.3840000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":14.778},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.1270000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.968},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":11.574}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.776999999999999},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.757}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.7729999999999997},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":12.170999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.754},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.968}]}]},{"osd":3,"up_from":21,"seq":90194313271,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.6790000000000003},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":3.9510000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.004},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":8.6820000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":7.4710000000000001},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.769}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.933999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":9.2460000000000004},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":12.351000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":14.898},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.654999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444084,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.0099999999999998,"5min":9.6430000000000007,"15min":10.153},"min":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.012,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.4889999999999999},{"interface":"front","average":{"1min":8.2750000000000004,"5min":8.7189999999999994,"15min":8.6509999999999998},"min":{"1min":5.9909999999999997,"5min":4.3559999999999999,"15min":4.3559999999999999},"max":{"1min":16.163,"5min":16.163,"15min":16.163},"last":11.525}]},{"osd":1,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":8.5649999999999995,"5min":12.112,"15min":13.564},"min":{"1min":4.5890000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":16.443000000000001,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.7720000000000002},{"interface":"front","average":{"1min":10.629,"5min":14.693,"15min":16.114999999999998},"min":{"1min":6.8869999999999996,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.897,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.9009999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.523},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.202}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.930999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.183}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:18:17 2024","interfaces":[{"interface":"back","average":{"1min":13.016999999999999,"5min":15.140000000000001,"15min":15.763999999999999},"min":{"1min":8.1170000000000009,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.917000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.811},{"interface":"front","average":{"1min":10.597,"5min":10.726000000000001,"15min":10.874000000000001},"min":{"1min":5.6260000000000003,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.680999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7220000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":8.0749999999999993},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":14.885999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444085,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.6870000000000003},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.0990000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.4559999999999995},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.916}]},{"osd":3,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":13.587,"5min":21.878,"15min":25.312999999999999},"min":{"1min":10.106,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":17.808,"5min":259.483,"15min":259.483},"last":12.077999999999999},{"interface":"front","average":{"1min":9.3390000000000004,"5min":17.062999999999999,"15min":20.114000000000001},"min":{"1min":5.9930000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.798,"5min":243.24199999999999,"15min":243.24199999999999},"last":11.032999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":10.456},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.704000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.0149999999999997},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":14.522}]},{"osd":7,"last update":"Sat Jan 20 11:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.593999999999999,"5min":24.456,"15min":29.670000000000002},"min":{"1min":8.5079999999999991,"5min":8.1940000000000008,"15min":8.1940000000000008},"max":{"1min":18.855,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.1219999999999999},{"interface":"front","average":{"1min":14.362,"5min":25.606000000000002,"15min":30.225999999999999},"min":{"1min":10.555,"5min":10.555,"15min":10.555},"max":{"1min":17.494,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.339}]}]},{"osd":0,"up_from":17,"seq":73014444085,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":6.6369999999999996},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":4.3040000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":13.349},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.5659999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":9.2829999999999995},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.976000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.614000000000001},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.1869999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":8.7390000000000008},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":12.638}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.912000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":14.662000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509494,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":4.1449999999999996},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":10.441000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":14.026999999999999},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":9.6449999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.105},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.776}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":7.4800000000000004},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.565}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.734},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.403}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.754999999999999},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":8.6850000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:26.613 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.714+0000 7fd231632640 10 monclient: get_monmap_and_config 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.714+0000 7fd231632640 10 monclient: build_initial_monmap 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.714+0000 7fd231632640 10 monclient: monmap: 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:26.714630+0000 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:26.714630+0000 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:26.715 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.714+0000 7fd231632640 1 Processor -- start 2024-01-20T11:19:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.714+0000 7fd231632640 1 -- start start 2024-01-20T11:19:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.714+0000 7fd231632640 10 monclient: init 2024-01-20T11:19:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-01-20T11:19:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd22c145760 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd22c0a9d80 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c13f8b0 0x7fd22c145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 10 monclient(hunting): picked mon.noname-b con 0x7fd22c13f8b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c146250 con 0x7fd22c145760 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c0b20c0 con 0x7fd22c0a9d80 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0ab050 con 0x7fd22c13f8b0 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd231632640 10 monclient(hunting): authenticate will time out at 1610.065435s 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd22b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c13f8b0 0x7fd22c145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd22affd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd22c145760 0x7fd22c13f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33934/0 (socket says 172.21.15.110:33934) 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.715+0000 7fd22affd640 1 -- 172.21.15.110:0/236337480 learned_addr learned my addr 172.21.15.110:0/236337480 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:26.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.716+0000 7fd22b7fe640 10 monclient(hunting): get_auth_request con 0x7fd22c13f8b0 auth_method 0 2024-01-20T11:19:26.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.716+0000 7fd22b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:26.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.716+0000 7fd22b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:26.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.716+0000 7fd22b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:26.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.716+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:26.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.716+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:26.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.716+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:26.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.718+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_done global_id 4853 payload 274 2024-01-20T11:19:26.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.718+0000 7fd22b7fe640 1 -- 172.21.15.110:0/236337480 >> v1:172.21.15.134:6789/0 conn(0x7fd22c0a9d80 legacy=0x7fd22c0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:26.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.718+0000 7fd22b7fe640 1 -- 172.21.15.110:0/236337480 >> v1:172.21.15.110:6789/0 conn(0x7fd22c145760 legacy=0x7fd22c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:26.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.718+0000 7fd22b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:26.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.718+0000 7fd22b7fe640 1 monclient: found mon.noname-b 2024-01-20T11:19:26.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.718+0000 7fd22b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:26.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.718+0000 7fd22b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:26.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.718+0000 7fd22b7fe640 1 -- 172.21.15.110:0/236337480 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0b2d80 con 0x7fd22c13f8b0 2024-01-20T11:19:26.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.720+0000 7fd22b7fe640 1 --2- 172.21.15.110:0/236337480 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c13f8b0 0x7fd22c145b40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd21c0097d0 tx=0x7fd21c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e19f42bc373337e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:26.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 1 -- 172.21.15.110:0/236337480 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c031890 con 0x7fd22c13f8b0 2024-01-20T11:19:26.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:26.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:26.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 10 monclient: dump: 2024-01-20T11:19:26.723 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:26.723 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 10 monclient: _finish_auth 0 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 1 -- 172.21.15.110:0/236337480 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c0319f0 con 0x7fd22c13f8b0 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 1 -- 172.21.15.110:0/236337480 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c031d80 con 0x7fd22c13f8b0 2024-01-20T11:19:26.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd231632640 5 monclient: authenticate success, global_id 4853 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd231632640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd231632640 4 set_mon_vals no callback set 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.723+0000 7fd229ffb640 10 monclient: dump: 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:26.725 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.724+0000 7fd231632640 10 monclient: shutdown 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.724+0000 7fd231632640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.724+0000 7fd231632640 1 -- 172.21.15.110:0/236337480 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c13f8b0 msgr2=0x7fd22c145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.724+0000 7fd231632640 1 --2- 172.21.15.110:0/236337480 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c13f8b0 0x7fd22c145b40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd21c0097d0 tx=0x7fd21c02b930 comp rx=0 tx=0).stop 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.724+0000 7fd231632640 1 -- 172.21.15.110:0/236337480 shutdown_connections 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.724+0000 7fd231632640 1 --2- 172.21.15.110:0/236337480 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c13f8b0 0x7fd22c145b40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.724+0000 7fd231632640 1 -- 172.21.15.110:0/236337480 >> 172.21.15.110:0/236337480 conn(0x7fd22c0663b0 msgr2=0x7fd22c0667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.724+0000 7fd231632640 1 -- 172.21.15.110:0/236337480 shutdown_connections 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.724+0000 7fd231632640 1 -- 172.21.15.110:0/236337480 wait complete. 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.725+0000 7fd231632640 10 monclient: build_initial_monmap 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.725+0000 7fd231632640 10 monclient: monmap: 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:26.730 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.725+0000 7fd231632640 1 librados: starting msgr at 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.725+0000 7fd231632640 1 librados: starting objecter 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.725+0000 7fd231632640 1 Processor -- start 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.725+0000 7fd231632640 1 -- start start 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.725+0000 7fd231632640 1 librados: setting wanted keys 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.725+0000 7fd231632640 1 librados: calling monclient init 2024-01-20T11:19:26.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.725+0000 7fd231632640 10 monclient: init 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd22c0a9d80 0x7fd22c0b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient(hunting): picked mon.noname-a con 0x7fd22c0a9d80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd22c13f8b0 0x7fd22c1420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient(hunting): picked mon.noname-b con 0x7fd22c13f8b0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c145760 0x7fd22c0b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient(hunting): picked mon.noname-c con 0x7fd22c145760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c137ad0 con 0x7fd22c0a9d80 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c142620 con 0x7fd22c13f8b0 2024-01-20T11:19:26.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c137cb0 con 0x7fd22c145760 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd231632640 10 monclient(hunting): authenticate will time out at 1610.076654s 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.726+0000 7fd22b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c145760 0x7fd22c0b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22a7fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd22c13f8b0 0x7fd22c1420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd22c0a9d80 0x7fd22c0b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c145760 0x7fd22c0b7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50696/0 (socket says 172.21.15.110:50696) 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22b7fe640 1 -- 172.21.15.110:0/453015076 learned_addr learned my addr 172.21.15.110:0/453015076 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22b7fe640 10 monclient(hunting): get_auth_request con 0x7fd22c145760 auth_method 0 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:26.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22affd640 10 monclient(hunting): get_auth_request con 0x7fd22c0a9d80 auth_method 0 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22a7fc640 10 monclient(hunting): get_auth_request con 0x7fd22c13f8b0 auth_method 0 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.727+0000 7fd22b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.728+0000 7fd22a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:26.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.728+0000 7fd22a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:26.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.728+0000 7fd22a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:26.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.728+0000 7fd22affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:26.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.728+0000 7fd22affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:26.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.728+0000 7fd22affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:26.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 10 monclient(hunting): handle_auth_done global_id 5110 payload 824 2024-01-20T11:19:26.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 1 -- 172.21.15.110:0/453015076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c145760 msgr2=0x7fd22c0b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:26.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c145760 0x7fd22c0b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:26.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 1 -- 172.21.15.110:0/453015076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd22c0a9d80 msgr2=0x7fd22c0b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:26.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd22c0a9d80 0x7fd22c0b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:26.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 10 monclient: _finish_hunting 0 2024-01-20T11:19:26.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 1 monclient: found mon.noname-b 2024-01-20T11:19:26.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:26.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:26.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.731+0000 7fd22a7fc640 1 -- 172.21.15.110:0/453015076 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21c009400 con 0x7fd22c13f8b0 2024-01-20T11:19:26.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.733+0000 7fd22a7fc640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd22c13f8b0 0x7fd22c1420e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd22000c700 tx=0x7fd22000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:26.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.733+0000 7fd2137fe640 10 client.?.objecter ms_handle_connect 0x7fd22c13f8b0 2024-01-20T11:19:26.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.733+0000 7fd2137fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:26.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 1 -- 172.21.15.110:0/453015076 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220006d50 con 0x7fd22c13f8b0 2024-01-20T11:19:26.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:26.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:26.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 10 monclient: dump: 2024-01-20T11:19:26.738 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:26.738 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:26.738 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 1 -- 172.21.15.110:0/453015076 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd220003970 con 0x7fd22c13f8b0 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:26.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 1 -- 172.21.15.110:0/453015076 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220003d20 con 0x7fd22c13f8b0 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2137fe640 10 monclient: dump: 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:26.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd231632640 5 monclient: authenticate success, global_id 5110 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd231632640 10 monclient: _renew_subs 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd231632640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c1380d0 con 0x7fd22c13f8b0 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd2297fa640 4 set_mon_vals no callback set 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd231632640 10 client.5110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd231632640 10 monclient: _renew_subs 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd231632640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c0b8250 con 0x7fd22c13f8b0 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.737+0000 7fd231632640 1 librados: init done 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.738+0000 7fd231632640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.738+0000 7fd231632640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.738+0000 7fd231632640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:26.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.738+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c1382f0 con 0x7fd22c13f8b0 2024-01-20T11:19:26.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.978+0000 7fd2137fe640 1 -- 172.21.15.110:0/453015076 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd2200050d0 con 0x7fd22c13f8b0 2024-01-20T11:19:26.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.978+0000 7fd2137fe640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd21806c8b0 0x7fd21806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:26.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.979+0000 7fd22affd640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd21806c8b0 0x7fd21806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:26.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.979+0000 7fd22affd640 10 monclient: get_auth_request con 0x7fd21806c8b0 auth_method 0 2024-01-20T11:19:26.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.979+0000 7fd22affd640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd21806c8b0 0x7fd21806ed70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd224007fd0 tx=0x7fd224007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:26.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.979+0000 7fd2137fe640 10 client.5110.objecter ms_handle_connect 0x7fd21806c8b0 2024-01-20T11:19:26.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.981+0000 7fd2137fe640 1 -- 172.21.15.110:0/453015076 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd220058a70 con 0x7fd22c13f8b0 2024-01-20T11:19:26.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.982+0000 7fd2137fe640 10 client.5110.objecter ms_dispatch 0x7fd22c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:26.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.982+0000 7fd2137fe640 3 client.5110.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:26.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.982+0000 7fd2137fe640 3 client.5110.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:26.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.982+0000 7fd2137fe640 20 client.5110.objecter dump_active .. 0 homeless 2024-01-20T11:19:26.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.991+0000 7fd2137fe640 1 -- 172.21.15.110:0/453015076 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd220054db0 con 0x7fd22c13f8b0 2024-01-20T11:19:26.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.991+0000 7fd2137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:26.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:26.991+0000 7fd2137fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:27.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.166+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd22c137e20 con 0x7fd21806c8b0 2024-01-20T11:19:27.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.169+0000 7fd2137fe640 1 -- 172.21.15.110:0/453015076 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64013 (secure 0 0 0) 0x7fd22c137e20 con 0x7fd21806c8b0 2024-01-20T11:19:27.170 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 10 librados: watch_flush enter 2024-01-20T11:19:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 10 librados: watch_flush exit 2024-01-20T11:19:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 20 client.5110.objecter shutdown clearing up homeless session... 2024-01-20T11:19:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 10 client.5110.objecter successfully canceled tick 2024-01-20T11:19:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd21806c8b0 msgr2=0x7fd21806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd21806c8b0 0x7fd21806ed70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd224007fd0 tx=0x7fd224007e80 comp rx=0 tx=0).stop 2024-01-20T11:19:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 10 monclient: shutdown 2024-01-20T11:19:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd22c13f8b0 msgr2=0x7fd22c1420e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd22c13f8b0 0x7fd22c1420e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd22000c700 tx=0x7fd22000cbc0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 shutdown_connections 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd21806c8b0 0x7fd21806ed70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd22c145760 0x7fd22c0b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd22c13f8b0 0x7fd22c1420e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 --2- 172.21.15.110:0/453015076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd22c0a9d80 0x7fd22c0b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 >> 172.21.15.110:0/453015076 conn(0x7fd22c0663b0 msgr2=0x7fd22c13cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 shutdown_connections 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 -- 172.21.15.110:0/453015076 wait complete. 2024-01-20T11:19:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.171+0000 7fd231632640 1 librados: shutdown 2024-01-20T11:19:27.182 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-01-20T11:19:26.758730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.3460000000000001},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":4.1929999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.2530000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":14.574999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.169},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.108000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.6689999999999996},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":14.962}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.413},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":17.018999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":6.931},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":11.698}]}]},{"osd":7,"up_from":21,"seq":90194313271,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.1879999999999997},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.7169999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":13.07},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.991}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.372999999999999},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.5860000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4450000000000003},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.747}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.484},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":15.105}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.353},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.736000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313272,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.3490000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.5589999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9250000000000007},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.4459999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":11.864000000000001},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.624000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":15.965999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.464}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":13.189},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":15.218999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":16.896000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.398}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444085,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.8550000000000004},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":8.968}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.6449999999999996},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.3540000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.987},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.262}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":11.638999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.912000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.25},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.547000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.301},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":12.228999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444086,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.7549999999999999},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":8.8179999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.919},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.5519999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":13.494},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.9729999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":12.56},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.829000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.4619999999999997},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":14.988}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.664},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.798}]}]},{"osd":0,"up_from":17,"seq":73014444086,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":11.606},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":7.5140000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.4670000000000005},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.9790000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":14.654},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.923999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.509},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.8109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":15.853999999999999},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":16.57}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":17.663},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":10.052}]}]},{"osd":6,"up_from":15,"seq":64424509495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.7919999999999998},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.6769999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.143000000000001},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":11.101000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.452},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.831}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.534000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.986000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.994999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":18.061}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":17.408000000000001},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":19.515000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:27.184 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:27.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.283+0000 7f6858032640 10 monclient: get_monmap_and_config 2024-01-20T11:19:27.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.283+0000 7f6858032640 10 monclient: build_initial_monmap 2024-01-20T11:19:27.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.283+0000 7f6858032640 10 monclient: monmap: 2024-01-20T11:19:27.283 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:27.283 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.283 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:27.284017+0000 2024-01-20T11:19:27.283 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:27.284017+0000 2024-01-20T11:19:27.284 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:27.284 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.284 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:27.284 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:27.284 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:27.284 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:27.284 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:27.284 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.283+0000 7f6858032640 1 Processor -- start 2024-01-20T11:19:27.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 1 -- start start 2024-01-20T11:19:27.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient: init 2024-01-20T11:19:27.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:27.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-01-20T11:19:27.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68500a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient(hunting): picked mon.noname-b con 0x7f68500a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6850142920 0x7f685014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient(hunting): picked mon.noname-a con 0x7f6850142920 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68500b25c0 con 0x7f68500a96f0 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68500aaea0 con 0x7f6850142920 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685013fbe0 con 0x7f68500a9ad0 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.284+0000 7f6858032640 10 monclient(hunting): authenticate will time out at 1610.634973s 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68555cd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68565cf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6850142920 0x7f685014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f6855dce640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f68500a96f0 0x7f685013f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:40648/0 (socket says 172.21.15.110:40648) 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f6855dce640 1 -- 172.21.15.110:0/3924232782 learned_addr learned my addr 172.21.15.110:0/3924232782 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68555cd640 10 monclient(hunting): get_auth_request con 0x7f68500a9ad0 auth_method 0 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68555cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68555cd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68555cd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68565cf640 10 monclient(hunting): get_auth_request con 0x7f6850142920 auth_method 0 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68565cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68565cf640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.285+0000 7f68565cf640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.286+0000 7f68555cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.286+0000 7f68555cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.286+0000 7f68555cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.286+0000 7f68565cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.286+0000 7f68565cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.286+0000 7f68565cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f6854dcc640 1 -- 172.21.15.110:0/3924232782 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870115894 0 0) 0x7f68500b25c0 con 0x7f68500a96f0 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f6854dcc640 10 monclient(hunting): _negotiate 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f6854dcc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f6854dcc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f6854dcc640 10 monclient(hunting): my global_id is 5113 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f6854dcc640 1 -- 172.21.15.110:0/3924232782 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f683c003520 con 0x7f68500a96f0 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f68555cd640 10 monclient(hunting): handle_auth_done global_id 4859 payload 274 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f68555cd640 1 -- 172.21.15.110:0/3924232782 >> v2:172.21.15.110:3300/0 conn(0x7f6850142920 msgr2=0x7f685014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f68555cd640 1 --2- 172.21.15.110:0/3924232782 >> v2:172.21.15.110:3300/0 conn(0x7f6850142920 0x7f685014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f68555cd640 1 -- 172.21.15.110:0/3924232782 >> v1:172.21.15.134:6789/0 conn(0x7f68500a96f0 legacy=0x7f685013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f68555cd640 10 monclient: _finish_hunting 0 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f68555cd640 1 monclient: found mon.noname-b 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f68555cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f68555cd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:27.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.287+0000 7f68555cd640 1 -- 172.21.15.110:0/3924232782 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68500ab0a0 con 0x7f68500a9ad0 2024-01-20T11:19:27.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.289+0000 7f68555cd640 1 --2- 172.21.15.110:0/3924232782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501423e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f68400097a0 tx=0x7f684002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b95f72396d0d3c75 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 1 -- 172.21.15.110:0/3924232782 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684002e070 con 0x7f68500a9ad0 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 10 monclient: dump: 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.294 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 10 monclient: _finish_auth 0 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 1 -- 172.21.15.110:0/3924232782 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68400315a0 con 0x7f68500a9ad0 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6858032640 5 monclient: authenticate success, global_id 4859 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6858032640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6858032640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 1 -- 172.21.15.110:0/3924232782 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840031930 con 0x7f68500a9ad0 2024-01-20T11:19:27.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6858032640 4 set_mon_vals no callback set 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.293+0000 7f6854dcc640 10 monclient: dump: 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 10 monclient: shutdown 2024-01-20T11:19:27.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:27.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 1 -- 172.21.15.110:0/3924232782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 msgr2=0x7f68501423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 1 --2- 172.21.15.110:0/3924232782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501423e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f68400097a0 tx=0x7f684002b840 comp rx=0 tx=0).stop 2024-01-20T11:19:27.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 1 -- 172.21.15.110:0/3924232782 shutdown_connections 2024-01-20T11:19:27.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 1 --2- 172.21.15.110:0/3924232782 >> v2:172.21.15.110:3300/0 conn(0x7f6850142920 0x7f685014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 1 --2- 172.21.15.110:0/3924232782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501423e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 1 -- 172.21.15.110:0/3924232782 >> 172.21.15.110:0/3924232782 conn(0x7f68500b7bc0 msgr2=0x7f68500b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:27.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 1 -- 172.21.15.110:0/3924232782 shutdown_connections 2024-01-20T11:19:27.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 1 -- 172.21.15.110:0/3924232782 wait complete. 2024-01-20T11:19:27.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 10 monclient: build_initial_monmap 2024-01-20T11:19:27.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.294+0000 7f6858032640 10 monclient: monmap: 2024-01-20T11:19:27.300 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:27.300 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 librados: starting msgr at 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 librados: starting objecter 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 Processor -- start 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 -- start start 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 librados: setting wanted keys 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 librados: calling monclient init 2024-01-20T11:19:27.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient: init 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68500a96f0 0x7f68501cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient(hunting): picked mon.noname-a con 0x7f68500a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501d7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient(hunting): picked mon.noname-c con 0x7f68500a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6850142920 0x7f68501d78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient(hunting): picked mon.noname-b con 0x7f6850142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685013a480 con 0x7f68500a96f0 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685013e4b0 con 0x7f6850142920 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685013a910 con 0x7f68500a9ad0 2024-01-20T11:19:27.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.295+0000 7f6858032640 10 monclient(hunting): authenticate will time out at 1610.646102s 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68555cd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501d7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68565cf640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6850142920 0x7f68501d78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f6855dce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68500a96f0 0x7f68501cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68555cd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501d7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50724/0 (socket says 172.21.15.110:50724) 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68555cd640 1 -- 172.21.15.110:0/1793587909 learned_addr learned my addr 172.21.15.110:0/1793587909 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68555cd640 10 monclient(hunting): get_auth_request con 0x7f68500a9ad0 auth_method 0 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68555cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68555cd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68555cd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68565cf640 10 monclient(hunting): get_auth_request con 0x7f6850142920 auth_method 0 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68565cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68565cf640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f68565cf640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f6855dce640 10 monclient(hunting): get_auth_request con 0x7f68500a96f0 auth_method 0 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f6855dce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f6855dce640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.296+0000 7f6855dce640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.297+0000 7f68555cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.297+0000 7f68555cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.297+0000 7f68555cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.297+0000 7f6855dce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.297+0000 7f6855dce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.297+0000 7f6855dce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.297+0000 7f68565cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.297+0000 7f68565cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.297+0000 7f68565cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.299+0000 7f68555cd640 10 monclient(hunting): handle_auth_done global_id 4862 payload 824 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.299+0000 7f68555cd640 1 -- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6850142920 msgr2=0x7f68501d78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.299+0000 7f68555cd640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6850142920 0x7f68501d78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.299+0000 7f68555cd640 1 -- 172.21.15.110:0/1793587909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68500a96f0 msgr2=0x7f68501cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.299+0000 7f68555cd640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68500a96f0 0x7f68501cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.300+0000 7f68555cd640 10 monclient: _finish_hunting 0 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.300+0000 7f68555cd640 1 monclient: found mon.noname-c 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.300+0000 7f68555cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.300+0000 7f68555cd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.300+0000 7f68555cd640 1 -- 172.21.15.110:0/1793587909 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6840009400 con 0x7f68500a9ad0 2024-01-20T11:19:27.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.300+0000 7f68565cf640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6850142920 0x7f68501d78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:27.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.302+0000 7f68555cd640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501d7370 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6840034040 tx=0x7f68400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:27.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.302+0000 7f68467fc640 10 client.?.objecter ms_handle_connect 0x7f68500a9ad0 2024-01-20T11:19:27.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.302+0000 7f68467fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:27.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 1 -- 172.21.15.110:0/1793587909 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840032630 con 0x7f68500a9ad0 2024-01-20T11:19:27.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 10 monclient: dump: 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6858032640 5 monclient: authenticate success, global_id 4862 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 1 -- 172.21.15.110:0/1793587909 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6840032790 con 0x7f68500a9ad0 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 1 -- 172.21.15.110:0/1793587909 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840032b40 con 0x7f68500a9ad0 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6858032640 10 monclient: _renew_subs 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6858032640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685013aad0 con 0x7f68500a9ad0 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6847fff640 4 set_mon_vals no callback set 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f68467fc640 10 monclient: dump: 2024-01-20T11:19:27.309 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:27.310 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.310 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.310 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.310 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:27.311 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.311 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:27.311 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:27.311 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:27.312 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6858032640 10 client.4862.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:27.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6858032640 10 monclient: _renew_subs 2024-01-20T11:19:27.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6858032640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:27.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685013c3c0 con 0x7f68500a9ad0 2024-01-20T11:19:27.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.306+0000 7f6858032640 1 librados: init done 2024-01-20T11:19:27.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.307+0000 7f6858032640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:27.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.307+0000 7f6858032640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:27.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.307+0000 7f6858032640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:27.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.307+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68500aa4a0 con 0x7f68500a9ad0 2024-01-20T11:19:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.546+0000 7f68467fc640 1 -- 172.21.15.110:0/1793587909 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f684004d3f0 con 0x7f68500a9ad0 2024-01-20T11:19:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.546+0000 7f68467fc640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683406c8b0 0x7f683406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.546+0000 7f6855dce640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683406c8b0 0x7f683406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.546+0000 7f6855dce640 10 monclient: get_auth_request con 0x7f683406c8b0 auth_method 0 2024-01-20T11:19:27.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.547+0000 7f6855dce640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683406c8b0 0x7f683406ed70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6848005880 tx=0x7f6848009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:27.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.547+0000 7f68467fc640 10 client.4862.objecter ms_handle_connect 0x7f683406c8b0 2024-01-20T11:19:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.558+0000 7f68467fc640 1 -- 172.21.15.110:0/1793587909 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f684007a020 con 0x7f68500a9ad0 2024-01-20T11:19:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.558+0000 7f68467fc640 10 client.4862.objecter ms_dispatch 0x7f6850000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.558+0000 7f68467fc640 3 client.4862.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.558+0000 7f68467fc640 3 client.4862.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.558+0000 7f68467fc640 20 client.4862.objecter dump_active .. 0 homeless 2024-01-20T11:19:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.558+0000 7f68467fc640 1 -- 172.21.15.110:0/1793587909 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6840076360 con 0x7f68500a9ad0 2024-01-20T11:19:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.558+0000 7f68467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.558+0000 7f68467fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.724+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68500b4400 con 0x7f683406c8b0 2024-01-20T11:19:27.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.727+0000 7f68467fc640 1 -- 172.21.15.110:0/1793587909 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64013 (secure 0 0 0) 0x7f6848001580 con 0x7f683406c8b0 2024-01-20T11:19:27.727 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:27.728 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 10 librados: watch_flush enter 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 10 librados: watch_flush exit 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 20 client.4862.objecter shutdown clearing up homeless session... 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 10 client.4862.objecter successfully canceled tick 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683406c8b0 msgr2=0x7f683406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683406c8b0 0x7f683406ed70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6848005880 tx=0x7f6848009040 comp rx=0 tx=0).stop 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 10 monclient: shutdown 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 msgr2=0x7f68501d7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501d7370 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6840034040 tx=0x7f68400046c0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 shutdown_connections 2024-01-20T11:19:27.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683406c8b0 0x7f683406ed70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6850142920 0x7f68501d78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68500a9ad0 0x7f68501d7370 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 --2- 172.21.15.110:0/1793587909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68500a96f0 0x7f68501cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 >> 172.21.15.110:0/1793587909 conn(0x7f68500b7bc0 msgr2=0x7f685013c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:27.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 shutdown_connections 2024-01-20T11:19:27.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 -- 172.21.15.110:0/1793587909 wait complete. 2024-01-20T11:19:27.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.729+0000 7f6858032640 1 librados: shutdown 2024-01-20T11:19:27.739 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-01-20T11:19:26.758730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.3460000000000001},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":4.1929999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.2530000000000001},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":14.574999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.169},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.108000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.6689999999999996},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":14.962}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.413},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":17.018999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":6.931},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":11.698}]}]},{"osd":7,"up_from":21,"seq":90194313271,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.1879999999999997},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.7169999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":13.07},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.991}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.372999999999999},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.5860000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4450000000000003},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.747}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.484},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":15.105}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.353},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.736000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313272,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.3490000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.5589999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9250000000000007},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.4459999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":11.864000000000001},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.624000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":15.965999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.464}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":13.189},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":15.218999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":16.896000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.398}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444085,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.8550000000000004},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":8.968}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.6449999999999996},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":7.3540000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.275,"5min":15.701000000000001,"15min":17.077000000000002},"min":{"1min":8.0709999999999997,"5min":8.0709999999999997,"15min":8.0709999999999997},"max":{"1min":16.021000000000001,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.987},{"interface":"front","average":{"1min":10.221,"5min":13.297000000000001,"15min":14.459},"min":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":17.103999999999999,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.262}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":11.638999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":12.912000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.25},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.547000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:25 2024","interfaces":[{"interface":"back","average":{"1min":12.211,"5min":13.241,"15min":13.641},"min":{"1min":8.7230000000000008,"5min":7.8680000000000003,"15min":7.8680000000000003},"max":{"1min":18.609000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.301},{"interface":"front","average":{"1min":14.167999999999999,"5min":16.847999999999999,"15min":17.675999999999998},"min":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.75,"5min":28.148,"15min":28.148},"last":12.228999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444086,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.7549999999999999},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":8.8179999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.919},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.5519999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":13.494},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.9729999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":12.56},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.829000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.4619999999999997},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":14.988}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.664},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.798}]}]},{"osd":0,"up_from":17,"seq":73014444086,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.6870000000000003,"5min":12.638,"15min":14.465},"min":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":12.894,"5min":114.843,"15min":114.843},"last":11.606},{"interface":"front","average":{"1min":8.8140000000000001,"5min":13.358000000000001,"15min":15.33},"min":{"1min":4.4400000000000004,"5min":4.4400000000000004,"15min":4.4400000000000004},"max":{"1min":15.550000000000001,"5min":121.044,"15min":121.044},"last":7.5140000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":7.5780000000000003,"5min":22.251999999999999,"15min":28.756},"min":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":5.2539999999999996},"max":{"1min":9.9039999999999999,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.4670000000000005},{"interface":"front","average":{"1min":11.042999999999999,"5min":24.558,"15min":30.521999999999998},"min":{"1min":6.1849999999999996,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":15.244999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":7.9790000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":14.654},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.923999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.509},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.8109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:19 2024","interfaces":[{"interface":"back","average":{"1min":14.231,"5min":27.280000000000001,"15min":33.189},"min":{"1min":9.6530000000000005,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.344000000000001,"5min":375.733,"15min":375.733},"last":15.853999999999999},{"interface":"front","average":{"1min":14.351000000000001,"5min":25.635999999999999,"15min":30.870999999999999},"min":{"1min":10.836,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":19.553000000000001,"5min":374.654,"15min":374.654},"last":16.57}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":17.663},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":10.052}]}]},{"osd":6,"up_from":15,"seq":64424509495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.7919999999999998},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.6769999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.143000000000001},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":11.101000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.452},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.831}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.534000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.986000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.994999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":18.061}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":17.408000000000001},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":19.515000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:27.743 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:27.743 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:27.743 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:27.743 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:27.743 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:27.743 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.859625+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:16:52.859625+0000', 'last_peered': '2024-01-20T11:16:52.859625+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:16:52.859625+0000', 'last_undegraded': '2024-01-20T11:16:52.859625+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:27.743 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:27.743 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:27.744 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.840+0000 7f26caf1d640 10 monclient: get_monmap_and_config 2024-01-20T11:19:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.840+0000 7f26caf1d640 10 monclient: build_initial_monmap 2024-01-20T11:19:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.840+0000 7f26caf1d640 10 monclient: monmap: 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:27.841155+0000 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:27.841155+0000 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:27.841 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 1 Processor -- start 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 1 -- start start 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient: init 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26c40b0ba0 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f26c40adc30 0x7f26c40ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f26c40adc30 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f26c40ac4d0 0x7f26c413bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f26c40ac4d0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c40ad530 con 0x7f26c40b0ba0 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c40af840 con 0x7f26c40adc30 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c40b0f80 con 0x7f26c40ac4d0 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.841+0000 7f26caf1d640 10 monclient(hunting): authenticate will time out at 1611.192066s 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.842+0000 7f26c3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f26c40adc30 0x7f26c40ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.842+0000 7f26c94ba640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f26c40ac4d0 0x7f26c413bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c8cb9640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f26c40b0ba0 0x7f26c40af130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33940/0 (socket says 172.21.15.110:33940) 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c8cb9640 1 -- 172.21.15.110:0/3000963134 learned_addr learned my addr 172.21.15.110:0/3000963134 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c3fff640 10 monclient(hunting): get_auth_request con 0x7f26c40adc30 auth_method 0 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c94ba640 10 monclient(hunting): get_auth_request con 0x7f26c40ac4d0 auth_method 0 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c94ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c94ba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c94ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c94ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c94ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.843+0000 7f26c94ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.844+0000 7f26c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.844+0000 7f26c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.844+0000 7f26c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.845+0000 7f26c94ba640 10 monclient(hunting): handle_auth_done global_id 5119 payload 274 2024-01-20T11:19:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.845+0000 7f26c94ba640 1 -- 172.21.15.110:0/3000963134 >> v2:172.21.15.110:3300/0 conn(0x7f26c40adc30 msgr2=0x7f26c40ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.845+0000 7f26c94ba640 1 --2- 172.21.15.110:0/3000963134 >> v2:172.21.15.110:3300/0 conn(0x7f26c40adc30 0x7f26c40ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.845+0000 7f26c94ba640 1 -- 172.21.15.110:0/3000963134 >> v1:172.21.15.110:6789/0 conn(0x7f26c40b0ba0 legacy=0x7f26c40af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.845+0000 7f26c94ba640 10 monclient: _finish_hunting 0 2024-01-20T11:19:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.845+0000 7f26c94ba640 1 monclient: found mon.noname-c 2024-01-20T11:19:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.845+0000 7f26c94ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.845+0000 7f26c94ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:19:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.845+0000 7f26c94ba640 1 -- 172.21.15.110:0/3000963134 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c40ad810 con 0x7f26c40ac4d0 2024-01-20T11:19:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.847+0000 7f26c94ba640 1 --2- 172.21.15.110:0/3000963134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40ac4d0 0x7f26c413bf30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f26c40ac980 tx=0x7f26bc02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f855bb7ca7bcfa00 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.849+0000 7f26c37fe640 1 -- 172.21.15.110:0/3000963134 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc02cbf0 con 0x7f26c40ac4d0 2024-01-20T11:19:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.849+0000 7f26c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.849+0000 7f26c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.849+0000 7f26c37fe640 10 monclient: dump: 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.849+0000 7f26c37fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.849+0000 7f26c37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.849+0000 7f26c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.849+0000 7f26caf1d640 5 monclient: authenticate success, global_id 5119 2024-01-20T11:19:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.849+0000 7f26caf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.851+0000 7f26c37fe640 1 -- 172.21.15.110:0/3000963134 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26bc02cd50 con 0x7f26c40ac4d0 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.851+0000 7f26c37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.851+0000 7f26c37fe640 1 -- 172.21.15.110:0/3000963134 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc034660 con 0x7f26c40ac4d0 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.851+0000 7f26caf1d640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.851+0000 7f26caf1d640 4 set_mon_vals no callback set 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.851+0000 7f26c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.851+0000 7f26c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.851+0000 7f26c37fe640 10 monclient: dump: 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 10 monclient: shutdown 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 1 -- 172.21.15.110:0/3000963134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40ac4d0 msgr2=0x7f26c413bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 1 --2- 172.21.15.110:0/3000963134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40ac4d0 0x7f26c413bf30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f26c40ac980 tx=0x7f26bc02be40 comp rx=0 tx=0).stop 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 1 -- 172.21.15.110:0/3000963134 shutdown_connections 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 1 --2- 172.21.15.110:0/3000963134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40ac4d0 0x7f26c413bf30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 1 --2- 172.21.15.110:0/3000963134 >> v2:172.21.15.110:3300/0 conn(0x7f26c40adc30 0x7f26c40ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 1 -- 172.21.15.110:0/3000963134 >> 172.21.15.110:0/3000963134 conn(0x7f26c40b3e60 msgr2=0x7f26c40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 1 -- 172.21.15.110:0/3000963134 shutdown_connections 2024-01-20T11:19:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 1 -- 172.21.15.110:0/3000963134 wait complete. 2024-01-20T11:19:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 10 monclient: build_initial_monmap 2024-01-20T11:19:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.852+0000 7f26caf1d640 10 monclient: monmap: 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 librados: starting msgr at 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 librados: starting objecter 2024-01-20T11:19:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 Processor -- start 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 -- start start 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 librados: setting wanted keys 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 librados: calling monclient init 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient: init 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26c40ac4d0 0x7f26c4131ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f26c40ac4d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40b0ba0 0x7f26c4140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f26c40b0ba0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26c40b26f0 0x7f26c41417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f26c40b26f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c4131370 con 0x7f26c40ac4d0 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c40b2b80 con 0x7f26c40b0ba0 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c4132c60 con 0x7f26c40b26f0 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.853+0000 7f26caf1d640 10 monclient(hunting): authenticate will time out at 1611.204055s 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c94ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26c40b26f0 0x7f26c41417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c3fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40b0ba0 0x7f26c4140e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c8cb9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26c40ac4d0 0x7f26c4131ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c3fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40b0ba0 0x7f26c4140e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:45570/0 (socket says 172.21.15.110:45570) 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c3fff640 1 -- 172.21.15.110:0/393015639 learned_addr learned my addr 172.21.15.110:0/393015639 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c3fff640 10 monclient(hunting): get_auth_request con 0x7f26c40b0ba0 auth_method 0 2024-01-20T11:19:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c94ba640 10 monclient(hunting): get_auth_request con 0x7f26c40b26f0 auth_method 0 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c94ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c94ba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c94ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c8cb9640 10 monclient(hunting): get_auth_request con 0x7f26c40ac4d0 auth_method 0 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c8cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c8cb9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.854+0000 7f26c8cb9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.855+0000 7f26c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.855+0000 7f26c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.855+0000 7f26c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.855+0000 7f26c8cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.855+0000 7f26c8cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.855+0000 7f26c8cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.855+0000 7f26c94ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.855+0000 7f26c94ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.855+0000 7f26c94ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 10 monclient(hunting): handle_auth_done global_id 5122 payload 824 2024-01-20T11:19:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 1 -- 172.21.15.110:0/393015639 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26c40b26f0 msgr2=0x7f26c41417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26c40b26f0 0x7f26c41417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 1 -- 172.21.15.110:0/393015639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26c40ac4d0 msgr2=0x7f26c4131ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26c40ac4d0 0x7f26c4131ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 1 monclient: found mon.noname-b 2024-01-20T11:19:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c3fff640 1 -- 172.21.15.110:0/393015639 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc009a70 con 0x7f26c40b0ba0 2024-01-20T11:19:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.858+0000 7f26c8cb9640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26c40ac4d0 0x7f26c4131ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.860+0000 7f26c3fff640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40b0ba0 0x7f26c4140e40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f26b000e9e0 tx=0x7f26b000eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.860+0000 7f26c17fa640 10 client.?.objecter ms_handle_connect 0x7f26c40b0ba0 2024-01-20T11:19:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.860+0000 7f26c17fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:27.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 1 -- 172.21.15.110:0/393015639 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b0009670 con 0x7f26c40b0ba0 2024-01-20T11:19:27.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 10 monclient: dump: 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 10 monclient: _finish_auth 0 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 10 monclient: _check_auth_tickets 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 1 -- 172.21.15.110:0/393015639 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b000c9d0 con 0x7f26c40b0ba0 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26caf1d640 5 monclient: authenticate success, global_id 5122 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26caf1d640 10 monclient: _renew_subs 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26caf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c4131570 con 0x7f26c40b0ba0 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 1 -- 172.21.15.110:0/393015639 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b000cd80 con 0x7f26c40b0ba0 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c17fa640 10 monclient: dump: 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26c2ffd640 4 set_mon_vals no callback set 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26caf1d640 10 client.5122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26caf1d640 10 monclient: _renew_subs 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26caf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c4131900 con 0x7f26c40b0ba0 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.864+0000 7f26caf1d640 1 librados: init done 2024-01-20T11:19:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.865+0000 7f26caf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:27.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.865+0000 7f26caf1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:27.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.865+0000 7f26caf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:27.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.865+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c4141fc0 con 0x7f26c40b0ba0 2024-01-20T11:19:27.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.883+0000 7f26c17fa640 1 -- 172.21.15.110:0/393015639 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f26b0013470 con 0x7f26c40b0ba0 2024-01-20T11:19:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.884+0000 7f26c17fa640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f26a806c8e0 0x7f26a806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.884+0000 7f26c8cb9640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f26a806c8e0 0x7f26a806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.884+0000 7f26c8cb9640 10 monclient: get_auth_request con 0x7f26a806c8e0 auth_method 0 2024-01-20T11:19:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.884+0000 7f26c8cb9640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f26a806c8e0 0x7f26a806eda0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f26b8007fd0 tx=0x7f26b8007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.884+0000 7f26c17fa640 10 client.5122.objecter ms_handle_connect 0x7f26a806c8e0 2024-01-20T11:19:27.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.887+0000 7f26c17fa640 1 -- 172.21.15.110:0/393015639 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f26b0058b90 con 0x7f26c40b0ba0 2024-01-20T11:19:27.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.887+0000 7f26c17fa640 10 client.5122.objecter ms_dispatch 0x7f26c4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:27.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.887+0000 7f26c17fa640 3 client.5122.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:27.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.887+0000 7f26c17fa640 3 client.5122.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:27.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:27.887+0000 7f26c17fa640 20 client.5122.objecter dump_active .. 0 homeless 2024-01-20T11:19:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.119+0000 7f26c17fa640 1 -- 172.21.15.110:0/393015639 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f26b0054ed0 con 0x7f26c40b0ba0 2024-01-20T11:19:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.119+0000 7f26c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.119+0000 7f26c17fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:28.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.391+0000 7f26caf1d640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:28.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.391+0000 7f26caf1d640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:28.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.391+0000 7f26caf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:28.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.391+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f26c40b2cc0 con 0x7f26c40b0ba0 2024-01-20T11:19:28.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.397+0000 7f26c17fa640 1 -- 172.21.15.110:0/393015639 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f26b0021520 con 0x7f26c40b0ba0 2024-01-20T11:19:28.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.397+0000 7f26c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:28.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.397+0000 7f26c17fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:28.397 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:28.397 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":302,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:28.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.398+0000 7f26caf1d640 10 librados: watch_flush enter 2024-01-20T11:19:28.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.398+0000 7f26caf1d640 10 librados: watch_flush exit 2024-01-20T11:19:28.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.398+0000 7f26caf1d640 20 client.5122.objecter shutdown clearing up homeless session... 2024-01-20T11:19:28.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.398+0000 7f26caf1d640 10 client.5122.objecter successfully canceled tick 2024-01-20T11:19:28.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f26a806c8e0 msgr2=0x7f26a806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f26a806c8e0 0x7f26a806eda0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f26b8007fd0 tx=0x7f26b8007e80 comp rx=0 tx=0).stop 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 10 monclient: shutdown 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40b0ba0 msgr2=0x7f26c4140e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40b0ba0 0x7f26c4140e40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f26b000e9e0 tx=0x7f26b000eea0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 shutdown_connections 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f26a806c8e0 0x7f26a806eda0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26c40b26f0 0x7f26c41417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26c40b0ba0 0x7f26c4140e40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 --2- 172.21.15.110:0/393015639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26c40ac4d0 0x7f26c4131ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 >> 172.21.15.110:0/393015639 conn(0x7f26c40b3e60 msgr2=0x7f26c40a9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 shutdown_connections 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 -- 172.21.15.110:0/393015639 wait complete. 2024-01-20T11:19:28.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.399+0000 7f26caf1d640 1 librados: shutdown 2024-01-20T11:19:28.410 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:28.410 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:28.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.513+0000 7f9dce333640 10 monclient: get_monmap_and_config 2024-01-20T11:19:28.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.513+0000 7f9dce333640 10 monclient: build_initial_monmap 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.513+0000 7f9dce333640 10 monclient: monmap: 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:28.514437+0000 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:28.514437+0000 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:28.514 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:28.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.514+0000 7f9dce333640 1 Processor -- start 2024-01-20T11:19:28.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.514+0000 7f9dce333640 1 -- start start 2024-01-20T11:19:28.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.514+0000 7f9dce333640 10 monclient: init 2024-01-20T11:19:28.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:28.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-01-20T11:19:28.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9dc806bb00 0x7f9dc80b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:28.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc806bb00 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:28.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9dc80af790 0x7f9dc8145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:28.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc80af790 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9dc80b0900 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc806c220 con 0x7f9dc80b0900 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc806bee0 con 0x7f9dc80af790 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc80ada30 con 0x7f9dc806bb00 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dce333640 10 monclient(hunting): authenticate will time out at 1611.865416s 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dc77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9dc80af790 0x7f9dc8145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.515+0000 7f9dc7fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9dc806bb00 0x7f9dc80b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9dc80af790 0x7f9dc8145770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51486/0 (socket says 172.21.15.110:51486) 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc77fe640 1 -- 172.21.15.110:0/263704188 learned_addr learned my addr 172.21.15.110:0/263704188 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc77fe640 10 monclient(hunting): get_auth_request con 0x7f9dc80af790 auth_method 0 2024-01-20T11:19:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc7fff640 10 monclient(hunting): get_auth_request con 0x7f9dc806bb00 auth_method 0 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.516+0000 7f9dc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.517+0000 7f9dc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.517+0000 7f9dc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.517+0000 7f9dc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.518+0000 7f9dc77fe640 10 monclient(hunting): handle_auth_done global_id 5049 payload 274 2024-01-20T11:19:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.518+0000 7f9dc77fe640 1 -- 172.21.15.110:0/263704188 >> v2:172.21.15.134:3300/0 conn(0x7f9dc806bb00 msgr2=0x7f9dc80b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.518+0000 7f9dc77fe640 1 --2- 172.21.15.110:0/263704188 >> v2:172.21.15.134:3300/0 conn(0x7f9dc806bb00 0x7f9dc80b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.518+0000 7f9dc77fe640 1 -- 172.21.15.110:0/263704188 >> v1:172.21.15.134:6789/0 conn(0x7f9dc80b0900 legacy=0x7f9dc8145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.518+0000 7f9dc77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.518+0000 7f9dc77fe640 1 monclient: found mon.noname-a 2024-01-20T11:19:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.518+0000 7f9dc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.518+0000 7f9dc77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.518+0000 7f9dc77fe640 1 -- 172.21.15.110:0/263704188 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc80add10 con 0x7f9dc80af790 2024-01-20T11:19:28.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.520+0000 7f9dc77fe640 1 --2- 172.21.15.110:0/263704188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc80af790 0x7f9dc8145770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80b6630 tx=0x7f9dbc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eae29efbdd28bc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:28.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 1 -- 172.21.15.110:0/263704188 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc004230 con 0x7f9dc80af790 2024-01-20T11:19:28.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:28.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:28.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 10 monclient: dump: 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:28.526 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 1 -- 172.21.15.110:0/263704188 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc004390 con 0x7f9dc80af790 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 5 monclient: authenticate success, global_id 5049 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 1 -- 172.21.15.110:0/263704188 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc003810 con 0x7f9dc80af790 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 4 set_mon_vals no callback set 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dc6ffd640 10 monclient: dump: 2024-01-20T11:19:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 10 monclient: shutdown 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 1 -- 172.21.15.110:0/263704188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc80af790 msgr2=0x7f9dc8145770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 1 --2- 172.21.15.110:0/263704188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc80af790 0x7f9dc8145770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80b6630 tx=0x7f9dbc02b8f0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 1 -- 172.21.15.110:0/263704188 shutdown_connections 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 1 --2- 172.21.15.110:0/263704188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc80af790 0x7f9dc8145770 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 1 --2- 172.21.15.110:0/263704188 >> v2:172.21.15.134:3300/0 conn(0x7f9dc806bb00 0x7f9dc80b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 1 -- 172.21.15.110:0/263704188 >> 172.21.15.110:0/263704188 conn(0x7f9dc8066d60 msgr2=0x7f9dc80b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 1 -- 172.21.15.110:0/263704188 shutdown_connections 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.524+0000 7f9dce333640 1 -- 172.21.15.110:0/263704188 wait complete. 2024-01-20T11:19:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.525+0000 7f9dce333640 10 monclient: build_initial_monmap 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.525+0000 7f9dce333640 10 monclient: monmap: 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:28.530 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.525+0000 7f9dce333640 1 librados: starting msgr at 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.525+0000 7f9dce333640 1 librados: starting objecter 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 Processor -- start 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 -- start start 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 librados: setting wanted keys 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 librados: calling monclient init 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient: init 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:19:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9dc806bb00 0x7f9dc811fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc80af790 0x7f9dc8120890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc80af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc80b0900 0x7f9dc81dab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient(hunting): picked mon.noname-b con 0x7f9dc80b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc81da6e0 con 0x7f9dc80af790 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc81203c0 con 0x7f9dc80b0900 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc811d5a0 con 0x7f9dc806bb00 2024-01-20T11:19:28.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.526+0000 7f9dce333640 10 monclient(hunting): authenticate will time out at 1611.877002s 2024-01-20T11:19:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9dc806bb00 0x7f9dc811fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc80af790 0x7f9dc8120890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dcc8d0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc80b0900 0x7f9dc81dab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9dc806bb00 0x7f9dc811fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50744/0 (socket says 172.21.15.110:50744) 2024-01-20T11:19:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc7fff640 1 -- 172.21.15.110:0/1437192336 learned_addr learned my addr 172.21.15.110:0/1437192336 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc7fff640 10 monclient(hunting): get_auth_request con 0x7f9dc806bb00 auth_method 0 2024-01-20T11:19:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dcc8d0640 10 monclient(hunting): get_auth_request con 0x7f9dc80b0900 auth_method 0 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dcc8d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dcc8d0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dcc8d0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc77fe640 10 monclient(hunting): get_auth_request con 0x7f9dc80af790 auth_method 0 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.527+0000 7f9dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.528+0000 7f9dcc8d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.528+0000 7f9dcc8d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.528+0000 7f9dcc8d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.528+0000 7f9dc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.528+0000 7f9dc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:28.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.528+0000 7f9dc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.528+0000 7f9dc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.528+0000 7f9dc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.528+0000 7f9dc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.531+0000 7f9dcc8d0640 10 monclient(hunting): handle_auth_done global_id 5128 payload 824 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.532+0000 7f9dcc8d0640 1 -- 172.21.15.110:0/1437192336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9dc806bb00 msgr2=0x7f9dc811fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.532+0000 7f9dcc8d0640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9dc806bb00 0x7f9dc811fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.532+0000 7f9dcc8d0640 1 -- 172.21.15.110:0/1437192336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc80af790 msgr2=0x7f9dc8120890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.532+0000 7f9dcc8d0640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc80af790 0x7f9dc8120890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.532+0000 7f9dcc8d0640 10 monclient: _finish_hunting 0 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.532+0000 7f9dcc8d0640 1 monclient: found mon.noname-b 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.532+0000 7f9dcc8d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.532+0000 7f9dcc8d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.532+0000 7f9dcc8d0640 1 -- 172.21.15.110:0/1437192336 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc009460 con 0x7f9dc80b0900 2024-01-20T11:19:28.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.534+0000 7f9dcc8d0640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc80b0900 0x7f9dc81dab70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000c700 tx=0x7f9dc000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:28.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.534+0000 7f9dc4ff9640 10 client.?.objecter ms_handle_connect 0x7f9dc80b0900 2024-01-20T11:19:28.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.534+0000 7f9dc4ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:28.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 1 -- 172.21.15.110:0/1437192336 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0003b70 con 0x7f9dc80b0900 2024-01-20T11:19:28.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:28.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 10 monclient: dump: 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 10 monclient: _finish_auth 0 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:19:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 1 -- 172.21.15.110:0/1437192336 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc0003cd0 con 0x7f9dc80b0900 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 1 -- 172.21.15.110:0/1437192336 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0005310 con 0x7f9dc80b0900 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dce333640 5 monclient: authenticate success, global_id 5128 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dce333640 10 monclient: _renew_subs 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dce333640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc8120dd0 con 0x7f9dc80b0900 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc4ff9640 10 monclient: dump: 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dc67fc640 4 set_mon_vals no callback set 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dce333640 10 client.5128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dce333640 10 monclient: _renew_subs 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dce333640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc81db290 con 0x7f9dc80b0900 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.538+0000 7f9dce333640 1 librados: init done 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.539+0000 7f9dce333640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.539+0000 7f9dce333640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.539+0000 7f9dce333640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:28.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.539+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc8075490 con 0x7f9dc80b0900 2024-01-20T11:19:28.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.556+0000 7f9dc4ff9640 1 -- 172.21.15.110:0/1437192336 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9dc0005470 con 0x7f9dc80b0900 2024-01-20T11:19:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.557+0000 7f9dc4ff9640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9db406c860 0x7f9db406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.557+0000 7f9dc7fff640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9db406c860 0x7f9db406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.557+0000 7f9dc7fff640 10 monclient: get_auth_request con 0x7f9db406c860 auth_method 0 2024-01-20T11:19:28.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.557+0000 7f9dc7fff640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9db406c860 0x7f9db406ed20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009c00 tx=0x7f9db8009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:28.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.557+0000 7f9dc4ff9640 10 client.5128.objecter ms_handle_connect 0x7f9db406c860 2024-01-20T11:19:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.783+0000 7f9dc4ff9640 1 -- 172.21.15.110:0/1437192336 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9dc0058900 con 0x7f9dc80b0900 2024-01-20T11:19:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.783+0000 7f9dc4ff9640 10 client.5128.objecter ms_dispatch 0x7f9dc8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.783+0000 7f9dc4ff9640 3 client.5128.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.783+0000 7f9dc4ff9640 3 client.5128.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:28.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.783+0000 7f9dc4ff9640 20 client.5128.objecter dump_active .. 0 homeless 2024-01-20T11:19:28.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.793+0000 7f9dc4ff9640 1 -- 172.21.15.110:0/1437192336 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9dc0054c40 con 0x7f9dc80b0900 2024-01-20T11:19:28.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.793+0000 7f9dc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:28.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.793+0000 7f9dc4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:28.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.986+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9dc811d760 con 0x7f9db406c860 2024-01-20T11:19:28.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.989+0000 7f9dc4ff9640 1 -- 172.21.15.110:0/1437192336 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63977 (secure 0 0 0) 0x7f9dc811d760 con 0x7f9db406c860 2024-01-20T11:19:28.989 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:28.989 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:28.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 10 librados: watch_flush enter 2024-01-20T11:19:28.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 10 librados: watch_flush exit 2024-01-20T11:19:28.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 20 client.5128.objecter shutdown clearing up homeless session... 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 10 client.5128.objecter successfully canceled tick 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9db406c860 msgr2=0x7f9db406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9db406c860 0x7f9db406ed20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009c00 tx=0x7f9db8009190 comp rx=0 tx=0).stop 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 10 monclient: shutdown 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc80b0900 msgr2=0x7f9dc81dab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc80b0900 0x7f9dc81dab70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000c700 tx=0x7f9dc000cbc0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 shutdown_connections 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9db406c860 0x7f9db406ed20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc80b0900 0x7f9dc81dab70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc80af790 0x7f9dc8120890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 --2- 172.21.15.110:0/1437192336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9dc806bb00 0x7f9dc811fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:28.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 >> 172.21.15.110:0/1437192336 conn(0x7f9dc8066d60 msgr2=0x7f9dc8144f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:28.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 shutdown_connections 2024-01-20T11:19:28.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 -- 172.21.15.110:0/1437192336 wait complete. 2024-01-20T11:19:28.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:28.991+0000 7f9dce333640 1 librados: shutdown 2024-01-20T11:19:29.001 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-01-20T11:19:28.759073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002765"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.2869999999999999},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":12.744999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":5.0830000000000002},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":13.358000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.4480000000000004},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":8.8789999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":14.647},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":8.6010000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":10.131},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":12.077}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":15.398},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":10.699}]}]},{"osd":7,"up_from":21,"seq":90194313271,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.1879999999999997},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.7169999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":13.07},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.991}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.372999999999999},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.5860000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4450000000000003},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.747}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.484},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":15.105}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.353},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.736000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313272,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.3490000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.5589999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9250000000000007},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.4459999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":11.864000000000001},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.624000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":15.965999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.464}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":13.189},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":15.218999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":16.896000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.398}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444086,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.025},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":8.3620000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.297000000000001},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.616}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.771000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.221}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":17.094999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.388000000000002},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7709999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.5779999999999994},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":18.126000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444087,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.5720000000000001},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.2560000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.335000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.503}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.913},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.032}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":9.4209999999999994},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":9.6950000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.544},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":13.023999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.380000000000001},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.619}]}]},{"osd":0,"up_from":17,"seq":73014444087,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":8.8160000000000007},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.417}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.5910000000000002},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.598000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":12.814},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.827}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.548},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.1079999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":12.263},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.65}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.845000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":8.0020000000000007}]}]},{"osd":6,"up_from":15,"seq":64424509495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.7919999999999998},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.6769999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.143000000000001},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":11.101000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.452},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.831}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.534000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.986000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.994999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":18.061}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":17.408000000000001},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":19.515000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:29.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~31s 2024-01-20T11:19:32.004 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:32.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.105+0000 7f1823747640 10 monclient: get_monmap_and_config 2024-01-20T11:19:32.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.105+0000 7f1823747640 10 monclient: build_initial_monmap 2024-01-20T11:19:32.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.105+0000 7f1823747640 10 monclient: monmap: 2024-01-20T11:19:32.105 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:32.105 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.105 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:32.105944+0000 2024-01-20T11:19:32.105 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:32.105944+0000 2024-01-20T11:19:32.105 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:32.105 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.106 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:32.106 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:32.106 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:32.106 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:32.106 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:32.106 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.105+0000 7f1823747640 1 Processor -- start 2024-01-20T11:19:32.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 1 -- start start 2024-01-20T11:19:32.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 10 monclient: init 2024-01-20T11:19:32.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:32.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f181c0b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f181c0b1ed0 0x7f181c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 10 monclient(hunting): picked mon.noname-c con 0x7f181c0b1ed0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f181c145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c0aade0 con 0x7f181c0b4d10 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c145460 con 0x7f181c145980 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c0b50f0 con 0x7f181c0b1ed0 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.106+0000 7f1823747640 10 monclient(hunting): authenticate will time out at 1615.457008s 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.107+0000 7f18214e3640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f181c0b4d10 0x7f181c0b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55510/0 (socket says 172.21.15.110:55510) 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.107+0000 7f18214e3640 1 -- 172.21.15.110:0/4256225171 learned_addr learned my addr 172.21.15.110:0/4256225171 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.107+0000 7f1820ce2640 1 --2- 172.21.15.110:0/4256225171 >> v2:172.21.15.134:3300/0 conn(0x7f181c0b1ed0 0x7f181c0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.108+0000 7f1820ce2640 10 monclient(hunting): get_auth_request con 0x7f181c0b1ed0 auth_method 0 2024-01-20T11:19:32.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.108+0000 7f1820ce2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:32.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.108+0000 7f1820ce2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.108+0000 7f1820ce2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.109+0000 7f1820ce2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:32.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.109+0000 7f1820ce2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:32.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.109+0000 7f1820ce2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:32.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.109+0000 7f1813fff640 1 -- 172.21.15.110:0/4256225171 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519524462 0 0) 0x7f181c0aade0 con 0x7f181c0b4d10 2024-01-20T11:19:32.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.109+0000 7f1813fff640 10 monclient(hunting): _negotiate 2024-01-20T11:19:32.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.109+0000 7f1813fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.109+0000 7f1813fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.109+0000 7f1813fff640 10 monclient(hunting): my global_id is 5055 2024-01-20T11:19:32.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.109+0000 7f1813fff640 1 -- 172.21.15.110:0/4256225171 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1808003520 con 0x7f181c0b4d10 2024-01-20T11:19:32.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.113+0000 7f1820ce2640 10 monclient(hunting): handle_auth_done global_id 5131 payload 274 2024-01-20T11:19:32.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.113+0000 7f1820ce2640 1 -- 172.21.15.110:0/4256225171 >> v1:172.21.15.134:6789/0 conn(0x7f181c145980 legacy=0x7f181c145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.113+0000 7f1820ce2640 1 -- 172.21.15.110:0/4256225171 >> v1:172.21.15.110:6789/0 conn(0x7f181c0b4d10 legacy=0x7f181c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.113+0000 7f1820ce2640 10 monclient: _finish_hunting 0 2024-01-20T11:19:32.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.113+0000 7f1820ce2640 1 monclient: found mon.noname-c 2024-01-20T11:19:32.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.113+0000 7f1820ce2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:32.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.113+0000 7f1820ce2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:19:32.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.113+0000 7f1820ce2640 1 -- 172.21.15.110:0/4256225171 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c0b2660 con 0x7f181c0b1ed0 2024-01-20T11:19:32.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.117+0000 7f1820ce2640 1 --2- 172.21.15.110:0/4256225171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 0x7f181c0b3da0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f180c0097a0 tx=0x7f180c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15e963f1439bd070 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:32.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 1 -- 172.21.15.110:0/4256225171 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c031830 con 0x7f181c0b1ed0 2024-01-20T11:19:32.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:32.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:32.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 10 monclient: dump: 2024-01-20T11:19:32.121 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 10 monclient: _finish_auth 0 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 1 -- 172.21.15.110:0/4256225171 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c031990 con 0x7f181c0b1ed0 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 1 -- 172.21.15.110:0/4256225171 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c031d20 con 0x7f181c0b1ed0 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1813fff640 10 monclient: dump: 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1823747640 5 monclient: authenticate success, global_id 5131 2024-01-20T11:19:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1823747640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:32.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.121+0000 7f1823747640 4 set_mon_vals no callback set 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.122+0000 7f1823747640 10 monclient: shutdown 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.122+0000 7f1823747640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.122+0000 7f1823747640 1 -- 172.21.15.110:0/4256225171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 msgr2=0x7f181c0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.122+0000 7f1823747640 1 --2- 172.21.15.110:0/4256225171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 0x7f181c0b3da0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f180c0097a0 tx=0x7f180c02b8d0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.122+0000 7f1823747640 1 -- 172.21.15.110:0/4256225171 shutdown_connections 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.122+0000 7f1823747640 1 --2- 172.21.15.110:0/4256225171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 0x7f181c0b3da0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.122+0000 7f1823747640 1 -- 172.21.15.110:0/4256225171 >> 172.21.15.110:0/4256225171 conn(0x7f181c066590 msgr2=0x7f181c0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.122+0000 7f1823747640 1 -- 172.21.15.110:0/4256225171 shutdown_connections 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.122+0000 7f1823747640 1 -- 172.21.15.110:0/4256225171 wait complete. 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.123+0000 7f1823747640 10 monclient: build_initial_monmap 2024-01-20T11:19:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.123+0000 7f1823747640 10 monclient: monmap: 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.123+0000 7f1823747640 1 librados: starting msgr at 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.123+0000 7f1823747640 1 librados: starting objecter 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.123+0000 7f1823747640 1 Processor -- start 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.123+0000 7f1823747640 1 -- start start 2024-01-20T11:19:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.123+0000 7f1823747640 1 librados: setting wanted keys 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.123+0000 7f1823747640 1 librados: calling monclient init 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.123+0000 7f1823747640 10 monclient: init 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 0x7f181c13dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient(hunting): picked mon.noname-b con 0x7f181c0b1ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f181c0b4d10 0x7f181c13e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient(hunting): picked mon.noname-c con 0x7f181c0b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c145980 0x7f181c13ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient(hunting): picked mon.noname-a con 0x7f181c145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c13d8d0 con 0x7f181c145980 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0ab880 con 0x7f181c0b1ed0 2024-01-20T11:19:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0abb60 con 0x7f181c0b4d10 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1823747640 10 monclient(hunting): authenticate will time out at 1615.474422s 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1820ce2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f181c0b4d10 0x7f181c13e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.124+0000 7f1821ce4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c145980 0x7f181c13ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f18214e3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 0x7f181c13dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1821ce4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c145980 0x7f181c13ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40562/0 (socket says 172.21.15.110:40562) 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1821ce4640 1 -- 172.21.15.110:0/717228529 learned_addr learned my addr 172.21.15.110:0/717228529 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1821ce4640 10 monclient(hunting): get_auth_request con 0x7f181c145980 auth_method 0 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1821ce4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1821ce4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1821ce4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1820ce2640 10 monclient(hunting): get_auth_request con 0x7f181c0b4d10 auth_method 0 2024-01-20T11:19:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1820ce2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1820ce2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1820ce2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f18214e3640 10 monclient(hunting): get_auth_request con 0x7f181c0b1ed0 auth_method 0 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f18214e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f18214e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f18214e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1821ce4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1821ce4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1821ce4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1820ce2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1820ce2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.125+0000 7f1820ce2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.126+0000 7f18214e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.126+0000 7f18214e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:32.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.126+0000 7f18214e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 10 monclient(hunting): handle_auth_done global_id 5137 payload 824 2024-01-20T11:19:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 1 -- 172.21.15.110:0/717228529 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f181c0b4d10 msgr2=0x7f181c13e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f181c0b4d10 0x7f181c13e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 1 -- 172.21.15.110:0/717228529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c145980 msgr2=0x7f181c13ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c145980 0x7f181c13ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 10 monclient: _finish_hunting 0 2024-01-20T11:19:32.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 1 monclient: found mon.noname-b 2024-01-20T11:19:32.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:32.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:32.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.129+0000 7f18214e3640 1 -- 172.21.15.110:0/717228529 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c009400 con 0x7f181c0b1ed0 2024-01-20T11:19:32.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.131+0000 7f18214e3640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 0x7f181c13dd90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f181400c400 tx=0x7f181400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:32.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.131+0000 7f1811ffb640 10 client.?.objecter ms_handle_connect 0x7f181c0b1ed0 2024-01-20T11:19:32.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.131+0000 7f1811ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 1 -- 172.21.15.110:0/717228529 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814005ce0 con 0x7f181c0b1ed0 2024-01-20T11:19:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 10 monclient: dump: 2024-01-20T11:19:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 10 monclient: _finish_auth 0 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 1 -- 172.21.15.110:0/717228529 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1814005e40 con 0x7f181c0b1ed0 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1823747640 5 monclient: authenticate success, global_id 5137 2024-01-20T11:19:32.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1823747640 10 monclient: _renew_subs 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1823747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c0abe40 con 0x7f181c0b1ed0 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 1 -- 172.21.15.110:0/717228529 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18140072f0 con 0x7f181c0b1ed0 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1811ffb640 10 monclient: dump: 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:32.138 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f18137fe640 4 set_mon_vals no callback set 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1823747640 10 client.5137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1823747640 10 monclient: _renew_subs 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1823747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c13b610 con 0x7f181c0b1ed0 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.135+0000 7f1823747640 1 librados: init done 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.136+0000 7f1823747640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.136+0000 7f1823747640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.136+0000 7f1823747640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.136+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c075490 con 0x7f181c0b1ed0 2024-01-20T11:19:32.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.158+0000 7f1811ffb640 1 -- 172.21.15.110:0/717228529 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1814007450 con 0x7f181c0b1ed0 2024-01-20T11:19:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.159+0000 7f1811ffb640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f180006c8b0 0x7f180006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.159+0000 7f1820ce2640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f180006c8b0 0x7f180006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.159+0000 7f1820ce2640 10 monclient: get_auth_request con 0x7f180006c8b0 auth_method 0 2024-01-20T11:19:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.160+0000 7f1820ce2640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f180006c8b0 0x7f180006ed70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f180c036000 tx=0x7f180c03c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.160+0000 7f1811ffb640 10 client.5137.objecter ms_handle_connect 0x7f180006c8b0 2024-01-20T11:19:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.162+0000 7f1811ffb640 1 -- 172.21.15.110:0/717228529 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1814057f20 con 0x7f181c0b1ed0 2024-01-20T11:19:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.162+0000 7f1811ffb640 10 client.5137.objecter ms_dispatch 0x7f181c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.162+0000 7f1811ffb640 3 client.5137.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.162+0000 7f1811ffb640 3 client.5137.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.162+0000 7f1811ffb640 20 client.5137.objecter dump_active .. 0 homeless 2024-01-20T11:19:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.401+0000 7f1811ffb640 1 -- 172.21.15.110:0/717228529 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1814054260 con 0x7f181c0b1ed0 2024-01-20T11:19:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.401+0000 7f1811ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.401+0000 7f1811ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.576+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f181c0a9cf0 con 0x7f180006c8b0 2024-01-20T11:19:32.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.579+0000 7f1811ffb640 1 -- 172.21.15.110:0/717228529 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63977 (secure 0 0 0) 0x7f180c03a420 con 0x7f180006c8b0 2024-01-20T11:19:32.580 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:32.580 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:32.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 10 librados: watch_flush enter 2024-01-20T11:19:32.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 10 librados: watch_flush exit 2024-01-20T11:19:32.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 20 client.5137.objecter shutdown clearing up homeless session... 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 10 client.5137.objecter successfully canceled tick 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f180006c8b0 msgr2=0x7f180006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f180006c8b0 0x7f180006ed70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f180c036000 tx=0x7f180c03c040 comp rx=0 tx=0).stop 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 10 monclient: shutdown 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 msgr2=0x7f181c13dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 0x7f181c13dd90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f181400c400 tx=0x7f181400c8c0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 shutdown_connections 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f180006c8b0 0x7f180006ed70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c145980 0x7f181c13ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f181c0b4d10 0x7f181c13e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 --2- 172.21.15.110:0/717228529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f181c0b1ed0 0x7f181c13dd90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 >> 172.21.15.110:0/717228529 conn(0x7f181c066590 msgr2=0x7f181c145e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:32.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 shutdown_connections 2024-01-20T11:19:32.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 -- 172.21.15.110:0/717228529 wait complete. 2024-01-20T11:19:32.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.581+0000 7f1823747640 1 librados: shutdown 2024-01-20T11:19:32.591 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-01-20T11:19:30.759470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002785"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.859625+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:16:52.859625+0000","last_peered":"2024-01-20T11:16:52.859625+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:16:52.859625+0000","last_undegraded":"2024-01-20T11:16:52.859625+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.2869999999999999},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":12.744999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":5.0830000000000002},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":13.358000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.4480000000000004},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":8.8789999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":14.647},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":8.6010000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":10.131},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":12.077}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":15.398},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":10.699}]}]},{"osd":7,"up_from":21,"seq":90194313271,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0980000000000008,"5min":9.4740000000000002,"15min":10.08},"min":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":15.970000000000001,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.1879999999999997},{"interface":"front","average":{"1min":8.7330000000000005,"5min":12.211,"15min":13.443},"min":{"1min":5.7279999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":12.315,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.7169999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":9.3049999999999997,"5min":10.089,"15min":10.464},"min":{"1min":6.0039999999999996,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":13.026,"5min":18.859000000000002,"15min":18.859000000000002},"last":13.07},{"interface":"front","average":{"1min":10.949999999999999,"5min":12.569000000000001,"15min":13.186},"min":{"1min":7.298,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.030999999999999,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.991}]},{"osd":2,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":12.404,"5min":15.680999999999999,"15min":16.890999999999998},"min":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.108000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.372999999999999},{"interface":"front","average":{"1min":11.737,"5min":15.064,"15min":16.462},"min":{"1min":6.2030000000000003,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":18.869,"5min":78.548000000000002,"15min":78.548000000000002},"last":9.5860000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":11.699999999999999,"5min":14.176,"15min":15.036},"min":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":18.474,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4450000000000003},{"interface":"front","average":{"1min":14.034000000000001,"5min":16.495999999999999,"15min":17.122},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":20.396999999999998,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.747}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.484},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":15.105}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:18:27 2024","interfaces":[{"interface":"back","average":{"1min":13.632,"5min":17.548999999999999,"15min":18.692},"min":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":9.5579999999999998},"max":{"1min":19.93,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.353},{"interface":"front","average":{"1min":15.124000000000001,"5min":18.151,"15min":19.056000000000001},"min":{"1min":11.132,"5min":11.132,"15min":11.132},"max":{"1min":19.315999999999999,"5min":53.398000000000003,"15min":53.398000000000003},"last":16.736000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313272,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":8.2520000000000007,"5min":13.82,"15min":16.224},"min":{"1min":3.9260000000000002,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":16.588000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.3490000000000002},{"interface":"front","average":{"1min":7.8540000000000001,"5min":12.269,"15min":14.279},"min":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":17.398,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.5589999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.175000000000001,"5min":14.156000000000001,"15min":15.800000000000001},"min":{"1min":7.3070000000000004,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":15.708,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.9250000000000007},{"interface":"front","average":{"1min":11.076000000000001,"5min":17.117999999999999,"15min":19.381},"min":{"1min":7.0309999999999997,"5min":7.0309999999999997,"15min":7.0309999999999997},"max":{"1min":17.704000000000001,"5min":155.91200000000001,"15min":155.91200000000001},"last":7.4459999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.359999999999999,"5min":15.85,"15min":18.079000000000001},"min":{"1min":7.1630000000000003,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":18.369,"5min":145.553,"15min":145.553},"last":11.864000000000001},{"interface":"front","average":{"1min":11.727,"5min":16.175999999999998,"15min":18.076000000000001},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":15.589,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.624000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":13.332000000000001,"5min":21.805,"15min":25.445},"min":{"1min":9.5999999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":17.469999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":15.965999999999999},{"interface":"front","average":{"1min":12.237,"5min":22.102,"15min":26.181999999999999},"min":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":18.117000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.464}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":10.75,"5min":17.530000000000001,"15min":20.277999999999999},"min":{"1min":7.633,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.585000000000001,"5min":146.744,"15min":146.744},"last":13.189},{"interface":"front","average":{"1min":14.096,"5min":18.937999999999999,"15min":21.030999999999999},"min":{"1min":10.731999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":18.326000000000001,"5min":150.542,"15min":150.542},"last":15.218999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":15.583,"5min":20.835000000000001,"15min":23.052},"min":{"1min":14.037000000000001,"5min":11.300000000000001,"15min":11.300000000000001},"max":{"1min":18.268999999999998,"5min":159.179,"15min":159.179},"last":16.896000000000001},{"interface":"front","average":{"1min":13.06,"5min":20.239999999999998,"15min":23.033000000000001},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.568999999999999,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.398}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444086,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.025},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":8.3620000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.297000000000001},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.616}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.771000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.221}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":17.094999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.388000000000002},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7709999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.5779999999999994},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":18.126000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444087,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.5720000000000001},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.2560000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.335000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.503}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.913},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.032}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":9.4209999999999994},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":9.6950000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.544},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":13.023999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.380000000000001},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.619}]}]},{"osd":0,"up_from":17,"seq":73014444087,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":8.8160000000000007},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.417}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.5910000000000002},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.598000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":12.814},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.827}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.548},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.1079999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":12.263},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.65}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.845000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":8.0020000000000007}]}]},{"osd":6,"up_from":15,"seq":64424509495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.7919999999999998},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.6769999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":13.143000000000001},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":11.101000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.452},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":14.831}]},{"osd":3,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":11.747,"5min":14.925000000000001,"15min":15.832000000000001},"min":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":15.340999999999999,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.534000000000001},{"interface":"front","average":{"1min":12.938000000000001,"5min":15.051,"15min":16.135999999999999},"min":{"1min":9.0679999999999996,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":16.033000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.986000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.994999999999999},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":18.061}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:18:23 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":15.919,"15min":16.475000000000001},"min":{"1min":9.7590000000000003,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":17.736000000000001,"5min":25.303999999999998,"15min":25.303999999999998},"last":17.408000000000001},{"interface":"front","average":{"1min":13.638,"5min":15.483000000000001,"15min":15.975},"min":{"1min":8,"5min":8,"15min":8},"max":{"1min":17.513999999999999,"5min":22.838999999999999,"15min":22.838999999999999},"last":19.515000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:32.593 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.691+0000 7fac61e68640 10 monclient: get_monmap_and_config 2024-01-20T11:19:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.691+0000 7fac61e68640 10 monclient: build_initial_monmap 2024-01-20T11:19:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.691+0000 7fac61e68640 10 monclient: monmap: 2024-01-20T11:19:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:32.692036+0000 2024-01-20T11:19:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:32.692036+0000 2024-01-20T11:19:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:32.693 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 1 Processor -- start 2024-01-20T11:19:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 1 -- start start 2024-01-20T11:19:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient: init 2024-01-20T11:19:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-01-20T11:19:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac5c06a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c143390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient(hunting): picked mon.noname-b con 0x7fac5c13d800 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fac5c0b0620 0x7fac5c1438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient(hunting): picked mon.noname-c con 0x7fac5c0b0620 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c0b10f0 con 0x7fac5c06a860 2024-01-20T11:19:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c06ac40 con 0x7fac5c0b0620 2024-01-20T11:19:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c072150 con 0x7fac5c13d800 2024-01-20T11:19:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.692+0000 7fac61e68640 10 monclient(hunting): authenticate will time out at 1616.043049s 2024-01-20T11:19:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.693+0000 7fac5affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.693+0000 7fac5bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fac5c0b0620 0x7fac5c1438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.693+0000 7fac5affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c143390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:34448/0 (socket says 172.21.15.110:34448) 2024-01-20T11:19:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.693+0000 7fac5affd640 1 -- 172.21.15.110:0/3346661781 learned_addr learned my addr 172.21.15.110:0/3346661781 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.693+0000 7fac5affd640 10 monclient(hunting): get_auth_request con 0x7fac5c13d800 auth_method 0 2024-01-20T11:19:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.693+0000 7fac5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.693+0000 7fac5affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.693+0000 7fac5affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5bfff640 10 monclient(hunting): get_auth_request con 0x7fac5c0b0620 auth_method 0 2024-01-20T11:19:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.694+0000 7fac5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.695+0000 7fac5affd640 10 monclient(hunting): handle_auth_done global_id 4874 payload 274 2024-01-20T11:19:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.695+0000 7fac5affd640 1 -- 172.21.15.110:0/3346661781 >> v2:172.21.15.134:3300/0 conn(0x7fac5c0b0620 msgr2=0x7fac5c1438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.695+0000 7fac5affd640 1 --2- 172.21.15.110:0/3346661781 >> v2:172.21.15.134:3300/0 conn(0x7fac5c0b0620 0x7fac5c1438d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.695+0000 7fac5affd640 1 -- 172.21.15.110:0/3346661781 >> v1:172.21.15.134:6789/0 conn(0x7fac5c06a860 legacy=0x7fac5c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.696+0000 7fac5affd640 10 monclient: _finish_hunting 0 2024-01-20T11:19:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.696+0000 7fac5affd640 1 monclient: found mon.noname-b 2024-01-20T11:19:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.696+0000 7fac5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.696+0000 7fac5affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.696+0000 7fac5affd640 1 -- 172.21.15.110:0/3346661781 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c071e70 con 0x7fac5c13d800 2024-01-20T11:19:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.697+0000 7fac5affd640 1 --2- 172.21.15.110:0/3346661781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c143390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac50009870 tx=0x7fac5002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8993556429a8ce server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 1 -- 172.21.15.110:0/3346661781 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50030840 con 0x7fac5c13d800 2024-01-20T11:19:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 10 monclient: dump: 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 1 -- 172.21.15.110:0/3346661781 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac500309a0 con 0x7fac5c13d800 2024-01-20T11:19:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac61e68640 5 monclient: authenticate success, global_id 4874 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac61e68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac61e68640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 1 -- 172.21.15.110:0/3346661781 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50030d30 con 0x7fac5c13d800 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac61e68640 4 set_mon_vals no callback set 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.701+0000 7fac5a7fc640 10 monclient: dump: 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 10 monclient: shutdown 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 1 -- 172.21.15.110:0/3346661781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 msgr2=0x7fac5c143390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 1 --2- 172.21.15.110:0/3346661781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c143390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac50009870 tx=0x7fac5002b980 comp rx=0 tx=0).stop 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 1 -- 172.21.15.110:0/3346661781 shutdown_connections 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 1 --2- 172.21.15.110:0/3346661781 >> v2:172.21.15.134:3300/0 conn(0x7fac5c0b0620 0x7fac5c1438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 1 --2- 172.21.15.110:0/3346661781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c143390 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 1 -- 172.21.15.110:0/3346661781 >> 172.21.15.110:0/3346661781 conn(0x7fac5c0b47d0 msgr2=0x7fac5c0b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 1 -- 172.21.15.110:0/3346661781 shutdown_connections 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 1 -- 172.21.15.110:0/3346661781 wait complete. 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 10 monclient: build_initial_monmap 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.702+0000 7fac61e68640 10 monclient: monmap: 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:32.721 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.703+0000 7fac61e68640 1 librados: starting msgr at 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.703+0000 7fac61e68640 1 librados: starting objecter 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.703+0000 7fac61e68640 1 Processor -- start 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.703+0000 7fac61e68640 1 -- start start 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.703+0000 7fac61e68640 1 librados: setting wanted keys 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.703+0000 7fac61e68640 1 librados: calling monclient init 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.703+0000 7fac61e68640 10 monclient: init 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.703+0000 7fac61e68640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac5c06a860 0x7fac5c138730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 10 monclient(hunting): picked mon.noname-b con 0x7fac5c06a860 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac5c0b0620 0x7fac5c130a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 10 monclient(hunting): picked mon.noname-a con 0x7fac5c0b0620 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c130f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 10 monclient(hunting): picked mon.noname-c con 0x7fac5c13d800 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c13deb0 con 0x7fac5c0b0620 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c1314c0 con 0x7fac5c06a860 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c139b30 con 0x7fac5c13d800 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac61e68640 10 monclient(hunting): authenticate will time out at 1616.054284s 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c130f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac5c06a860 0x7fac5c138730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac5c0b0620 0x7fac5c130a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c130f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:34460/0 (socket says 172.21.15.110:34460) 2024-01-20T11:19:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5bfff640 1 -- 172.21.15.110:0/38565303 learned_addr learned my addr 172.21.15.110:0/38565303 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5bfff640 10 monclient(hunting): get_auth_request con 0x7fac5c13d800 auth_method 0 2024-01-20T11:19:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5b7fe640 10 monclient(hunting): get_auth_request con 0x7fac5c06a860 auth_method 0 2024-01-20T11:19:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.704+0000 7fac5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5affd640 10 monclient(hunting): get_auth_request con 0x7fac5c0b0620 auth_method 0 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.705+0000 7fac5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 10 monclient(hunting): handle_auth_done global_id 5146 payload 824 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 1 -- 172.21.15.110:0/38565303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 msgr2=0x7fac5c130f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c130f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 1 -- 172.21.15.110:0/38565303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac5c0b0620 msgr2=0x7fac5c130a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac5c0b0620 0x7fac5c130a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 1 monclient: found mon.noname-b 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.708+0000 7fac5b7fe640 1 -- 172.21.15.110:0/38565303 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac500094d0 con 0x7fac5c06a860 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.710+0000 7fac5b7fe640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac5c06a860 0x7fac5c138730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00c3c0 tx=0x7fac4c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.710+0000 7fac43fff640 10 client.?.objecter ms_handle_connect 0x7fac5c06a860 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.710+0000 7fac43fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.713+0000 7fac43fff640 1 -- 172.21.15.110:0/38565303 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c0052d0 con 0x7fac5c06a860 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.713+0000 7fac43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.713+0000 7fac43fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.713+0000 7fac43fff640 10 monclient: dump: 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.713+0000 7fac43fff640 10 monclient: _finish_auth 0 2024-01-20T11:19:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.713+0000 7fac43fff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac43fff640 1 -- 172.21.15.110:0/38565303 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c005430 con 0x7fac5c06a860 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac61e68640 5 monclient: authenticate success, global_id 5146 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac61e68640 10 monclient: _renew_subs 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac61e68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c139610 con 0x7fac5c06a860 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac43fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac43fff640 1 -- 172.21.15.110:0/38565303 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c003830 con 0x7fac5c06a860 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac43fff640 10 monclient: dump: 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:32.729 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac59ffb640 4 set_mon_vals no callback set 2024-01-20T11:19:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac61e68640 10 client.5146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac61e68640 10 monclient: _renew_subs 2024-01-20T11:19:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac61e68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c0ae4d0 con 0x7fac5c06a860 2024-01-20T11:19:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.714+0000 7fac61e68640 1 librados: init done 2024-01-20T11:19:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.715+0000 7fac61e68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.715+0000 7fac61e68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.715+0000 7fac61e68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.715+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c139850 con 0x7fac5c06a860 2024-01-20T11:19:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.733+0000 7fac43fff640 1 -- 172.21.15.110:0/38565303 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fac4c003990 con 0x7fac5c06a860 2024-01-20T11:19:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.733+0000 7fac43fff640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac4806c8b0 0x7fac4806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.734+0000 7fac5affd640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac4806c8b0 0x7fac4806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.734+0000 7fac5affd640 10 monclient: get_auth_request con 0x7fac4806c8b0 auth_method 0 2024-01-20T11:19:32.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.734+0000 7fac5affd640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac4806c8b0 0x7fac4806ed70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fac50036000 tx=0x7fac50008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:32.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.734+0000 7fac43fff640 10 client.5146.objecter ms_handle_connect 0x7fac4806c8b0 2024-01-20T11:19:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.736+0000 7fac43fff640 1 -- 172.21.15.110:0/38565303 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fac4c057730 con 0x7fac5c06a860 2024-01-20T11:19:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.736+0000 7fac43fff640 10 client.5146.objecter ms_dispatch 0x7fac5c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.736+0000 7fac43fff640 3 client.5146.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.736+0000 7fac43fff640 3 client.5146.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.736+0000 7fac43fff640 20 client.5146.objecter dump_active .. 0 homeless 2024-01-20T11:19:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.969+0000 7fac43fff640 1 -- 172.21.15.110:0/38565303 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fac4c053a70 con 0x7fac5c06a860 2024-01-20T11:19:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.969+0000 7fac43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:32.969+0000 7fac43fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:33.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.136+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fac5c143e10 con 0x7fac4806c8b0 2024-01-20T11:19:33.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.138+0000 7fac43fff640 1 -- 172.21.15.110:0/38565303 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63977 (secure 0 0 0) 0x7fac50039450 con 0x7fac4806c8b0 2024-01-20T11:19:33.139 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:33.139 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:33.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 10 librados: watch_flush enter 2024-01-20T11:19:33.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 10 librados: watch_flush exit 2024-01-20T11:19:33.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 20 client.5146.objecter shutdown clearing up homeless session... 2024-01-20T11:19:33.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 10 client.5146.objecter successfully canceled tick 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac4806c8b0 msgr2=0x7fac4806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac4806c8b0 0x7fac4806ed70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fac50036000 tx=0x7fac50008030 comp rx=0 tx=0).stop 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 10 monclient: shutdown 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac5c06a860 msgr2=0x7fac5c138730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.140+0000 7fac61e68640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac5c06a860 0x7fac5c138730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00c3c0 tx=0x7fac4c00c880 comp rx=0 tx=0).stop 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.141+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 shutdown_connections 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.141+0000 7fac61e68640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac4806c8b0 0x7fac4806ed70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.141+0000 7fac61e68640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac5c13d800 0x7fac5c130f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.141+0000 7fac61e68640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac5c0b0620 0x7fac5c130a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.141+0000 7fac61e68640 1 --2- 172.21.15.110:0/38565303 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac5c06a860 0x7fac5c138730 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.141+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 >> 172.21.15.110:0/38565303 conn(0x7fac5c0b47d0 msgr2=0x7fac5c13d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.141+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 shutdown_connections 2024-01-20T11:19:33.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.141+0000 7fac61e68640 1 -- 172.21.15.110:0/38565303 wait complete. 2024-01-20T11:19:33.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.141+0000 7fac61e68640 1 librados: shutdown 2024-01-20T11:19:33.151 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-01-20T11:19:32.759865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.2869999999999999},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":12.744999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":5.0830000000000002},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":13.358000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.4480000000000004},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":8.8789999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":14.647},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":8.6010000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":10.131},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":12.077}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":15.398},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":10.699}]}]},{"osd":7,"up_from":21,"seq":90194313272,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.383},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.9989999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.0690000000000008},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.436999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.58},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.945}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.7919999999999998},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.202999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.249000000000001},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.353999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.853},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.747}]}]},{"osd":3,"up_from":21,"seq":90194313273,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.9240000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":12.83}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.2759999999999998},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.412000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.5670000000000002},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.6739999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.422000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":14.706}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":15.840999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":12.099}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":14.105},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.917999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444086,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.025},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":8.3620000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.297000000000001},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.616}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.771000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.221}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":17.094999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.388000000000002},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7709999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.5779999999999994},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":18.126000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444087,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.5720000000000001},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.2560000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.335000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.503}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.913},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.032}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":9.4209999999999994},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":9.6950000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.544},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":13.023999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.380000000000001},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.619}]}]},{"osd":0,"up_from":17,"seq":73014444087,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":8.8160000000000007},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.417}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.5910000000000002},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.598000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":12.814},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.827}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.548},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.1079999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":12.263},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.65}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.845000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":8.0020000000000007}]}]},{"osd":6,"up_from":15,"seq":64424509496,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.4160000000000004},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.7289999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.5119999999999996},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":9.9390000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":6.3360000000000003},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.085000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.993},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.632}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.157},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.673}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.236000000000001},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.503}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:33.153 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:33.153 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:33.153 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:33.153 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:33.153 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:33.153 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:33.153 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:33.153 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:33.154 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:33.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.253+0000 7f1477261640 10 monclient: get_monmap_and_config 2024-01-20T11:19:33.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.253+0000 7f1477261640 10 monclient: build_initial_monmap 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.253+0000 7f1477261640 10 monclient: monmap: 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:33.254525+0000 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:33.254525+0000 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:33.254 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.254+0000 7f1477261640 1 Processor -- start 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.254+0000 7f1477261640 1 -- start start 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.254+0000 7f1477261640 10 monclient: init 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f147006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 10 monclient(hunting): picked mon.noname-b con 0x7f147006f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f147013d230 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1470144060 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700715b0 con 0x7f147013d230 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1470071890 con 0x7f1470144060 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1470071b70 con 0x7f147006f920 2024-01-20T11:19:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1477261640 10 monclient(hunting): authenticate will time out at 1616.605560s 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f1474ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f147006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.255+0000 7f146ffff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f147013d230 0x7f1470143950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55516/0 (socket says 172.21.15.110:55516) 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.256+0000 7f146ffff640 1 -- 172.21.15.110:0/169452757 learned_addr learned my addr 172.21.15.110:0/169452757 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.256+0000 7f1474ffd640 10 monclient(hunting): get_auth_request con 0x7f147006f920 auth_method 0 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.256+0000 7f1474ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.256+0000 7f1474ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.256+0000 7f1474ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.256+0000 7f1474ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.256+0000 7f1474ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.257+0000 7f1474ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 1 -- 172.21.15.110:0/169452757 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353775901 0 0) 0x7f1470071890 con 0x7f1470144060 2024-01-20T11:19:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 10 monclient(hunting): my global_id is 5149 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 1 -- 172.21.15.110:0/169452757 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f145c003520 con 0x7f1470144060 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 1 -- 172.21.15.110:0/169452757 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566411648 0 0) 0x7f14700715b0 con 0x7f147013d230 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 10 monclient(hunting): my global_id is 5064 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f146f7fe640 1 -- 172.21.15.110:0/169452757 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470071890 con 0x7f147013d230 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f1474ffd640 10 monclient(hunting): handle_auth_done global_id 4880 payload 274 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f1474ffd640 1 -- 172.21.15.110:0/169452757 >> v1:172.21.15.134:6789/0 conn(0x7f1470144060 legacy=0x7f147014a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f1474ffd640 1 -- 172.21.15.110:0/169452757 >> v1:172.21.15.110:6789/0 conn(0x7f147013d230 legacy=0x7f1470143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f1474ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:19:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f1474ffd640 1 monclient: found mon.noname-b 2024-01-20T11:19:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f1474ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f1474ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.258+0000 7f1474ffd640 1 -- 172.21.15.110:0/169452757 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470143430 con 0x7f147006f920 2024-01-20T11:19:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.260+0000 7f1474ffd640 1 --2- 172.21.15.110:0/169452757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f147006fd00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1468009dc0 tx=0x7f146802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cc5aefa59b01942 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:33.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.263+0000 7f146f7fe640 1 -- 172.21.15.110:0/169452757 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146802cc40 con 0x7f147006f920 2024-01-20T11:19:33.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.263+0000 7f146f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.263+0000 7f146f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.263+0000 7f146f7fe640 10 monclient: dump: 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.263+0000 7f146f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:33.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.263+0000 7f146f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f146f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f146f7fe640 1 -- 172.21.15.110:0/169452757 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146802cda0 con 0x7f147006f920 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f146f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f146f7fe640 1 -- 172.21.15.110:0/169452757 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468034660 con 0x7f147006f920 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f146f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f146f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f146f7fe640 10 monclient: dump: 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:33.266 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 5 monclient: authenticate success, global_id 4880 2024-01-20T11:19:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:33.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 4 set_mon_vals no callback set 2024-01-20T11:19:33.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 10 monclient: shutdown 2024-01-20T11:19:33.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:33.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 1 -- 172.21.15.110:0/169452757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 msgr2=0x7f147006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 1 --2- 172.21.15.110:0/169452757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f147006fd00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1468009dc0 tx=0x7f146802be90 comp rx=0 tx=0).stop 2024-01-20T11:19:33.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 1 -- 172.21.15.110:0/169452757 shutdown_connections 2024-01-20T11:19:33.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 1 --2- 172.21.15.110:0/169452757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f147006fd00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 1 -- 172.21.15.110:0/169452757 >> 172.21.15.110:0/169452757 conn(0x7f1470066d60 msgr2=0x7f14700b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:33.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 1 -- 172.21.15.110:0/169452757 shutdown_connections 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.264+0000 7f1477261640 1 -- 172.21.15.110:0/169452757 wait complete. 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.265+0000 7f1477261640 10 monclient: build_initial_monmap 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.265+0000 7f1477261640 10 monclient: monmap: 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.265+0000 7f1477261640 1 librados: starting msgr at 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.265+0000 7f1477261640 1 librados: starting objecter 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.265+0000 7f1477261640 1 Processor -- start 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.265+0000 7f1477261640 1 -- start start 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.265+0000 7f1477261640 1 librados: setting wanted keys 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.265+0000 7f1477261640 1 librados: calling monclient init 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.265+0000 7f1477261640 10 monclient: init 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f1470142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient(hunting): picked mon.noname-c con 0x7f147006f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f147013d230 0x7f1470132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient(hunting): picked mon.noname-b con 0x7f147013d230 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:33.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1470144060 0x7f1470133010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient(hunting): picked mon.noname-a con 0x7f1470144060 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470133550 con 0x7f1470144060 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147013f950 con 0x7f147013d230 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1470138c90 con 0x7f147006f920 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1477261640 10 monclient(hunting): authenticate will time out at 1616.616701s 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.266+0000 7f1474ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f1470142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f14757fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1470144060 0x7f1470133010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f146ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f147013d230 0x7f1470132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f1474ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f1470142dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:34474/0 (socket says 172.21.15.110:34474) 2024-01-20T11:19:33.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f1474ffd640 1 -- 172.21.15.110:0/1191701883 learned_addr learned my addr 172.21.15.110:0/1191701883 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f1474ffd640 10 monclient(hunting): get_auth_request con 0x7f147006f920 auth_method 0 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f1474ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f1474ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f1474ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f14757fe640 10 monclient(hunting): get_auth_request con 0x7f1470144060 auth_method 0 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f14757fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f14757fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f14757fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f146ffff640 10 monclient(hunting): get_auth_request con 0x7f147013d230 auth_method 0 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f146ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f146ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f146ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f1474ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f1474ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:33.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.267+0000 7f1474ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:33.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.268+0000 7f14757fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:33.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.268+0000 7f14757fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:33.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.268+0000 7f14757fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:33.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.268+0000 7f146ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:33.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.268+0000 7f146ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:33.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.268+0000 7f146ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 10 monclient(hunting): handle_auth_done global_id 5152 payload 824 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 1 -- 172.21.15.110:0/1191701883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 msgr2=0x7f1470142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f1470142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 1 -- 172.21.15.110:0/1191701883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1470144060 msgr2=0x7f1470133010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1470144060 0x7f1470133010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 1 monclient: found mon.noname-b 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.270+0000 7f146ffff640 1 -- 172.21.15.110:0/1191701883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1468009a70 con 0x7f147013d230 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.273+0000 7f146ffff640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f147013d230 0x7f1470132ad0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1460001000 tx=0x7f146000cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.273+0000 7f146d7fa640 10 client.?.objecter ms_handle_connect 0x7f147013d230 2024-01-20T11:19:33.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.273+0000 7f146d7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 1 -- 172.21.15.110:0/1191701883 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460007070 con 0x7f147013d230 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 10 monclient: dump: 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:33.278 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 10 monclient: _finish_auth 0 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 1 -- 172.21.15.110:0/1191701883 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14600071f0 con 0x7f147013d230 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f1477261640 5 monclient: authenticate success, global_id 5152 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f1477261640 10 monclient: _renew_subs 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f1477261640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14701391b0 con 0x7f147013d230 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 1 -- 172.21.15.110:0/1191701883 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14600075a0 con 0x7f147013d230 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:33.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146d7fa640 10 monclient: dump: 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.276+0000 7f146effd640 4 set_mon_vals no callback set 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.277+0000 7f1477261640 10 client.5152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.277+0000 7f1477261640 10 monclient: _renew_subs 2024-01-20T11:19:33.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.277+0000 7f1477261640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:33.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.277+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147013de50 con 0x7f147013d230 2024-01-20T11:19:33.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.277+0000 7f1477261640 1 librados: init done 2024-01-20T11:19:33.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.277+0000 7f1477261640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:33.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.277+0000 7f1477261640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:33.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.277+0000 7f1477261640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:33.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.277+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1470075490 con 0x7f147013d230 2024-01-20T11:19:33.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.517+0000 7f146d7fa640 1 -- 172.21.15.110:0/1191701883 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f14600057a0 con 0x7f147013d230 2024-01-20T11:19:33.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.517+0000 7f146d7fa640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f145006c860 0x7f145006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.517+0000 7f1474ffd640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f145006c860 0x7f145006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.517+0000 7f1474ffd640 10 monclient: get_auth_request con 0x7f145006c860 auth_method 0 2024-01-20T11:19:33.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.518+0000 7f1474ffd640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f145006c860 0x7f145006ed20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1468000f80 tx=0x7f1468030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:33.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.518+0000 7f146d7fa640 10 client.5152.objecter ms_handle_connect 0x7f145006c860 2024-01-20T11:19:33.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.520+0000 7f146d7fa640 1 -- 172.21.15.110:0/1191701883 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1460057b20 con 0x7f147013d230 2024-01-20T11:19:33.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.520+0000 7f146d7fa640 10 client.5152.objecter ms_dispatch 0x7f1470000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:33.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.520+0000 7f146d7fa640 3 client.5152.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:33.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.520+0000 7f146d7fa640 3 client.5152.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:33.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.520+0000 7f146d7fa640 20 client.5152.objecter dump_active .. 0 homeless 2024-01-20T11:19:33.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.531+0000 7f146d7fa640 1 -- 172.21.15.110:0/1191701883 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1460053e60 con 0x7f147013d230 2024-01-20T11:19:33.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.531+0000 7f146d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:33.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.531+0000 7f146d7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:33.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.780+0000 7f1477261640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:33.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.780+0000 7f1477261640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:33.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.780+0000 7f1477261640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:33.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.780+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1470138eb0 con 0x7f147013d230 2024-01-20T11:19:33.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.787+0000 7f146d7fa640 1 -- 172.21.15.110:0/1191701883 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f1460017020 con 0x7f147013d230 2024-01-20T11:19:33.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.787+0000 7f146d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:33.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.787+0000 7f146d7fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:33.788 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:33.788 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":308,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:33.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.789+0000 7f1477261640 10 librados: watch_flush enter 2024-01-20T11:19:33.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.789+0000 7f1477261640 10 librados: watch_flush exit 2024-01-20T11:19:33.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 20 client.5152.objecter shutdown clearing up homeless session... 2024-01-20T11:19:33.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 10 client.5152.objecter successfully canceled tick 2024-01-20T11:19:33.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f145006c860 msgr2=0x7f145006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f145006c860 0x7f145006ed20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1468000f80 tx=0x7f1468030000 comp rx=0 tx=0).stop 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 10 monclient: shutdown 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f147013d230 msgr2=0x7f1470132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f147013d230 0x7f1470132ad0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1460001000 tx=0x7f146000cb90 comp rx=0 tx=0).stop 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 shutdown_connections 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f145006c860 0x7f145006ed20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1470144060 0x7f1470133010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f147013d230 0x7f1470132ad0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 --2- 172.21.15.110:0/1191701883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f147006f920 0x7f1470142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 >> 172.21.15.110:0/1191701883 conn(0x7f1470066d60 msgr2=0x7f14700b4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 shutdown_connections 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 -- 172.21.15.110:0/1191701883 wait complete. 2024-01-20T11:19:33.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.790+0000 7f1477261640 1 librados: shutdown 2024-01-20T11:19:33.801 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:33.801 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:33.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.899+0000 7fe126b7c640 10 monclient: get_monmap_and_config 2024-01-20T11:19:33.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.899+0000 7fe126b7c640 10 monclient: build_initial_monmap 2024-01-20T11:19:33.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.899+0000 7fe126b7c640 10 monclient: monmap: 2024-01-20T11:19:33.899 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:33.899 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.899 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:33.900093+0000 2024-01-20T11:19:33.900 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:33.900093+0000 2024-01-20T11:19:33.900 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:33.900 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.900 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:33.900 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:33.900 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:33.900 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:33.900 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:33.900 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 1 Processor -- start 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 1 -- start start 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient: init 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe12006a860 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe12013d800 0x7fe120143390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient(hunting): picked mon.noname-a con 0x7fe12013d800 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1200b0620 0x7fe1201438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient(hunting): picked mon.noname-b con 0x7fe1200b0620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1200b10f0 con 0x7fe12006a860 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12006ac40 con 0x7fe12013d800 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe120072150 con 0x7fe1200b0620 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.900+0000 7fe126b7c640 10 monclient(hunting): authenticate will time out at 1617.251098s 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.901+0000 7fe125119640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1200b0620 0x7fe1201438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.901+0000 7fe11ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe12013d800 0x7fe120143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.901+0000 7fe125119640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1200b0620 0x7fe1201438d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:34486/0 (socket says 172.21.15.110:34486) 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.901+0000 7fe125119640 1 -- 172.21.15.110:0/4078583468 learned_addr learned my addr 172.21.15.110:0/4078583468 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.901+0000 7fe125119640 10 monclient(hunting): get_auth_request con 0x7fe1200b0620 auth_method 0 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.901+0000 7fe125119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.901+0000 7fe125119640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.901+0000 7fe125119640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe11ffff640 10 monclient(hunting): get_auth_request con 0x7fe12013d800 auth_method 0 2024-01-20T11:19:33.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe11ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:33.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe11ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe11ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe125119640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:33.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe125119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:33.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe125119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:33.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe11ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:33.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe11ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:33.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.902+0000 7fe11ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:33.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.904+0000 7fe125119640 10 monclient(hunting): handle_auth_done global_id 4886 payload 274 2024-01-20T11:19:33.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.904+0000 7fe125119640 1 -- 172.21.15.110:0/4078583468 >> v2:172.21.15.110:3300/0 conn(0x7fe12013d800 msgr2=0x7fe120143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:33.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.904+0000 7fe125119640 1 --2- 172.21.15.110:0/4078583468 >> v2:172.21.15.110:3300/0 conn(0x7fe12013d800 0x7fe120143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.904+0000 7fe125119640 1 -- 172.21.15.110:0/4078583468 >> v1:172.21.15.110:6789/0 conn(0x7fe12006a860 legacy=0x7fe12013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.904+0000 7fe125119640 10 monclient: _finish_hunting 0 2024-01-20T11:19:33.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.904+0000 7fe125119640 1 monclient: found mon.noname-b 2024-01-20T11:19:33.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.904+0000 7fe125119640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:33.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.904+0000 7fe125119640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:33.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.904+0000 7fe125119640 1 -- 172.21.15.110:0/4078583468 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe120071e70 con 0x7fe1200b0620 2024-01-20T11:19:33.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.906+0000 7fe125119640 1 --2- 172.21.15.110:0/4078583468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1200b0620 0x7fe1201438d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe118009e50 tx=0x7fe11802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78ebc8ef9bfc6802 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 1 -- 172.21.15.110:0/4078583468 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11802cb00 con 0x7fe1200b0620 2024-01-20T11:19:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 10 monclient: dump: 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 1 -- 172.21.15.110:0/4078583468 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11802cc60 con 0x7fe1200b0620 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 1 -- 172.21.15.110:0/4078583468 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1180343f0 con 0x7fe1200b0620 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe126b7c640 5 monclient: authenticate success, global_id 4886 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe126b7c640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe126b7c640 4 set_mon_vals no callback set 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.909+0000 7fe11f7fe640 10 monclient: dump: 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:33.912 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:33.912 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:33.912 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 10 monclient: shutdown 2024-01-20T11:19:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 1 -- 172.21.15.110:0/4078583468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1200b0620 msgr2=0x7fe1201438d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 1 --2- 172.21.15.110:0/4078583468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1200b0620 0x7fe1201438d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe118009e50 tx=0x7fe11802bf60 comp rx=0 tx=0).stop 2024-01-20T11:19:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 1 -- 172.21.15.110:0/4078583468 shutdown_connections 2024-01-20T11:19:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 1 --2- 172.21.15.110:0/4078583468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1200b0620 0x7fe1201438d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 1 --2- 172.21.15.110:0/4078583468 >> v2:172.21.15.110:3300/0 conn(0x7fe12013d800 0x7fe120143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 1 -- 172.21.15.110:0/4078583468 >> 172.21.15.110:0/4078583468 conn(0x7fe1200b47d0 msgr2=0x7fe1200b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 1 -- 172.21.15.110:0/4078583468 shutdown_connections 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.910+0000 7fe126b7c640 1 -- 172.21.15.110:0/4078583468 wait complete. 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.911+0000 7fe126b7c640 10 monclient: build_initial_monmap 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.911+0000 7fe126b7c640 10 monclient: monmap: 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.916 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.911+0000 7fe126b7c640 1 librados: starting msgr at 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.911+0000 7fe126b7c640 1 librados: starting objecter 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.911+0000 7fe126b7c640 1 Processor -- start 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.911+0000 7fe126b7c640 1 -- start start 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.911+0000 7fe126b7c640 1 librados: setting wanted keys 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.911+0000 7fe126b7c640 1 librados: calling monclient init 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.911+0000 7fe126b7c640 10 monclient: init 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe12006a860 0x7fe1201346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient(hunting): picked mon.noname-b con 0x7fe12006a860 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:33.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1200b0620 0x7fe1201385e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient(hunting): picked mon.noname-a con 0x7fe1200b0620 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe12013d800 0x7fe120138c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient(hunting): picked mon.noname-c con 0x7fe12013d800 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe120139350 con 0x7fe1200b0620 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200adb90 con 0x7fe12006a860 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200add50 con 0x7fe12013d800 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe126b7c640 10 monclient(hunting): authenticate will time out at 1617.262734s 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.912+0000 7fe125119640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe12013d800 0x7fe120138c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe11ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1200b0620 0x7fe1201385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe124918640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe12006a860 0x7fe1201346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe125119640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe12013d800 0x7fe120138c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:34494/0 (socket says 172.21.15.110:34494) 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe125119640 1 -- 172.21.15.110:0/746875137 learned_addr learned my addr 172.21.15.110:0/746875137 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe125119640 10 monclient(hunting): get_auth_request con 0x7fe12013d800 auth_method 0 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe125119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe125119640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe125119640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe11ffff640 10 monclient(hunting): get_auth_request con 0x7fe1200b0620 auth_method 0 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe11ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe11ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe11ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe124918640 10 monclient(hunting): get_auth_request con 0x7fe12006a860 auth_method 0 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe124918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe124918640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe124918640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe125119640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe125119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe125119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe11ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe11ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe11ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe124918640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe124918640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:33.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.913+0000 7fe124918640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:33.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 10 monclient(hunting): handle_auth_done global_id 5073 payload 824 2024-01-20T11:19:33.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 1 -- 172.21.15.110:0/746875137 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe12013d800 msgr2=0x7fe120138c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe12013d800 0x7fe120138c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 1 -- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe12006a860 msgr2=0x7fe1201346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:33.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe12006a860 0x7fe1201346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 1 monclient: found mon.noname-a 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe11ffff640 1 -- 172.21.15.110:0/746875137 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118009b30 con 0x7fe1200b0620 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.916+0000 7fe124918640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe12006a860 0x7fe1201346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.918+0000 7fe11ffff640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1200b0620 0x7fe1201385e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00e870 tx=0x7fe10c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.918+0000 7fe11d7fa640 10 client.?.objecter ms_handle_connect 0x7fe1200b0620 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.918+0000 7fe11d7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 1 -- 172.21.15.110:0/746875137 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00ee60 con 0x7fe1200b0620 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 10 monclient: dump: 2024-01-20T11:19:33.923 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 10 monclient: _finish_auth 0 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 1 -- 172.21.15.110:0/746875137 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c00c940 con 0x7fe1200b0620 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 1 -- 172.21.15.110:0/746875137 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00ccd0 con 0x7fe1200b0620 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11d7fa640 10 monclient: dump: 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe126b7c640 5 monclient: authenticate success, global_id 5073 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe126b7c640 10 monclient: _renew_subs 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe11effd640 4 set_mon_vals no callback set 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe126b7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1200adfd0 con 0x7fe1200b0620 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe126b7c640 10 client.5073.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe126b7c640 10 monclient: _renew_subs 2024-01-20T11:19:33.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe126b7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1200ae490 con 0x7fe1200b0620 2024-01-20T11:19:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.922+0000 7fe126b7c640 1 librados: init done 2024-01-20T11:19:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.923+0000 7fe126b7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.923+0000 7fe126b7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.923+0000 7fe126b7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:33.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:33.923+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1201391c0 con 0x7fe1200b0620 2024-01-20T11:19:34.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.161+0000 7fe11d7fa640 1 -- 172.21.15.110:0/746875137 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe10c013470 con 0x7fe1200b0620 2024-01-20T11:19:34.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.161+0000 7fe11d7fa640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe10406c8b0 0x7fe10406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:34.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.161+0000 7fe124918640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe10406c8b0 0x7fe10406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:34.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.161+0000 7fe124918640 10 monclient: get_auth_request con 0x7fe10406c8b0 auth_method 0 2024-01-20T11:19:34.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.162+0000 7fe11d7fa640 10 client.5073.objecter ms_handle_connect 0x7fe10406c8b0 2024-01-20T11:19:34.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.162+0000 7fe124918640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe10406c8b0 0x7fe10406ed70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe114001b70 tx=0x7fe114009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:34.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.175+0000 7fe11d7fa640 1 -- 172.21.15.110:0/746875137 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe10c00ce30 con 0x7fe1200b0620 2024-01-20T11:19:34.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.175+0000 7fe11d7fa640 10 client.5073.objecter ms_dispatch 0x7fe120000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:34.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.175+0000 7fe11d7fa640 3 client.5073.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:34.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.175+0000 7fe11d7fa640 3 client.5073.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:34.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.175+0000 7fe11d7fa640 20 client.5073.objecter dump_active .. 0 homeless 2024-01-20T11:19:34.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.175+0000 7fe11d7fa640 1 -- 172.21.15.110:0/746875137 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe10c054cb0 con 0x7fe1200b0620 2024-01-20T11:19:34.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.175+0000 7fe11d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:34.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.175+0000 7fe11d7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:34.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.350+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe120143e10 con 0x7fe10406c8b0 2024-01-20T11:19:34.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.352+0000 7fe11d7fa640 1 -- 172.21.15.110:0/746875137 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63977 (secure 0 0 0) 0x7fe11400fe00 con 0x7fe10406c8b0 2024-01-20T11:19:34.353 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:34.353 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:34.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 10 librados: watch_flush enter 2024-01-20T11:19:34.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 10 librados: watch_flush exit 2024-01-20T11:19:34.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 20 client.5073.objecter shutdown clearing up homeless session... 2024-01-20T11:19:34.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 10 client.5073.objecter successfully canceled tick 2024-01-20T11:19:34.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe10406c8b0 msgr2=0x7fe10406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:34.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe10406c8b0 0x7fe10406ed70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe114001b70 tx=0x7fe114009040 comp rx=0 tx=0).stop 2024-01-20T11:19:34.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 10 monclient: shutdown 2024-01-20T11:19:34.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1200b0620 msgr2=0x7fe1201385e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.354+0000 7fe126b7c640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1200b0620 0x7fe1201385e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00e870 tx=0x7fe10c00ed30 comp rx=0 tx=0).stop 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.355+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 shutdown_connections 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.355+0000 7fe126b7c640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe10406c8b0 0x7fe10406ed70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.355+0000 7fe126b7c640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe12013d800 0x7fe120138c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.355+0000 7fe126b7c640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1200b0620 0x7fe1201385e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.355+0000 7fe126b7c640 1 --2- 172.21.15.110:0/746875137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe12006a860 0x7fe1201346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.355+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 >> 172.21.15.110:0/746875137 conn(0x7fe1200b47d0 msgr2=0x7fe12013b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.355+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 shutdown_connections 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.355+0000 7fe126b7c640 1 -- 172.21.15.110:0/746875137 wait complete. 2024-01-20T11:19:34.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:34.355+0000 7fe126b7c640 1 librados: shutdown 2024-01-20T11:19:34.365 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-01-20T11:19:32.759865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":7.6130000000000004,"5min":12.478,"15min":14.398999999999999},"min":{"1min":4.4640000000000004,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":14.779,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.2869999999999999},{"interface":"front","average":{"1min":7.0579999999999998,"5min":13.198,"15min":15.576000000000001},"min":{"1min":4.298,"5min":4.298,"15min":4.298},"max":{"1min":10.298,"5min":91.789000000000001,"15min":91.789000000000001},"last":12.744999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.170999999999999,"5min":15.305999999999999,"15min":17.34},"min":{"1min":6.2549999999999999,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":15.750999999999999,"5min":98.700000000000003,"15min":98.700000000000003},"last":5.0830000000000002},{"interface":"front","average":{"1min":10.692,"5min":14.734999999999999,"15min":16.283000000000001},"min":{"1min":6.258,"5min":6.258,"15min":6.258},"max":{"1min":15.481,"5min":95.603999999999999,"15min":95.603999999999999},"last":13.358000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.789,"5min":16.452000000000002,"15min":18.364999999999998},"min":{"1min":8.8330000000000002,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":16.492999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.4480000000000004},{"interface":"front","average":{"1min":12.555,"5min":16.408999999999999,"15min":17.620000000000001},"min":{"1min":8.9619999999999997,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":17.042999999999999,"5min":105.43899999999999,"15min":105.43899999999999},"last":8.8789999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":11.189,"5min":18.149999999999999,"15min":20.655999999999999},"min":{"1min":7.1349999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":17.420999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":14.647},{"interface":"front","average":{"1min":14.026,"5min":20.335999999999999,"15min":22.512},"min":{"1min":9.6069999999999993,"5min":8.2799999999999994,"15min":8.2799999999999994},"max":{"1min":18.768999999999998,"5min":100.633,"15min":100.633},"last":8.6010000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":15.117000000000001,"5min":20.279,"15min":22.100999999999999},"min":{"1min":6.8070000000000004,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":19.931999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":10.131},{"interface":"front","average":{"1min":13.571,"5min":18.881,"15min":20.803000000000001},"min":{"1min":8.8079999999999998,"5min":8.8079999999999998,"15min":8.8079999999999998},"max":{"1min":17.181000000000001,"5min":107.962,"15min":107.962},"last":12.077}]},{"osd":7,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":12.093999999999999,"5min":17.363,"15min":19.309000000000001},"min":{"1min":8.0990000000000002,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.016999999999999,"5min":96.911000000000001,"15min":96.911000000000001},"last":15.398},{"interface":"front","average":{"1min":16.606999999999999,"5min":21.315999999999999,"15min":22.969000000000001},"min":{"1min":13.186,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":21.094000000000001,"5min":129.828,"15min":129.828},"last":10.699}]}]},{"osd":7,"up_from":21,"seq":90194313272,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.383},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.9989999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":8.0690000000000008},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.436999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.58},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.945}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.7919999999999998},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.202999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":16.433,"15min":17.635000000000002},"min":{"1min":8.2270000000000003,"5min":8.2270000000000003,"15min":8.2270000000000003},"max":{"1min":18.178999999999998,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.249000000000001},{"interface":"front","average":{"1min":11.619,"5min":16.224,"15min":17.619},"min":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":16.835000000000001,"5min":92.613,"15min":92.613},"last":10.353999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.853},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.747}]}]},{"osd":3,"up_from":21,"seq":90194313273,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.9240000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":12.83}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.2759999999999998},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.412000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.5670000000000002},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.6739999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.422000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":14.706}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":15.840999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":12.099}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":14.105},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.917999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444086,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.025},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":8.3620000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.297000000000001},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.616}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.771000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.221}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":17.094999999999999},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.388000000000002},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.7709999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.5779999999999994},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":18.126000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444087,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.5720000000000001},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.2560000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.335000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.503}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.913},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.032}]},{"osd":4,"last update":"Sat Jan 20 11:18:30 2024","interfaces":[{"interface":"back","average":{"1min":11.824999999999999,"5min":13.536,"15min":14.041},"min":{"1min":7.8200000000000003,"5min":7.8200000000000003,"15min":7.8200000000000003},"max":{"1min":15.241,"5min":25.541,"15min":25.541},"last":9.4209999999999994},{"interface":"front","average":{"1min":13.750999999999999,"5min":15.118,"15min":15.263999999999999},"min":{"1min":10.727,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":16.777000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":9.6950000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.544},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":13.023999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.380000000000001},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.619}]}]},{"osd":0,"up_from":17,"seq":73014444087,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":8.8160000000000007},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.417}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.5910000000000002},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.598000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":11.773,"5min":22.888999999999999,"15min":27.629000000000001},"min":{"1min":8.1319999999999997,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.619999999999999,"5min":358.738,"15min":358.738},"last":12.814},{"interface":"front","average":{"1min":10.794,"5min":12.286,"15min":12.932},"min":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":19.866,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.827}]},{"osd":4,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":12.750999999999999,"5min":24.635000000000002,"15min":29.887},"min":{"1min":8.7319999999999993,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":16.937999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.548},{"interface":"front","average":{"1min":10.680999999999999,"5min":22.082999999999998,"15min":27.131},"min":{"1min":5.9480000000000004,"5min":5.9480000000000004,"15min":5.9480000000000004},"max":{"1min":16.239000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.1079999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":12.263},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.65}]},{"osd":7,"last update":"Sat Jan 20 11:18:29 2024","interfaces":[{"interface":"back","average":{"1min":14.664999999999999,"5min":16.486000000000001,"15min":17.183},"min":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":19.030999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.845000000000001},{"interface":"front","average":{"1min":11.811,"5min":12.384,"15min":12.436999999999999},"min":{"1min":8.0589999999999993,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":16.681999999999999,"5min":21.529,"15min":21.529},"last":8.0020000000000007}]}]},{"osd":6,"up_from":15,"seq":64424509496,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.4160000000000004},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.7289999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.5119999999999996},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":9.9390000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":6.3360000000000003},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.085000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.993},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.632}]},{"osd":4,"last update":"Sat Jan 20 11:18:28 2024","interfaces":[{"interface":"back","average":{"1min":14.032,"5min":15.978999999999999,"15min":16.972000000000001},"min":{"1min":9.9329999999999998,"5min":8.5890000000000004,"15min":8.5890000000000004},"max":{"1min":18.038,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.157},{"interface":"front","average":{"1min":12.808999999999999,"5min":15.667,"15min":16.896999999999998},"min":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"max":{"1min":18.062000000000001,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.673}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.236000000000001},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.503}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:35.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~37s 2024-01-20T11:19:37.367 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.468+0000 7f7b21e90640 10 monclient: get_monmap_and_config 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.468+0000 7f7b21e90640 10 monclient: build_initial_monmap 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.468+0000 7f7b21e90640 10 monclient: monmap: 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:37.468681+0000 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:37.468681+0000 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:37.468 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:37.469 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:37.469 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:37.469 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.468+0000 7f7b21e90640 1 Processor -- start 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 1 -- start start 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient: init 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b1c0a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b1c0a9ad0 0x7f7b1c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient(hunting): picked mon.noname-a con 0x7f7b1c0a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c142920 0x7f7b1c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:37.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient(hunting): picked mon.noname-b con 0x7f7b1c142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c0b25c0 con 0x7f7b1c0a96f0 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b1c0aaea0 con 0x7f7b1c0a9ad0 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c13fbe0 con 0x7f7b1c142920 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.469+0000 7f7b21e90640 10 monclient(hunting): authenticate will time out at 1620.819851s 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.470+0000 7f7b13fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b1c0a9ad0 0x7f7b1c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.470+0000 7f7b1bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c142920 0x7f7b1c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.470+0000 7f7b1bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c142920 0x7f7b1c14c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:34508/0 (socket says 172.21.15.110:34508) 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.470+0000 7f7b1bfff640 1 -- 172.21.15.110:0/3377590262 learned_addr learned my addr 172.21.15.110:0/3377590262 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.470+0000 7f7b1bfff640 10 monclient(hunting): get_auth_request con 0x7f7b1c142920 auth_method 0 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.470+0000 7f7b1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.470+0000 7f7b1bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:37.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.470+0000 7f7b1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.471+0000 7f7b13fff640 10 monclient(hunting): get_auth_request con 0x7f7b1c0a9ad0 auth_method 0 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.471+0000 7f7b13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.471+0000 7f7b13fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.471+0000 7f7b13fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.471+0000 7f7b1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.471+0000 7f7b1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.471+0000 7f7b1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b1bfff640 10 monclient(hunting): handle_auth_done global_id 4892 payload 274 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b1bfff640 1 -- 172.21.15.110:0/3377590262 >> v2:172.21.15.110:3300/0 conn(0x7f7b1c0a9ad0 msgr2=0x7f7b1c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b1bfff640 1 --2- 172.21.15.110:0/3377590262 >> v2:172.21.15.110:3300/0 conn(0x7f7b1c0a9ad0 0x7f7b1c1423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b1bfff640 1 -- 172.21.15.110:0/3377590262 >> v1:172.21.15.110:6789/0 conn(0x7f7b1c0a96f0 legacy=0x7f7b1c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:37.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b1bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:37.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b1bfff640 1 monclient: found mon.noname-b 2024-01-20T11:19:37.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:37.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b1bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:37.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.472+0000 7f7b1bfff640 1 -- 172.21.15.110:0/3377590262 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c0ab0a0 con 0x7f7b1c142920 2024-01-20T11:19:37.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.474+0000 7f7b1bfff640 1 --2- 172.21.15.110:0/3377590262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c142920 0x7f7b1c14c850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009e50 tx=0x7f7b1402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9ea840ebf61abd3 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:37.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 1 -- 172.21.15.110:0/3377590262 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1402ca70 con 0x7f7b1c142920 2024-01-20T11:19:37.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:37.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:37.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 10 monclient: dump: 2024-01-20T11:19:37.479 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 10 monclient: _finish_auth 0 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:37.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 1 -- 172.21.15.110:0/3377590262 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1402cbd0 con 0x7f7b1c142920 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b21e90640 5 monclient: authenticate success, global_id 4892 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b21e90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b21e90640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 1 -- 172.21.15.110:0/3377590262 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b140343f0 con 0x7f7b1c142920 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b21e90640 4 set_mon_vals no callback set 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.479+0000 7f7b1affd640 10 monclient: dump: 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:37.481 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:37.482 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:37.482 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:37.482 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:37.482 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:37.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 10 monclient: shutdown 2024-01-20T11:19:37.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 1 -- 172.21.15.110:0/3377590262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c142920 msgr2=0x7f7b1c14c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 1 --2- 172.21.15.110:0/3377590262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c142920 0x7f7b1c14c850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009e50 tx=0x7f7b1402bed0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 1 -- 172.21.15.110:0/3377590262 shutdown_connections 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 1 --2- 172.21.15.110:0/3377590262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c142920 0x7f7b1c14c850 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 1 --2- 172.21.15.110:0/3377590262 >> v2:172.21.15.110:3300/0 conn(0x7f7b1c0a9ad0 0x7f7b1c1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 1 -- 172.21.15.110:0/3377590262 >> 172.21.15.110:0/3377590262 conn(0x7f7b1c0b7bc0 msgr2=0x7f7b1c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 1 -- 172.21.15.110:0/3377590262 shutdown_connections 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.480+0000 7f7b21e90640 1 -- 172.21.15.110:0/3377590262 wait complete. 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.481+0000 7f7b21e90640 10 monclient: build_initial_monmap 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.481+0000 7f7b21e90640 10 monclient: monmap: 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.481+0000 7f7b21e90640 1 librados: starting msgr at 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.481+0000 7f7b21e90640 1 librados: starting objecter 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.481+0000 7f7b21e90640 1 Processor -- start 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.481+0000 7f7b21e90640 1 -- start start 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.481+0000 7f7b21e90640 1 librados: setting wanted keys 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.481+0000 7f7b21e90640 1 librados: calling monclient init 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.481+0000 7f7b21e90640 10 monclient: init 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c0a96f0 0x7f7b1c1dab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient(hunting): picked mon.noname-c con 0x7f7b1c0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b1c0a9ad0 0x7f7b1c0aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient(hunting): picked mon.noname-a con 0x7f7b1c0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b1c142920 0x7f7b1c0af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient(hunting): picked mon.noname-b con 0x7f7b1c142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c1db490 con 0x7f7b1c0a9ad0 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c0b1390 con 0x7f7b1c142920 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c139890 con 0x7f7b1c0a96f0 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.482+0000 7f7b21e90640 10 monclient(hunting): authenticate will time out at 1620.832514s 2024-01-20T11:19:37.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.483+0000 7f7b1bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b1c142920 0x7f7b1c0af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:37.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.483+0000 7f7b1bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b1c142920 0x7f7b1c0af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:60268/0 (socket says 172.21.15.110:60268) 2024-01-20T11:19:37.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.483+0000 7f7b1bfff640 1 -- 172.21.15.110:0/1776477643 learned_addr learned my addr 172.21.15.110:0/1776477643 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:37.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.483+0000 7f7b1bfff640 10 monclient(hunting): get_auth_request con 0x7f7b1c142920 auth_method 0 2024-01-20T11:19:37.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.483+0000 7f7b1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:37.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.483+0000 7f7b1bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:37.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.483+0000 7f7b1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:37.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.483+0000 7f7b13fff640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b1c0a9ad0 0x7f7b1c0aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1b7fe640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c0a96f0 0x7f7b1c1dab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1b7fe640 10 monclient(hunting): get_auth_request con 0x7f7b1c0a96f0 auth_method 0 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:37.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:37.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.484+0000 7f7b1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:37.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.486+0000 7f7b13fff640 10 monclient(hunting): get_auth_request con 0x7f7b1c0a9ad0 auth_method 0 2024-01-20T11:19:37.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.486+0000 7f7b13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:37.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.486+0000 7f7b13fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:37.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.486+0000 7f7b13fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 10 monclient(hunting): handle_auth_done global_id 5158 payload 824 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 1 -- 172.21.15.110:0/1776477643 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c0a96f0 msgr2=0x7f7b1c1dab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c0a96f0 0x7f7b1c1dab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 1 -- 172.21.15.110:0/1776477643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b1c0a9ad0 msgr2=0x7f7b1c0aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b1c0a9ad0 0x7f7b1c0aefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:37.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 1 monclient: found mon.noname-b 2024-01-20T11:19:37.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:37.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:37.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.487+0000 7f7b1bfff640 1 -- 172.21.15.110:0/1776477643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b14009b30 con 0x7f7b1c142920 2024-01-20T11:19:37.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.490+0000 7f7b1bfff640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b1c142920 0x7f7b1c0af4e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009e50 tx=0x7f7b14004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:37.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.490+0000 7f7b18ff9640 10 client.?.objecter ms_handle_connect 0x7f7b1c142920 2024-01-20T11:19:37.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.490+0000 7f7b18ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:37.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 1 -- 172.21.15.110:0/1776477643 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b14009510 con 0x7f7b1c142920 2024-01-20T11:19:37.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:37.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:37.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 10 monclient: dump: 2024-01-20T11:19:37.495 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 10 monclient: _finish_auth 0 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b21e90640 5 monclient: authenticate success, global_id 5158 2024-01-20T11:19:37.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 1 -- 172.21.15.110:0/1776477643 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1403e450 con 0x7f7b1c142920 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b21e90640 10 monclient: _renew_subs 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 1 -- 172.21.15.110:0/1776477643 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1403e800 con 0x7f7b1c142920 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b21e90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1c1db090 con 0x7f7b1c142920 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b18ff9640 10 monclient: dump: 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:37.497 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b1a7fc640 4 set_mon_vals no callback set 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b21e90640 10 client.5158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b21e90640 10 monclient: _renew_subs 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b21e90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1c138df0 con 0x7f7b1c142920 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.494+0000 7f7b21e90640 1 librados: init done 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.495+0000 7f7b21e90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.495+0000 7f7b21e90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.495+0000 7f7b21e90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:37.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.495+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1c1db640 con 0x7f7b1c142920 2024-01-20T11:19:37.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.514+0000 7f7b18ff9640 1 -- 172.21.15.110:0/1776477643 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7b1403e960 con 0x7f7b1c142920 2024-01-20T11:19:37.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.515+0000 7f7b18ff9640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7afc06c860 0x7f7afc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:37.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.515+0000 7f7b1b7fe640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7afc06c860 0x7f7afc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:37.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.515+0000 7f7b1b7fe640 10 monclient: get_auth_request con 0x7f7afc06c860 auth_method 0 2024-01-20T11:19:37.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.516+0000 7f7b18ff9640 10 client.5158.objecter ms_handle_connect 0x7f7afc06c860 2024-01-20T11:19:37.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.516+0000 7f7b1b7fe640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7afc06c860 0x7f7afc06ed20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c001780 tx=0x7f7b0c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:37.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.517+0000 7f7b18ff9640 1 -- 172.21.15.110:0/1776477643 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7b1407b0c0 con 0x7f7b1c142920 2024-01-20T11:19:37.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.517+0000 7f7b18ff9640 10 client.5158.objecter ms_dispatch 0x7f7b1c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:37.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.517+0000 7f7b18ff9640 3 client.5158.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:37.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.517+0000 7f7b18ff9640 3 client.5158.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:37.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.517+0000 7f7b18ff9640 20 client.5158.objecter dump_active .. 0 homeless 2024-01-20T11:19:37.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.750+0000 7f7b18ff9640 1 -- 172.21.15.110:0/1776477643 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7b14077400 con 0x7f7b1c142920 2024-01-20T11:19:37.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.750+0000 7f7b18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:37.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.750+0000 7f7b18ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.920+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b1c1db300 con 0x7f7afc06c860 2024-01-20T11:19:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.923+0000 7f7b18ff9640 1 -- 172.21.15.110:0/1776477643 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64041 (secure 0 0 0) 0x7f7b1c1db300 con 0x7f7afc06c860 2024-01-20T11:19:37.923 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.924+0000 7f7b21e90640 10 librados: watch_flush enter 2024-01-20T11:19:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.924+0000 7f7b21e90640 10 librados: watch_flush exit 2024-01-20T11:19:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.924+0000 7f7b21e90640 20 client.5158.objecter shutdown clearing up homeless session... 2024-01-20T11:19:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.924+0000 7f7b21e90640 10 client.5158.objecter successfully canceled tick 2024-01-20T11:19:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7afc06c860 msgr2=0x7f7afc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7afc06c860 0x7f7afc06ed20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c001780 tx=0x7f7b0c009040 comp rx=0 tx=0).stop 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 10 monclient: shutdown 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b1c142920 msgr2=0x7f7b1c0af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b1c142920 0x7f7b1c0af4e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009e50 tx=0x7f7b14004a40 comp rx=0 tx=0).stop 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 shutdown_connections 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7afc06c860 0x7f7afc06ed20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b1c142920 0x7f7b1c0af4e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b1c0a9ad0 0x7f7b1c0aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 --2- 172.21.15.110:0/1776477643 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b1c0a96f0 0x7f7b1c1dab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 >> 172.21.15.110:0/1776477643 conn(0x7f7b1c0b7bc0 msgr2=0x7f7b1c0b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 shutdown_connections 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 -- 172.21.15.110:0/1776477643 wait complete. 2024-01-20T11:19:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:37.925+0000 7f7b21e90640 1 librados: shutdown 2024-01-20T11:19:37.935 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-01-20T11:19:36.760553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002434"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.6210000000000004},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.3109999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.6389999999999993},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":13.706}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.65},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.862}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":15.443},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":9.2720000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":17.920999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":12.013999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":19.039000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":11.093999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313273,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.4269999999999996},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":10.565}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":12.585000000000001},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.9260000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.0570000000000004},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.321}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.183999999999999},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.9670000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.388},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":15.997999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.173},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":7.6790000000000003}]}]},{"osd":3,"up_from":21,"seq":90194313274,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.6820000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6429999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":13.377000000000001},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.042999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":9.2270000000000003},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.106}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.757},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":10.888999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.986000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.782},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.074999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444087,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.016},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":15.246}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":15.747},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.3399999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.568},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.6669999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.766},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.898999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.602},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.9610000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.755000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":13.335000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444088,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.7919999999999998},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":8.5079999999999991}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.016},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.7480000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.432},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.872999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":11.438000000000001},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.175999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.5},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.603999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.6980000000000004},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.173}]}]},{"osd":0,"up_from":17,"seq":73014444088,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.5389999999999997},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":5.4989999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":12.337},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.1999999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":9.5210000000000008},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.249000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.045},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.4980000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":10.124000000000001},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":14.933}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.717000000000001},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":11.090999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.516},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.7309999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.423},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":10.218999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":14.609999999999999},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.156000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.992000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.3800000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.156000000000001},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.1079999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.834},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.272}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:37.937 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:38.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.038+0000 7fe4db6da640 10 monclient: get_monmap_and_config 2024-01-20T11:19:38.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.038+0000 7fe4db6da640 10 monclient: build_initial_monmap 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.038+0000 7fe4db6da640 10 monclient: monmap: 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:38.038838+0000 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:38.038838+0000 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.038+0000 7fe4db6da640 1 Processor -- start 2024-01-20T11:19:38.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 1 -- start start 2024-01-20T11:19:38.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 10 monclient: init 2024-01-20T11:19:38.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe4d40a96f0 0x7fe4d413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d40a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4d413fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4d40a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40b25c0 con 0x7fe4d413fa10 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40ab0a0 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d40aaea0 con 0x7fe4d40a96f0 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.039+0000 7fe4db6da640 10 monclient(hunting): authenticate will time out at 1621.389790s 2024-01-20T11:19:38.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.040+0000 7fe4d9476640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe4d40a96f0 0x7fe4d413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.040+0000 7fe4d9c77640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fe4d40a9ad0 0x7fe4d414c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:53290/0 (socket says 172.21.15.110:53290) 2024-01-20T11:19:38.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.040+0000 7fe4d9c77640 1 -- 172.21.15.110:0/808952181 learned_addr learned my addr 172.21.15.110:0/808952181 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:38.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.040+0000 7fe4d9476640 10 monclient(hunting): get_auth_request con 0x7fe4d40a96f0 auth_method 0 2024-01-20T11:19:38.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.040+0000 7fe4d9476640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:38.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.040+0000 7fe4d9476640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:38.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.040+0000 7fe4d9476640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:38.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.041+0000 7fe4d9476640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:38.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.041+0000 7fe4d9476640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:38.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.041+0000 7fe4d9476640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:38.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.042+0000 7fe4cbfff640 1 -- 172.21.15.110:0/808952181 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228346706 0 0) 0x7fe4d40ab0a0 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.042+0000 7fe4cbfff640 10 monclient(hunting): _negotiate 2024-01-20T11:19:38.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.042+0000 7fe4cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.042+0000 7fe4cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.042+0000 7fe4cbfff640 10 monclient(hunting): my global_id is 5161 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.042+0000 7fe4cbfff640 1 -- 172.21.15.110:0/808952181 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c0003520 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.043+0000 7fe4d9476640 10 monclient(hunting): handle_auth_done global_id 5085 payload 274 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.043+0000 7fe4d9476640 1 -- 172.21.15.110:0/808952181 >> v1:172.21.15.134:6789/0 conn(0x7fe4d40a9ad0 legacy=0x7fe4d414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.043+0000 7fe4d9476640 1 -- 172.21.15.110:0/808952181 >> v1:172.21.15.110:6789/0 conn(0x7fe4d413fa10 legacy=0x7fe4d41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.043+0000 7fe4d9476640 10 monclient: _finish_hunting 0 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.043+0000 7fe4d9476640 1 monclient: found mon.noname-a 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.043+0000 7fe4d9476640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.043+0000 7fe4d9476640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:38.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.043+0000 7fe4d9476640 1 -- 172.21.15.110:0/808952181 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d40b3fb0 con 0x7fe4d40a96f0 2024-01-20T11:19:38.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.047+0000 7fe4d9476640 1 --2- 172.21.15.110:0/808952181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 0x7fe4d413f4d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009d90 tx=0x7fe4cc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a056e919d26a1f87 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 1 -- 172.21.15.110:0/808952181 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc02cc60 con 0x7fe4d40a96f0 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 10 monclient: dump: 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 10 monclient: _finish_auth 0 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 1 -- 172.21.15.110:0/808952181 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc02cdc0 con 0x7fe4d40a96f0 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 1 -- 172.21.15.110:0/808952181 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc034660 con 0x7fe4d40a96f0 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4cbfff640 10 monclient: dump: 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4db6da640 5 monclient: authenticate success, global_id 5085 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4db6da640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4db6da640 4 set_mon_vals no callback set 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4db6da640 10 monclient: shutdown 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4db6da640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4db6da640 1 -- 172.21.15.110:0/808952181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 msgr2=0x7fe4d413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.052+0000 7fe4db6da640 1 --2- 172.21.15.110:0/808952181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 0x7fe4d413f4d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009d90 tx=0x7fe4cc02beb0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.053+0000 7fe4db6da640 1 -- 172.21.15.110:0/808952181 shutdown_connections 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.053+0000 7fe4db6da640 1 --2- 172.21.15.110:0/808952181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 0x7fe4d413f4d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.053+0000 7fe4db6da640 1 -- 172.21.15.110:0/808952181 >> 172.21.15.110:0/808952181 conn(0x7fe4d40b7bc0 msgr2=0x7fe4d40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.053+0000 7fe4db6da640 1 -- 172.21.15.110:0/808952181 shutdown_connections 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.053+0000 7fe4db6da640 1 -- 172.21.15.110:0/808952181 wait complete. 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.053+0000 7fe4db6da640 10 monclient: build_initial_monmap 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.053+0000 7fe4db6da640 10 monclient: monmap: 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:38.056 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.057 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:38.057 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:38.057 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:38.057 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 librados: starting msgr at 2024-01-20T11:19:38.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 librados: starting objecter 2024-01-20T11:19:38.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 Processor -- start 2024-01-20T11:19:38.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 -- start start 2024-01-20T11:19:38.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 librados: setting wanted keys 2024-01-20T11:19:38.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 librados: calling monclient init 2024-01-20T11:19:38.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient: init 2024-01-20T11:19:38.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:38.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 0x7fe4d41dabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d40a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4d40a9ad0 0x7fe4d41db130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d40a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4d41391f0 0x7fe4d40af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d41391f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d411f880 con 0x7fe4d40a96f0 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d41db7c0 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40afeb0 con 0x7fe4d41391f0 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:38.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.054+0000 7fe4db6da640 10 monclient(hunting): authenticate will time out at 1621.405023s 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9476640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 0x7fe4d41dabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d8c75640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4d40a9ad0 0x7fe4d41db130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9c77640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4d41391f0 0x7fe4d40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9476640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 0x7fe4d41dabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40638/0 (socket says 172.21.15.110:40638) 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9476640 1 -- 172.21.15.110:0/2213626162 learned_addr learned my addr 172.21.15.110:0/2213626162 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9476640 10 monclient(hunting): get_auth_request con 0x7fe4d40a96f0 auth_method 0 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9476640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9476640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9476640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d8c75640 10 monclient(hunting): get_auth_request con 0x7fe4d40a9ad0 auth_method 0 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d8c75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:38.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d8c75640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d8c75640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9c77640 10 monclient(hunting): get_auth_request con 0x7fe4d41391f0 auth_method 0 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9c77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9c77640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.055+0000 7fe4d9c77640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.056+0000 7fe4d8c75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.056+0000 7fe4d8c75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.056+0000 7fe4d8c75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.056+0000 7fe4d9476640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.056+0000 7fe4d9476640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.056+0000 7fe4d9476640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.056+0000 7fe4d9c77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.056+0000 7fe4d9c77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:38.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.056+0000 7fe4d9c77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.058+0000 7fe4d8c75640 10 monclient(hunting): handle_auth_done global_id 5164 payload 824 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.058+0000 7fe4d8c75640 1 -- 172.21.15.110:0/2213626162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4d41391f0 msgr2=0x7fe4d40af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.058+0000 7fe4d8c75640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4d41391f0 0x7fe4d40af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.059+0000 7fe4d8c75640 1 -- 172.21.15.110:0/2213626162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 msgr2=0x7fe4d41dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.059+0000 7fe4d8c75640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 0x7fe4d41dabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.059+0000 7fe4d8c75640 10 monclient: _finish_hunting 0 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.059+0000 7fe4d8c75640 1 monclient: found mon.noname-b 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.059+0000 7fe4d8c75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.059+0000 7fe4d8c75640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.059+0000 7fe4d8c75640 1 -- 172.21.15.110:0/2213626162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc009a70 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.061+0000 7fe4d8c75640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4d40a9ad0 0x7fe4d41db130 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40aac70 tx=0x7fe4c400c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.061+0000 7fe4c9ffb640 10 client.?.objecter ms_handle_connect 0x7fe4d40a9ad0 2024-01-20T11:19:38.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.061+0000 7fe4c9ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:38.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.064+0000 7fe4c9ffb640 1 -- 172.21.15.110:0/2213626162 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40052d0 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.064+0000 7fe4c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:38.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.064+0000 7fe4c9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:38.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.064+0000 7fe4c9ffb640 10 monclient: dump: 2024-01-20T11:19:38.067 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:38.067 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.064+0000 7fe4c9ffb640 10 monclient: _finish_auth 0 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.064+0000 7fe4c9ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.064+0000 7fe4c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4db6da640 5 monclient: authenticate success, global_id 5164 2024-01-20T11:19:38.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.064+0000 7fe4c9ffb640 1 -- 172.21.15.110:0/2213626162 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c4005450 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4db6da640 10 monclient: _renew_subs 2024-01-20T11:19:38.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4db6da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:38.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d40b0020 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:38.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4c9ffb640 1 -- 172.21.15.110:0/2213626162 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4007260 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:38.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:38.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4c9ffb640 10 monclient: dump: 2024-01-20T11:19:38.069 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4caffd640 4 set_mon_vals no callback set 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4db6da640 10 client.5164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4db6da640 10 monclient: _renew_subs 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4db6da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:38.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d40b0580 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.065+0000 7fe4db6da640 1 librados: init done 2024-01-20T11:19:38.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.066+0000 7fe4db6da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:38.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.066+0000 7fe4db6da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:38.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.066+0000 7fe4db6da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:38.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.066+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d4075490 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.085+0000 7fe4c9ffb640 1 -- 172.21.15.110:0/2213626162 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe4c40073c0 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.085+0000 7fe4c9ffb640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe4b806c890 0x7fe4b806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.085+0000 7fe4d9476640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe4b806c890 0x7fe4b806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.085+0000 7fe4d9476640 10 monclient: get_auth_request con 0x7fe4b806c890 auth_method 0 2024-01-20T11:19:38.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.086+0000 7fe4c9ffb640 10 client.5164.objecter ms_handle_connect 0x7fe4b806c890 2024-01-20T11:19:38.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.086+0000 7fe4d9476640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe4b806c890 0x7fe4b806ed50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc000f80 tx=0x7fe4cc030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:38.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.312+0000 7fe4c9ffb640 1 -- 172.21.15.110:0/2213626162 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe4c4057c80 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.312+0000 7fe4c9ffb640 10 client.5164.objecter ms_dispatch 0x7fe4d4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:38.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.312+0000 7fe4c9ffb640 3 client.5164.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:38.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.312+0000 7fe4c9ffb640 3 client.5164.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:38.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.312+0000 7fe4c9ffb640 20 client.5164.objecter dump_active .. 0 homeless 2024-01-20T11:19:38.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.324+0000 7fe4c9ffb640 1 -- 172.21.15.110:0/2213626162 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe4c4053fc0 con 0x7fe4d40a9ad0 2024-01-20T11:19:38.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.324+0000 7fe4c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:38.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.324+0000 7fe4c9ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:38.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.498+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4d413a8a0 con 0x7fe4b806c890 2024-01-20T11:19:38.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.500+0000 7fe4c9ffb640 1 -- 172.21.15.110:0/2213626162 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64041 (secure 0 0 0) 0x7fe4d413a8a0 con 0x7fe4b806c890 2024-01-20T11:19:38.501 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:38.501 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 10 librados: watch_flush enter 2024-01-20T11:19:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 10 librados: watch_flush exit 2024-01-20T11:19:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 20 client.5164.objecter shutdown clearing up homeless session... 2024-01-20T11:19:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 10 client.5164.objecter successfully canceled tick 2024-01-20T11:19:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe4b806c890 msgr2=0x7fe4b806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe4b806c890 0x7fe4b806ed50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc000f80 tx=0x7fe4cc030000 comp rx=0 tx=0).stop 2024-01-20T11:19:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 10 monclient: shutdown 2024-01-20T11:19:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4d40a9ad0 msgr2=0x7fe4d41db130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.502+0000 7fe4db6da640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4d40a9ad0 0x7fe4d41db130 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40aac70 tx=0x7fe4c400c820 comp rx=0 tx=0).stop 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.503+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 shutdown_connections 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.503+0000 7fe4db6da640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe4b806c890 0x7fe4b806ed50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.503+0000 7fe4db6da640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4d41391f0 0x7fe4d40af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.503+0000 7fe4db6da640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4d40a9ad0 0x7fe4d41db130 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.503+0000 7fe4db6da640 1 --2- 172.21.15.110:0/2213626162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4d40a96f0 0x7fe4d41dabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.503+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 >> 172.21.15.110:0/2213626162 conn(0x7fe4d40b7bc0 msgr2=0x7fe4d40b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.503+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 shutdown_connections 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.503+0000 7fe4db6da640 1 -- 172.21.15.110:0/2213626162 wait complete. 2024-01-20T11:19:38.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.503+0000 7fe4db6da640 1 librados: shutdown 2024-01-20T11:19:38.513 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-01-20T11:19:36.760553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002434"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.6210000000000004},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.3109999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.6389999999999993},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":13.706}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.65},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.862}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":15.443},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":9.2720000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":17.920999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":12.013999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":19.039000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":11.093999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313273,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.4269999999999996},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":10.565}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":12.585000000000001},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.9260000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.0570000000000004},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.321}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.183999999999999},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.9670000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.388},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":15.997999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.173},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":7.6790000000000003}]}]},{"osd":3,"up_from":21,"seq":90194313274,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.6820000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6429999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":13.377000000000001},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.042999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":9.2270000000000003},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.106}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.757},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":10.888999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.986000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.782},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.074999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444087,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.016},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":15.246}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":15.747},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.3399999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.568},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.6669999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:18:31 2024","interfaces":[{"interface":"back","average":{"1min":12.817,"5min":16.341000000000001,"15min":17.462},"min":{"1min":7.3869999999999996,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":15.111000000000001,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.766},{"interface":"front","average":{"1min":12.095000000000001,"5min":13.83,"15min":14.510999999999999},"min":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"max":{"1min":17.638000000000002,"5min":48.299999999999997,"15min":48.299999999999997},"last":11.898999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.602},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.9610000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.755000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":13.335000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444088,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.7919999999999998},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":8.5079999999999991}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.016},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.7480000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.432},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.872999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":11.438000000000001},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.175999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.5},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.603999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.6980000000000004},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.173}]}]},{"osd":0,"up_from":17,"seq":73014444088,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.5389999999999997},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":5.4989999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":12.337},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.1999999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":9.5210000000000008},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.249000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.045},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.4980000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":10.124000000000001},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":14.933}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.717000000000001},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":11.090999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.516},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.7309999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.423},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":10.218999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":14.609999999999999},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.156000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.992000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.3800000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.156000000000001},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.1079999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.834},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.272}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:38.514 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:38.514 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:38.515 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:38.515 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:53.007633+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:16:53.007633+0000', 'last_peered': '2024-01-20T11:16:53.007633+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:16:53.007633+0000', 'last_undegraded': '2024-01-20T11:16:53.007633+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:38.515 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:38.515 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:38.515 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:38.515 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:38.515 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:38.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.613+0000 7f4a0e22b640 10 monclient: get_monmap_and_config 2024-01-20T11:19:38.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.613+0000 7f4a0e22b640 10 monclient: build_initial_monmap 2024-01-20T11:19:38.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.613+0000 7f4a0e22b640 10 monclient: monmap: 2024-01-20T11:19:38.613 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:38.613 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.613 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:38.614091+0000 2024-01-20T11:19:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:38.614091+0000 2024-01-20T11:19:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:38.614 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 1 Processor -- start 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 1 -- start start 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient: init 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a080a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4a080a9ad0 0x7f4a081423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient(hunting): picked mon.noname-c con 0x7f4a080a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a08142920 0x7f4a0814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient(hunting): picked mon.noname-b con 0x7f4a08142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a080b25c0 con 0x7f4a080a96f0 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a080aaea0 con 0x7f4a080a9ad0 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0813fbe0 con 0x7f4a08142920 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.614+0000 7f4a0e22b640 10 monclient(hunting): authenticate will time out at 1621.965134s 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.615+0000 7f4a077fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4a080a96f0 0x7f4a0813f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55566/0 (socket says 172.21.15.110:55566) 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.615+0000 7f4a077fe640 1 -- 172.21.15.110:0/930762690 learned_addr learned my addr 172.21.15.110:0/930762690 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.615+0000 7f4a07fff640 1 --2- 172.21.15.110:0/930762690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a08142920 0x7f4a0814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a06ffd640 1 --2- 172.21.15.110:0/930762690 >> v2:172.21.15.134:3300/0 conn(0x7f4a080a9ad0 0x7f4a081423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a07fff640 10 monclient(hunting): get_auth_request con 0x7f4a08142920 auth_method 0 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a07fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a07fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:38.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a06ffd640 10 monclient(hunting): get_auth_request con 0x7f4a080a9ad0 auth_method 0 2024-01-20T11:19:38.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:38.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a06ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:38.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:38.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:38.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:38.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:38.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:38.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:38.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.616+0000 7f4a06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:38.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.618+0000 7f4a07fff640 10 monclient(hunting): handle_auth_done global_id 4901 payload 274 2024-01-20T11:19:38.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.618+0000 7f4a07fff640 1 -- 172.21.15.110:0/930762690 >> v2:172.21.15.134:3300/0 conn(0x7f4a080a9ad0 msgr2=0x7f4a081423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:38.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.618+0000 7f4a07fff640 1 --2- 172.21.15.110:0/930762690 >> v2:172.21.15.134:3300/0 conn(0x7f4a080a9ad0 0x7f4a081423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.618+0000 7f4a07fff640 1 -- 172.21.15.110:0/930762690 >> v1:172.21.15.110:6789/0 conn(0x7f4a080a96f0 legacy=0x7f4a0813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.618+0000 7f4a07fff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:38.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.618+0000 7f4a07fff640 1 monclient: found mon.noname-b 2024-01-20T11:19:38.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.618+0000 7f4a07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:38.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.618+0000 7f4a07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:38.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.618+0000 7f4a07fff640 1 -- 172.21.15.110:0/930762690 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a080ab0a0 con 0x7f4a08142920 2024-01-20T11:19:38.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.619+0000 7f4a07fff640 1 --2- 172.21.15.110:0/930762690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a08142920 0x7f4a0814c850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009d90 tx=0x7f4a0002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95b7eda6e5c98c64 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:38.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 1 -- 172.21.15.110:0/930762690 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0002cbc0 con 0x7f4a08142920 2024-01-20T11:19:38.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:38.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:38.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 10 monclient: dump: 2024-01-20T11:19:38.624 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 1 -- 172.21.15.110:0/930762690 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0002cd20 con 0x7f4a08142920 2024-01-20T11:19:38.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a0e22b640 5 monclient: authenticate success, global_id 4901 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a0e22b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a0e22b640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 1 -- 172.21.15.110:0/930762690 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a00034660 con 0x7f4a08142920 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a0e22b640 4 set_mon_vals no callback set 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.624+0000 7f4a067fc640 10 monclient: dump: 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.626 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:38.627 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:38.627 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:38.627 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 10 monclient: shutdown 2024-01-20T11:19:38.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:38.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 1 -- 172.21.15.110:0/930762690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a08142920 msgr2=0x7f4a0814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 1 --2- 172.21.15.110:0/930762690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a08142920 0x7f4a0814c850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009d90 tx=0x7f4a0002be10 comp rx=0 tx=0).stop 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 1 -- 172.21.15.110:0/930762690 shutdown_connections 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 1 --2- 172.21.15.110:0/930762690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a08142920 0x7f4a0814c850 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 1 --2- 172.21.15.110:0/930762690 >> v2:172.21.15.134:3300/0 conn(0x7f4a080a9ad0 0x7f4a081423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 1 -- 172.21.15.110:0/930762690 >> 172.21.15.110:0/930762690 conn(0x7f4a080b7bc0 msgr2=0x7f4a080b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 1 -- 172.21.15.110:0/930762690 shutdown_connections 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 1 -- 172.21.15.110:0/930762690 wait complete. 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 10 monclient: build_initial_monmap 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.625+0000 7f4a0e22b640 10 monclient: monmap: 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:38.629 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:38.630 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:38.630 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:38.630 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:38.630 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.630 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:38.630 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:38.630 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:38.630 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.626+0000 7f4a0e22b640 1 librados: starting msgr at 2024-01-20T11:19:38.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.626+0000 7f4a0e22b640 1 librados: starting objecter 2024-01-20T11:19:38.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.626+0000 7f4a0e22b640 1 Processor -- start 2024-01-20T11:19:38.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.626+0000 7f4a0e22b640 1 -- start start 2024-01-20T11:19:38.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.626+0000 7f4a0e22b640 1 librados: setting wanted keys 2024-01-20T11:19:38.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.626+0000 7f4a0e22b640 1 librados: calling monclient init 2024-01-20T11:19:38.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.626+0000 7f4a0e22b640 10 monclient: init 2024-01-20T11:19:38.631 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:29.651 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:38.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:38.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a080a96f0 0x7f4a08138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient(hunting): picked mon.noname-c con 0x7f4a080a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a080a9ad0 0x7f4a081dac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient(hunting): picked mon.noname-b con 0x7f4a080a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a08142920 0x7f4a080af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient(hunting): picked mon.noname-a con 0x7f4a08142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a081db920 con 0x7f4a08142920 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a081db640 con 0x7f4a080a9ad0 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a081db1a0 con 0x7f4a080a96f0 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a0e22b640 10 monclient(hunting): authenticate will time out at 1621.977491s 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.627+0000 7f4a077fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a080a96f0 0x7f4a08138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a077fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a080a96f0 0x7f4a08138fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:34540/0 (socket says 172.21.15.110:34540) 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a077fe640 1 -- 172.21.15.110:0/3096019971 learned_addr learned my addr 172.21.15.110:0/3096019971 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a077fe640 10 monclient(hunting): get_auth_request con 0x7f4a080a96f0 auth_method 0 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a077fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.628+0000 7f4a06ffd640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a080a9ad0 0x7f4a081dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.629+0000 7f4a06ffd640 10 monclient(hunting): get_auth_request con 0x7f4a080a9ad0 auth_method 0 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.629+0000 7f4a06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:38.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.629+0000 7f4a06ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.629+0000 7f4a06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 10 monclient(hunting): handle_auth_done global_id 4904 payload 824 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 1 -- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a080a9ad0 msgr2=0x7f4a081dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a080a9ad0 0x7f4a081dac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 1 -- 172.21.15.110:0/3096019971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a08142920 msgr2=0x7f4a080af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a08142920 0x7f4a080af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 1 monclient: found mon.noname-c 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.631+0000 7f4a077fe640 1 -- 172.21.15.110:0/3096019971 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a00009a70 con 0x7f4a080a96f0 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.633+0000 7f4a077fe640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a080a96f0 0x7f4a08138fd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f49f800c430 tx=0x7f49f800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.633+0000 7f49effff640 10 client.?.objecter ms_handle_connect 0x7f4a080a96f0 2024-01-20T11:19:38.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.633+0000 7f49effff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:38.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 1 -- 172.21.15.110:0/3096019971 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8005ce0 con 0x7f4a080a96f0 2024-01-20T11:19:38.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:38.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:38.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 10 monclient: dump: 2024-01-20T11:19:38.639 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:38.639 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.639 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.639 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 10 monclient: _finish_auth 0 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:38.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 1 -- 172.21.15.110:0/3096019971 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f8005e40 con 0x7f4a080a96f0 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a0e22b640 5 monclient: authenticate success, global_id 4904 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a0e22b640 10 monclient: _renew_subs 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a0e22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a081db420 con 0x7f4a080a96f0 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 1 -- 172.21.15.110:0/3096019971 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f80072f0 con 0x7f4a080a96f0 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f49effff640 10 monclient: dump: 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:38.641 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a05ffb640 4 set_mon_vals no callback set 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a0e22b640 10 client.4904.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a0e22b640 10 monclient: _renew_subs 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a0e22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a080afc10 con 0x7f4a080a96f0 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.636+0000 7f4a0e22b640 1 librados: init done 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.637+0000 7f4a0e22b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.637+0000 7f4a0e22b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.637+0000 7f4a0e22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:38.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.637+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a080b3da0 con 0x7f4a080a96f0 2024-01-20T11:19:38.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.656+0000 7f49effff640 1 -- 172.21.15.110:0/3096019971 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f49f8007510 con 0x7f4a080a96f0 2024-01-20T11:19:38.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.656+0000 7f49effff640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f49f406c860 0x7f49f406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:38.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.657+0000 7f4a06ffd640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f49f406c860 0x7f49f406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:38.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.657+0000 7f4a06ffd640 10 monclient: get_auth_request con 0x7f49f406c860 auth_method 0 2024-01-20T11:19:38.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.657+0000 7f4a06ffd640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f49f406c860 0x7f49f406ed20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4a080af030 tx=0x7f49fc008040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:38.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.657+0000 7f49effff640 10 client.4904.objecter ms_handle_connect 0x7f49f406c860 2024-01-20T11:19:38.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.660+0000 7f49effff640 1 -- 172.21.15.110:0/3096019971 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f49f8058350 con 0x7f4a080a96f0 2024-01-20T11:19:38.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.660+0000 7f49effff640 10 client.4904.objecter ms_dispatch 0x7f4a08000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:38.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.660+0000 7f49effff640 3 client.4904.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:38.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.660+0000 7f49effff640 3 client.4904.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:38.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.660+0000 7f49effff640 20 client.4904.objecter dump_active .. 0 homeless 2024-01-20T11:19:38.675 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:05:29.681 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:38.676 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:05:29.683 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:38.677 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:29.701 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:38.694 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:05:29.700 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:38.696 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:29.721 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:38.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.899+0000 7f49effff640 1 -- 172.21.15.110:0/3096019971 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f49f8053f90 con 0x7f4a080a96f0 2024-01-20T11:19:38.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.899+0000 7f49effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:38.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:38.899+0000 7f49effff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:38.955 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:29.979 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:38.957 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:29.982 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:39.048 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:30.072 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:39.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.162+0000 7f4a0e22b640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:39.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.162+0000 7f4a0e22b640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:39.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.162+0000 7f4a0e22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:39.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.162+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f4a08075490 con 0x7f4a080a96f0 2024-01-20T11:19:39.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.179+0000 7f49effff640 1 -- 172.21.15.110:0/3096019971 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f49f8017020 con 0x7f4a080a96f0 2024-01-20T11:19:39.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.179+0000 7f49effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:39.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.179+0000 7f49effff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:39.179 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:39.179 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":313,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:39.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 10 librados: watch_flush enter 2024-01-20T11:19:39.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 10 librados: watch_flush exit 2024-01-20T11:19:39.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 20 client.4904.objecter shutdown clearing up homeless session... 2024-01-20T11:19:39.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 10 client.4904.objecter successfully canceled tick 2024-01-20T11:19:39.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f49f406c860 msgr2=0x7f49f406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:39.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f49f406c860 0x7f49f406ed20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4a080af030 tx=0x7f49fc008040 comp rx=0 tx=0).stop 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 10 monclient: shutdown 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a080a96f0 msgr2=0x7f4a08138fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a080a96f0 0x7f4a08138fd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f49f800c430 tx=0x7f49f800c8f0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 shutdown_connections 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f49f406c860 0x7f49f406ed20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a08142920 0x7f4a080af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a080a9ad0 0x7f4a081dac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 --2- 172.21.15.110:0/3096019971 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a080a96f0 0x7f4a08138fd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 >> 172.21.15.110:0/3096019971 conn(0x7f4a080b7bc0 msgr2=0x7f4a080b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 shutdown_connections 2024-01-20T11:19:39.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.181+0000 7f4a0e22b640 1 -- 172.21.15.110:0/3096019971 wait complete. 2024-01-20T11:19:39.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.182+0000 7f4a0e22b640 1 librados: shutdown 2024-01-20T11:19:39.193 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:39.193 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:39.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.297+0000 7f5c8f84f640 10 monclient: get_monmap_and_config 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.297+0000 7f5c8f84f640 10 monclient: build_initial_monmap 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.297+0000 7f5c8f84f640 10 monclient: monmap: 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:39.297680+0000 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:39.297680+0000 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:39.297 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:39.298 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:39.298 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:39.298 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.297+0000 7f5c8f84f640 1 Processor -- start 2024-01-20T11:19:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.297+0000 7f5c8f84f640 1 -- start start 2024-01-20T11:19:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.297+0000 7f5c8f84f640 10 monclient: init 2024-01-20T11:19:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-01-20T11:19:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5c880a96f0 0x7f5c8813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:39.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 10 monclient(hunting): picked mon.noname-c con 0x7f5c880a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c8813fa10 0x7f5c881423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 10 monclient(hunting): picked mon.noname-a con 0x7f5c8813fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c880a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c880b25c0 con 0x7f5c880a9ad0 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c880aaea0 con 0x7f5c8813fa10 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c880ab0a0 con 0x7f5c880a96f0 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.298+0000 7f5c8f84f640 10 monclient(hunting): authenticate will time out at 1622.648724s 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8cdea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c8813fa10 0x7f5c881423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8d5eb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5c880a96f0 0x7f5c8813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:39.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8d5eb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5c880a96f0 0x7f5c8813f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:60302/0 (socket says 172.21.15.110:60302) 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8d5eb640 1 -- 172.21.15.110:0/3566910 learned_addr learned my addr 172.21.15.110:0/3566910 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8d5eb640 10 monclient(hunting): get_auth_request con 0x7f5c880a96f0 auth_method 0 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8d5eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8d5eb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8d5eb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8cdea640 10 monclient(hunting): get_auth_request con 0x7f5c8813fa10 auth_method 0 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8cdea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8cdea640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.299+0000 7f5c8cdea640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.300+0000 7f5c8cdea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.300+0000 7f5c8cdea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:39.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.300+0000 7f5c8cdea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:39.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.300+0000 7f5c8d5eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:39.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.300+0000 7f5c8d5eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:39.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.300+0000 7f5c8d5eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.301+0000 7f5c8cdea640 10 monclient(hunting): handle_auth_done global_id 5097 payload 274 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.301+0000 7f5c8cdea640 1 -- 172.21.15.110:0/3566910 >> v2:172.21.15.134:3300/0 conn(0x7f5c880a96f0 msgr2=0x7f5c8813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.301+0000 7f5c8cdea640 1 --2- 172.21.15.110:0/3566910 >> v2:172.21.15.134:3300/0 conn(0x7f5c880a96f0 0x7f5c8813f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.301+0000 7f5c8cdea640 1 -- 172.21.15.110:0/3566910 >> v1:172.21.15.134:6789/0 conn(0x7f5c880a9ad0 legacy=0x7f5c88142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.302+0000 7f5c8cdea640 10 monclient: _finish_hunting 0 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.302+0000 7f5c8cdea640 1 monclient: found mon.noname-a 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.302+0000 7f5c8cdea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.302+0000 7f5c8cdea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.302+0000 7f5c8cdea640 1 -- 172.21.15.110:0/3566910 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880b3fb0 con 0x7f5c8813fa10 2024-01-20T11:19:39.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.303+0000 7f5c8cdea640 1 --2- 172.21.15.110:0/3566910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c8813fa10 0x7f5c881423e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5c880b5720 tx=0x7f5c7802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5039f688dc3e2c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:39.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 1 -- 172.21.15.110:0/3566910 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7802e070 con 0x7f5c8813fa10 2024-01-20T11:19:39.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:39.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:39.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 10 monclient: dump: 2024-01-20T11:19:39.307 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 10 monclient: _finish_auth 0 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:39.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 1 -- 172.21.15.110:0/3566910 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c780315a0 con 0x7f5c8813fa10 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c8f84f640 5 monclient: authenticate success, global_id 5097 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c8f84f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c8f84f640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c8f84f640 4 set_mon_vals no callback set 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 1 -- 172.21.15.110:0/3566910 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c78031930 con 0x7f5c8813fa10 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.307+0000 7f5c7ffff640 10 monclient: dump: 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:39.309 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:39.310 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:39.310 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:39.310 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:39.310 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:39.310 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:39.310 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:39.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 10 monclient: shutdown 2024-01-20T11:19:39.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:39.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3566910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c8813fa10 msgr2=0x7f5c881423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:39.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 1 --2- 172.21.15.110:0/3566910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c8813fa10 0x7f5c881423e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5c880b5720 tx=0x7f5c7802b840 comp rx=0 tx=0).stop 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3566910 shutdown_connections 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 1 --2- 172.21.15.110:0/3566910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c8813fa10 0x7f5c881423e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 1 --2- 172.21.15.110:0/3566910 >> v2:172.21.15.134:3300/0 conn(0x7f5c880a96f0 0x7f5c8813f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3566910 >> 172.21.15.110:0/3566910 conn(0x7f5c880b7bc0 msgr2=0x7f5c880b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3566910 shutdown_connections 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.308+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3566910 wait complete. 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.309+0000 7f5c8f84f640 10 monclient: build_initial_monmap 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.309+0000 7f5c8f84f640 10 monclient: monmap: 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:39.314 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.309+0000 7f5c8f84f640 1 librados: starting msgr at 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.309+0000 7f5c8f84f640 1 librados: starting objecter 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.309+0000 7f5c8f84f640 1 Processor -- start 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.309+0000 7f5c8f84f640 1 -- start start 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.309+0000 7f5c8f84f640 1 librados: setting wanted keys 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.309+0000 7f5c8f84f640 1 librados: calling monclient init 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.309+0000 7f5c8f84f640 10 monclient: init 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:39.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c880a96f0 0x7f5c88137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient(hunting): picked mon.noname-b con 0x7f5c880a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c880a9ad0 0x7f5c8813a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient(hunting): picked mon.noname-c con 0x7f5c880a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c88135d70 0x7f5c8812f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient(hunting): picked mon.noname-a con 0x7f5c88135d70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c88139580 con 0x7f5c88135d70 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c88139780 con 0x7f5c880a96f0 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:39.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c88139a60 con 0x7f5c880a9ad0 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8f84f640 10 monclient(hunting): authenticate will time out at 1622.660398s 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8ddec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c88135d70 0x7f5c8812f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8cdea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c880a9ad0 0x7f5c8813a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8ddec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c88135d70 0x7f5c8812f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40654/0 (socket says 172.21.15.110:40654) 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8ddec640 1 -- 172.21.15.110:0/3696384937 learned_addr learned my addr 172.21.15.110:0/3696384937 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8ddec640 10 monclient(hunting): get_auth_request con 0x7f5c88135d70 auth_method 0 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8ddec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8ddec640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8ddec640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.310+0000 7f5c8d5eb640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c880a96f0 0x7f5c88137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8cdea640 10 monclient(hunting): get_auth_request con 0x7f5c880a9ad0 auth_method 0 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8cdea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:39.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8cdea640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8cdea640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8d5eb640 10 monclient(hunting): get_auth_request con 0x7f5c880a96f0 auth_method 0 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8d5eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8d5eb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8d5eb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8ddec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8ddec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8ddec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8d5eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8d5eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8d5eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8cdea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8cdea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:39.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.311+0000 7f5c8cdea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:39.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 10 monclient(hunting): handle_auth_done global_id 5179 payload 824 2024-01-20T11:19:39.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 1 -- 172.21.15.110:0/3696384937 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c880a9ad0 msgr2=0x7f5c8813a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:39.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c880a9ad0 0x7f5c8813a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 1 -- 172.21.15.110:0/3696384937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c88135d70 msgr2=0x7f5c8812f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:39.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c88135d70 0x7f5c8812f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 10 monclient: _finish_hunting 0 2024-01-20T11:19:39.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 1 monclient: found mon.noname-b 2024-01-20T11:19:39.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:39.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:39.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8d5eb640 1 -- 172.21.15.110:0/3696384937 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78009400 con 0x7f5c880a96f0 2024-01-20T11:19:39.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.314+0000 7f5c8ddec640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c88135d70 0x7f5c8812f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:39.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.316+0000 7f5c8d5eb640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c880a96f0 0x7f5c88137f80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5c80009db0 tx=0x7f5c8000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.316+0000 7f5c7dffb640 10 client.?.objecter ms_handle_connect 0x7f5c880a96f0 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.316+0000 7f5c7dffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.319+0000 7f5c7dffb640 1 -- 172.21.15.110:0/3696384937 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8000cda0 con 0x7f5c880a96f0 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.319+0000 7f5c7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.319+0000 7f5c7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.319+0000 7f5c7dffb640 10 monclient: dump: 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:39.323 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7dffb640 10 monclient: _finish_auth 0 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7dffb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7dffb640 1 -- 172.21.15.110:0/3696384937 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8000f040 con 0x7f5c880a96f0 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c8f84f640 5 monclient: authenticate success, global_id 5179 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7dffb640 1 -- 172.21.15.110:0/3696384937 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c80014680 con 0x7f5c880a96f0 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c8f84f640 10 monclient: _renew_subs 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c8f84f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:39.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c88139d40 con 0x7f5c880a96f0 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7f7fe640 4 set_mon_vals no callback set 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c7dffb640 10 monclient: dump: 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:39.325 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:39.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c8f84f640 10 client.5179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:39.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c8f84f640 10 monclient: _renew_subs 2024-01-20T11:19:39.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c8f84f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:39.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c881375b0 con 0x7f5c880a96f0 2024-01-20T11:19:39.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.320+0000 7f5c8f84f640 1 librados: init done 2024-01-20T11:19:39.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.321+0000 7f5c8f84f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:39.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.321+0000 7f5c8f84f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:39.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.321+0000 7f5c8f84f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:39.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.321+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c881430c0 con 0x7f5c880a96f0 2024-01-20T11:19:39.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.555+0000 7f5c7dffb640 1 -- 172.21.15.110:0/3696384937 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5c800147e0 con 0x7f5c880a96f0 2024-01-20T11:19:39.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.555+0000 7f5c7dffb640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6c06c8b0 0x7f5c6c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:39.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.555+0000 7f5c8cdea640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6c06c8b0 0x7f5c6c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:39.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.555+0000 7f5c8cdea640 10 monclient: get_auth_request con 0x7f5c6c06c8b0 auth_method 0 2024-01-20T11:19:39.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.556+0000 7f5c8cdea640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6c06c8b0 0x7f5c6c06ed70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5c78036000 tx=0x7f5c7803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:39.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.556+0000 7f5c7dffb640 10 client.5179.objecter ms_handle_connect 0x7f5c6c06c8b0 2024-01-20T11:19:39.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.559+0000 7f5c7dffb640 1 -- 172.21.15.110:0/3696384937 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5c800593c0 con 0x7f5c880a96f0 2024-01-20T11:19:39.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.559+0000 7f5c7dffb640 10 client.5179.objecter ms_dispatch 0x7f5c88000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:39.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.559+0000 7f5c7dffb640 3 client.5179.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:39.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.559+0000 7f5c7dffb640 3 client.5179.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:39.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.559+0000 7f5c7dffb640 20 client.5179.objecter dump_active .. 0 homeless 2024-01-20T11:19:39.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.569+0000 7f5c7dffb640 1 -- 172.21.15.110:0/3696384937 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5c80055700 con 0x7f5c880a96f0 2024-01-20T11:19:39.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.569+0000 7f5c7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:39.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.569+0000 7f5c7dffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:39.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.736+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5c88075490 con 0x7f5c6c06c8b0 2024-01-20T11:19:39.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.739+0000 7f5c7dffb640 1 -- 172.21.15.110:0/3696384937 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63953 (secure 0 0 0) 0x7f5c88075490 con 0x7f5c6c06c8b0 2024-01-20T11:19:39.739 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:39.740 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:39.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.740+0000 7f5c8f84f640 10 librados: watch_flush enter 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.740+0000 7f5c8f84f640 10 librados: watch_flush exit 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.740+0000 7f5c8f84f640 20 client.5179.objecter shutdown clearing up homeless session... 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.740+0000 7f5c8f84f640 10 client.5179.objecter successfully canceled tick 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6c06c8b0 msgr2=0x7f5c6c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6c06c8b0 0x7f5c6c06ed70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5c78036000 tx=0x7f5c7803d040 comp rx=0 tx=0).stop 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 10 monclient: shutdown 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c880a96f0 msgr2=0x7f5c88137f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c880a96f0 0x7f5c88137f80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5c80009db0 tx=0x7f5c8000c490 comp rx=0 tx=0).stop 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 shutdown_connections 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6c06c8b0 0x7f5c6c06ed70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c88135d70 0x7f5c8812f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c880a9ad0 0x7f5c8813a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 --2- 172.21.15.110:0/3696384937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c880a96f0 0x7f5c88137f80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:39.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 >> 172.21.15.110:0/3696384937 conn(0x7f5c880b7bc0 msgr2=0x7f5c880b6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:39.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 shutdown_connections 2024-01-20T11:19:39.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 -- 172.21.15.110:0/3696384937 wait complete. 2024-01-20T11:19:39.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:39.741+0000 7f5c8f84f640 1 librados: shutdown 2024-01-20T11:19:39.751 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-01-20T11:19:38.761134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002404"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.872506+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:16:52.872506+0000","last_peered":"2024-01-20T11:16:52.872506+0000","last_clean":"2024-01-20T11:16:52.872506+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:16:52.872506+0000","last_undegraded":"2024-01-20T11:16:52.872506+0000","last_fullsized":"2024-01-20T11:16:52.872506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.843947+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:16:52.843947+0000","last_peered":"2024-01-20T11:16:52.843947+0000","last_clean":"2024-01-20T11:16:52.843947+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:16:52.843947+0000","last_undegraded":"2024-01-20T11:16:52.843947+0000","last_fullsized":"2024-01-20T11:16:52.843947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:53.007633+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:16:53.007633+0000","last_peered":"2024-01-20T11:16:53.007633+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:16:53.007633+0000","last_undegraded":"2024-01-20T11:16:53.007633+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.848285+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:16:52.848285+0000","last_peered":"2024-01-20T11:16:52.848285+0000","last_clean":"2024-01-20T11:16:52.848285+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:16:52.848285+0000","last_undegraded":"2024-01-20T11:16:52.848285+0000","last_fullsized":"2024-01-20T11:16:52.848285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280571,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":10.397},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.766}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.0630000000000006},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":15.558999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.054},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.262}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.569000000000001},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":8.6359999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.167999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":6.9939999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":12.971},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":9.766}]}]},{"osd":7,"up_from":21,"seq":90194313273,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.4269999999999996},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":10.565}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":12.585000000000001},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.9260000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.0570000000000004},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.321}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.183999999999999},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.9670000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.388},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":15.997999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.173},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":7.6790000000000003}]}]},{"osd":3,"up_from":21,"seq":90194313274,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.6820000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6429999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":13.377000000000001},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.042999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":9.2270000000000003},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.106}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.757},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":10.888999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.986000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.782},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.074999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444088,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.9900000000000002},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":5.7460000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":13.677},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.1790000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.903},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.988}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.367000000000001},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.363}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.292},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":8.4990000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":16.218},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":12.427}]}]},{"osd":1,"up_from":17,"seq":73014444089,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.7919999999999998},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":8.5079999999999991}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.016},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.7480000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.432},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.872999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":11.438000000000001},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.175999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.5},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.603999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.6980000000000004},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.173}]}]},{"osd":0,"up_from":17,"seq":73014444089,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.609},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":7.0250000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":10.725},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.202}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":8.8089999999999993},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.6500000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.680999999999999},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.448}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":9.423},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":14.17}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.925000000000001},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":12.015000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.516},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.7309999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.423},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":10.218999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":14.609999999999999},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":12.156000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.992000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.3800000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.156000000000001},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.1079999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.834},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.272}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:41.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~43s 2024-01-20T11:19:42.754 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:42.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.854+0000 7faf397dc640 10 monclient: get_monmap_and_config 2024-01-20T11:19:42.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.854+0000 7faf397dc640 10 monclient: build_initial_monmap 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.854+0000 7faf397dc640 10 monclient: monmap: 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:42.855401+0000 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:42.855401+0000 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:42.855 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:42.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.855+0000 7faf397dc640 1 Processor -- start 2024-01-20T11:19:42.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.855+0000 7faf397dc640 1 -- start start 2024-01-20T11:19:42.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.855+0000 7faf397dc640 10 monclient: init 2024-01-20T11:19:42.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.855+0000 7faf397dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:42.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.855+0000 7faf397dc640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-01-20T11:19:42.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.855+0000 7faf397dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf340b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:42.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf397dc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf340b1ed0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:42.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf397dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf34146090 0x7faf340b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf397dc640 10 monclient(hunting): picked mon.noname-b con 0x7faf34146090 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf397dc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf340aade0 con 0x7faf340b4d10 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf397dc640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf34145460 con 0x7faf340b1ed0 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf397dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf397dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf340b50f0 con 0x7faf34146090 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf397dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf397dc640 10 monclient(hunting): authenticate will time out at 1626.206233s 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf337fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf34146090 0x7faf340b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf32ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7faf340b4d10 0x7faf340b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46614/0 (socket says 172.21.15.110:46614) 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.856+0000 7faf32ffd640 1 -- 172.21.15.110:0/3323719093 learned_addr learned my addr 172.21.15.110:0/3323719093 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.857+0000 7faf337fe640 10 monclient(hunting): get_auth_request con 0x7faf34146090 auth_method 0 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.857+0000 7faf337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.857+0000 7faf337fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.857+0000 7faf337fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:42.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.857+0000 7faf337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:42.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.857+0000 7faf337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:42.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.857+0000 7faf337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf31ffb640 1 -- 172.21.15.110:0/3323719093 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998785477 0 0) 0x7faf34145460 con 0x7faf340b1ed0 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf31ffb640 10 monclient(hunting): _negotiate 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf31ffb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf31ffb640 10 monclient(hunting): my global_id is 5182 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf31ffb640 1 -- 172.21.15.110:0/3323719093 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1c003520 con 0x7faf340b1ed0 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf337fe640 10 monclient(hunting): handle_auth_done global_id 4910 payload 274 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf337fe640 1 -- 172.21.15.110:0/3323719093 >> v1:172.21.15.134:6789/0 conn(0x7faf340b1ed0 legacy=0x7faf34145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf337fe640 1 -- 172.21.15.110:0/3323719093 >> v1:172.21.15.110:6789/0 conn(0x7faf340b4d10 legacy=0x7faf340b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:42.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf31ffb640 1 -- 172.21.15.110:0/3323719093 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501975579 0 0) 0x7faf340aade0 con 0x7faf340b4d10 2024-01-20T11:19:42.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf337fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:42.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf337fe640 1 monclient: found mon.noname-b 2024-01-20T11:19:42.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:42.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:42.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf337fe640 1 -- 172.21.15.110:0/3323719093 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf340b2660 con 0x7faf34146090 2024-01-20T11:19:42.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.859+0000 7faf31ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:19:42.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.861+0000 7faf337fe640 1 --2- 172.21.15.110:0/3323719093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf34146090 0x7faf340b3da0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009df0 tx=0x7faf2c02be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75003eff7250b49d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 1 -- 172.21.15.110:0/3323719093 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c02cc20 con 0x7faf34146090 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 10 monclient: dump: 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:42.865 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 10 monclient: _finish_auth 0 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 1 -- 172.21.15.110:0/3323719093 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2c02cd80 con 0x7faf34146090 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf397dc640 5 monclient: authenticate success, global_id 4910 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf397dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf397dc640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:42.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 1 -- 172.21.15.110:0/3323719093 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c034660 con 0x7faf34146090 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf397dc640 4 set_mon_vals no callback set 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf31ffb640 10 monclient: dump: 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:42.867 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:42.868 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:42.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf397dc640 10 monclient: shutdown 2024-01-20T11:19:42.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf397dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:42.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf397dc640 1 -- 172.21.15.110:0/3323719093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf34146090 msgr2=0x7faf340b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:42.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.865+0000 7faf397dc640 1 --2- 172.21.15.110:0/3323719093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf34146090 0x7faf340b3da0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009df0 tx=0x7faf2c02be70 comp rx=0 tx=0).stop 2024-01-20T11:19:42.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.866+0000 7faf397dc640 1 -- 172.21.15.110:0/3323719093 shutdown_connections 2024-01-20T11:19:42.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.866+0000 7faf397dc640 1 --2- 172.21.15.110:0/3323719093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf34146090 0x7faf340b3da0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:42.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.866+0000 7faf397dc640 1 -- 172.21.15.110:0/3323719093 >> 172.21.15.110:0/3323719093 conn(0x7faf34066590 msgr2=0x7faf340669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.866+0000 7faf397dc640 1 -- 172.21.15.110:0/3323719093 shutdown_connections 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.866+0000 7faf397dc640 1 -- 172.21.15.110:0/3323719093 wait complete. 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.866+0000 7faf397dc640 10 monclient: build_initial_monmap 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.866+0000 7faf397dc640 10 monclient: monmap: 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:42.872 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.867+0000 7faf397dc640 1 librados: starting msgr at 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.867+0000 7faf397dc640 1 librados: starting objecter 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.867+0000 7faf397dc640 1 Processor -- start 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.867+0000 7faf397dc640 1 -- start start 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.867+0000 7faf397dc640 1 librados: setting wanted keys 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.867+0000 7faf397dc640 1 librados: calling monclient init 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.867+0000 7faf397dc640 10 monclient: init 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf340b1ed0 0x7faf340ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient(hunting): picked mon.noname-c con 0x7faf340b1ed0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf340b4d10 0x7faf340ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:42.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient(hunting): picked mon.noname-a con 0x7faf340b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf34146090 0x7faf340ac600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient(hunting): picked mon.noname-b con 0x7faf34146090 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf34138bd0 con 0x7faf340b4d10 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf340ae180 con 0x7faf34146090 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf340ad7a0 con 0x7faf340b1ed0 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf397dc640 10 monclient(hunting): authenticate will time out at 1626.218559s 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf327fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf340b4d10 0x7faf340ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.868+0000 7faf32ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf340b1ed0 0x7faf340ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:42.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf337fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf34146090 0x7faf340ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf327fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf340b4d10 0x7faf340ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53500/0 (socket says 172.21.15.110:53500) 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf327fc640 1 -- 172.21.15.110:0/2377706844 learned_addr learned my addr 172.21.15.110:0/2377706844 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf327fc640 10 monclient(hunting): get_auth_request con 0x7faf340b4d10 auth_method 0 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf327fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf327fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf32ffd640 10 monclient(hunting): get_auth_request con 0x7faf340b1ed0 auth_method 0 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf32ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf337fe640 10 monclient(hunting): get_auth_request con 0x7faf34146090 auth_method 0 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf337fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf337fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:42.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:42.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:42.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:42.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:42.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:42.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:42.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:42.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.869+0000 7faf337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 10 monclient(hunting): handle_auth_done global_id 5185 payload 824 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 1 -- 172.21.15.110:0/2377706844 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf340b1ed0 msgr2=0x7faf340ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf340b1ed0 0x7faf340ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 1 -- 172.21.15.110:0/2377706844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf340b4d10 msgr2=0x7faf340ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf340b4d10 0x7faf340ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 1 monclient: found mon.noname-b 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:42.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf337fe640 1 -- 172.21.15.110:0/2377706844 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c009a70 con 0x7faf34146090 2024-01-20T11:19:42.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.872+0000 7faf327fc640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf340b4d10 0x7faf340ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:42.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.875+0000 7faf337fe640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf34146090 0x7faf340ac600 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faf2c033000 tx=0x7faf2c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:42.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.875+0000 7faf1b7fe640 10 client.?.objecter ms_handle_connect 0x7faf34146090 2024-01-20T11:19:42.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.875+0000 7faf1b7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:42.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 1 -- 172.21.15.110:0/2377706844 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c03e610 con 0x7faf34146090 2024-01-20T11:19:42.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:42.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:42.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 10 monclient: dump: 2024-01-20T11:19:42.879 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:42.879 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:42.879 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:42.879 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:42.879 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 1 -- 172.21.15.110:0/2377706844 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2c03e770 con 0x7faf34146090 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 1 -- 172.21.15.110:0/2377706844 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c03eb20 con 0x7faf34146090 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf397dc640 5 monclient: authenticate success, global_id 5185 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf397dc640 10 monclient: _renew_subs 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf397dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:42.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf340ada80 con 0x7faf34146090 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf1b7fe640 10 monclient: dump: 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf317fa640 4 set_mon_vals no callback set 2024-01-20T11:19:42.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf397dc640 10 client.5185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:42.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf397dc640 10 monclient: _renew_subs 2024-01-20T11:19:42.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf397dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:42.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf340abe40 con 0x7faf34146090 2024-01-20T11:19:42.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.879+0000 7faf397dc640 1 librados: init done 2024-01-20T11:19:42.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.880+0000 7faf397dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:42.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.880+0000 7faf397dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:42.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.880+0000 7faf397dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:42.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.880+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3413e8c0 con 0x7faf34146090 2024-01-20T11:19:42.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.898+0000 7faf1b7fe640 1 -- 172.21.15.110:0/2377706844 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7faf2c03b050 con 0x7faf34146090 2024-01-20T11:19:42.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.898+0000 7faf1b7fe640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf2006c8b0 0x7faf2006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:42.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.898+0000 7faf32ffd640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf2006c8b0 0x7faf2006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:42.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.898+0000 7faf32ffd640 10 monclient: get_auth_request con 0x7faf2006c8b0 auth_method 0 2024-01-20T11:19:42.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.899+0000 7faf1b7fe640 10 client.5185.objecter ms_handle_connect 0x7faf2006c8b0 2024-01-20T11:19:42.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.899+0000 7faf32ffd640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf2006c8b0 0x7faf2006ed70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7faf24000f60 tx=0x7faf24007d50 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:42.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.900+0000 7faf1b7fe640 1 -- 172.21.15.110:0/2377706844 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faf2c07abc0 con 0x7faf34146090 2024-01-20T11:19:42.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.900+0000 7faf1b7fe640 10 client.5185.objecter ms_dispatch 0x7faf34000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:42.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.900+0000 7faf1b7fe640 3 client.5185.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:42.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.900+0000 7faf1b7fe640 3 client.5185.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:42.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:42.900+0000 7faf1b7fe640 20 client.5185.objecter dump_active .. 0 homeless 2024-01-20T11:19:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.130+0000 7faf1b7fe640 1 -- 172.21.15.110:0/2377706844 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7faf2c076f00 con 0x7faf34146090 2024-01-20T11:19:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.130+0000 7faf1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.130+0000 7faf1b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:43.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.309+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faf3413e8c0 con 0x7faf2006c8b0 2024-01-20T11:19:43.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.312+0000 7faf1b7fe640 1 -- 172.21.15.110:0/2377706844 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63893 (secure 0 0 0) 0x7faf3413e8c0 con 0x7faf2006c8b0 2024-01-20T11:19:43.312 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:43.313 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 10 librados: watch_flush enter 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 10 librados: watch_flush exit 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 20 client.5185.objecter shutdown clearing up homeless session... 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 10 client.5185.objecter successfully canceled tick 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf2006c8b0 msgr2=0x7faf2006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf2006c8b0 0x7faf2006ed70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7faf24000f60 tx=0x7faf24007d50 comp rx=0 tx=0).stop 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 10 monclient: shutdown 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf34146090 msgr2=0x7faf340ac600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:43.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf34146090 0x7faf340ac600 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faf2c033000 tx=0x7faf2c004a40 comp rx=0 tx=0).stop 2024-01-20T11:19:43.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 shutdown_connections 2024-01-20T11:19:43.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf2006c8b0 0x7faf2006ed70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf34146090 0x7faf340ac600 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf340b4d10 0x7faf340ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 --2- 172.21.15.110:0/2377706844 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf340b1ed0 0x7faf340ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 >> 172.21.15.110:0/2377706844 conn(0x7faf34066590 msgr2=0x7faf340b83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:43.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 shutdown_connections 2024-01-20T11:19:43.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 -- 172.21.15.110:0/2377706844 wait complete. 2024-01-20T11:19:43.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.314+0000 7faf397dc640 1 librados: shutdown 2024-01-20T11:19:43.325 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-01-20T11:19:42.761922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280571,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":10.397},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.766}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.0630000000000006},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":15.558999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.054},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.262}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.569000000000001},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":8.6359999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.167999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":6.9939999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":12.971},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":9.766}]}]},{"osd":7,"up_from":21,"seq":90194313274,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6070000000000002},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":4.0730000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.685},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.013}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.7200000000000006},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.0129999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.558},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.643000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.353999999999999},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":7.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.741},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.122999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313275,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1170000000000009},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.9669999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.384},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.262}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":9.8539999999999992},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.423}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.693},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":7.117},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.167999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.067},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.698}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444088,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.9900000000000002},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":5.7460000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":13.677},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.1790000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.903},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.988}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.367000000000001},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.363}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.292},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":8.4990000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":16.218},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":12.427}]}]},{"osd":1,"up_from":17,"seq":73014444089,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.7919999999999998},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":8.5079999999999991}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.016},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.7480000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.432},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.872999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":11.438000000000001},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.175999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.5},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.603999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.6980000000000004},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.173}]}]},{"osd":0,"up_from":17,"seq":73014444089,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.609},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":7.0250000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":10.725},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.202}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":8.8089999999999993},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.6500000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.680999999999999},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.448}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":9.423},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":14.17}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.925000000000001},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":12.015000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.2510000000000003},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.4619999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.2729999999999997},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":10.125}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.385},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":6.9349999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.718999999999999},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.411}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.978999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.1519999999999992},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.792999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:43.326 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:43.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.429+0000 7fd027a90640 10 monclient: get_monmap_and_config 2024-01-20T11:19:43.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.429+0000 7fd027a90640 10 monclient: build_initial_monmap 2024-01-20T11:19:43.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.429+0000 7fd027a90640 10 monclient: monmap: 2024-01-20T11:19:43.431 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:43.431 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:43.431 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:43.429870+0000 2024-01-20T11:19:43.431 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:43.429870+0000 2024-01-20T11:19:43.431 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:43.431 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.429+0000 7fd027a90640 1 Processor -- start 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 1 -- start start 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 10 monclient: init 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:43.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd020066620 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd02006df70 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd020145b50 0x7fd02014d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 10 monclient(hunting): picked mon.noname-a con 0x7fd020145b50 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0200adf80 con 0x7fd020066620 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02006f650 con 0x7fd02006df70 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd020072880 con 0x7fd020145b50 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.430+0000 7fd027a90640 10 monclient(hunting): authenticate will time out at 1626.780838s 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.431+0000 7fd02602d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd020145b50 0x7fd02014d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:43.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.431+0000 7fd02502b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd02006df70 0x7fd020145440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:59890/0 (socket says 172.21.15.110:59890) 2024-01-20T11:19:43.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.431+0000 7fd02502b640 1 -- 172.21.15.110:0/1473849262 learned_addr learned my addr 172.21.15.110:0/1473849262 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:43.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.431+0000 7fd02602d640 10 monclient(hunting): get_auth_request con 0x7fd020145b50 auth_method 0 2024-01-20T11:19:43.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.431+0000 7fd02602d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:43.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.431+0000 7fd02602d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:43.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.431+0000 7fd02602d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:43.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.432+0000 7fd02602d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:43.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.432+0000 7fd02602d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.432+0000 7fd02602d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.433+0000 7fd02482a640 1 -- 172.21.15.110:0/1473849262 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135955100 0 0) 0x7fd02006f650 con 0x7fd02006df70 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.433+0000 7fd02482a640 10 monclient(hunting): _negotiate 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.433+0000 7fd02482a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.433+0000 7fd02482a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.433+0000 7fd02482a640 10 monclient(hunting): my global_id is 5188 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.433+0000 7fd02482a640 1 -- 172.21.15.110:0/1473849262 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd00c003520 con 0x7fd02006df70 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.434+0000 7fd02602d640 10 monclient(hunting): handle_auth_done global_id 5109 payload 274 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.434+0000 7fd02602d640 1 -- 172.21.15.110:0/1473849262 >> v1:172.21.15.134:6789/0 conn(0x7fd02006df70 legacy=0x7fd020145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.434+0000 7fd02602d640 1 -- 172.21.15.110:0/1473849262 >> v1:172.21.15.110:6789/0 conn(0x7fd020066620 legacy=0x7fd02013bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:43.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.434+0000 7fd02602d640 10 monclient: _finish_hunting 0 2024-01-20T11:19:43.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.434+0000 7fd02602d640 1 monclient: found mon.noname-a 2024-01-20T11:19:43.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.434+0000 7fd02602d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:43.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.434+0000 7fd02602d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:43.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.434+0000 7fd02602d640 1 -- 172.21.15.110:0/1473849262 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd020072be0 con 0x7fd020145b50 2024-01-20T11:19:43.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.438+0000 7fd02602d640 1 --2- 172.21.15.110:0/1473849262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 0x7fd02014d7e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd018009d90 tx=0x7fd01802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf00e2c9ea8a490 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:43.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 1 -- 172.21.15.110:0/1473849262 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01802cc60 con 0x7fd020145b50 2024-01-20T11:19:43.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 10 monclient: dump: 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:43.443 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 10 monclient: _finish_auth 0 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 10 monclient: _check_auth_tickets 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 1 -- 172.21.15.110:0/1473849262 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01802cdc0 con 0x7fd020145b50 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 1 -- 172.21.15.110:0/1473849262 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018034660 con 0x7fd020145b50 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd027a90640 5 monclient: authenticate success, global_id 5109 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd027a90640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd027a90640 4 set_mon_vals no callback set 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:43.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.442+0000 7fd02482a640 10 monclient: dump: 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:43.445 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.443+0000 7fd027a90640 10 monclient: shutdown 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.443+0000 7fd027a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.443+0000 7fd027a90640 1 -- 172.21.15.110:0/1473849262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 msgr2=0x7fd02014d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.443+0000 7fd027a90640 1 --2- 172.21.15.110:0/1473849262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 0x7fd02014d7e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd018009d90 tx=0x7fd01802beb0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.443+0000 7fd027a90640 1 -- 172.21.15.110:0/1473849262 shutdown_connections 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.443+0000 7fd027a90640 1 --2- 172.21.15.110:0/1473849262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 0x7fd02014d7e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.443+0000 7fd027a90640 1 -- 172.21.15.110:0/1473849262 >> 172.21.15.110:0/1473849262 conn(0x7fd0200b09e0 msgr2=0x7fd0200b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.443+0000 7fd027a90640 1 -- 172.21.15.110:0/1473849262 shutdown_connections 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.443+0000 7fd027a90640 1 -- 172.21.15.110:0/1473849262 wait complete. 2024-01-20T11:19:43.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.444+0000 7fd027a90640 10 monclient: build_initial_monmap 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.444+0000 7fd027a90640 10 monclient: monmap: 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.444+0000 7fd027a90640 1 librados: starting msgr at 2024-01-20T11:19:43.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.444+0000 7fd027a90640 1 librados: starting objecter 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.444+0000 7fd027a90640 1 Processor -- start 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.444+0000 7fd027a90640 1 -- start start 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.444+0000 7fd027a90640 1 librados: setting wanted keys 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.444+0000 7fd027a90640 1 librados: calling monclient init 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.444+0000 7fd027a90640 10 monclient: init 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd020066620 0x7fd0201dae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient(hunting): picked mon.noname-c con 0x7fd020066620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd02006df70 0x7fd020140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient(hunting): picked mon.noname-b con 0x7fd02006df70 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:43.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 0x7fd020141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient(hunting): picked mon.noname-a con 0x7fd020145b50 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0201db520 con 0x7fd020145b50 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd020142450 con 0x7fd02006df70 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd020148a40 con 0x7fd020066620 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd027a90640 10 monclient(hunting): authenticate will time out at 1626.795402s 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd02602d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 0x7fd020141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd02582c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd020066620 0x7fd0201dae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.445+0000 7fd02502b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd02006df70 0x7fd020140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:43.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02602d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 0x7fd020141350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53512/0 (socket says 172.21.15.110:53512) 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02502b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd02006df70 0x7fd020140e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:47936/0 (socket says 172.21.15.110:47936) 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02602d640 1 -- 172.21.15.110:0/2558049637 learned_addr learned my addr 172.21.15.110:0/2558049637 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02602d640 10 monclient(hunting): get_auth_request con 0x7fd020145b50 auth_method 0 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02602d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02602d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02602d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02502b640 10 monclient(hunting): get_auth_request con 0x7fd02006df70 auth_method 0 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02502b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02502b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02502b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02582c640 10 monclient(hunting): get_auth_request con 0x7fd020066620 auth_method 0 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02582c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:43.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02582c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02582c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02502b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02502b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02502b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02602d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02602d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02602d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02582c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02582c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.446+0000 7fd02582c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 10 monclient(hunting): handle_auth_done global_id 5191 payload 824 2024-01-20T11:19:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 1 -- 172.21.15.110:0/2558049637 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd020066620 msgr2=0x7fd0201dae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:43.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd020066620 0x7fd0201dae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 1 -- 172.21.15.110:0/2558049637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 msgr2=0x7fd020141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:43.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 0x7fd020141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 10 monclient: _finish_hunting 0 2024-01-20T11:19:43.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 1 monclient: found mon.noname-b 2024-01-20T11:19:43.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:43.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:43.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.449+0000 7fd02502b640 1 -- 172.21.15.110:0/2558049637 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd018009a70 con 0x7fd02006df70 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.451+0000 7fd02502b640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd02006df70 0x7fd020140e10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c430 tx=0x7fd01000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.451+0000 7fd0167fc640 10 client.?.objecter ms_handle_connect 0x7fd02006df70 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.451+0000 7fd0167fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 1 -- 172.21.15.110:0/2558049637 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd010005ce0 con 0x7fd02006df70 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 10 monclient: dump: 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:43.458 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 1 -- 172.21.15.110:0/2558049637 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd010005e40 con 0x7fd02006df70 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:43.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 1 -- 172.21.15.110:0/2558049637 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0100072d0 con 0x7fd02006df70 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd027a90640 5 monclient: authenticate success, global_id 5191 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd027a90640 10 monclient: _renew_subs 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd027a90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd020148d20 con 0x7fd02006df70 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd017fff640 4 set_mon_vals no callback set 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd0167fc640 10 monclient: dump: 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:43.460 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd027a90640 10 client.5191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd027a90640 10 monclient: _renew_subs 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd027a90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd020136c70 con 0x7fd02006df70 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.454+0000 7fd027a90640 1 librados: init done 2024-01-20T11:19:43.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.455+0000 7fd027a90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:43.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.455+0000 7fd027a90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:43.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.455+0000 7fd027a90640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:43.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.455+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0201db6d0 con 0x7fd02006df70 2024-01-20T11:19:43.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.693+0000 7fd0167fc640 1 -- 172.21.15.110:0/2558049637 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd010007430 con 0x7fd02006df70 2024-01-20T11:19:43.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.694+0000 7fd0167fc640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd00406c8b0 0x7fd00406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:43.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.694+0000 7fd02582c640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd00406c8b0 0x7fd00406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:43.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.694+0000 7fd02582c640 10 monclient: get_auth_request con 0x7fd00406c8b0 auth_method 0 2024-01-20T11:19:43.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.694+0000 7fd0167fc640 10 client.5191.objecter ms_handle_connect 0x7fd00406c8b0 2024-01-20T11:19:43.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.694+0000 7fd02582c640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd00406c8b0 0x7fd00406ed70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0016e0 tx=0x7fd01c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:43.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.697+0000 7fd0167fc640 1 -- 172.21.15.110:0/2558049637 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd010057ba0 con 0x7fd02006df70 2024-01-20T11:19:43.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.697+0000 7fd0167fc640 10 client.5191.objecter ms_dispatch 0x7fd020000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:43.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.697+0000 7fd0167fc640 3 client.5191.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:43.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.697+0000 7fd0167fc640 3 client.5191.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:43.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.697+0000 7fd0167fc640 20 client.5191.objecter dump_active .. 0 homeless 2024-01-20T11:19:43.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.708+0000 7fd0167fc640 1 -- 172.21.15.110:0/2558049637 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd010053ee0 con 0x7fd02006df70 2024-01-20T11:19:43.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.708+0000 7fd0167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:43.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.708+0000 7fd0167fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:43.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.887+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd020066a00 con 0x7fd00406c8b0 2024-01-20T11:19:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.890+0000 7fd0167fc640 1 -- 172.21.15.110:0/2558049637 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63893 (secure 0 0 0) 0x7fd01c001cc0 con 0x7fd00406c8b0 2024-01-20T11:19:43.890 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 10 librados: watch_flush enter 2024-01-20T11:19:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 10 librados: watch_flush exit 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 20 client.5191.objecter shutdown clearing up homeless session... 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 10 client.5191.objecter successfully canceled tick 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd00406c8b0 msgr2=0x7fd00406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd00406c8b0 0x7fd00406ed70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0016e0 tx=0x7fd01c009040 comp rx=0 tx=0).stop 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 10 monclient: shutdown 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd02006df70 msgr2=0x7fd020140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd02006df70 0x7fd020140e10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c430 tx=0x7fd01000c8f0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 shutdown_connections 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd00406c8b0 0x7fd00406ed70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd020145b50 0x7fd020141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd02006df70 0x7fd020140e10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 --2- 172.21.15.110:0/2558049637 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd020066620 0x7fd0201dae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 >> 172.21.15.110:0/2558049637 conn(0x7fd0200b09e0 msgr2=0x7fd0200b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 shutdown_connections 2024-01-20T11:19:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 -- 172.21.15.110:0/2558049637 wait complete. 2024-01-20T11:19:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:43.892+0000 7fd027a90640 1 librados: shutdown 2024-01-20T11:19:43.902 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-01-20T11:19:42.761922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280571,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":10.397},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.766}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.0630000000000006},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":15.558999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.054},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.262}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.569000000000001},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":8.6359999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.167999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":6.9939999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":12.971},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":9.766}]}]},{"osd":7,"up_from":21,"seq":90194313274,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6070000000000002},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":4.0730000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.685},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.013}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.7200000000000006},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.0129999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.558},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.643000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.353999999999999},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":7.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.741},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.122999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313275,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1170000000000009},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.9669999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.384},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.262}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":9.8539999999999992},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.423}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.693},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":7.117},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.167999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.067},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.698}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444088,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.9900000000000002},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":5.7460000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":13.677},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.1790000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.903},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.988}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.367000000000001},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.363}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.292},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":8.4990000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":16.218},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":12.427}]}]},{"osd":1,"up_from":17,"seq":73014444089,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.7919999999999998},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":8.5079999999999991}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.016},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.7480000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.432},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.872999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":11.438000000000001},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.175999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.5},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.603999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.6980000000000004},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.173}]}]},{"osd":0,"up_from":17,"seq":73014444089,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.609},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":7.0250000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":10.725},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.202}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":8.8089999999999993},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.6500000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.680999999999999},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.448}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":9.423},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":14.17}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.925000000000001},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":12.015000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.2510000000000003},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.4619999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.2729999999999997},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":10.125}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.385},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":6.9349999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.718999999999999},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.411}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.978999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.1519999999999992},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.792999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:43.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:43.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:43.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:43.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:43.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:43.905 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:43.905 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:43.905 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:43.905 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.004+0000 7fda2e415640 10 monclient: get_monmap_and_config 2024-01-20T11:19:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.004+0000 7fda2e415640 10 monclient: build_initial_monmap 2024-01-20T11:19:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.004+0000 7fda2e415640 10 monclient: monmap: 2024-01-20T11:19:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:44.005023+0000 2024-01-20T11:19:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:44.005023+0000 2024-01-20T11:19:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:44.005 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 1 Processor -- start 2024-01-20T11:19:44.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 1 -- start start 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient: init 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fda280b0c00 0x7fda280accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient(hunting): picked mon.noname-a con 0x7fda280b0c00 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda2813d0e0 0x7fda2813d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient(hunting): picked mon.noname-b con 0x7fda2813d0e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda28143200 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2806dcc0 con 0x7fda28143200 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda280ad450 con 0x7fda280b0c00 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda280ad8e0 con 0x7fda2813d0e0 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.005+0000 7fda2e415640 10 monclient(hunting): authenticate will time out at 1627.355941s 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.006+0000 7fda27fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fda280b0c00 0x7fda280accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.006+0000 7fda2c9b2640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fda28143200 0x7fda28143690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:59894/0 (socket says 172.21.15.110:59894) 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.006+0000 7fda2c9b2640 1 -- 172.21.15.110:0/2029947772 learned_addr learned my addr 172.21.15.110:0/2029947772 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.006+0000 7fda277fe640 1 --2- 172.21.15.110:0/2029947772 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda2813d0e0 0x7fda2813d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.006+0000 7fda277fe640 10 monclient(hunting): get_auth_request con 0x7fda2813d0e0 auth_method 0 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.006+0000 7fda277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.006+0000 7fda277fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.006+0000 7fda277fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda27fff640 10 monclient(hunting): get_auth_request con 0x7fda280b0c00 auth_method 0 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda27fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda27fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.007+0000 7fda27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.008+0000 7fda26ffd640 1 -- 172.21.15.110:0/2029947772 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079893723 0 0) 0x7fda2806dcc0 con 0x7fda28143200 2024-01-20T11:19:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.008+0000 7fda26ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:19:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.008+0000 7fda26ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.008+0000 7fda26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.008+0000 7fda26ffd640 10 monclient(hunting): my global_id is 5194 2024-01-20T11:19:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.008+0000 7fda26ffd640 1 -- 172.21.15.110:0/2029947772 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda10003520 con 0x7fda28143200 2024-01-20T11:19:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.009+0000 7fda277fe640 10 monclient(hunting): handle_auth_done global_id 4919 payload 274 2024-01-20T11:19:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.009+0000 7fda277fe640 1 -- 172.21.15.110:0/2029947772 >> v2:172.21.15.110:3300/0 conn(0x7fda280b0c00 msgr2=0x7fda280accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.009+0000 7fda277fe640 1 --2- 172.21.15.110:0/2029947772 >> v2:172.21.15.110:3300/0 conn(0x7fda280b0c00 0x7fda280accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.009+0000 7fda277fe640 1 -- 172.21.15.110:0/2029947772 >> v1:172.21.15.134:6789/0 conn(0x7fda28143200 legacy=0x7fda28143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.009+0000 7fda277fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.009+0000 7fda277fe640 1 monclient: found mon.noname-b 2024-01-20T11:19:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.009+0000 7fda277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.009+0000 7fda277fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.009+0000 7fda277fe640 1 -- 172.21.15.110:0/2029947772 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda28143da0 con 0x7fda2813d0e0 2024-01-20T11:19:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.011+0000 7fda277fe640 1 --2- 172.21.15.110:0/2029947772 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda2813d0e0 0x7fda2813d520 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0097a0 tx=0x7fda1c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5356e2b60d2206b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 1 -- 172.21.15.110:0/2029947772 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c02e070 con 0x7fda2813d0e0 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 10 monclient: dump: 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.015 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 10 monclient: _finish_auth 0 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 1 -- 172.21.15.110:0/2029947772 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c0315a0 con 0x7fda2813d0e0 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda2e415640 5 monclient: authenticate success, global_id 4919 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda2e415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda2e415640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 1 -- 172.21.15.110:0/2029947772 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c031930 con 0x7fda2813d0e0 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda2e415640 4 set_mon_vals no callback set 2024-01-20T11:19:44.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.015+0000 7fda26ffd640 10 monclient: dump: 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:44.017 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 10 monclient: shutdown 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 1 -- 172.21.15.110:0/2029947772 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda2813d0e0 msgr2=0x7fda2813d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 1 --2- 172.21.15.110:0/2029947772 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda2813d0e0 0x7fda2813d520 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0097a0 tx=0x7fda1c02b840 comp rx=0 tx=0).stop 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 1 -- 172.21.15.110:0/2029947772 shutdown_connections 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 1 --2- 172.21.15.110:0/2029947772 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda2813d0e0 0x7fda2813d520 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 1 --2- 172.21.15.110:0/2029947772 >> v2:172.21.15.110:3300/0 conn(0x7fda280b0c00 0x7fda280accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 1 -- 172.21.15.110:0/2029947772 >> 172.21.15.110:0/2029947772 conn(0x7fda280b3e60 msgr2=0x7fda280b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 1 -- 172.21.15.110:0/2029947772 shutdown_connections 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.016+0000 7fda2e415640 1 -- 172.21.15.110:0/2029947772 wait complete. 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.017+0000 7fda2e415640 10 monclient: build_initial_monmap 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.017+0000 7fda2e415640 10 monclient: monmap: 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.017+0000 7fda2e415640 1 librados: starting msgr at 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.017+0000 7fda2e415640 1 librados: starting objecter 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.017+0000 7fda2e415640 1 Processor -- start 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 1 -- start start 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 1 librados: setting wanted keys 2024-01-20T11:19:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 1 librados: calling monclient init 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient: init 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda280b0c00 0x7fda28132a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient(hunting): picked mon.noname-b con 0x7fda280b0c00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda2813d0e0 0x7fda28134490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient(hunting): picked mon.noname-a con 0x7fda2813d0e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda28143200 0x7fda281daaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient(hunting): picked mon.noname-c con 0x7fda28143200 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda281349d0 con 0x7fda2813d0e0 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda281da380 con 0x7fda280b0c00 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda281da660 con 0x7fda28143200 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.018+0000 7fda2e415640 10 monclient(hunting): authenticate will time out at 1627.368791s 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda280b0c00 0x7fda28132a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda277fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda2813d0e0 0x7fda28134490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda2c9b2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda28143200 0x7fda281daaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda280b0c00 0x7fda28132a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:47942/0 (socket says 172.21.15.110:47942) 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 1 -- 172.21.15.110:0/25582109 learned_addr learned my addr 172.21.15.110:0/25582109 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 10 monclient(hunting): get_auth_request con 0x7fda280b0c00 auth_method 0 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda277fe640 10 monclient(hunting): get_auth_request con 0x7fda2813d0e0 auth_method 0 2024-01-20T11:19:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda277fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda277fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda2c9b2640 10 monclient(hunting): get_auth_request con 0x7fda28143200 auth_method 0 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda2c9b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda2c9b2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda2c9b2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.019+0000 7fda27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.020+0000 7fda277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.020+0000 7fda277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.020+0000 7fda277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.020+0000 7fda2c9b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.020+0000 7fda2c9b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.020+0000 7fda2c9b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 10 monclient(hunting): handle_auth_done global_id 5121 payload 824 2024-01-20T11:19:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 1 -- 172.21.15.110:0/25582109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda28143200 msgr2=0x7fda281daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda28143200 0x7fda281daaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 1 -- 172.21.15.110:0/25582109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda280b0c00 msgr2=0x7fda28132a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda280b0c00 0x7fda28132a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 1 monclient: found mon.noname-a 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.022+0000 7fda277fe640 1 -- 172.21.15.110:0/25582109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c009400 con 0x7fda2813d0e0 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.025+0000 7fda277fe640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda2813d0e0 0x7fda28134490 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fda1c033000 tx=0x7fda1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.025+0000 7fda24ff9640 10 client.?.objecter ms_handle_connect 0x7fda2813d0e0 2024-01-20T11:19:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.025+0000 7fda24ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 1 -- 172.21.15.110:0/25582109 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c032630 con 0x7fda2813d0e0 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 10 monclient: dump: 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 10 monclient: _finish_auth 0 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 1 -- 172.21.15.110:0/25582109 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c0327b0 con 0x7fda2813d0e0 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 1 -- 172.21.15.110:0/25582109 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c032b60 con 0x7fda2813d0e0 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda24ff9640 10 monclient: dump: 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda2e415640 5 monclient: authenticate success, global_id 5121 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda2e415640 10 monclient: _renew_subs 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda2e415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda28133d90 con 0x7fda2813d0e0 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda267fc640 4 set_mon_vals no callback set 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda2e415640 10 client.5121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda2e415640 10 monclient: _renew_subs 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda2e415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda281db210 con 0x7fda2813d0e0 2024-01-20T11:19:44.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.028+0000 7fda2e415640 1 librados: init done 2024-01-20T11:19:44.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.029+0000 7fda2e415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:44.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.029+0000 7fda2e415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:44.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.029+0000 7fda2e415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:44.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.029+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda281da7f0 con 0x7fda2813d0e0 2024-01-20T11:19:44.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.266+0000 7fda24ff9640 1 -- 172.21.15.110:0/25582109 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fda1c04e430 con 0x7fda2813d0e0 2024-01-20T11:19:44.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.266+0000 7fda24ff9640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda1406c8e0 0x7fda1406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.266+0000 7fda27fff640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda1406c8e0 0x7fda1406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.266+0000 7fda27fff640 10 monclient: get_auth_request con 0x7fda1406c8e0 auth_method 0 2024-01-20T11:19:44.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.267+0000 7fda24ff9640 10 client.5121.objecter ms_handle_connect 0x7fda1406c8e0 2024-01-20T11:19:44.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.267+0000 7fda27fff640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda1406c8e0 0x7fda1406eda0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fda18009c00 tx=0x7fda18009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:44.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.277+0000 7fda24ff9640 1 -- 172.21.15.110:0/25582109 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fda1c07a6b0 con 0x7fda2813d0e0 2024-01-20T11:19:44.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.277+0000 7fda24ff9640 10 client.5121.objecter ms_dispatch 0x7fda28000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:44.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.277+0000 7fda24ff9640 3 client.5121.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:44.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.277+0000 7fda24ff9640 3 client.5121.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:44.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.277+0000 7fda24ff9640 20 client.5121.objecter dump_active .. 0 homeless 2024-01-20T11:19:44.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.277+0000 7fda24ff9640 1 -- 172.21.15.110:0/25582109 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fda1c0769f0 con 0x7fda2813d0e0 2024-01-20T11:19:44.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.277+0000 7fda24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:44.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.277+0000 7fda24ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:44.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.540+0000 7fda2e415640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:44.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.540+0000 7fda2e415640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:44.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.540+0000 7fda2e415640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:44.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.540+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fda28075490 con 0x7fda2813d0e0 2024-01-20T11:19:44.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.561+0000 7fda24ff9640 1 -- 172.21.15.110:0/25582109 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fda1c07a010 con 0x7fda2813d0e0 2024-01-20T11:19:44.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.561+0000 7fda24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:44.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.561+0000 7fda24ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:44.561 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:44.561 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":319,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:44.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.562+0000 7fda2e415640 10 librados: watch_flush enter 2024-01-20T11:19:44.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.562+0000 7fda2e415640 10 librados: watch_flush exit 2024-01-20T11:19:44.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.562+0000 7fda2e415640 20 client.5121.objecter shutdown clearing up homeless session... 2024-01-20T11:19:44.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.562+0000 7fda2e415640 10 client.5121.objecter successfully canceled tick 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda1406c8e0 msgr2=0x7fda1406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda1406c8e0 0x7fda1406eda0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fda18009c00 tx=0x7fda18009190 comp rx=0 tx=0).stop 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 10 monclient: shutdown 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda2813d0e0 msgr2=0x7fda28134490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda2813d0e0 0x7fda28134490 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fda1c033000 tx=0x7fda1c0046c0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 shutdown_connections 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda1406c8e0 0x7fda1406eda0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda28143200 0x7fda281daaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda2813d0e0 0x7fda28134490 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 --2- 172.21.15.110:0/25582109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda280b0c00 0x7fda28132a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 >> 172.21.15.110:0/25582109 conn(0x7fda280b3e60 msgr2=0x7fda2813c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 shutdown_connections 2024-01-20T11:19:44.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 -- 172.21.15.110:0/25582109 wait complete. 2024-01-20T11:19:44.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.563+0000 7fda2e415640 1 librados: shutdown 2024-01-20T11:19:44.573 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:44.574 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:44.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.672+0000 7f33fdaa9640 10 monclient: get_monmap_and_config 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.672+0000 7f33fdaa9640 10 monclient: build_initial_monmap 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.672+0000 7f33fdaa9640 10 monclient: monmap: 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:44.673336+0000 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:44.673336+0000 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:44.673 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.673+0000 7f33fdaa9640 1 Processor -- start 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.673+0000 7f33fdaa9640 1 -- start start 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.673+0000 7f33fdaa9640 10 monclient: init 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f33f80a96f0 0x7f33f813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 10 monclient(hunting): picked mon.noname-c con 0x7f33f80a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33f813fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f33f80a9ad0 0x7f33f8142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 10 monclient(hunting): picked mon.noname-a con 0x7f33f80a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f80b25c0 con 0x7f33f813fa10 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f80aaea0 con 0x7f33f80a9ad0 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f80ab0a0 con 0x7f33f80a96f0 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33fdaa9640 10 monclient(hunting): authenticate will time out at 1628.024308s 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33f7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f33f80a9ad0 0x7f33f8142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.674+0000 7f33f77fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f33f80a96f0 0x7f33f813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f77fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f33f80a96f0 0x7f33f813f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:47956/0 (socket says 172.21.15.110:47956) 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f77fe640 1 -- 172.21.15.110:0/3337561992 learned_addr learned my addr 172.21.15.110:0/3337561992 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f77fe640 10 monclient(hunting): get_auth_request con 0x7f33f80a96f0 auth_method 0 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f7fff640 10 monclient(hunting): get_auth_request con 0x7f33f80a9ad0 auth_method 0 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.675+0000 7f33f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.677+0000 7f33f77fe640 10 monclient(hunting): handle_auth_done global_id 5200 payload 274 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.677+0000 7f33f77fe640 1 -- 172.21.15.110:0/3337561992 >> v2:172.21.15.110:3300/0 conn(0x7f33f80a9ad0 msgr2=0x7f33f8142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.677+0000 7f33f77fe640 1 --2- 172.21.15.110:0/3337561992 >> v2:172.21.15.110:3300/0 conn(0x7f33f80a9ad0 0x7f33f8142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.677+0000 7f33f77fe640 1 -- 172.21.15.110:0/3337561992 >> v1:172.21.15.110:6789/0 conn(0x7f33f813fa10 legacy=0x7f33f81423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.677+0000 7f33f77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.677+0000 7f33f77fe640 1 monclient: found mon.noname-c 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.677+0000 7f33f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.677+0000 7f33f77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.677+0000 7f33f77fe640 1 -- 172.21.15.110:0/3337561992 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f80b3fb0 con 0x7f33f80a96f0 2024-01-20T11:19:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.679+0000 7f33f77fe640 1 --2- 172.21.15.110:0/3337561992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f80a96f0 0x7f33f813f4d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f33f80b5720 tx=0x7f33e802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff5d63781ae94269 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.682+0000 7f33f67fc640 1 -- 172.21.15.110:0/3337561992 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8031870 con 0x7f33f80a96f0 2024-01-20T11:19:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.682+0000 7f33f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.682+0000 7f33f67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.682+0000 7f33f67fc640 10 monclient: dump: 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33f67fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33f67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33f67fc640 1 -- 172.21.15.110:0/3337561992 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e80319d0 con 0x7f33f80a96f0 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33f67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33f67fc640 1 -- 172.21.15.110:0/3337561992 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8031d60 con 0x7f33f80a96f0 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33f67fc640 10 monclient: dump: 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:44.684 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:44.685 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:44.685 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 5 monclient: authenticate success, global_id 5200 2024-01-20T11:19:44.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:44.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 4 set_mon_vals no callback set 2024-01-20T11:19:44.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 10 monclient: shutdown 2024-01-20T11:19:44.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:44.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 1 -- 172.21.15.110:0/3337561992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f80a96f0 msgr2=0x7f33f813f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 1 --2- 172.21.15.110:0/3337561992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f80a96f0 0x7f33f813f4d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f33f80b5720 tx=0x7f33e802b910 comp rx=0 tx=0).stop 2024-01-20T11:19:44.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 1 -- 172.21.15.110:0/3337561992 shutdown_connections 2024-01-20T11:19:44.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 1 --2- 172.21.15.110:0/3337561992 >> v2:172.21.15.110:3300/0 conn(0x7f33f80a9ad0 0x7f33f8142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 1 --2- 172.21.15.110:0/3337561992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f80a96f0 0x7f33f813f4d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 1 -- 172.21.15.110:0/3337561992 >> 172.21.15.110:0/3337561992 conn(0x7f33f80b7bc0 msgr2=0x7f33f80b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:44.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 1 -- 172.21.15.110:0/3337561992 shutdown_connections 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.683+0000 7f33fdaa9640 1 -- 172.21.15.110:0/3337561992 wait complete. 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.684+0000 7f33fdaa9640 10 monclient: build_initial_monmap 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.684+0000 7f33fdaa9640 10 monclient: monmap: 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:44.689 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.684+0000 7f33fdaa9640 1 librados: starting msgr at 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.684+0000 7f33fdaa9640 1 librados: starting objecter 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.684+0000 7f33fdaa9640 1 Processor -- start 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 1 -- start start 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 1 librados: setting wanted keys 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 1 librados: calling monclient init 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient: init 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33f80a96f0 0x7f33f81375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient(hunting): picked mon.noname-c con 0x7f33f80a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f80a9ad0 0x7f33f8134e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient(hunting): picked mon.noname-a con 0x7f33f80a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f8135b10 0x7f33f8135380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient(hunting): picked mon.noname-b con 0x7f33f8135b10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f8134680 con 0x7f33f80a9ad0 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f8131fb0 con 0x7f33f8135b10 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f81358c0 con 0x7f33f80a96f0 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.685+0000 7f33fdaa9640 10 monclient(hunting): authenticate will time out at 1628.035772s 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f6ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f80a9ad0 0x7f33f8134e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33f80a96f0 0x7f33f81375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f8135b10 0x7f33f8135380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f8135b10 0x7f33f8135380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:47960/0 (socket says 172.21.15.110:47960) 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 1 -- 172.21.15.110:0/2194341014 learned_addr learned my addr 172.21.15.110:0/2194341014 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 10 monclient(hunting): get_auth_request con 0x7f33f8135b10 auth_method 0 2024-01-20T11:19:44.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f6ffd640 10 monclient(hunting): get_auth_request con 0x7f33f80a9ad0 auth_method 0 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f77fe640 10 monclient(hunting): get_auth_request con 0x7f33f80a96f0 auth_method 0 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.686+0000 7f33f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.687+0000 7f33f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.687+0000 7f33f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.687+0000 7f33f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.687+0000 7f33f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:44.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.687+0000 7f33f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:44.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.687+0000 7f33f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 10 monclient(hunting): handle_auth_done global_id 5203 payload 824 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 1 -- 172.21.15.110:0/2194341014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33f80a96f0 msgr2=0x7f33f81375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33f80a96f0 0x7f33f81375e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 1 -- 172.21.15.110:0/2194341014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f80a9ad0 msgr2=0x7f33f8134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f80a9ad0 0x7f33f8134e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 1 monclient: found mon.noname-b 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.689+0000 7f33f7fff640 1 -- 172.21.15.110:0/2194341014 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e80094d0 con 0x7f33f8135b10 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.691+0000 7f33f7fff640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f8135b10 0x7f33f8135380 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f33f000efc0 tx=0x7f33f000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.691+0000 7f33dffff640 10 client.?.objecter ms_handle_connect 0x7f33f8135b10 2024-01-20T11:19:44.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.691+0000 7f33dffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.694+0000 7f33dffff640 1 -- 172.21.15.110:0/2194341014 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0009ca0 con 0x7f33f8135b10 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 10 monclient: dump: 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.696 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 10 monclient: _finish_auth 0 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 1 -- 172.21.15.110:0/2194341014 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f000f040 con 0x7f33f8135b10 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33fdaa9640 5 monclient: authenticate success, global_id 5203 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33fdaa9640 10 monclient: _renew_subs 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 1 -- 172.21.15.110:0/2194341014 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f00146c0 con 0x7f33f8135b10 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33fdaa9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f8136b70 con 0x7f33f8135b10 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:44.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33dffff640 10 monclient: dump: 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33f57fa640 4 set_mon_vals no callback set 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33fdaa9640 10 client.5203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33fdaa9640 10 monclient: _renew_subs 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33fdaa9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f8137b20 con 0x7f33f8135b10 2024-01-20T11:19:44.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.695+0000 7f33fdaa9640 1 librados: init done 2024-01-20T11:19:44.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.696+0000 7f33fdaa9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:44.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.696+0000 7f33fdaa9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:44.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.696+0000 7f33fdaa9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:44.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.696+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f8134450 con 0x7f33f8135b10 2024-01-20T11:19:44.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.714+0000 7f33dffff640 1 -- 172.21.15.110:0/2194341014 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f33f0014820 con 0x7f33f8135b10 2024-01-20T11:19:44.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.715+0000 7f33dffff640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f33e406c8b0 0x7f33e406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:44.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.715+0000 7f33f77fe640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f33e406c8b0 0x7f33e406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:44.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.715+0000 7f33f77fe640 10 monclient: get_auth_request con 0x7f33e406c8b0 auth_method 0 2024-01-20T11:19:44.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.715+0000 7f33f77fe640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f33e406c8b0 0x7f33e406ed70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f33e8036000 tx=0x7f33e803c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:44.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.715+0000 7f33dffff640 10 client.5203.objecter ms_handle_connect 0x7f33e406c8b0 2024-01-20T11:19:44.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.717+0000 7f33dffff640 1 -- 172.21.15.110:0/2194341014 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f33f0059420 con 0x7f33f8135b10 2024-01-20T11:19:44.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.717+0000 7f33dffff640 10 client.5203.objecter ms_dispatch 0x7f33f8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:44.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.717+0000 7f33dffff640 3 client.5203.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:44.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.717+0000 7f33dffff640 3 client.5203.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:44.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.717+0000 7f33dffff640 20 client.5203.objecter dump_active .. 0 homeless 2024-01-20T11:19:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.945+0000 7f33dffff640 1 -- 172.21.15.110:0/2194341014 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f33f0055760 con 0x7f33f8135b10 2024-01-20T11:19:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.945+0000 7f33dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:44.945+0000 7f33dffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.124+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33f8134450 con 0x7f33e406c8b0 2024-01-20T11:19:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.127+0000 7f33dffff640 1 -- 172.21.15.110:0/2194341014 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63968 (secure 0 0 0) 0x7f33f8134450 con 0x7f33e406c8b0 2024-01-20T11:19:45.127 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:45.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 10 librados: watch_flush enter 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 10 librados: watch_flush exit 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 20 client.5203.objecter shutdown clearing up homeless session... 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 10 client.5203.objecter successfully canceled tick 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f33e406c8b0 msgr2=0x7f33e406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f33e406c8b0 0x7f33e406ed70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f33e8036000 tx=0x7f33e803c040 comp rx=0 tx=0).stop 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 10 monclient: shutdown 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f8135b10 msgr2=0x7f33f8135380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f8135b10 0x7f33f8135380 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f33f000efc0 tx=0x7f33f000c490 comp rx=0 tx=0).stop 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 shutdown_connections 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f33e406c8b0 0x7f33e406ed70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33f8135b10 0x7f33f8135380 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f80a9ad0 0x7f33f8134e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 --2- 172.21.15.110:0/2194341014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33f80a96f0 0x7f33f81375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 >> 172.21.15.110:0/2194341014 conn(0x7f33f80b7bc0 msgr2=0x7f33f80b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 shutdown_connections 2024-01-20T11:19:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 -- 172.21.15.110:0/2194341014 wait complete. 2024-01-20T11:19:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:45.129+0000 7f33fdaa9640 1 librados: shutdown 2024-01-20T11:19:45.139 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-01-20T11:19:44.762339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002474"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.748553+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:16:52.748553+0000","last_peered":"2024-01-20T11:16:52.748553+0000","last_clean":"2024-01-20T11:16:52.748553+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:16:52.748553+0000","last_undegraded":"2024-01-20T11:16:52.748553+0000","last_fullsized":"2024-01-20T11:16:52.748553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.9119999999999999},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.4459999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.862},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.233000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.236000000000001},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.1679999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.46},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":12.65}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.853},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":10.805}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.4169999999999998},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":14.51}]}]},{"osd":7,"up_from":21,"seq":90194313274,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.6070000000000002},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":4.0730000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.685},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.013}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.7200000000000006},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.0129999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.558},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.643000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.353999999999999},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":7.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.741},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.122999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313275,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1170000000000009},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.9669999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.384},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.262}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":9.8539999999999992},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.423}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.693},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":7.117},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.167999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.067},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.698}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444089,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":4.2640000000000002},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.4249999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.1229999999999993},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.307}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.7609999999999992},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":7.2149999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.750999999999999},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.718}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":12.994999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.342000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.9659999999999993},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":14.859999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.335000000000001},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":10.02}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":16.916},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.496}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":10.603999999999999},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.0289999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":13.095000000000001},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.939}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.5229999999999997},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":14.367000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.308},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.692}]}]},{"osd":0,"up_from":17,"seq":73014444090,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":7.2229999999999999},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":5.9260000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.6829999999999998},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.689}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":9.2300000000000004},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.0500000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.554},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.6349999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":11.202999999999999},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":11.731}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.851000000000001},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":9.8550000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.2510000000000003},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":7.4619999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.2729999999999997},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":10.125}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.385},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":6.9349999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.718999999999999},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.411}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.978999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.1519999999999992},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.792999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:47.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~50s 2024-01-20T11:19:48.142 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.242+0000 7f82c5f41640 10 monclient: get_monmap_and_config 2024-01-20T11:19:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.242+0000 7f82c5f41640 10 monclient: build_initial_monmap 2024-01-20T11:19:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.242+0000 7f82c5f41640 10 monclient: monmap: 2024-01-20T11:19:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:48.242694+0000 2024-01-20T11:19:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:48.242694+0000 2024-01-20T11:19:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:48.243 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.242+0000 7f82c5f41640 1 Processor -- start 2024-01-20T11:19:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.242+0000 7f82c5f41640 1 -- start start 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.242+0000 7f82c5f41640 10 monclient: init 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82c00a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82c00a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c014d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 10 monclient(hunting): picked mon.noname-b con 0x7f82c0142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c00b25c0 con 0x7f82c00a9ad0 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c0142f50 con 0x7f82c00a96f0 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c00aaea0 con 0x7f82c0142af0 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.243+0000 7f82c5f41640 10 monclient(hunting): authenticate will time out at 1631.593683s 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82bffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c014d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82beffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f82c00a9ad0 0x7f82c01423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46636/0 (socket says 172.21.15.110:46636) 2024-01-20T11:19:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82beffd640 1 -- 172.21.15.110:0/893919811 learned_addr learned my addr 172.21.15.110:0/893919811 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:48.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82bffff640 10 monclient(hunting): get_auth_request con 0x7f82c0142af0 auth_method 0 2024-01-20T11:19:48.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:48.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82bffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:48.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82bffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:48.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:48.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:48.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.244+0000 7f82bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:48.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.246+0000 7f82bffff640 10 monclient(hunting): handle_auth_done global_id 4928 payload 274 2024-01-20T11:19:48.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.246+0000 7f82bffff640 1 -- 172.21.15.110:0/893919811 >> v1:172.21.15.134:6789/0 conn(0x7f82c00a96f0 legacy=0x7f82c013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.246+0000 7f82bffff640 1 -- 172.21.15.110:0/893919811 >> v1:172.21.15.110:6789/0 conn(0x7f82c00a9ad0 legacy=0x7f82c01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.246+0000 7f82bffff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.246+0000 7f82bffff640 1 monclient: found mon.noname-b 2024-01-20T11:19:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.246+0000 7f82bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.246+0000 7f82bffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.246+0000 7f82bffff640 1 -- 172.21.15.110:0/893919811 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c013fbe0 con 0x7f82c0142af0 2024-01-20T11:19:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.248+0000 7f82bffff640 1 --2- 172.21.15.110:0/893919811 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c014d860 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009dc0 tx=0x7f82b802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d0b3ab164499451 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:48.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 1 -- 172.21.15.110:0/893919811 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b802cc40 con 0x7f82c0142af0 2024-01-20T11:19:48.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:48.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 10 monclient: dump: 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 10 monclient: _finish_auth 0 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:19:48.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 1 -- 172.21.15.110:0/893919811 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b802cda0 con 0x7f82c0142af0 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 1 -- 172.21.15.110:0/893919811 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b8034660 con 0x7f82c0142af0 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82c5f41640 5 monclient: authenticate success, global_id 4928 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82c5f41640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82c5f41640 4 set_mon_vals no callback set 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.251+0000 7f82be7fc640 10 monclient: dump: 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:48.253 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.254 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:48.254 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:48.254 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:48.254 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.252+0000 7f82c5f41640 10 monclient: shutdown 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.252+0000 7f82c5f41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.252+0000 7f82c5f41640 1 -- 172.21.15.110:0/893919811 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 msgr2=0x7f82c014d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.252+0000 7f82c5f41640 1 --2- 172.21.15.110:0/893919811 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c014d860 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009dc0 tx=0x7f82b802be90 comp rx=0 tx=0).stop 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.252+0000 7f82c5f41640 1 -- 172.21.15.110:0/893919811 shutdown_connections 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.252+0000 7f82c5f41640 1 --2- 172.21.15.110:0/893919811 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c014d860 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.252+0000 7f82c5f41640 1 -- 172.21.15.110:0/893919811 >> 172.21.15.110:0/893919811 conn(0x7f82c00b7bc0 msgr2=0x7f82c00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.252+0000 7f82c5f41640 1 -- 172.21.15.110:0/893919811 shutdown_connections 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.252+0000 7f82c5f41640 1 -- 172.21.15.110:0/893919811 wait complete. 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.253+0000 7f82c5f41640 10 monclient: build_initial_monmap 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.253+0000 7f82c5f41640 10 monclient: monmap: 2024-01-20T11:19:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.253+0000 7f82c5f41640 1 librados: starting msgr at 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.253+0000 7f82c5f41640 1 librados: starting objecter 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.253+0000 7f82c5f41640 1 Processor -- start 2024-01-20T11:19:48.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 1 -- start start 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 1 librados: setting wanted keys 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 1 librados: calling monclient init 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient: init 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82c00a96f0 0x7f82c01d7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient(hunting): picked mon.noname-b con 0x7f82c00a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82c00a9ad0 0x7f82c01351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient(hunting): picked mon.noname-a con 0x7f82c00a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c0135f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient(hunting): picked mon.noname-c con 0x7f82c0142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c01359e0 con 0x7f82c00a9ad0 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c0135700 con 0x7f82c00a96f0 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c0133030 con 0x7f82c0142af0 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.254+0000 7f82c5f41640 10 monclient(hunting): authenticate will time out at 1631.604713s 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c0135f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82beffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82c00a9ad0 0x7f82c01351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c0135f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:59136/0 (socket says 172.21.15.110:59136) 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 1 -- 172.21.15.110:0/635585516 learned_addr learned my addr 172.21.15.110:0/635585516 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 10 monclient(hunting): get_auth_request con 0x7f82c0142af0 auth_method 0 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82beffd640 10 monclient(hunting): get_auth_request con 0x7f82c00a9ad0 auth_method 0 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82beffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82beffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bf7fe640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82c00a96f0 0x7f82c01d7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bf7fe640 10 monclient(hunting): get_auth_request con 0x7f82c00a96f0 auth_method 0 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.255+0000 7f82bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.256+0000 7f82beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.256+0000 7f82beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.256+0000 7f82beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.256+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.256+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.256+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 10 monclient(hunting): handle_auth_done global_id 4931 payload 824 2024-01-20T11:19:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 1 -- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82c00a96f0 msgr2=0x7f82c01d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82c00a96f0 0x7f82c01d7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 1 -- 172.21.15.110:0/635585516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82c00a9ad0 msgr2=0x7f82c01351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82c00a9ad0 0x7f82c01351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 1 monclient: found mon.noname-c 2024-01-20T11:19:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.258+0000 7f82bffff640 1 -- 172.21.15.110:0/635585516 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8009a70 con 0x7f82c0142af0 2024-01-20T11:19:48.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.260+0000 7f82bffff640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c0135f00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f82b8037040 tx=0x7f82b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.260+0000 7f82a7fff640 10 client.?.objecter ms_handle_connect 0x7f82c0142af0 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.260+0000 7f82a7fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 1 -- 172.21.15.110:0/635585516 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b8040650 con 0x7f82c0142af0 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 10 monclient: dump: 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 10 monclient: _finish_auth 0 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 10 monclient: _check_auth_tickets 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 1 -- 172.21.15.110:0/635585516 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b80407b0 con 0x7f82c0142af0 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 1 -- 172.21.15.110:0/635585516 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b8040b60 con 0x7f82c0142af0 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82c5f41640 5 monclient: authenticate success, global_id 4931 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82c5f41640 10 monclient: _renew_subs 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82c5f41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c01331f0 con 0x7f82c0142af0 2024-01-20T11:19:48.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82bdffb640 4 set_mon_vals no callback set 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82a7fff640 10 monclient: dump: 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82c5f41640 10 client.4931.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82c5f41640 10 monclient: _renew_subs 2024-01-20T11:19:48.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82c5f41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:48.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c01d80b0 con 0x7f82c0142af0 2024-01-20T11:19:48.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.264+0000 7f82c5f41640 1 librados: init done 2024-01-20T11:19:48.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.265+0000 7f82c5f41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:48.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.265+0000 7f82c5f41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:48.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.265+0000 7f82c5f41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:48.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.265+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c014dd20 con 0x7f82c0142af0 2024-01-20T11:19:48.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.507+0000 7f82a7fff640 1 -- 172.21.15.110:0/635585516 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f82b8040cc0 con 0x7f82c0142af0 2024-01-20T11:19:48.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.508+0000 7f82a7fff640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f82ac06c8b0 0x7f82ac06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.508+0000 7f82bf7fe640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f82ac06c8b0 0x7f82ac06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.508+0000 7f82bf7fe640 10 monclient: get_auth_request con 0x7f82ac06c8b0 auth_method 0 2024-01-20T11:19:48.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.508+0000 7f82bf7fe640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f82ac06c8b0 0x7f82ac06ed70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f82b0001780 tx=0x7f82b0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:48.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.508+0000 7f82a7fff640 10 client.4931.objecter ms_handle_connect 0x7f82ac06c8b0 2024-01-20T11:19:48.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.536+0000 7f82a7fff640 1 -- 172.21.15.110:0/635585516 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f82b807b3d0 con 0x7f82c0142af0 2024-01-20T11:19:48.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.536+0000 7f82a7fff640 10 client.4931.objecter ms_dispatch 0x7f82c0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:48.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.536+0000 7f82a7fff640 3 client.4931.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:48.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.536+0000 7f82a7fff640 3 client.4931.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:48.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.536+0000 7f82a7fff640 20 client.4931.objecter dump_active .. 0 homeless 2024-01-20T11:19:48.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.536+0000 7f82a7fff640 1 -- 172.21.15.110:0/635585516 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f82b8077710 con 0x7f82c0142af0 2024-01-20T11:19:48.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.536+0000 7f82a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:48.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.536+0000 7f82a7fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:48.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.762+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82c00b4400 con 0x7f82ac06c8b0 2024-01-20T11:19:48.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.765+0000 7f82a7fff640 1 -- 172.21.15.110:0/635585516 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64022 (secure 0 0 0) 0x7f82b0001420 con 0x7f82ac06c8b0 2024-01-20T11:19:48.766 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:48.766 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:48.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.767+0000 7f82c5f41640 10 librados: watch_flush enter 2024-01-20T11:19:48.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.767+0000 7f82c5f41640 10 librados: watch_flush exit 2024-01-20T11:19:48.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.767+0000 7f82c5f41640 20 client.4931.objecter shutdown clearing up homeless session... 2024-01-20T11:19:48.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.767+0000 7f82c5f41640 10 client.4931.objecter successfully canceled tick 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f82ac06c8b0 msgr2=0x7f82ac06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f82ac06c8b0 0x7f82ac06ed70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f82b0001780 tx=0x7f82b0009040 comp rx=0 tx=0).stop 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 10 monclient: shutdown 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 msgr2=0x7f82c0135f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c0135f00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f82b8037040 tx=0x7f82b8004a40 comp rx=0 tx=0).stop 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 shutdown_connections 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f82ac06c8b0 0x7f82ac06ed70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82c0142af0 0x7f82c0135f00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82c00a9ad0 0x7f82c01351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 --2- 172.21.15.110:0/635585516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82c00a96f0 0x7f82c01d7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 >> 172.21.15.110:0/635585516 conn(0x7f82c00b7bc0 msgr2=0x7f82c00b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 shutdown_connections 2024-01-20T11:19:48.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 -- 172.21.15.110:0/635585516 wait complete. 2024-01-20T11:19:48.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.768+0000 7f82c5f41640 1 librados: shutdown 2024-01-20T11:19:48.779 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-01-20T11:19:48.763211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002657"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280573,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.0590000000000002},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.945}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.532999999999999},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.6440000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":8.0280000000000005},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.191000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.708},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.823}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":9.4600000000000009}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":7.2709999999999999},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":14.302}]}]},{"osd":7,"up_from":21,"seq":90194313275,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.4770000000000003},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":10.009}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":10.744},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":6.2240000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.5629999999999997},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":15.755000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":15.039},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":8.7799999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.076000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":13.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.443},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.869999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313276,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1170000000000009},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.9669999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.384},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.262}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":9.8539999999999992},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.423}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.693},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":7.117},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.167999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.067},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.698}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.9020000000000001},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.2679999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.3280000000000003},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.882999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.500999999999999},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.3889999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.112},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8209999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.680999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.131}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.959},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.388000000000002}]}]},{"osd":1,"up_from":17,"seq":73014444091,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.3849999999999998},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":9.7249999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.688000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.9749999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":9.0960000000000001},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.468999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":12.298999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.013}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.715},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.855}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.260999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.116}]}]},{"osd":0,"up_from":17,"seq":73014444091,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":4.3490000000000002},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.608000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":13.637},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.983000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":8.0060000000000002},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.3920000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.565},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.9390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":8.7300000000000004},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":16.207000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.324},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":11.616}]}]},{"osd":6,"up_from":15,"seq":64424509499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":7.0339999999999998},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.3479999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.3200000000000003},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":8.1270000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.188000000000001},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.223000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.952},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.343999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.805999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.340999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.728},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.774999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:48.781 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:48.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.880+0000 7f6bd167c640 10 monclient: get_monmap_and_config 2024-01-20T11:19:48.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.880+0000 7f6bd167c640 10 monclient: build_initial_monmap 2024-01-20T11:19:48.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.880+0000 7f6bd167c640 10 monclient: monmap: 2024-01-20T11:19:48.880 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:48.880 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.880 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:48.880715+0000 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:48.880715+0000 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.880+0000 7f6bd167c640 1 Processor -- start 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.880+0000 7f6bd167c640 1 -- start start 2024-01-20T11:19:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.880+0000 7f6bd167c640 10 monclient: init 2024-01-20T11:19:48.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:48.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-01-20T11:19:48.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a96f0 0x7f6bcc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 10 monclient(hunting): picked mon.noname-b con 0x7f6bcc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:48.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6bcc13fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:48.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bcc0a9ad0 0x7f6bcc142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 10 monclient(hunting): picked mon.noname-a con 0x7f6bcc0a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc0b25c0 con 0x7f6bcc13fa10 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bcc0aaea0 con 0x7f6bcc0a9ad0 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0ab0a0 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bd167c640 10 monclient(hunting): authenticate will time out at 1632.231499s 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bcaffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a96f0 0x7f6bcc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.881+0000 7f6bcb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bcc0a9ad0 0x7f6bcc142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bca7fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6bcc13fa10 0x7f6bcc1423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:59916/0 (socket says 172.21.15.110:59916) 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bca7fc640 1 -- 172.21.15.110:0/2775271556 learned_addr learned my addr 172.21.15.110:0/2775271556 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcaffd640 10 monclient(hunting): get_auth_request con 0x7f6bcc0a96f0 auth_method 0 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:48.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcb7fe640 10 monclient(hunting): get_auth_request con 0x7f6bcc0a9ad0 auth_method 0 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:48.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.882+0000 7f6bcb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:48.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcb7fe640 10 monclient(hunting): handle_auth_done global_id 5136 payload 274 2024-01-20T11:19:48.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcb7fe640 1 -- 172.21.15.110:0/2775271556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a96f0 msgr2=0x7f6bcc13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:48.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcb7fe640 1 --2- 172.21.15.110:0/2775271556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a96f0 0x7f6bcc13f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcb7fe640 1 -- 172.21.15.110:0/2775271556 >> v1:172.21.15.134:6789/0 conn(0x7f6bcc13fa10 legacy=0x7f6bcc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcb7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:19:48.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcb7fe640 1 monclient: found mon.noname-a 2024-01-20T11:19:48.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:48.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:48.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcb7fe640 1 -- 172.21.15.110:0/2775271556 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0b3fb0 con 0x7f6bcc0a9ad0 2024-01-20T11:19:48.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.884+0000 7f6bcaffd640 1 --2- 172.21.15.110:0/2775271556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a96f0 0x7f6bcc13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:48.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.886+0000 7f6bcb7fe640 1 --2- 172.21.15.110:0/2775271556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a9ad0 0x7f6bcc142af0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40097a0 tx=0x7f6bc402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b183dd994a865e26 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:48.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 1 -- 172.21.15.110:0/2775271556 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4031780 con 0x7f6bcc0a9ad0 2024-01-20T11:19:48.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:48.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:48.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 10 monclient: dump: 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 10 monclient: _finish_auth 0 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 1 -- 172.21.15.110:0/2775271556 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc40318e0 con 0x7f6bcc0a9ad0 2024-01-20T11:19:48.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 5 monclient: authenticate success, global_id 5136 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 4 set_mon_vals no callback set 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 1 -- 172.21.15.110:0/2775271556 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4031c70 con 0x7f6bcc0a9ad0 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bc9ffb640 10 monclient: dump: 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.892 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 10 monclient: shutdown 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 1 -- 172.21.15.110:0/2775271556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a9ad0 msgr2=0x7f6bcc142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 1 --2- 172.21.15.110:0/2775271556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a9ad0 0x7f6bcc142af0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40097a0 tx=0x7f6bc402b820 comp rx=0 tx=0).stop 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 1 -- 172.21.15.110:0/2775271556 shutdown_connections 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 1 --2- 172.21.15.110:0/2775271556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a9ad0 0x7f6bcc142af0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 1 --2- 172.21.15.110:0/2775271556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a96f0 0x7f6bcc13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 1 -- 172.21.15.110:0/2775271556 >> 172.21.15.110:0/2775271556 conn(0x7f6bcc0b7bc0 msgr2=0x7f6bcc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 1 -- 172.21.15.110:0/2775271556 shutdown_connections 2024-01-20T11:19:48.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.890+0000 7f6bd167c640 1 -- 172.21.15.110:0/2775271556 wait complete. 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.891+0000 7f6bd167c640 10 monclient: build_initial_monmap 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.891+0000 7f6bd167c640 10 monclient: monmap: 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.891+0000 7f6bd167c640 1 librados: starting msgr at 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.891+0000 7f6bd167c640 1 librados: starting objecter 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.891+0000 7f6bd167c640 1 Processor -- start 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 1 -- start start 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 1 librados: setting wanted keys 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 1 librados: calling monclient init 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient: init 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a96f0 0x7f6bcc12ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient(hunting): picked mon.noname-a con 0x7f6bcc0a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a9ad0 0x7f6bcc13d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient(hunting): picked mon.noname-c con 0x7f6bcc0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bcc141610 0x7f6bcc1d8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient(hunting): picked mon.noname-b con 0x7f6bcc141610 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc130de0 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc13de00 con 0x7f6bcc141610 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc1d8540 con 0x7f6bcc0a9ad0 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bd167c640 10 monclient(hunting): authenticate will time out at 1632.242750s 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.892+0000 7f6bcaffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a96f0 0x7f6bcc12ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bca7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a9ad0 0x7f6bcc13d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcaffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a96f0 0x7f6bcc12ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53576/0 (socket says 172.21.15.110:53576) 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcaffd640 1 -- 172.21.15.110:0/1967119441 learned_addr learned my addr 172.21.15.110:0/1967119441 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcaffd640 10 monclient(hunting): get_auth_request con 0x7f6bcc0a96f0 auth_method 0 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcb7fe640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bcc141610 0x7f6bcc1d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bca7fc640 10 monclient(hunting): get_auth_request con 0x7f6bcc0a9ad0 auth_method 0 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bca7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcb7fe640 10 monclient(hunting): get_auth_request con 0x7f6bcc141610 auth_method 0 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.893+0000 7f6bcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.894+0000 7f6bca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.894+0000 7f6bca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.894+0000 7f6bca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.894+0000 7f6bcb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.894+0000 7f6bcb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.894+0000 7f6bcb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 10 monclient(hunting): handle_auth_done global_id 5139 payload 824 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 1 -- 172.21.15.110:0/1967119441 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a9ad0 msgr2=0x7f6bcc13d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a9ad0 0x7f6bcc13d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 1 -- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bcc141610 msgr2=0x7f6bcc1d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bcc141610 0x7f6bcc1d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 10 monclient: _finish_hunting 0 2024-01-20T11:19:48.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 1 monclient: found mon.noname-a 2024-01-20T11:19:48.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:48.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:48.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.896+0000 7f6bcaffd640 1 -- 172.21.15.110:0/1967119441 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc009a70 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.899+0000 7f6bcaffd640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a96f0 0x7f6bcc12ffc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00efc0 tx=0x7f6bbc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.899+0000 7f6bb37fe640 10 client.?.objecter ms_handle_connect 0x7f6bcc0a96f0 2024-01-20T11:19:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.899+0000 7f6bb37fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 1 -- 172.21.15.110:0/1967119441 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc014680 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 10 monclient: dump: 2024-01-20T11:19:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 1 -- 172.21.15.110:0/1967119441 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc00f040 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:48.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 1 -- 172.21.15.110:0/1967119441 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc014a00 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bb37fe640 10 monclient: dump: 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:48.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bd167c640 5 monclient: authenticate success, global_id 5139 2024-01-20T11:19:48.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bd167c640 10 monclient: _renew_subs 2024-01-20T11:19:48.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bd167c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:48.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bc97fa640 4 set_mon_vals no callback set 2024-01-20T11:19:48.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.902+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc4009400 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.903+0000 7f6bd167c640 10 client.5139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:48.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.903+0000 7f6bd167c640 10 monclient: _renew_subs 2024-01-20T11:19:48.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.903+0000 7f6bd167c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:48.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.903+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bcc1d8a90 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.903+0000 7f6bd167c640 1 librados: init done 2024-01-20T11:19:48.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.903+0000 7f6bd167c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:48.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.903+0000 7f6bd167c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:48.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.903+0000 7f6bd167c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:48.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.903+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bcc1d8d10 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.923+0000 7f6bb37fe640 1 -- 172.21.15.110:0/1967119441 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6bbc014b60 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.923+0000 7f6bb37fe640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bb806c8b0 0x7f6bb806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:48.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.924+0000 7f6bca7fc640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bb806c8b0 0x7f6bb806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:48.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.924+0000 7f6bca7fc640 10 monclient: get_auth_request con 0x7f6bb806c8b0 auth_method 0 2024-01-20T11:19:48.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.924+0000 7f6bca7fc640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bb806c8b0 0x7f6bb806ed70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc1306a0 tx=0x7f6bc0007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:48.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.924+0000 7f6bb37fe640 10 client.5139.objecter ms_handle_connect 0x7f6bb806c8b0 2024-01-20T11:19:48.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.926+0000 7f6bb37fe640 1 -- 172.21.15.110:0/1967119441 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6bbc059570 con 0x7f6bcc0a96f0 2024-01-20T11:19:48.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.926+0000 7f6bb37fe640 10 client.5139.objecter ms_dispatch 0x7f6bcc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:48.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.926+0000 7f6bb37fe640 3 client.5139.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:48.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.926+0000 7f6bb37fe640 3 client.5139.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:48.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:48.926+0000 7f6bb37fe640 20 client.5139.objecter dump_active .. 0 homeless 2024-01-20T11:19:49.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.157+0000 7f6bb37fe640 1 -- 172.21.15.110:0/1967119441 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6bbc088e60 con 0x7f6bcc0a96f0 2024-01-20T11:19:49.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.157+0000 7f6bb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:49.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.157+0000 7f6bb37fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:49.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.332+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6bcc1d8d10 con 0x7f6bb806c8b0 2024-01-20T11:19:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.335+0000 7f6bb37fe640 1 -- 172.21.15.110:0/1967119441 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64022 (secure 0 0 0) 0x7f6bcc1d8d10 con 0x7f6bb806c8b0 2024-01-20T11:19:49.335 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 10 librados: watch_flush enter 2024-01-20T11:19:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 10 librados: watch_flush exit 2024-01-20T11:19:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 20 client.5139.objecter shutdown clearing up homeless session... 2024-01-20T11:19:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 10 client.5139.objecter successfully canceled tick 2024-01-20T11:19:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bb806c8b0 msgr2=0x7f6bb806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bb806c8b0 0x7f6bb806ed70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc1306a0 tx=0x7f6bc0007e80 comp rx=0 tx=0).stop 2024-01-20T11:19:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 10 monclient: shutdown 2024-01-20T11:19:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a96f0 msgr2=0x7f6bcc12ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a96f0 0x7f6bcc12ffc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00efc0 tx=0x7f6bbc00c490 comp rx=0 tx=0).stop 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 shutdown_connections 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6bb806c8b0 0x7f6bb806ed70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bcc141610 0x7f6bcc1d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bcc0a9ad0 0x7f6bcc13d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 --2- 172.21.15.110:0/1967119441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bcc0a96f0 0x7f6bcc12ffc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 >> 172.21.15.110:0/1967119441 conn(0x7f6bcc0b7bc0 msgr2=0x7f6bcc137380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 shutdown_connections 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 -- 172.21.15.110:0/1967119441 wait complete. 2024-01-20T11:19:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.337+0000 7f6bd167c640 1 librados: shutdown 2024-01-20T11:19:49.348 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-01-20T11:19:48.763211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002657"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280573,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.0590000000000002},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.945}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.532999999999999},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.6440000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":8.0280000000000005},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.191000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.708},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.823}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":9.4600000000000009}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":7.2709999999999999},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":14.302}]}]},{"osd":7,"up_from":21,"seq":90194313275,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.4770000000000003},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":10.009}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":10.744},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":6.2240000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.5629999999999997},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":15.755000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":15.039},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":8.7799999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.076000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":13.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.443},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.869999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313276,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1170000000000009},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.9669999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.384},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.262}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":9.8539999999999992},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.423}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.693},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":7.117},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.167999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.067},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.698}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.9020000000000001},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.2679999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.3280000000000003},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.882999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.500999999999999},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.3889999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.112},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8209999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.680999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.131}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.959},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.388000000000002}]}]},{"osd":1,"up_from":17,"seq":73014444091,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.3849999999999998},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":9.7249999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.688000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.9749999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":9.0960000000000001},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.468999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":12.298999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.013}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.715},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.855}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.260999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.116}]}]},{"osd":0,"up_from":17,"seq":73014444091,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":4.3490000000000002},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.608000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":13.637},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.983000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":8.0060000000000002},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.3920000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.565},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.9390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":8.7300000000000004},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":16.207000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.324},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":11.616}]}]},{"osd":6,"up_from":15,"seq":64424509499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":7.0339999999999998},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.3479999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.3200000000000003},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":8.1270000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.188000000000001},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.223000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.952},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.343999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.805999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.340999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.728},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.774999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:49.349 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:49.350 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:49.350 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:49.350 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:49.350 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:49.350 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:49.350 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:49.350 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:49.350 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:49.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.448+0000 7fc41c76c640 10 monclient: get_monmap_and_config 2024-01-20T11:19:49.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.448+0000 7fc41c76c640 10 monclient: build_initial_monmap 2024-01-20T11:19:49.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.448+0000 7fc41c76c640 10 monclient: monmap: 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:49.449328+0000 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:49.449328+0000 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:49.449 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.449+0000 7fc41c76c640 1 Processor -- start 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.449+0000 7fc41c76c640 1 -- start start 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.449+0000 7fc41c76c640 10 monclient: init 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc41406a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc41413d800 0x7fc414143390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 10 monclient(hunting): picked mon.noname-a con 0x7fc41413d800 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4140b0620 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4140b10f0 con 0x7fc4140b0620 2024-01-20T11:19:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc414072150 con 0x7fc41406a860 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41406ac40 con 0x7fc41413d800 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41c76c640 10 monclient(hunting): authenticate will time out at 1632.800369s 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc419d07640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc41413d800 0x7fc414143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41a508640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc41406a860 0x7fc41413d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:55400/0 (socket says 172.21.15.110:55400) 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.450+0000 7fc41a508640 1 -- 172.21.15.110:0/3414102245 learned_addr learned my addr 172.21.15.110:0/3414102245 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.451+0000 7fc419d07640 10 monclient(hunting): get_auth_request con 0x7fc41413d800 auth_method 0 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.451+0000 7fc419d07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.451+0000 7fc419d07640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.451+0000 7fc419d07640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.452+0000 7fc419d07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.452+0000 7fc419d07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.452+0000 7fc419d07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.453+0000 7fc419506640 1 -- 172.21.15.110:0/3414102245 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405220535 0 0) 0x7fc414072150 con 0x7fc41406a860 2024-01-20T11:19:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.453+0000 7fc419506640 10 monclient(hunting): _negotiate 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.453+0000 7fc419506640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.453+0000 7fc419506640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.453+0000 7fc419506640 10 monclient(hunting): my global_id is 5218 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.453+0000 7fc419506640 1 -- 172.21.15.110:0/3414102245 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc400003520 con 0x7fc41406a860 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.453+0000 7fc419d07640 10 monclient(hunting): handle_auth_done global_id 5142 payload 274 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.453+0000 7fc419d07640 1 -- 172.21.15.110:0/3414102245 >> v1:172.21.15.134:6789/0 conn(0x7fc41406a860 legacy=0x7fc41413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.453+0000 7fc419d07640 1 -- 172.21.15.110:0/3414102245 >> v1:172.21.15.110:6789/0 conn(0x7fc4140b0620 legacy=0x7fc4141438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.454+0000 7fc419d07640 10 monclient: _finish_hunting 0 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.454+0000 7fc419d07640 1 monclient: found mon.noname-a 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.454+0000 7fc419d07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.454+0000 7fc419d07640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.454+0000 7fc419d07640 1 -- 172.21.15.110:0/3414102245 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414071e70 con 0x7fc41413d800 2024-01-20T11:19:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.456+0000 7fc419d07640 1 --2- 172.21.15.110:0/3414102245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 0x7fc414143390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc4040097a0 tx=0x7fc40402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d08e6ef32c1183b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 1 -- 172.21.15.110:0/3414102245 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404030610 con 0x7fc41413d800 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 10 monclient: dump: 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 10 monclient: _finish_auth 0 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 10 monclient: _check_auth_tickets 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 1 -- 172.21.15.110:0/3414102245 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc404030770 con 0x7fc41413d800 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 1 -- 172.21.15.110:0/3414102245 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404030b00 con 0x7fc41413d800 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc419506640 10 monclient: dump: 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc41c76c640 5 monclient: authenticate success, global_id 5142 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc41c76c640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.461+0000 7fc41c76c640 4 set_mon_vals no callback set 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 10 monclient: shutdown 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 1 -- 172.21.15.110:0/3414102245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 msgr2=0x7fc414143390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 1 --2- 172.21.15.110:0/3414102245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 0x7fc414143390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc4040097a0 tx=0x7fc40402b960 comp rx=0 tx=0).stop 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 1 -- 172.21.15.110:0/3414102245 shutdown_connections 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 1 --2- 172.21.15.110:0/3414102245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 0x7fc414143390 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 1 -- 172.21.15.110:0/3414102245 >> 172.21.15.110:0/3414102245 conn(0x7fc4140b47d0 msgr2=0x7fc4140b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 1 -- 172.21.15.110:0/3414102245 shutdown_connections 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 1 -- 172.21.15.110:0/3414102245 wait complete. 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 10 monclient: build_initial_monmap 2024-01-20T11:19:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.462+0000 7fc41c76c640 10 monclient: monmap: 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 librados: starting msgr at 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 librados: starting objecter 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 Processor -- start 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 -- start start 2024-01-20T11:19:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 librados: setting wanted keys 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 librados: calling monclient init 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient: init 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41406a860 0x7fc4140adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient(hunting): picked mon.noname-b con 0x7fc41406a860 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4140b0620 0x7fc41406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient(hunting): picked mon.noname-c con 0x7fc4140b0620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 0x7fc41412e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient(hunting): picked mon.noname-a con 0x7fc41413d800 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc414070040 con 0x7fc41413d800 2024-01-20T11:19:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41406ed70 con 0x7fc41406a860 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41406eee0 con 0x7fc4140b0620 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.463+0000 7fc41c76c640 10 monclient(hunting): authenticate will time out at 1632.814044s 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41ad09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 0x7fc41412e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc419d07640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4140b0620 0x7fc41406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41a508640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41406a860 0x7fc4140adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41ad09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 0x7fc41412e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54536/0 (socket says 172.21.15.110:54536) 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41ad09640 1 -- 172.21.15.110:0/1134628841 learned_addr learned my addr 172.21.15.110:0/1134628841 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41ad09640 10 monclient(hunting): get_auth_request con 0x7fc41413d800 auth_method 0 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41ad09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41ad09640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41ad09640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc419d07640 10 monclient(hunting): get_auth_request con 0x7fc4140b0620 auth_method 0 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc419d07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc419d07640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc419d07640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41a508640 10 monclient(hunting): get_auth_request con 0x7fc41406a860 auth_method 0 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41a508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41a508640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.464+0000 7fc41a508640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.465+0000 7fc41ad09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.465+0000 7fc41ad09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.465+0000 7fc41ad09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.465+0000 7fc419d07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.465+0000 7fc419d07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.465+0000 7fc419d07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.465+0000 7fc41a508640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.465+0000 7fc41a508640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.465+0000 7fc41a508640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 10 monclient(hunting): handle_auth_done global_id 4940 payload 824 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 1 -- 172.21.15.110:0/1134628841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41406a860 msgr2=0x7fc4140adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41406a860 0x7fc4140adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 1 -- 172.21.15.110:0/1134628841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 msgr2=0x7fc41412e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 0x7fc41412e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 10 monclient: _finish_hunting 0 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 1 monclient: found mon.noname-c 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.468+0000 7fc419d07640 1 -- 172.21.15.110:0/1134628841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404009400 con 0x7fc4140b0620 2024-01-20T11:19:49.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.470+0000 7fc419d07640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4140b0620 0x7fc41406f5a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc41406af00 tx=0x7fc4040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:49.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.470+0000 7fc40affd640 10 client.?.objecter ms_handle_connect 0x7fc4140b0620 2024-01-20T11:19:49.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.470+0000 7fc40affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:49.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 1 -- 172.21.15.110:0/1134628841 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40403e070 con 0x7fc4140b0620 2024-01-20T11:19:49.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:49.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:49.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 10 monclient: dump: 2024-01-20T11:19:49.475 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:49.475 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 10 monclient: _finish_auth 0 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 1 -- 172.21.15.110:0/1134628841 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4040397a0 con 0x7fc4140b0620 2024-01-20T11:19:49.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 1 -- 172.21.15.110:0/1134628841 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404039b30 con 0x7fc4140b0620 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc40affd640 10 monclient: dump: 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:49.477 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc41c76c640 5 monclient: authenticate success, global_id 4940 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc41c76c640 10 monclient: _renew_subs 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.473+0000 7fc418d05640 4 set_mon_vals no callback set 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.474+0000 7fc41c76c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.474+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41406fae0 con 0x7fc4140b0620 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.474+0000 7fc41c76c640 10 client.4940.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.474+0000 7fc41c76c640 10 monclient: _renew_subs 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.474+0000 7fc41c76c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.474+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41412f040 con 0x7fc4140b0620 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.474+0000 7fc41c76c640 1 librados: init done 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.475+0000 7fc41c76c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.475+0000 7fc41c76c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.475+0000 7fc41c76c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:49.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.475+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4140718b0 con 0x7fc4140b0620 2024-01-20T11:19:49.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.494+0000 7fc40affd640 1 -- 172.21.15.110:0/1134628841 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc404039c90 con 0x7fc4140b0620 2024-01-20T11:19:49.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.494+0000 7fc40affd640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3f806c8b0 0x7fc3f806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:49.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.495+0000 7fc41a508640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3f806c8b0 0x7fc3f806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:49.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.495+0000 7fc41a508640 10 monclient: get_auth_request con 0x7fc3f806c8b0 auth_method 0 2024-01-20T11:19:49.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.495+0000 7fc41a508640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3f806c8b0 0x7fc3f806ed70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc4100011e0 tx=0x7fc410007d30 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:49.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.495+0000 7fc40affd640 10 client.4940.objecter ms_handle_connect 0x7fc3f806c8b0 2024-01-20T11:19:49.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.735+0000 7fc40affd640 1 -- 172.21.15.110:0/1134628841 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc40407aeb0 con 0x7fc4140b0620 2024-01-20T11:19:49.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.735+0000 7fc40affd640 10 client.4940.objecter ms_dispatch 0x7fc414000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:49.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.735+0000 7fc40affd640 3 client.4940.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:49.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.735+0000 7fc40affd640 3 client.4940.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:49.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.735+0000 7fc40affd640 20 client.4940.objecter dump_active .. 0 homeless 2024-01-20T11:19:49.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.735+0000 7fc40affd640 1 -- 172.21.15.110:0/1134628841 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc4040771f0 con 0x7fc4140b0620 2024-01-20T11:19:49.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.735+0000 7fc40affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:49.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.735+0000 7fc40affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:49.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.991+0000 7fc41c76c640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:49.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.992+0000 7fc41c76c640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:49.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.992+0000 7fc41c76c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:49.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.992+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc41413a950 con 0x7fc4140b0620 2024-01-20T11:19:49.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.997+0000 7fc40affd640 1 -- 172.21.15.110:0/1134628841 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fc404034520 con 0x7fc4140b0620 2024-01-20T11:19:49.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.997+0000 7fc40affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:49.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.997+0000 7fc40affd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:49.997 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:49.997 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":324,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:50.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:49.999+0000 7fc41c76c640 10 librados: watch_flush enter 2024-01-20T11:19:50.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 10 librados: watch_flush exit 2024-01-20T11:19:50.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 20 client.4940.objecter shutdown clearing up homeless session... 2024-01-20T11:19:50.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 10 client.4940.objecter successfully canceled tick 2024-01-20T11:19:50.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3f806c8b0 msgr2=0x7fc3f806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:50.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3f806c8b0 0x7fc3f806ed70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc4100011e0 tx=0x7fc410007d30 comp rx=0 tx=0).stop 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 10 monclient: shutdown 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4140b0620 msgr2=0x7fc41406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4140b0620 0x7fc41406f5a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc41406af00 tx=0x7fc4040046c0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 shutdown_connections 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3f806c8b0 0x7fc3f806ed70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc41413d800 0x7fc41412e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc4140b0620 0x7fc41406f5a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 --2- 172.21.15.110:0/1134628841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc41406a860 0x7fc4140adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 >> 172.21.15.110:0/1134628841 conn(0x7fc4140b47d0 msgr2=0x7fc41413d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 shutdown_connections 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 -- 172.21.15.110:0/1134628841 wait complete. 2024-01-20T11:19:50.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.000+0000 7fc41c76c640 1 librados: shutdown 2024-01-20T11:19:50.011 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:50.011 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:50.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.109+0000 7f119117d640 10 monclient: get_monmap_and_config 2024-01-20T11:19:50.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.109+0000 7f119117d640 10 monclient: build_initial_monmap 2024-01-20T11:19:50.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.109+0000 7f119117d640 10 monclient: monmap: 2024-01-20T11:19:50.109 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:50.110408+0000 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:50.110408+0000 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:50.110 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.110+0000 7f119117d640 1 Processor -- start 2024-01-20T11:19:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.110+0000 7f119117d640 1 -- start start 2024-01-20T11:19:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.110+0000 7f119117d640 10 monclient: init 2024-01-20T11:19:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-01-20T11:19:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f118c06f920 0x7f118c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 10 monclient(hunting): picked mon.noname-a con 0x7f118c06f920 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f118c13d230 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f118c144060 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0715b0 con 0x7f118c13d230 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c071890 con 0x7f118c144060 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118c071b70 con 0x7f118c06f920 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f119117d640 10 monclient(hunting): authenticate will time out at 1633.461413s 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f118ad9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f118c06f920 0x7f118c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f118b59e640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f118c144060 0x7f118c14a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:55414/0 (socket says 172.21.15.110:55414) 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.111+0000 7f118b59e640 1 -- 172.21.15.110:0/3916375219 learned_addr learned my addr 172.21.15.110:0/3916375219 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.112+0000 7f118ad9d640 10 monclient(hunting): get_auth_request con 0x7f118c06f920 auth_method 0 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.112+0000 7f118ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.112+0000 7f118ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:50.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.112+0000 7f118ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.113+0000 7f118ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.113+0000 7f118ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.113+0000 7f118ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.114+0000 7f1189d9b640 1 -- 172.21.15.110:0/3916375219 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206131305 0 0) 0x7f118c071890 con 0x7f118c144060 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.114+0000 7f1189d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.114+0000 7f1189d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.114+0000 7f1189d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.114+0000 7f1189d9b640 10 monclient(hunting): my global_id is 5224 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.114+0000 7f1189d9b640 1 -- 172.21.15.110:0/3916375219 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1174003520 con 0x7f118c144060 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.114+0000 7f118ad9d640 10 monclient(hunting): handle_auth_done global_id 5148 payload 274 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.115+0000 7f118ad9d640 1 -- 172.21.15.110:0/3916375219 >> v1:172.21.15.134:6789/0 conn(0x7f118c144060 legacy=0x7f118c14a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.115+0000 7f118ad9d640 1 -- 172.21.15.110:0/3916375219 >> v1:172.21.15.110:6789/0 conn(0x7f118c13d230 legacy=0x7f118c143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.115+0000 7f118ad9d640 10 monclient: _finish_hunting 0 2024-01-20T11:19:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.115+0000 7f118ad9d640 1 monclient: found mon.noname-a 2024-01-20T11:19:50.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.115+0000 7f118ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:50.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.115+0000 7f118ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:50.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.115+0000 7f118ad9d640 1 -- 172.21.15.110:0/3916375219 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c143430 con 0x7f118c06f920 2024-01-20T11:19:50.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.117+0000 7f118ad9d640 1 --2- 172.21.15.110:0/3916375219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 0x7f118c06fd00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f117c0097a0 tx=0x7f117c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2851fb059ef857 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 1 -- 172.21.15.110:0/3916375219 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c031830 con 0x7f118c06f920 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 10 monclient: dump: 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 10 monclient: _finish_auth 0 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f119117d640 5 monclient: authenticate success, global_id 5148 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 1 -- 172.21.15.110:0/3916375219 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c031990 con 0x7f118c06f920 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f119117d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f119117d640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f119117d640 4 set_mon_vals no callback set 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 1 -- 172.21.15.110:0/3916375219 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c031d20 con 0x7f118c06f920 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f1189d9b640 10 monclient: dump: 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:50.123 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:50.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f119117d640 10 monclient: shutdown 2024-01-20T11:19:50.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f119117d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:50.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f119117d640 1 -- 172.21.15.110:0/3916375219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 msgr2=0x7f118c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:50.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.121+0000 7f119117d640 1 --2- 172.21.15.110:0/3916375219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 0x7f118c06fd00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f117c0097a0 tx=0x7f117c02b8d0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.122+0000 7f119117d640 1 -- 172.21.15.110:0/3916375219 shutdown_connections 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.122+0000 7f119117d640 1 --2- 172.21.15.110:0/3916375219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 0x7f118c06fd00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.122+0000 7f119117d640 1 -- 172.21.15.110:0/3916375219 >> 172.21.15.110:0/3916375219 conn(0x7f118c066d60 msgr2=0x7f118c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.122+0000 7f119117d640 1 -- 172.21.15.110:0/3916375219 shutdown_connections 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.122+0000 7f119117d640 1 -- 172.21.15.110:0/3916375219 wait complete. 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.122+0000 7f119117d640 10 monclient: build_initial_monmap 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.122+0000 7f119117d640 10 monclient: monmap: 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.123+0000 7f119117d640 1 librados: starting msgr at 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.123+0000 7f119117d640 1 librados: starting objecter 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.123+0000 7f119117d640 1 Processor -- start 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.123+0000 7f119117d640 1 -- start start 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.123+0000 7f119117d640 1 librados: setting wanted keys 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.123+0000 7f119117d640 1 librados: calling monclient init 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.123+0000 7f119117d640 10 monclient: init 2024-01-20T11:19:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.123+0000 7f119117d640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 0x7f118c06c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 10 monclient(hunting): picked mon.noname-a con 0x7f118c06f920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f118c13d230 0x7f118c13efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 10 monclient(hunting): picked mon.noname-c con 0x7f118c13d230 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f118c144060 0x7f118c13f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 10 monclient(hunting): picked mon.noname-b con 0x7f118c144060 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c06ce80 con 0x7f118c06f920 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c13fa60 con 0x7f118c144060 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c13e8e0 con 0x7f118c13d230 2024-01-20T11:19:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f119117d640 10 monclient(hunting): authenticate will time out at 1633.474277s 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f118ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 0x7f118c06c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f118b59e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f118c144060 0x7f118c13f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f118a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f118c13d230 0x7f118c13efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f118ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 0x7f118c06c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54560/0 (socket says 172.21.15.110:54560) 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f118ad9d640 1 -- 172.21.15.110:0/1884374549 learned_addr learned my addr 172.21.15.110:0/1884374549 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f118ad9d640 10 monclient(hunting): get_auth_request con 0x7f118c06f920 auth_method 0 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f118ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f118ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.124+0000 7f118ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118a59c640 10 monclient(hunting): get_auth_request con 0x7f118c13d230 auth_method 0 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118b59e640 10 monclient(hunting): get_auth_request con 0x7f118c144060 auth_method 0 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118b59e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118b59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.125+0000 7f118a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 10 monclient(hunting): handle_auth_done global_id 5227 payload 824 2024-01-20T11:19:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 1 -- 172.21.15.110:0/1884374549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f118c13d230 msgr2=0x7f118c13efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f118c13d230 0x7f118c13efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 1 -- 172.21.15.110:0/1884374549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 msgr2=0x7f118c06c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 0x7f118c06c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 10 monclient: _finish_hunting 0 2024-01-20T11:19:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 1 monclient: found mon.noname-b 2024-01-20T11:19:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.128+0000 7f118b59e640 1 -- 172.21.15.110:0/1884374549 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c009400 con 0x7f118c144060 2024-01-20T11:19:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.130+0000 7f118b59e640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f118c144060 0x7f118c13f520 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f118400caa0 tx=0x7f118400cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.130+0000 7f11737fe640 10 client.?.objecter ms_handle_connect 0x7f118c144060 2024-01-20T11:19:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.130+0000 7f11737fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.133+0000 7f11737fe640 1 -- 172.21.15.110:0/1884374549 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118400ab40 con 0x7f118c144060 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 10 monclient: dump: 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 1 -- 172.21.15.110:0/1884374549 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118400aca0 con 0x7f118c144060 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 1 -- 172.21.15.110:0/1884374549 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184012600 con 0x7f118c144060 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f119117d640 5 monclient: authenticate success, global_id 5227 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f119117d640 10 monclient: _renew_subs 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f119117d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c13ec20 con 0x7f118c144060 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f11737fe640 10 monclient: dump: 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f118959a640 4 set_mon_vals no callback set 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f119117d640 10 client.5227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f119117d640 10 monclient: _renew_subs 2024-01-20T11:19:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f119117d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:50.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c06c9f0 con 0x7f118c144060 2024-01-20T11:19:50.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.134+0000 7f119117d640 1 librados: init done 2024-01-20T11:19:50.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.135+0000 7f119117d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:50.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.135+0000 7f119117d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:50.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.135+0000 7f119117d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:50.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.135+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c075490 con 0x7f118c144060 2024-01-20T11:19:50.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.153+0000 7f11737fe640 1 -- 172.21.15.110:0/1884374549 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1184007070 con 0x7f118c144060 2024-01-20T11:19:50.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.153+0000 7f11737fe640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f117806c8b0 0x7f117806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:50.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.153+0000 7f118ad9d640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f117806c8b0 0x7f117806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:50.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.153+0000 7f118ad9d640 10 monclient: get_auth_request con 0x7f117806c8b0 auth_method 0 2024-01-20T11:19:50.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.154+0000 7f118ad9d640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f117806c8b0 0x7f117806ed70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f117c02b910 tx=0x7f117c03c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:50.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.154+0000 7f11737fe640 10 client.5227.objecter ms_handle_connect 0x7f117806c8b0 2024-01-20T11:19:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.155+0000 7f11737fe640 1 -- 172.21.15.110:0/1884374549 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f11840585c0 con 0x7f118c144060 2024-01-20T11:19:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.155+0000 7f11737fe640 10 client.5227.objecter ms_dispatch 0x7f118c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.155+0000 7f11737fe640 3 client.5227.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.155+0000 7f11737fe640 3 client.5227.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.155+0000 7f11737fe640 20 client.5227.objecter dump_active .. 0 homeless 2024-01-20T11:19:50.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.391+0000 7f11737fe640 1 -- 172.21.15.110:0/1884374549 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1184054900 con 0x7f118c144060 2024-01-20T11:19:50.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.391+0000 7f11737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:50.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.391+0000 7f11737fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:50.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.559+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f118c0b00a0 con 0x7f117806c8b0 2024-01-20T11:19:50.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.562+0000 7f11737fe640 1 -- 172.21.15.110:0/1884374549 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64022 (secure 0 0 0) 0x7f118c0b00a0 con 0x7f117806c8b0 2024-01-20T11:19:50.562 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:50.562 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:50.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 10 librados: watch_flush enter 2024-01-20T11:19:50.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 10 librados: watch_flush exit 2024-01-20T11:19:50.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 20 client.5227.objecter shutdown clearing up homeless session... 2024-01-20T11:19:50.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 10 client.5227.objecter successfully canceled tick 2024-01-20T11:19:50.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f117806c8b0 msgr2=0x7f117806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:50.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f117806c8b0 0x7f117806ed70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f117c02b910 tx=0x7f117c03c040 comp rx=0 tx=0).stop 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 10 monclient: shutdown 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f118c144060 msgr2=0x7f118c13f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f118c144060 0x7f118c13f520 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f118400caa0 tx=0x7f118400cea0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 shutdown_connections 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f117806c8b0 0x7f117806ed70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f118c144060 0x7f118c13f520 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f118c13d230 0x7f118c13efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 --2- 172.21.15.110:0/1884374549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f118c06f920 0x7f118c06c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 >> 172.21.15.110:0/1884374549 conn(0x7f118c066d60 msgr2=0x7f118c0b4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 shutdown_connections 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 -- 172.21.15.110:0/1884374549 wait complete. 2024-01-20T11:19:50.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:50.564+0000 7f119117d640 1 librados: shutdown 2024-01-20T11:19:50.574 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-01-20T11:19:48.763211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002657"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:16:52.877315+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:16:52.877315+0000","last_peered":"2024-01-20T11:16:52.877315+0000","last_clean":"2024-01-20T11:16:52.877315+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:16:52.877315+0000","last_undegraded":"2024-01-20T11:16:52.877315+0000","last_fullsized":"2024-01-20T11:16:52.877315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280573,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.0590000000000002},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.945}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.532999999999999},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.6440000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":8.0280000000000005},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.191000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.708},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.823}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":9.4600000000000009}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":7.2709999999999999},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":14.302}]}]},{"osd":7,"up_from":21,"seq":90194313275,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.4770000000000003},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":10.009}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":10.744},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":6.2240000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.5629999999999997},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":15.755000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":15.039},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":8.7799999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.076000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":13.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.443},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.869999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313276,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.1170000000000009},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.9669999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.384},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.262}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":9.8539999999999992},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.423}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.693},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":7.117},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.167999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.067},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.698}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.9020000000000001},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.2679999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.3280000000000003},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.882999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.500999999999999},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.3889999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.112},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8209999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.680999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.131}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.959},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.388000000000002}]}]},{"osd":1,"up_from":17,"seq":73014444091,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.3849999999999998},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":9.7249999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.688000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.9749999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":9.0960000000000001},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.468999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":12.298999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.013}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.715},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.855}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.260999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.116}]}]},{"osd":0,"up_from":17,"seq":73014444091,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":4.3490000000000002},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.608000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":13.637},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.983000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":8.0060000000000002},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.3920000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.565},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.9390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":8.7300000000000004},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":16.207000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.324},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":11.616}]}]},{"osd":6,"up_from":15,"seq":64424509499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":7.0339999999999998},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.3479999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.3200000000000003},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":8.1270000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.188000000000001},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.223000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.952},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.343999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.805999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.340999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.728},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.774999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:53.577 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:53.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.689+0000 7f2368941640 10 monclient: get_monmap_and_config 2024-01-20T11:19:53.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.689+0000 7f2368941640 10 monclient: build_initial_monmap 2024-01-20T11:19:53.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.689+0000 7f2368941640 10 monclient: monmap: 2024-01-20T11:19:53.689 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:53.689 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:53.689 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:53.690140+0000 2024-01-20T11:19:53.689 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:53.690140+0000 2024-01-20T11:19:53.690 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:53.690 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:53.690 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:53.690 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:53.690 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:53.690 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:53.690 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:53.690 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:53.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 1 Processor -- start 2024-01-20T11:19:53.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 1 -- start start 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 10 monclient: init 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23640a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f23640a9ad0 0x7f23641423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 10 monclient(hunting): picked mon.noname-c con 0x7f23640a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2364142920 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23640b25c0 con 0x7f2364142920 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236413fbe0 con 0x7f23640a96f0 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23640aaea0 con 0x7f23640a9ad0 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.690+0000 7f2368941640 10 monclient(hunting): authenticate will time out at 1637.041091s 2024-01-20T11:19:53.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.691+0000 7f235bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f23640a9ad0 0x7f23641423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:53.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.692+0000 7f235bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f23640a9ad0 0x7f23641423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:60216/0 (socket says 172.21.15.110:60216) 2024-01-20T11:19:53.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.692+0000 7f235bfff640 1 -- 172.21.15.110:0/584043260 learned_addr learned my addr 172.21.15.110:0/584043260 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:53.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.692+0000 7f235bfff640 10 monclient(hunting): get_auth_request con 0x7f23640a9ad0 auth_method 0 2024-01-20T11:19:53.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.692+0000 7f235bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:53.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.692+0000 7f235bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:53.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.692+0000 7f235bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:53.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.692+0000 7f235bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:53.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.692+0000 7f235bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:53.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.692+0000 7f235bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:53.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.694+0000 7f235bfff640 10 monclient(hunting): handle_auth_done global_id 5230 payload 274 2024-01-20T11:19:53.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.694+0000 7f235bfff640 1 -- 172.21.15.110:0/584043260 >> v1:172.21.15.134:6789/0 conn(0x7f23640a96f0 legacy=0x7f236413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:53.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.694+0000 7f235bfff640 1 -- 172.21.15.110:0/584043260 >> v1:172.21.15.110:6789/0 conn(0x7f2364142920 legacy=0x7f236414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:53.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.694+0000 7f235bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:53.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.694+0000 7f235bfff640 1 monclient: found mon.noname-c 2024-01-20T11:19:53.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.694+0000 7f235bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:53.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.694+0000 7f235bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:19:53.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.694+0000 7f235bfff640 1 -- 172.21.15.110:0/584043260 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23640ab0a0 con 0x7f23640a9ad0 2024-01-20T11:19:53.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.698+0000 7f235bfff640 1 --2- 172.21.15.110:0/584043260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a9ad0 0x7f23641423e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f234c009870 tx=0x7f234c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e3c21d122b06c23 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:53.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 1 -- 172.21.15.110:0/584043260 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c0317e0 con 0x7f23640a9ad0 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 10 monclient: dump: 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:53.702 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 10 monclient: _finish_auth 0 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 1 -- 172.21.15.110:0/584043260 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234c031940 con 0x7f23640a9ad0 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 1 -- 172.21.15.110:0/584043260 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c031cd0 con 0x7f23640a9ad0 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 5 monclient: authenticate success, global_id 5230 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 4 set_mon_vals no callback set 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2361d9b640 10 monclient: dump: 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:53.703 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:53.704 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 10 monclient: shutdown 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 1 -- 172.21.15.110:0/584043260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a9ad0 msgr2=0x7f23641423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 1 --2- 172.21.15.110:0/584043260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a9ad0 0x7f23641423e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f234c009870 tx=0x7f234c02b880 comp rx=0 tx=0).stop 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 1 -- 172.21.15.110:0/584043260 shutdown_connections 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 1 --2- 172.21.15.110:0/584043260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a9ad0 0x7f23641423e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 1 -- 172.21.15.110:0/584043260 >> 172.21.15.110:0/584043260 conn(0x7f23640b7bc0 msgr2=0x7f23640b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 1 -- 172.21.15.110:0/584043260 shutdown_connections 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.702+0000 7f2368941640 1 -- 172.21.15.110:0/584043260 wait complete. 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.703+0000 7f2368941640 10 monclient: build_initial_monmap 2024-01-20T11:19:53.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.703+0000 7f2368941640 10 monclient: monmap: 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.703+0000 7f2368941640 1 librados: starting msgr at 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.703+0000 7f2368941640 1 librados: starting objecter 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 Processor -- start 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 -- start start 2024-01-20T11:19:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 librados: setting wanted keys 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 librados: calling monclient init 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient: init 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a96f0 0x7f2364139730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient(hunting): picked mon.noname-b con 0x7f23640a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23640a9ad0 0x7f23641dac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient(hunting): picked mon.noname-a con 0x7f23640a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2364142920 0x7f23641d7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient(hunting): picked mon.noname-c con 0x7f2364142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23641db7c0 con 0x7f23640a9ad0 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23640af8c0 con 0x7f23640a96f0 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23641db1d0 con 0x7f2364142920 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.704+0000 7f2368941640 10 monclient(hunting): authenticate will time out at 1637.054777s 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f2362d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2364142920 0x7f23641d7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f236259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a96f0 0x7f2364139730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f2362d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2364142920 0x7f23641d7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52552/0 (socket says 172.21.15.110:52552) 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f2362d9d640 1 -- 172.21.15.110:0/2297083921 learned_addr learned my addr 172.21.15.110:0/2297083921 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f2362d9d640 10 monclient(hunting): get_auth_request con 0x7f2364142920 auth_method 0 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f2362d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f2362d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f2362d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f236259c640 10 monclient(hunting): get_auth_request con 0x7f23640a96f0 auth_method 0 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f236259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f236259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:53.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.705+0000 7f236259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:53.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.706+0000 7f236259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:53.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.706+0000 7f236259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:53.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.706+0000 7f236259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:53.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.706+0000 7f2362d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:53.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.706+0000 7f2362d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:53.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.706+0000 7f2362d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:53.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.708+0000 7f235bfff640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23640a9ad0 0x7f23641dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:53.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 10 monclient(hunting): handle_auth_done global_id 5233 payload 824 2024-01-20T11:19:53.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 1 -- 172.21.15.110:0/2297083921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2364142920 msgr2=0x7f23641d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2364142920 0x7f23641d7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 1 -- 172.21.15.110:0/2297083921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23640a9ad0 msgr2=0x7f23641dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23640a9ad0 0x7f23641dac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 10 monclient: _finish_hunting 0 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 1 monclient: found mon.noname-b 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f236259c640 1 -- 172.21.15.110:0/2297083921 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c0094d0 con 0x7f23640a96f0 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f235bfff640 10 monclient: get_auth_request con 0x7f23640a9ad0 auth_method 0 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.709+0000 7f235bfff640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23640a9ad0 0x7f23641dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.712+0000 7f236259c640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a96f0 0x7f2364139730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f235400b3e0 tx=0x7f235400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.712+0000 7f235affd640 10 client.?.objecter ms_handle_connect 0x7f23640a96f0 2024-01-20T11:19:53.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.712+0000 7f235affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 1 -- 172.21.15.110:0/2297083921 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354011830 con 0x7f23640a96f0 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 10 monclient: dump: 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 10 monclient: _finish_auth 0 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 1 -- 172.21.15.110:0/2297083921 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2354011990 con 0x7f23640a96f0 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 1 -- 172.21.15.110:0/2297083921 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354011d40 con 0x7f23640a96f0 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f235affd640 10 monclient: dump: 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f2368941640 5 monclient: authenticate success, global_id 5233 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f2368941640 10 monclient: _renew_subs 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f2368941640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23641db3f0 con 0x7f23640a96f0 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f236159a640 4 set_mon_vals no callback set 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f2368941640 10 client.5233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f2368941640 10 monclient: _renew_subs 2024-01-20T11:19:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f2368941640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:53.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23641d7eb0 con 0x7f23640a96f0 2024-01-20T11:19:53.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.765+0000 7f2368941640 1 librados: init done 2024-01-20T11:19:53.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.766+0000 7f2368941640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:53.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.766+0000 7f2368941640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:53.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.766+0000 7f2368941640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:53.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:53.766+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23640b6280 con 0x7f23640a96f0 2024-01-20T11:19:53.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~56s 2024-01-20T11:19:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.007+0000 7f235affd640 1 -- 172.21.15.110:0/2297083921 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f235401a400 con 0x7f23640a96f0 2024-01-20T11:19:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.007+0000 7f235affd640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f234406c8b0 0x7f234406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.008+0000 7f235bfff640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f234406c8b0 0x7f234406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.008+0000 7f235bfff640 10 monclient: get_auth_request con 0x7f234406c8b0 auth_method 0 2024-01-20T11:19:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.008+0000 7f235affd640 10 client.5233.objecter ms_handle_connect 0x7f234406c8b0 2024-01-20T11:19:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.008+0000 7f235bfff640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f234406c8b0 0x7f234406ed70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f234c036000 tx=0x7f234c03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.012+0000 7f235affd640 1 -- 172.21.15.110:0/2297083921 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2354058f20 con 0x7f23640a96f0 2024-01-20T11:19:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.012+0000 7f235affd640 10 client.5233.objecter ms_dispatch 0x7f2364000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.012+0000 7f235affd640 3 client.5233.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.012+0000 7f235affd640 3 client.5233.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:54.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.012+0000 7f235affd640 20 client.5233.objecter dump_active .. 0 homeless 2024-01-20T11:19:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.022+0000 7f235affd640 1 -- 172.21.15.110:0/2297083921 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2354054b60 con 0x7f23640a96f0 2024-01-20T11:19:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.022+0000 7f235affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.022+0000 7f235affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:54.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.196+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23640b4400 con 0x7f234406c8b0 2024-01-20T11:19:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.199+0000 7f235affd640 1 -- 172.21.15.110:0/2297083921 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64045 (secure 0 0 0) 0x7f234c031c30 con 0x7f234406c8b0 2024-01-20T11:19:54.200 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 10 librados: watch_flush enter 2024-01-20T11:19:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 10 librados: watch_flush exit 2024-01-20T11:19:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 20 client.5233.objecter shutdown clearing up homeless session... 2024-01-20T11:19:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 10 client.5233.objecter successfully canceled tick 2024-01-20T11:19:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f234406c8b0 msgr2=0x7f234406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f234406c8b0 0x7f234406ed70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f234c036000 tx=0x7f234c03d040 comp rx=0 tx=0).stop 2024-01-20T11:19:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 10 monclient: shutdown 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a96f0 msgr2=0x7f2364139730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a96f0 0x7f2364139730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f235400b3e0 tx=0x7f235400b8a0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 shutdown_connections 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f234406c8b0 0x7f234406ed70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2364142920 0x7f23641d7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23640a9ad0 0x7f23641dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 --2- 172.21.15.110:0/2297083921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23640a96f0 0x7f2364139730 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 >> 172.21.15.110:0/2297083921 conn(0x7f23640b7bc0 msgr2=0x7f23640b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 shutdown_connections 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 -- 172.21.15.110:0/2297083921 wait complete. 2024-01-20T11:19:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.201+0000 7f2368941640 1 librados: shutdown 2024-01-20T11:19:54.211 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-01-20T11:19:52.764109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002566"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280573,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.0590000000000002},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.945}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.532999999999999},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.6440000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":8.0280000000000005},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.191000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.708},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.823}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.058999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":9.4600000000000009}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":7.2709999999999999},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":14.302}]}]},{"osd":7,"up_from":21,"seq":90194313276,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2149999999999999},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.9429999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.6749999999999998},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.6869999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.92},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":7.2720000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.125},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.52}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.337},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":10.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.157},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.895}]}]},{"osd":3,"up_from":21,"seq":90194313277,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.9189999999999996},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6390000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.9000000000000004},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":8.4900000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":14.903},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.272}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.262},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.1479999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":9.7420000000000009},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":12.861000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":11.696999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.002000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.9020000000000001},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.2679999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.3280000000000003},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.882999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.500999999999999},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.3889999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.112},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8209999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.680999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":13.131}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.959},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.388000000000002}]}]},{"osd":1,"up_from":17,"seq":73014444091,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.3849999999999998},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":9.7249999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.688000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.9749999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":9.0960000000000001},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.468999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":12.298999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.013}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.715},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.855}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.260999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.116}]}]},{"osd":0,"up_from":17,"seq":73014444091,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":4.3490000000000002},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.608000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":13.637},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":14.983000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":8.0060000000000002},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.3920000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.565},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.9390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":8.7300000000000004},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":16.207000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.324},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":11.616}]}]},{"osd":6,"up_from":15,"seq":64424509500,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":6.3600000000000003},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.8179999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.4560000000000004},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":10.553000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.081},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3670000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":11.679},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.724}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.869},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.326000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.164999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.059999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:54.213 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:54.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.313+0000 7ff1acf47640 10 monclient: get_monmap_and_config 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.313+0000 7ff1acf47640 10 monclient: build_initial_monmap 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.313+0000 7ff1acf47640 10 monclient: monmap: 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:54.313627+0000 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:54.313627+0000 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:54.313 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:54.314 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:54.314 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.313+0000 7ff1acf47640 1 Processor -- start 2024-01-20T11:19:54.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.313+0000 7ff1acf47640 1 -- start start 2024-01-20T11:19:54.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.313+0000 7ff1acf47640 10 monclient: init 2024-01-20T11:19:54.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:54.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-01-20T11:19:54.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff1a8145760 0x7ff1a8145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient(hunting): picked mon.noname-a con 0x7ff1a8145760 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a80a9d80 0x7ff1a813f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient(hunting): picked mon.noname-b con 0x7ff1a80a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff1a813f6e0 0x7ff1a80b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient(hunting): picked mon.noname-c con 0x7ff1a813f6e0 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a80ab050 con 0x7ff1a8145760 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a8146250 con 0x7ff1a813f6e0 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a813fb90 con 0x7ff1a80a9d80 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1acf47640 10 monclient(hunting): authenticate will time out at 1637.664587s 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1a5d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a80a9d80 0x7ff1a813f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.314+0000 7ff1a659c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff1a8145760 0x7ff1a8145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a6d9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff1a813f6e0 0x7ff1a80b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a5d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a80a9d80 0x7ff1a813f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52560/0 (socket says 172.21.15.110:52560) 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a5d9b640 1 -- 172.21.15.110:0/3247296841 learned_addr learned my addr 172.21.15.110:0/3247296841 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a5d9b640 10 monclient(hunting): get_auth_request con 0x7ff1a80a9d80 auth_method 0 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a5d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a659c640 10 monclient(hunting): get_auth_request con 0x7ff1a8145760 auth_method 0 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a6d9d640 10 monclient(hunting): get_auth_request con 0x7ff1a813f6e0 auth_method 0 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a6d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.315+0000 7ff1a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.316+0000 7ff1a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.316+0000 7ff1a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.316+0000 7ff1a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.316+0000 7ff1a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.316+0000 7ff1a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.316+0000 7ff1a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.316+0000 7ff1a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 10 monclient(hunting): handle_auth_done global_id 5154 payload 274 2024-01-20T11:19:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 1 -- 172.21.15.110:0/3247296841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a80a9d80 msgr2=0x7ff1a813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 1 --2- 172.21.15.110:0/3247296841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a80a9d80 0x7ff1a813f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 1 -- 172.21.15.110:0/3247296841 >> v2:172.21.15.134:3300/0 conn(0x7ff1a813f6e0 msgr2=0x7ff1a80b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 1 --2- 172.21.15.110:0/3247296841 >> v2:172.21.15.134:3300/0 conn(0x7ff1a813f6e0 0x7ff1a80b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 10 monclient: _finish_hunting 0 2024-01-20T11:19:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 1 monclient: found mon.noname-a 2024-01-20T11:19:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:54.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.317+0000 7ff1a659c640 1 -- 172.21.15.110:0/3247296841 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a80b20c0 con 0x7ff1a8145760 2024-01-20T11:19:54.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.319+0000 7ff1a659c640 1 --2- 172.21.15.110:0/3247296841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 0x7ff1a8145b40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8067330 tx=0x7ff19802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad49479096e39be server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:54.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 1 -- 172.21.15.110:0/3247296841 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19802cbd0 con 0x7ff1a8145760 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 10 monclient: dump: 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 10 monclient: _finish_auth 0 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 10 monclient: _check_auth_tickets 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 1 -- 172.21.15.110:0/3247296841 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19802cd30 con 0x7ff1a8145760 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 1 -- 172.21.15.110:0/3247296841 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19802cea0 con 0x7ff1a8145760 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 5 monclient: authenticate success, global_id 5154 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 4 set_mon_vals no callback set 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1a559a640 10 monclient: dump: 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:54.325 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 10 monclient: shutdown 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 1 -- 172.21.15.110:0/3247296841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 msgr2=0x7ff1a8145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3247296841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 0x7ff1a8145b40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8067330 tx=0x7ff19802be20 comp rx=0 tx=0).stop 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 1 -- 172.21.15.110:0/3247296841 shutdown_connections 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3247296841 >> v2:172.21.15.134:3300/0 conn(0x7ff1a813f6e0 0x7ff1a80b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3247296841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a80a9d80 0x7ff1a813f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3247296841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 0x7ff1a8145b40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 1 -- 172.21.15.110:0/3247296841 >> 172.21.15.110:0/3247296841 conn(0x7ff1a80663b0 msgr2=0x7ff1a80667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 1 -- 172.21.15.110:0/3247296841 shutdown_connections 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.323+0000 7ff1acf47640 1 -- 172.21.15.110:0/3247296841 wait complete. 2024-01-20T11:19:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.324+0000 7ff1acf47640 10 monclient: build_initial_monmap 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.324+0000 7ff1acf47640 10 monclient: monmap: 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 librados: starting msgr at 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 librados: starting objecter 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 Processor -- start 2024-01-20T11:19:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 -- start start 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 librados: setting wanted keys 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 librados: calling monclient init 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient: init 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1a80a9d80 0x7ff1a81dab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient(hunting): picked mon.noname-b con 0x7ff1a80a9d80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a813f6e0 0x7ff1a80b7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient(hunting): picked mon.noname-c con 0x7ff1a813f6e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 0x7ff1a80b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient(hunting): picked mon.noname-a con 0x7ff1a8145760 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80b6630 con 0x7ff1a8145760 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80b56b0 con 0x7ff1a80a9d80 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80b5870 con 0x7ff1a813f6e0 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.325+0000 7ff1acf47640 10 monclient(hunting): authenticate will time out at 1637.675978s 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.326+0000 7ff1a6d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 0x7ff1a80b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.326+0000 7ff1a6d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 0x7ff1a80b7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54588/0 (socket says 172.21.15.110:54588) 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.326+0000 7ff1a6d9d640 1 -- 172.21.15.110:0/3174462578 learned_addr learned my addr 172.21.15.110:0/3174462578 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.326+0000 7ff1a6d9d640 10 monclient(hunting): get_auth_request con 0x7ff1a8145760 auth_method 0 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.326+0000 7ff1a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.326+0000 7ff1a6d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.326+0000 7ff1a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.326+0000 7ff1a659c640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1a80a9d80 0x7ff1a81dab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a659c640 10 monclient(hunting): get_auth_request con 0x7ff1a80a9d80 auth_method 0 2024-01-20T11:19:54.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.327+0000 7ff1a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 10 monclient(hunting): handle_auth_done global_id 5157 payload 824 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 1 -- 172.21.15.110:0/3174462578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a813f6e0 msgr2=0x7ff1a80b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a813f6e0 0x7ff1a80b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 1 -- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1a80a9d80 msgr2=0x7ff1a81dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1a80a9d80 0x7ff1a81dab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 1 monclient: found mon.noname-a 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:19:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.329+0000 7ff1a6d9d640 1 -- 172.21.15.110:0/3174462578 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff198009a70 con 0x7ff1a8145760 2024-01-20T11:19:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.332+0000 7ff1a6d9d640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 0x7ff1a80b7750 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000e960 tx=0x7ff1a000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.332+0000 7ff196ffd640 10 client.?.objecter ms_handle_connect 0x7ff1a8145760 2024-01-20T11:19:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.332+0000 7ff196ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.335+0000 7ff196ffd640 1 -- 172.21.15.110:0/3174462578 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0009690 con 0x7ff1a8145760 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 10 monclient: dump: 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 10 monclient: _finish_auth 0 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 1 -- 172.21.15.110:0/3174462578 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a000c980 con 0x7ff1a8145760 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 1 -- 172.21.15.110:0/3174462578 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a000cd30 con 0x7ff1a8145760 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1acf47640 5 monclient: authenticate success, global_id 5157 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff196ffd640 10 monclient: dump: 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1acf47640 10 monclient: _renew_subs 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1a4d99640 4 set_mon_vals no callback set 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1acf47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a80b7c90 con 0x7ff1a8145760 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1acf47640 10 client.5157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1acf47640 10 monclient: _renew_subs 2024-01-20T11:19:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1acf47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a81db700 con 0x7ff1a8145760 2024-01-20T11:19:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.336+0000 7ff1acf47640 1 librados: init done 2024-01-20T11:19:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.337+0000 7ff1acf47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.337+0000 7ff1acf47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.337+0000 7ff1acf47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:19:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.337+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a8075490 con 0x7ff1a8145760 2024-01-20T11:19:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.355+0000 7ff196ffd640 1 -- 172.21.15.110:0/3174462578 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff1a0013470 con 0x7ff1a8145760 2024-01-20T11:19:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.355+0000 7ff196ffd640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff18806c8b0 0x7ff18806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.355+0000 7ff1a659c640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff18806c8b0 0x7ff18806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.355+0000 7ff1a659c640 10 monclient: get_auth_request con 0x7ff18806c8b0 auth_method 0 2024-01-20T11:19:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.356+0000 7ff196ffd640 10 client.5157.objecter ms_handle_connect 0x7ff18806c8b0 2024-01-20T11:19:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.356+0000 7ff1a659c640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff18806c8b0 0x7ff18806ed70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff198036000 tx=0x7ff198030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.358+0000 7ff196ffd640 1 -- 172.21.15.110:0/3174462578 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff1a0058b90 con 0x7ff1a8145760 2024-01-20T11:19:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.358+0000 7ff196ffd640 10 client.5157.objecter ms_dispatch 0x7ff1a8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.358+0000 7ff196ffd640 3 client.5157.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.359+0000 7ff196ffd640 3 client.5157.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.359+0000 7ff196ffd640 20 client.5157.objecter dump_active .. 0 homeless 2024-01-20T11:19:54.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.604+0000 7ff196ffd640 1 -- 172.21.15.110:0/3174462578 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff1a0054ed0 con 0x7ff1a8145760 2024-01-20T11:19:54.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.604+0000 7ff196ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:54.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.604+0000 7ff196ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:54.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.773+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1a80b7e80 con 0x7ff18806c8b0 2024-01-20T11:19:54.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.776+0000 7ff196ffd640 1 -- 172.21.15.110:0/3174462578 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63978 (secure 0 0 0) 0x7ff1a80b7e80 con 0x7ff18806c8b0 2024-01-20T11:19:54.776 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:54.776 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:54.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.777+0000 7ff1acf47640 10 librados: watch_flush enter 2024-01-20T11:19:54.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.777+0000 7ff1acf47640 10 librados: watch_flush exit 2024-01-20T11:19:54.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.777+0000 7ff1acf47640 20 client.5157.objecter shutdown clearing up homeless session... 2024-01-20T11:19:54.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.777+0000 7ff1acf47640 10 client.5157.objecter successfully canceled tick 2024-01-20T11:19:54.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff18806c8b0 msgr2=0x7ff18806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff18806c8b0 0x7ff18806ed70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff198036000 tx=0x7ff198030000 comp rx=0 tx=0).stop 2024-01-20T11:19:54.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 10 monclient: shutdown 2024-01-20T11:19:54.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 msgr2=0x7ff1a80b7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 0x7ff1a80b7750 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000e960 tx=0x7ff1a000ee20 comp rx=0 tx=0).stop 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 shutdown_connections 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff18806c8b0 0x7ff18806ed70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1a8145760 0x7ff1a80b7750 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1a813f6e0 0x7ff1a80b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 --2- 172.21.15.110:0/3174462578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1a80a9d80 0x7ff1a81dab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 >> 172.21.15.110:0/3174462578 conn(0x7ff1a80663b0 msgr2=0x7ff1a8139d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 shutdown_connections 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 -- 172.21.15.110:0/3174462578 wait complete. 2024-01-20T11:19:54.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.778+0000 7ff1acf47640 1 librados: shutdown 2024-01-20T11:19:54.788 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-01-20T11:19:54.764478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002556"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280574,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.0020000000000007},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.6070000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.903},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":16.905999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.231},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.082999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.337},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":9.2439999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.273999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":17.809999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.960000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":14.974}]}]},{"osd":7,"up_from":21,"seq":90194313276,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2149999999999999},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.9429999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.6749999999999998},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.6869999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.92},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":7.2720000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.125},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.52}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.337},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":10.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.157},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.895}]}]},{"osd":3,"up_from":21,"seq":90194313277,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.9189999999999996},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6390000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.9000000000000004},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":8.4900000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":14.903},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.272}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.262},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.1479999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":9.7420000000000009},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":12.861000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":11.696999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.002000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444091,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.9210000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.09},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.460000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.064},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":7.577}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.872999999999999},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.8089999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":14.500999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.587999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.162000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":15.221}]}]},{"osd":1,"up_from":17,"seq":73014444092,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.4790000000000001},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":11.869}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":13.246},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":6.3230000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":9.4019999999999992},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.7859999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":14.077},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":8.1660000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":10.708},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":15.013999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.284000000000001},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.808}]}]},{"osd":0,"up_from":17,"seq":73014444092,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":5.3810000000000002},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":8.5869999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":11.257},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":6.1159999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":9.2919999999999998},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":14.683}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.323},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.0250000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":7.2800000000000002},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":13.118}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.911},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":10.667}]}]},{"osd":6,"up_from":15,"seq":64424509500,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":6.3600000000000003},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.8179999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.4560000000000004},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":10.553000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.081},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3670000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":11.679},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.724}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.869},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.326000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.164999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.059999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:54.790 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:19:54.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:19:54.790 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:19:54.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:19:54.790 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:19:54.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:19:54.790 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:19:54.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:19:54.791 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:19:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.890+0000 7f397d061640 10 monclient: get_monmap_and_config 2024-01-20T11:19:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.890+0000 7f397d061640 10 monclient: build_initial_monmap 2024-01-20T11:19:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.890+0000 7f397d061640 10 monclient: monmap: 2024-01-20T11:19:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:54.891224+0000 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:54.891224+0000 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:54.891 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.891+0000 7f397d061640 1 Processor -- start 2024-01-20T11:19:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.891+0000 7f397d061640 1 -- start start 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.891+0000 7f397d061640 10 monclient: init 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.891+0000 7f397d061640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.891+0000 7f397d061640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.891+0000 7f397d061640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f397806bb00 0x7f39780b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 10 monclient(hunting): picked mon.noname-c con 0x7f397806bb00 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39780af790 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f3978145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 10 monclient(hunting): picked mon.noname-b con 0x7f39780b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397806c220 con 0x7f39780af790 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397806bee0 con 0x7f397806bb00 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39780ada30 con 0x7f39780b0900 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397d061640 10 monclient(hunting): authenticate will time out at 1638.242230s 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397759e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f3978145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f3976d9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f397806bb00 0x7f39780b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397759e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f3978145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52578/0 (socket says 172.21.15.110:52578) 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397759e640 1 -- 172.21.15.110:0/2423334572 learned_addr learned my addr 172.21.15.110:0/2423334572 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397759e640 10 monclient(hunting): get_auth_request con 0x7f39780b0900 auth_method 0 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397759e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.892+0000 7f397759e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f3976d9d640 10 monclient(hunting): get_auth_request con 0x7f397806bb00 auth_method 0 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f3976d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f3976d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f3976d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f397759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f397759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f397759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f3976d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f3976d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.893+0000 7f3976d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.895+0000 7f397759e640 10 monclient(hunting): handle_auth_done global_id 4952 payload 274 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.895+0000 7f397759e640 1 -- 172.21.15.110:0/2423334572 >> v2:172.21.15.134:3300/0 conn(0x7f397806bb00 msgr2=0x7f39780b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.895+0000 7f397759e640 1 --2- 172.21.15.110:0/2423334572 >> v2:172.21.15.134:3300/0 conn(0x7f397806bb00 0x7f39780b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.895+0000 7f397759e640 1 -- 172.21.15.110:0/2423334572 >> v1:172.21.15.134:6789/0 conn(0x7f39780af790 legacy=0x7f3978145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.895+0000 7f397759e640 10 monclient: _finish_hunting 0 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.895+0000 7f397759e640 1 monclient: found mon.noname-b 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.895+0000 7f397759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.895+0000 7f397759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.895+0000 7f397759e640 1 -- 172.21.15.110:0/2423334572 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39780add10 con 0x7f39780b0900 2024-01-20T11:19:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.896+0000 7f397759e640 1 --2- 172.21.15.110:0/2423334572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f3978145e80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3970009e50 tx=0x7f397002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb8e20c291f854b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:54.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 1 -- 172.21.15.110:0/2423334572 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397002c9e0 con 0x7f39780b0900 2024-01-20T11:19:54.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:54.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:54.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 10 monclient: dump: 2024-01-20T11:19:54.900 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 10 monclient: _finish_auth 0 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 10 monclient: _check_auth_tickets 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 1 -- 172.21.15.110:0/2423334572 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397002cb40 con 0x7f39780b0900 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 5 monclient: authenticate success, global_id 4952 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 1 -- 172.21.15.110:0/2423334572 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39700343f0 con 0x7f39780b0900 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 4 set_mon_vals no callback set 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397659c640 10 monclient: dump: 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:54.902 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.903 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.903 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.903 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:54.903 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.903 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:54.903 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:54.903 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:54.903 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 10 monclient: shutdown 2024-01-20T11:19:54.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:54.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 1 -- 172.21.15.110:0/2423334572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 msgr2=0x7f3978145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 1 --2- 172.21.15.110:0/2423334572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f3978145e80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3970009e50 tx=0x7f397002be40 comp rx=0 tx=0).stop 2024-01-20T11:19:54.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 1 -- 172.21.15.110:0/2423334572 shutdown_connections 2024-01-20T11:19:54.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 1 --2- 172.21.15.110:0/2423334572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f3978145e80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 1 --2- 172.21.15.110:0/2423334572 >> v2:172.21.15.134:3300/0 conn(0x7f397806bb00 0x7f39780b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.900+0000 7f397d061640 1 -- 172.21.15.110:0/2423334572 >> 172.21.15.110:0/2423334572 conn(0x7f3978066d60 msgr2=0x7f39780b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.901+0000 7f397d061640 1 -- 172.21.15.110:0/2423334572 shutdown_connections 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.901+0000 7f397d061640 1 -- 172.21.15.110:0/2423334572 wait complete. 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.901+0000 7f397d061640 10 monclient: build_initial_monmap 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.901+0000 7f397d061640 10 monclient: monmap: 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:54.908 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 librados: starting msgr at 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 librados: starting objecter 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 Processor -- start 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 -- start start 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 librados: setting wanted keys 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 librados: calling monclient init 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient: init 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397806bb00 0x7f3978141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient(hunting): picked mon.noname-a con 0x7f397806bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:54.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39780af790 0x7f397811f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient(hunting): picked mon.noname-b con 0x7f39780af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f39781da3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient(hunting): picked mon.noname-c con 0x7f39780b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397811e7b0 con 0x7f397806bb00 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397811fe10 con 0x7f39780af790 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397811ea00 con 0x7f39780b0900 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.902+0000 7f397d061640 10 monclient(hunting): authenticate will time out at 1638.253116s 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f397759e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f39781da3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f3976d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397806bb00 0x7f3978141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f396ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39780af790 0x7f397811f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f397759e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f39781da3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52592/0 (socket says 172.21.15.110:52592) 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f397759e640 1 -- 172.21.15.110:0/3711588579 learned_addr learned my addr 172.21.15.110:0/3711588579 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f397759e640 10 monclient(hunting): get_auth_request con 0x7f39780b0900 auth_method 0 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f397759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f397759e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f397759e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f396ffff640 10 monclient(hunting): get_auth_request con 0x7f39780af790 auth_method 0 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f396ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f396ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f396ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f3976d9d640 10 monclient(hunting): get_auth_request con 0x7f397806bb00 auth_method 0 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f3976d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f3976d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.903+0000 7f3976d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:54.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.904+0000 7f397759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.904+0000 7f397759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.904+0000 7f397759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.904+0000 7f396ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.904+0000 7f396ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.904+0000 7f396ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.904+0000 7f3976d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.904+0000 7f3976d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.904+0000 7f3976d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.906+0000 7f397759e640 10 monclient(hunting): handle_auth_done global_id 4955 payload 824 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.906+0000 7f397759e640 1 -- 172.21.15.110:0/3711588579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39780af790 msgr2=0x7f397811f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.906+0000 7f397759e640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39780af790 0x7f397811f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.906+0000 7f397759e640 1 -- 172.21.15.110:0/3711588579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397806bb00 msgr2=0x7f3978141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.906+0000 7f397759e640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397806bb00 0x7f3978141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:54.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.906+0000 7f397759e640 10 monclient: _finish_hunting 0 2024-01-20T11:19:54.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.906+0000 7f397759e640 1 monclient: found mon.noname-c 2024-01-20T11:19:54.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.907+0000 7f397759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:54.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.907+0000 7f397759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:19:54.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.907+0000 7f397759e640 1 -- 172.21.15.110:0/3711588579 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3970009b30 con 0x7f39780b0900 2024-01-20T11:19:54.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.909+0000 7f397759e640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f39781da3c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3970036040 tx=0x7f3970004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:54.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.909+0000 7f396f7fe640 10 client.?.objecter ms_handle_connect 0x7f39780b0900 2024-01-20T11:19:54.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.909+0000 7f396f7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:54.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 1 -- 172.21.15.110:0/3711588579 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970009510 con 0x7f39780b0900 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 10 monclient: dump: 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:54.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 1 -- 172.21.15.110:0/3711588579 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397003e450 con 0x7f39780b0900 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f397d061640 5 monclient: authenticate success, global_id 4955 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f397d061640 10 monclient: _renew_subs 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f397d061640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397813e6b0 con 0x7f39780b0900 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 1 -- 172.21.15.110:0/3711588579 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397003e7e0 con 0x7f39780b0900 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f396f7fe640 10 monclient: dump: 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:54.916 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f3975d9b640 4 set_mon_vals no callback set 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f397d061640 10 client.4955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f397d061640 10 monclient: _renew_subs 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f397d061640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397811f100 con 0x7f39780b0900 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.912+0000 7f397d061640 1 librados: init done 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.913+0000 7f397d061640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:54.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.913+0000 7f397d061640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:54.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.913+0000 7f397d061640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:54.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:54.913+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397811ec20 con 0x7f39780b0900 2024-01-20T11:19:55.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.154+0000 7f396f7fe640 1 -- 172.21.15.110:0/3711588579 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f397003e940 con 0x7f39780b0900 2024-01-20T11:19:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.155+0000 7f396f7fe640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f395806c8b0 0x7f395806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.155+0000 7f3976d9d640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f395806c8b0 0x7f395806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.155+0000 7f3976d9d640 10 monclient: get_auth_request con 0x7f395806c8b0 auth_method 0 2024-01-20T11:19:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.156+0000 7f396f7fe640 10 client.4955.objecter ms_handle_connect 0x7f395806c8b0 2024-01-20T11:19:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.156+0000 7f3976d9d640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f395806c8b0 0x7f395806ed70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3968009b40 tx=0x7f3968009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:55.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.168+0000 7f396f7fe640 1 -- 172.21.15.110:0/3711588579 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f397007b5d0 con 0x7f39780b0900 2024-01-20T11:19:55.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.168+0000 7f396f7fe640 10 client.4955.objecter ms_dispatch 0x7f3978000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:55.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.168+0000 7f396f7fe640 3 client.4955.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:55.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.168+0000 7f396f7fe640 3 client.4955.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:55.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.168+0000 7f396f7fe640 20 client.4955.objecter dump_active .. 0 homeless 2024-01-20T11:19:55.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.168+0000 7f396f7fe640 1 -- 172.21.15.110:0/3711588579 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f3970077910 con 0x7f39780b0900 2024-01-20T11:19:55.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.168+0000 7f396f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:55.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.168+0000 7f396f7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:55.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.422+0000 7f397d061640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:19:55.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.422+0000 7f397d061640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:55.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.422+0000 7f397d061640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:19:55.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.422+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3978075490 con 0x7f39780b0900 2024-01-20T11:19:55.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.428+0000 7f396f7fe640 1 -- 172.21.15.110:0/3711588579 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f3970043520 con 0x7f39780b0900 2024-01-20T11:19:55.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.428+0000 7f396f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:19:55.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.428+0000 7f396f7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:19:55.428 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:55.428 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":329,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:19:55.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 10 librados: watch_flush enter 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 10 librados: watch_flush exit 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 20 client.4955.objecter shutdown clearing up homeless session... 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 10 client.4955.objecter successfully canceled tick 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f395806c8b0 msgr2=0x7f395806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f395806c8b0 0x7f395806ed70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3968009b40 tx=0x7f3968009190 comp rx=0 tx=0).stop 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 10 monclient: shutdown 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 msgr2=0x7f39781da3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f39781da3c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3970036040 tx=0x7f3970004a40 comp rx=0 tx=0).stop 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 shutdown_connections 2024-01-20T11:19:55.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f395806c8b0 0x7f395806ed70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39780b0900 0x7f39781da3c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39780af790 0x7f397811f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 --2- 172.21.15.110:0/3711588579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397806bb00 0x7f3978141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 >> 172.21.15.110:0/3711588579 conn(0x7f3978066d60 msgr2=0x7f3978142950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:55.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 shutdown_connections 2024-01-20T11:19:55.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 -- 172.21.15.110:0/3711588579 wait complete. 2024-01-20T11:19:55.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.430+0000 7f397d061640 1 librados: shutdown 2024-01-20T11:19:55.441 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:19:55.441 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:55.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.541+0000 7f4c30f3c640 10 monclient: get_monmap_and_config 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.541+0000 7f4c30f3c640 10 monclient: build_initial_monmap 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.541+0000 7f4c30f3c640 10 monclient: monmap: 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:55.542460+0000 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:55.542460+0000 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:55.542 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:55.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.542+0000 7f4c30f3c640 1 Processor -- start 2024-01-20T11:19:55.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.542+0000 7f4c30f3c640 1 -- start start 2024-01-20T11:19:55.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.542+0000 7f4c30f3c640 10 monclient: init 2024-01-20T11:19:55.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:55.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-01-20T11:19:55.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c2c06bb00 0x7f4c2c0b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:55.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c06bb00 addr v2:172.21.15.110:3300/0 2024-01-20T11:19:55.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0af790 0x7f4c2c145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c0af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4c2c0b0900 0x7f4c2c145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c0b0900 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c06bee0 con 0x7f4c2c06bb00 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c0ada30 con 0x7f4c2c0b0900 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0add10 con 0x7f4c2c0af790 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c30f3c640 10 monclient(hunting): authenticate will time out at 1638.893486s 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c29d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0af790 0x7f4c2c145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:55.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.543+0000 7f4c2a59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c2c06bb00 0x7f4c2c0b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2ad9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4c2c0b0900 0x7f4c2c145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2a59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c2c06bb00 0x7f4c2c0b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54598/0 (socket says 172.21.15.110:54598) 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2a59c640 1 -- 172.21.15.110:0/1255020941 learned_addr learned my addr 172.21.15.110:0/1255020941 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2a59c640 10 monclient(hunting): get_auth_request con 0x7f4c2c06bb00 auth_method 0 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c29d9b640 10 monclient(hunting): get_auth_request con 0x7f4c2c0af790 auth_method 0 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c29d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2ad9d640 10 monclient(hunting): get_auth_request con 0x7f4c2c0b0900 auth_method 0 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:55.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:55.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:55.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:55.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:55.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.544+0000 7f4c29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:55.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.545+0000 7f4c29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:55.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.545+0000 7f4c29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:55.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.545+0000 7f4c2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:55.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.545+0000 7f4c2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:55.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.545+0000 7f4c2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:55.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 10 monclient(hunting): handle_auth_done global_id 5248 payload 274 2024-01-20T11:19:55.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 1 -- 172.21.15.110:0/1255020941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0af790 msgr2=0x7f4c2c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:55.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 1 --2- 172.21.15.110:0/1255020941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0af790 0x7f4c2c145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 1 -- 172.21.15.110:0/1255020941 >> v2:172.21.15.110:3300/0 conn(0x7f4c2c06bb00 msgr2=0x7f4c2c0b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:55.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 1 --2- 172.21.15.110:0/1255020941 >> v2:172.21.15.110:3300/0 conn(0x7f4c2c06bb00 0x7f4c2c0b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 10 monclient: _finish_hunting 0 2024-01-20T11:19:55.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 1 monclient: found mon.noname-c 2024-01-20T11:19:55.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:55.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:19:55.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.546+0000 7f4c2ad9d640 1 -- 172.21.15.110:0/1255020941 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0adff0 con 0x7f4c2c0b0900 2024-01-20T11:19:55.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.548+0000 7f4c2ad9d640 1 --2- 172.21.15.110:0/1255020941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0b0900 0x7f4c2c145cb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c06c190 tx=0x7f4c2402bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3da68a010ac36b79 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:55.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 1 -- 172.21.15.110:0/1255020941 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2402cb40 con 0x7f4c2c0b0900 2024-01-20T11:19:55.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:55.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:55.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 10 monclient: dump: 2024-01-20T11:19:55.552 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:55.552 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 10 monclient: _finish_auth 0 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 10 monclient: _check_auth_tickets 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 1 -- 172.21.15.110:0/1255020941 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2402cca0 con 0x7f4c2c0b0900 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 1 -- 172.21.15.110:0/1255020941 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24034660 con 0x7f4c2c0b0900 2024-01-20T11:19:55.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c2959a640 10 monclient: dump: 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c30f3c640 5 monclient: authenticate success, global_id 5248 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c30f3c640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:55.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.552+0000 7f4c30f3c640 4 set_mon_vals no callback set 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 10 monclient: shutdown 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 1 -- 172.21.15.110:0/1255020941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0b0900 msgr2=0x7f4c2c145cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/1255020941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0b0900 0x7f4c2c145cb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c06c190 tx=0x7f4c2402bd90 comp rx=0 tx=0).stop 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 1 -- 172.21.15.110:0/1255020941 shutdown_connections 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/1255020941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0b0900 0x7f4c2c145cb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/1255020941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0af790 0x7f4c2c145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/1255020941 >> v2:172.21.15.110:3300/0 conn(0x7f4c2c06bb00 0x7f4c2c0b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 1 -- 172.21.15.110:0/1255020941 >> 172.21.15.110:0/1255020941 conn(0x7f4c2c066d60 msgr2=0x7f4c2c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 1 -- 172.21.15.110:0/1255020941 shutdown_connections 2024-01-20T11:19:55.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 1 -- 172.21.15.110:0/1255020941 wait complete. 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 10 monclient: build_initial_monmap 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.553+0000 7f4c30f3c640 10 monclient: monmap: 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.554+0000 7f4c30f3c640 1 librados: starting msgr at 2024-01-20T11:19:55.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.554+0000 7f4c30f3c640 1 librados: starting objecter 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.554+0000 7f4c30f3c640 1 Processor -- start 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.554+0000 7f4c30f3c640 1 -- start start 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.554+0000 7f4c30f3c640 1 librados: setting wanted keys 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.554+0000 7f4c30f3c640 1 librados: calling monclient init 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.554+0000 7f4c30f3c640 10 monclient: init 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.554+0000 7f4c30f3c640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.554+0000 7f4c30f3c640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06bb00 0x7f4c2c0b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c06bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0af790 0x7f4c2c11da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c0af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0b0900 0x7f4c2c13bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:55.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c0b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c11f0b0 con 0x7f4c2c06bb00 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c141b60 con 0x7f4c2c0af790 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c142080 con 0x7f4c2c0b0900 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c30f3c640 10 monclient(hunting): authenticate will time out at 1638.905255s 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c29d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0af790 0x7f4c2c11da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:55.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c2a59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06bb00 0x7f4c2c0b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c2ad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0b0900 0x7f4c2c13bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.555+0000 7f4c29d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0af790 0x7f4c2c11da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:60270/0 (socket says 172.21.15.110:60270) 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c29d9b640 1 -- 172.21.15.110:0/4192289353 learned_addr learned my addr 172.21.15.110:0/4192289353 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c29d9b640 10 monclient(hunting): get_auth_request con 0x7f4c2c0af790 auth_method 0 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c29d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2ad9d640 10 monclient(hunting): get_auth_request con 0x7f4c2c0b0900 auth_method 0 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2a59c640 10 monclient(hunting): get_auth_request con 0x7f4c2c06bb00 auth_method 0 2024-01-20T11:19:55.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:55.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.556+0000 7f4c2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 10 monclient(hunting): handle_auth_done global_id 5251 payload 824 2024-01-20T11:19:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 1 -- 172.21.15.110:0/4192289353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0b0900 msgr2=0x7f4c2c13bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0b0900 0x7f4c2c13bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 1 -- 172.21.15.110:0/4192289353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06bb00 msgr2=0x7f4c2c0b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06bb00 0x7f4c2c0b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 1 monclient: found mon.noname-b 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.559+0000 7f4c29d9b640 1 -- 172.21.15.110:0/4192289353 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c24009a70 con 0x7f4c2c0af790 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.561+0000 7f4c29d9b640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0af790 0x7f4c2c11da60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000e930 tx=0x7f4c2000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.561+0000 7f4c1affd640 10 client.?.objecter ms_handle_connect 0x7f4c2c0af790 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.561+0000 7f4c1affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 1 -- 172.21.15.110:0/4192289353 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20009660 con 0x7f4c2c0af790 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 10 monclient: dump: 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:55.566 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 10 monclient: _finish_auth 0 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 1 -- 172.21.15.110:0/4192289353 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2000ca00 con 0x7f4c2c0af790 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 1 -- 172.21.15.110:0/4192289353 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2000cd70 con 0x7f4c2c0af790 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:55.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c1affd640 10 monclient: dump: 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c30f3c640 5 monclient: authenticate success, global_id 5251 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c28d99640 4 set_mon_vals no callback set 2024-01-20T11:19:55.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c30f3c640 10 monclient: _renew_subs 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c30f3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c141780 con 0x7f4c2c0af790 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c30f3c640 10 client.5251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c30f3c640 10 monclient: _renew_subs 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c30f3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c13c350 con 0x7f4c2c0af790 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.564+0000 7f4c30f3c640 1 librados: init done 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.565+0000 7f4c30f3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.565+0000 7f4c30f3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.565+0000 7f4c30f3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:55.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.565+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c141d20 con 0x7f4c2c0af790 2024-01-20T11:19:55.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.802+0000 7f4c1affd640 1 -- 172.21.15.110:0/4192289353 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4c20026480 con 0x7f4c2c0af790 2024-01-20T11:19:55.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.803+0000 7f4c1affd640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c0c06c8b0 0x7f4c0c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:55.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.803+0000 7f4c2a59c640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c0c06c8b0 0x7f4c0c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:55.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.803+0000 7f4c2a59c640 10 monclient: get_auth_request con 0x7f4c0c06c8b0 auth_method 0 2024-01-20T11:19:55.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.804+0000 7f4c1affd640 10 client.5251.objecter ms_handle_connect 0x7f4c0c06c8b0 2024-01-20T11:19:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.804+0000 7f4c2a59c640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c0c06c8b0 0x7f4c0c06ed70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009bd0 tx=0x7f4c1c009240 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:55.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.805+0000 7f4c1affd640 1 -- 172.21.15.110:0/4192289353 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4c20058c40 con 0x7f4c2c0af790 2024-01-20T11:19:55.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.805+0000 7f4c1affd640 10 client.5251.objecter ms_dispatch 0x7f4c2c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:55.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.805+0000 7f4c1affd640 3 client.5251.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:55.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.805+0000 7f4c1affd640 3 client.5251.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:55.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.805+0000 7f4c1affd640 20 client.5251.objecter dump_active .. 0 homeless 2024-01-20T11:19:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.815+0000 7f4c1affd640 1 -- 172.21.15.110:0/4192289353 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4c20054f80 con 0x7f4c2c0af790 2024-01-20T11:19:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.815+0000 7f4c1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.815+0000 7f4c1affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:55.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.983+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c2c1461f0 con 0x7f4c0c06c8b0 2024-01-20T11:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.986+0000 7f4c1affd640 1 -- 172.21.15.110:0/4192289353 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63978 (secure 0 0 0) 0x7f4c1c011af0 con 0x7f4c0c06c8b0 2024-01-20T11:19:55.986 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 10 librados: watch_flush enter 2024-01-20T11:19:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 10 librados: watch_flush exit 2024-01-20T11:19:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 20 client.5251.objecter shutdown clearing up homeless session... 2024-01-20T11:19:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 10 client.5251.objecter successfully canceled tick 2024-01-20T11:19:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c0c06c8b0 msgr2=0x7f4c0c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c0c06c8b0 0x7f4c0c06ed70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009bd0 tx=0x7f4c1c009240 comp rx=0 tx=0).stop 2024-01-20T11:19:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 10 monclient: shutdown 2024-01-20T11:19:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0af790 msgr2=0x7f4c2c11da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0af790 0x7f4c2c11da60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000e930 tx=0x7f4c2000edf0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 shutdown_connections 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c0c06c8b0 0x7f4c0c06ed70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c0b0900 0x7f4c2c13bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c0af790 0x7f4c2c11da60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 --2- 172.21.15.110:0/4192289353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06bb00 0x7f4c2c0b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 >> 172.21.15.110:0/4192289353 conn(0x7f4c2c066d60 msgr2=0x7f4c2c0b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 shutdown_connections 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 -- 172.21.15.110:0/4192289353 wait complete. 2024-01-20T11:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:55.988+0000 7f4c30f3c640 1 librados: shutdown 2024-01-20T11:19:55.999 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-01-20T11:19:54.764478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002556"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280574,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.0020000000000007},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.6070000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":13.903},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":16.905999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.231},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":16.082999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.337},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":9.2439999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.273999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":17.809999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.960000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":14.974}]}]},{"osd":7,"up_from":21,"seq":90194313276,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.2149999999999999},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.9429999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.6749999999999998},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.6869999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":15.92},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":7.2720000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.125},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.52}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.337},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":10.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.157},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.895}]}]},{"osd":3,"up_from":21,"seq":90194313277,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.9189999999999996},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6390000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.9000000000000004},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":8.4900000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":14.903},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.272}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.262},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.1479999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":9.7420000000000009},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":12.861000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":11.696999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.002000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444091,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.9210000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.09},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.460000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.064},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":7.577}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.872999999999999},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.8089999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":14.500999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.587999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.162000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":15.221}]}]},{"osd":1,"up_from":17,"seq":73014444092,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.4790000000000001},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":11.869}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":13.246},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":6.3230000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":9.4019999999999992},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.7859999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":14.077},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":8.1660000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":10.708},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":15.013999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.284000000000001},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.808}]}]},{"osd":0,"up_from":17,"seq":73014444092,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":5.3810000000000002},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":8.5869999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":11.257},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":6.1159999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":9.2919999999999998},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":14.683}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.323},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.0250000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":7.2800000000000002},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":13.118}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.911},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":10.667}]}]},{"osd":6,"up_from":15,"seq":64424509500,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":6.3600000000000003},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.8179999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.4560000000000004},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":10.553000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.081},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3670000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":11.679},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.724}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.869},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.326000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.164999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.059999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:58.718 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:49.742 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:58.719 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:49.744 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:58.733 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:49.758 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:58.761 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:05:49.767 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:58.764 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:05:49.771 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:58.765 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:49.789 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:58.766 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:49.791 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:58.779 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:05:49.786 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:58.783 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:49.808 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:19:59.002 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.102+0000 7f499e51d640 10 monclient: get_monmap_and_config 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.102+0000 7f499e51d640 10 monclient: build_initial_monmap 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.102+0000 7f499e51d640 10 monclient: monmap: 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:59.102696+0000 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:59.102696+0000 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:59.103 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.102+0000 7f499e51d640 1 Processor -- start 2024-01-20T11:19:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.102+0000 7f499e51d640 1 -- start start 2024-01-20T11:19:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.102+0000 7f499e51d640 10 monclient: init 2024-01-20T11:19:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-01-20T11:19:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4998145760 0x7f4998145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 10 monclient(hunting): picked mon.noname-c con 0x7f4998145760 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49980a9d80 addr v1:172.21.15.110:6789/0 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f499813f8b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4998146250 con 0x7f49980a9d80 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49980b20c0 con 0x7f499813f8b0 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49980ab050 con 0x7f4998145760 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.103+0000 7f499e51d640 10 monclient(hunting): authenticate will time out at 1642.453633s 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.104+0000 7f4997fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4998145760 0x7f4998145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.104+0000 7f4997fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4998145760 0x7f4998145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:60278/0 (socket says 172.21.15.110:60278) 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.104+0000 7f4997fff640 1 -- 172.21.15.110:0/491638584 learned_addr learned my addr 172.21.15.110:0/491638584 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.104+0000 7f4997fff640 10 monclient(hunting): get_auth_request con 0x7f4998145760 auth_method 0 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.104+0000 7f4997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.104+0000 7f4997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.104+0000 7f4997fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:59.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.105+0000 7f4997fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:59.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.105+0000 7f4997fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:59.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.105+0000 7f4997fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:59.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.106+0000 7f4997fff640 10 monclient(hunting): handle_auth_done global_id 5254 payload 274 2024-01-20T11:19:59.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.106+0000 7f4997fff640 1 -- 172.21.15.110:0/491638584 >> v1:172.21.15.134:6789/0 conn(0x7f499813f8b0 legacy=0x7f49980b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:59.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.106+0000 7f4997fff640 1 -- 172.21.15.110:0/491638584 >> v1:172.21.15.110:6789/0 conn(0x7f49980a9d80 legacy=0x7f499813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:59.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.106+0000 7f4997fff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:59.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.106+0000 7f4997fff640 1 monclient: found mon.noname-c 2024-01-20T11:19:59.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.106+0000 7f4997fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:59.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.106+0000 7f4997fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:19:59.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.106+0000 7f4997fff640 1 -- 172.21.15.110:0/491638584 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49980b2d80 con 0x7f4998145760 2024-01-20T11:19:59.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.108+0000 7f4997fff640 1 --2- 172.21.15.110:0/491638584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 0x7f4998145b40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f498c0098a0 tx=0x7f498c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2f98642abd8815 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 1 -- 172.21.15.110:0/491638584 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c030840 con 0x7f4998145760 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 10 monclient: dump: 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:59.113 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 10 monclient: _finish_auth 0 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 1 -- 172.21.15.110:0/491638584 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c0309a0 con 0x7f4998145760 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 1 -- 172.21.15.110:0/491638584 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c030d30 con 0x7f4998145760 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 5 monclient: authenticate success, global_id 5254 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 4 set_mon_vals no callback set 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f4996ffd640 10 monclient: dump: 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.115 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.115 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.115 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:59.115 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.115 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:59.115 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:59.115 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:59.115 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 10 monclient: shutdown 2024-01-20T11:19:59.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:59.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 1 -- 172.21.15.110:0/491638584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 msgr2=0x7f4998145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:59.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 1 --2- 172.21.15.110:0/491638584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 0x7f4998145b40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f498c0098a0 tx=0x7f498c02b980 comp rx=0 tx=0).stop 2024-01-20T11:19:59.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 1 -- 172.21.15.110:0/491638584 shutdown_connections 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 1 --2- 172.21.15.110:0/491638584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 0x7f4998145b40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 1 -- 172.21.15.110:0/491638584 >> 172.21.15.110:0/491638584 conn(0x7f49980663b0 msgr2=0x7f49980667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 1 -- 172.21.15.110:0/491638584 shutdown_connections 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.113+0000 7f499e51d640 1 -- 172.21.15.110:0/491638584 wait complete. 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.114+0000 7f499e51d640 10 monclient: build_initial_monmap 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.114+0000 7f499e51d640 10 monclient: monmap: 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.114+0000 7f499e51d640 1 librados: starting msgr at 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.114+0000 7f499e51d640 1 librados: starting objecter 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.114+0000 7f499e51d640 1 Processor -- start 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 1 -- start start 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 1 librados: setting wanted keys 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 1 librados: calling monclient init 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient: init 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49980a9d80 0x7f4998142650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient(hunting): picked mon.noname-c con 0x7f49980a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f499813f8b0 0x7f49981dabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient(hunting): picked mon.noname-a con 0x7f499813f8b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 0x7f49981db0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient(hunting): picked mon.noname-b con 0x7f4998145760 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4998137da0 con 0x7f499813f8b0 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4998138080 con 0x7f4998145760 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4998136320 con 0x7f49980a9d80 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.115+0000 7f499e51d640 10 monclient(hunting): authenticate will time out at 1642.465874s 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f49977fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f499813f8b0 0x7f49981dabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f4997fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49980a9d80 0x7f4998142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f499caba640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 0x7f49981db0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f49977fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f499813f8b0 0x7f49981dabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54616/0 (socket says 172.21.15.110:54616) 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f49977fe640 1 -- 172.21.15.110:0/13544773 learned_addr learned my addr 172.21.15.110:0/13544773 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f49977fe640 10 monclient(hunting): get_auth_request con 0x7f499813f8b0 auth_method 0 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f49977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f49977fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f49977fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f4997fff640 10 monclient(hunting): get_auth_request con 0x7f49980a9d80 auth_method 0 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f4997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f4997fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f4997fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f499caba640 10 monclient(hunting): get_auth_request con 0x7f4998145760 auth_method 0 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f499caba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f499caba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.116+0000 7f499caba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.117+0000 7f49977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.117+0000 7f49977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.117+0000 7f49977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.117+0000 7f499caba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.117+0000 7f499caba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.117+0000 7f499caba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.117+0000 7f4997fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.117+0000 7f4997fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.117+0000 7f4997fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 10 monclient(hunting): handle_auth_done global_id 5257 payload 824 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 1 -- 172.21.15.110:0/13544773 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49980a9d80 msgr2=0x7f4998142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49980a9d80 0x7f4998142650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 1 -- 172.21.15.110:0/13544773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f499813f8b0 msgr2=0x7f49981dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f499813f8b0 0x7f49981dabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 10 monclient: _finish_hunting 0 2024-01-20T11:19:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 1 monclient: found mon.noname-b 2024-01-20T11:19:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.119+0000 7f499caba640 1 -- 172.21.15.110:0/13544773 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c0094d0 con 0x7f4998145760 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.121+0000 7f499caba640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 0x7f49981db0f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f499000c750 tx=0x7f499000cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.121+0000 7f4994ff9640 10 client.?.objecter ms_handle_connect 0x7f4998145760 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.122+0000 7f4994ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 1 -- 172.21.15.110:0/13544773 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990003b70 con 0x7f4998145760 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 10 monclient: dump: 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 10 monclient: _finish_auth 0 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 1 -- 172.21.15.110:0/13544773 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4990003cf0 con 0x7f4998145760 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 1 -- 172.21.15.110:0/13544773 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990003ea0 con 0x7f4998145760 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4994ff9640 10 monclient: dump: 2024-01-20T11:19:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f499e51d640 5 monclient: authenticate success, global_id 5257 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f499e51d640 10 monclient: _renew_subs 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f499e51d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f4995ffb640 4 set_mon_vals no callback set 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4998136840 con 0x7f4998145760 2024-01-20T11:19:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f499e51d640 10 client.5257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f499e51d640 10 monclient: _renew_subs 2024-01-20T11:19:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f499e51d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4998137170 con 0x7f4998145760 2024-01-20T11:19:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.125+0000 7f499e51d640 1 librados: init done 2024-01-20T11:19:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.126+0000 7f499e51d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.126+0000 7f499e51d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.126+0000 7f499e51d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.126+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4998075490 con 0x7f4998145760 2024-01-20T11:19:59.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.367+0000 7f4994ff9640 1 -- 172.21.15.110:0/13544773 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4990006b10 con 0x7f4998145760 2024-01-20T11:19:59.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.367+0000 7f4994ff9640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f498406c860 0x7f498406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.367+0000 7f4997fff640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f498406c860 0x7f498406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.367+0000 7f4997fff640 10 monclient: get_auth_request con 0x7f498406c860 auth_method 0 2024-01-20T11:19:59.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.368+0000 7f4994ff9640 10 client.5257.objecter ms_handle_connect 0x7f498406c860 2024-01-20T11:19:59.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.368+0000 7f4997fff640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f498406c860 0x7f498406ed20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f498c036000 tx=0x7f498c008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:59.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.370+0000 7f4994ff9640 1 -- 172.21.15.110:0/13544773 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4990057930 con 0x7f4998145760 2024-01-20T11:19:59.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.370+0000 7f4994ff9640 10 client.5257.objecter ms_dispatch 0x7f4998000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:59.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.370+0000 7f4994ff9640 3 client.5257.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:59.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.370+0000 7f4994ff9640 3 client.5257.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:59.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.370+0000 7f4994ff9640 20 client.5257.objecter dump_active .. 0 homeless 2024-01-20T11:19:59.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.384+0000 7f4994ff9640 1 -- 172.21.15.110:0/13544773 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4990053c70 con 0x7f4998145760 2024-01-20T11:19:59.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.384+0000 7f4994ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:59.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.384+0000 7f4994ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:19:59.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.551+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49980b3340 con 0x7f498406c860 2024-01-20T11:19:59.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.554+0000 7f4994ff9640 1 -- 172.21.15.110:0/13544773 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63935 (secure 0 0 0) 0x7f498c030c10 con 0x7f498406c860 2024-01-20T11:19:59.554 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:19:59.555 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:19:59.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 10 librados: watch_flush enter 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 10 librados: watch_flush exit 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 20 client.5257.objecter shutdown clearing up homeless session... 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 10 client.5257.objecter successfully canceled tick 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f498406c860 msgr2=0x7f498406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f498406c860 0x7f498406ed20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f498c036000 tx=0x7f498c008030 comp rx=0 tx=0).stop 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 10 monclient: shutdown 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 msgr2=0x7f49981db0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 0x7f49981db0f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f499000c750 tx=0x7f499000cc10 comp rx=0 tx=0).stop 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 shutdown_connections 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f498406c860 0x7f498406ed20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4998145760 0x7f49981db0f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f499813f8b0 0x7f49981dabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 --2- 172.21.15.110:0/13544773 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49980a9d80 0x7f4998142650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 >> 172.21.15.110:0/13544773 conn(0x7f49980663b0 msgr2=0x7f49980b7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:59.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 shutdown_connections 2024-01-20T11:19:59.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 -- 172.21.15.110:0/13544773 wait complete. 2024-01-20T11:19:59.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.556+0000 7f499e51d640 1 librados: shutdown 2024-01-20T11:19:59.566 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-01-20T11:19:58.765168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":13.589},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.0619999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.119},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.2110000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":5.7240000000000002},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.693}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.7479999999999993},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":14.867000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.619},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":15.606}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7460000000000004},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":13.265000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313277,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.8979999999999997},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.0389999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.8419999999999996},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.519}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.782},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.6760000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.9569999999999999},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.359}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.483000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.3149999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.119999999999999},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.427}]}]},{"osd":3,"up_from":21,"seq":90194313278,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.5830000000000002},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.3130000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.6400000000000006},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.081}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":11.207000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":16.100999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.206},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.1}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":7.1360000000000001},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.879}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":17.378},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.083}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444092,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":10.474},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":7.3719999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.5389999999999997},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.09}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.874000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.8520000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.827},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.6470000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.506},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.173999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.678000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.16}]}]},{"osd":1,"up_from":17,"seq":73014444093,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.6699999999999999},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.9610000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.132},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.9030000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":11.708},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.1639999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":9.6859999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":10.254},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":14.242000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.962999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.992000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444093,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":11.095000000000001},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":5.0419999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.6890000000000001},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.476000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":16.576000000000001},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.911}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.637},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.0030000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.885},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.657}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":17.866},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":10.653}]}]},{"osd":6,"up_from":15,"seq":64424509501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":12.169},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.6280000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":10.831},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":14.772}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.7140000000000004},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.3630000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":9.3900000000000006},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.901}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.509},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.599}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.919},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.132999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:19:59.568 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:19:59.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.667+0000 7f31aaf1d640 10 monclient: get_monmap_and_config 2024-01-20T11:19:59.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.667+0000 7f31aaf1d640 10 monclient: build_initial_monmap 2024-01-20T11:19:59.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.667+0000 7f31aaf1d640 10 monclient: monmap: 2024-01-20T11:19:59.667 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:59.667 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:19:59.668089+0000 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:19:59.668089+0000 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:19:59.668 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 1 Processor -- start 2024-01-20T11:19:59.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 1 -- start start 2024-01-20T11:19:59.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient: init 2024-01-20T11:19:59.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:59.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-01-20T11:19:59.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f31a4145760 0x7f31a4145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f31a4145760 addr v2:172.21.15.134:3300/0 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a40a9d80 0x7f31a413f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f31a40a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31a413f6e0 addr v1:172.21.15.134:6789/0 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a4146250 con 0x7f31a413f6e0 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a40ab050 con 0x7f31a4145760 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a413fb90 con 0x7f31a40a9d80 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.668+0000 7f31aaf1d640 10 monclient(hunting): authenticate will time out at 1643.019113s 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a40a9d80 0x7f31a413f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a8cb9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f31a4145760 0x7f31a4145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a40a9d80 0x7f31a413f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57908/0 (socket says 172.21.15.110:57908) 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a3fff640 1 -- 172.21.15.110:0/3598772291 learned_addr learned my addr 172.21.15.110:0/3598772291 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a3fff640 10 monclient(hunting): get_auth_request con 0x7f31a40a9d80 auth_method 0 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a8cb9640 10 monclient(hunting): get_auth_request con 0x7f31a4145760 auth_method 0 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a8cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a8cb9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.669+0000 7f31a8cb9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.670+0000 7f31a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.670+0000 7f31a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.670+0000 7f31a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.670+0000 7f31a8cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.670+0000 7f31a8cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:59.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.670+0000 7f31a8cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:59.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.672+0000 7f31a3fff640 10 monclient(hunting): handle_auth_done global_id 4967 payload 274 2024-01-20T11:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.672+0000 7f31a3fff640 1 -- 172.21.15.110:0/3598772291 >> v2:172.21.15.134:3300/0 conn(0x7f31a4145760 msgr2=0x7f31a4145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.672+0000 7f31a3fff640 1 --2- 172.21.15.110:0/3598772291 >> v2:172.21.15.134:3300/0 conn(0x7f31a4145760 0x7f31a4145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.672+0000 7f31a3fff640 1 -- 172.21.15.110:0/3598772291 >> v1:172.21.15.134:6789/0 conn(0x7f31a413f6e0 legacy=0x7f31a40b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.672+0000 7f31a3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.672+0000 7f31a3fff640 1 monclient: found mon.noname-b 2024-01-20T11:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.672+0000 7f31a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.672+0000 7f31a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.672+0000 7f31a3fff640 1 -- 172.21.15.110:0/3598772291 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a40b20c0 con 0x7f31a40a9d80 2024-01-20T11:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.673+0000 7f31a3fff640 1 --2- 172.21.15.110:0/3598772291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a40a9d80 0x7f31a413f1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3190009870 tx=0x7f319002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cad309809ee0e12e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 1 -- 172.21.15.110:0/3598772291 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190031640 con 0x7f31a40a9d80 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 10 monclient: dump: 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:59.679 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 10 monclient: _finish_auth 0 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 1 -- 172.21.15.110:0/3598772291 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31900317a0 con 0x7f31a40a9d80 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31aaf1d640 5 monclient: authenticate success, global_id 4967 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31aaf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31aaf1d640 10 monclient: get_monmap_and_config success 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 1 -- 172.21.15.110:0/3598772291 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190031b30 con 0x7f31a40a9d80 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31aaf1d640 4 set_mon_vals no callback set 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.677+0000 7f31a37fe640 10 monclient: dump: 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 10 monclient: shutdown 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3598772291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a40a9d80 msgr2=0x7f31a413f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 1 --2- 172.21.15.110:0/3598772291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a40a9d80 0x7f31a413f1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3190009870 tx=0x7f319002b8f0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3598772291 shutdown_connections 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 1 --2- 172.21.15.110:0/3598772291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a40a9d80 0x7f31a413f1a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 1 --2- 172.21.15.110:0/3598772291 >> v2:172.21.15.134:3300/0 conn(0x7f31a4145760 0x7f31a4145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3598772291 >> 172.21.15.110:0/3598772291 conn(0x7f31a40663b0 msgr2=0x7f31a40667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3598772291 shutdown_connections 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3598772291 wait complete. 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 10 monclient: build_initial_monmap 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.678+0000 7f31aaf1d640 10 monclient: monmap: 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:19:59.682 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:19:59.683 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:19:59.683 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.679+0000 7f31aaf1d640 1 librados: starting msgr at 2024-01-20T11:19:59.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.679+0000 7f31aaf1d640 1 librados: starting objecter 2024-01-20T11:19:59.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.679+0000 7f31aaf1d640 1 Processor -- start 2024-01-20T11:19:59.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.679+0000 7f31aaf1d640 1 -- start start 2024-01-20T11:19:59.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.679+0000 7f31aaf1d640 1 librados: setting wanted keys 2024-01-20T11:19:59.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.679+0000 7f31aaf1d640 1 librados: calling monclient init 2024-01-20T11:19:59.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.679+0000 7f31aaf1d640 10 monclient: init 2024-01-20T11:19:59.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient: _reopen_session rank -1 2024-01-20T11:19:59.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a40a9d80 0x7f31a4142580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f31a40a9d80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31a413f6e0 0x7f31a41daa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f31a413f6e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a4145760 0x7f31a41dafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f31a4145760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a413a6f0 con 0x7f31a413f6e0 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40b64a0 con 0x7f31a40a9d80 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40b6710 con 0x7f31a4145760 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient(hunting): _renew_subs 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31aaf1d640 10 monclient(hunting): authenticate will time out at 1643.030274s 2024-01-20T11:19:59.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31a3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31a413f6e0 0x7f31a41daa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31a94ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a4145760 0x7f31a41dafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.680+0000 7f31a8cb9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a40a9d80 0x7f31a4142580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31a413f6e0 0x7f31a41daa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36176/0 (socket says 172.21.15.110:36176) 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a3fff640 1 -- 172.21.15.110:0/3191157830 learned_addr learned my addr 172.21.15.110:0/3191157830 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a3fff640 10 monclient(hunting): get_auth_request con 0x7f31a413f6e0 auth_method 0 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a94ba640 10 monclient(hunting): get_auth_request con 0x7f31a4145760 auth_method 0 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a94ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a94ba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a94ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a8cb9640 10 monclient(hunting): get_auth_request con 0x7f31a40a9d80 auth_method 0 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a8cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:19:59.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a8cb9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a8cb9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a94ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a94ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a94ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a8cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a8cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.681+0000 7f31a8cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 10 monclient(hunting): handle_auth_done global_id 5263 payload 824 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 1 -- 172.21.15.110:0/3191157830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a4145760 msgr2=0x7f31a41dafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a4145760 0x7f31a41dafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 1 -- 172.21.15.110:0/3191157830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31a413f6e0 msgr2=0x7f31a41daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:19:59.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31a413f6e0 0x7f31a41daa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:19:59.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 10 monclient: _finish_hunting 0 2024-01-20T11:19:59.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 1 monclient: found mon.noname-b 2024-01-20T11:19:59.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:19:59.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:19:59.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a8cb9640 1 -- 172.21.15.110:0/3191157830 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31900094d0 con 0x7f31a40a9d80 2024-01-20T11:19:59.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.684+0000 7f31a3fff640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31a413f6e0 0x7f31a41daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:19:59.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.686+0000 7f31a8cb9640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a40a9d80 0x7f31a4142580 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f319800e870 tx=0x7f319800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:59.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.686+0000 7f31a17fa640 10 client.?.objecter ms_handle_connect 0x7f31a40a9d80 2024-01-20T11:19:59.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.686+0000 7f31a17fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:19:59.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 1 -- 172.21.15.110:0/3191157830 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319800ee60 con 0x7f31a40a9d80 2024-01-20T11:19:59.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:59.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 10 monclient: dump: 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 10 monclient: _finish_auth 0 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 10 monclient: _check_auth_tickets 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:19:59.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 1 -- 172.21.15.110:0/3191157830 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319800c940 con 0x7f31a40a9d80 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31aaf1d640 5 monclient: authenticate success, global_id 5263 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31aaf1d640 10 monclient: _renew_subs 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31aaf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a413ab80 con 0x7f31a40a9d80 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 1 -- 172.21.15.110:0/3191157830 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319800ccd0 con 0x7f31a40a9d80 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a17fa640 10 monclient: dump: 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:19:59.692 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31a2ffd640 4 set_mon_vals no callback set 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31aaf1d640 10 client.5263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31aaf1d640 10 monclient: _renew_subs 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31aaf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a40b56b0 con 0x7f31a40a9d80 2024-01-20T11:19:59.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.690+0000 7f31aaf1d640 1 librados: init done 2024-01-20T11:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.691+0000 7f31aaf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.691+0000 7f31aaf1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.691+0000 7f31aaf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.691+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a4075490 con 0x7f31a40a9d80 2024-01-20T11:19:59.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.931+0000 7f31a17fa640 1 -- 172.21.15.110:0/3191157830 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3198013470 con 0x7f31a40a9d80 2024-01-20T11:19:59.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.931+0000 7f31a17fa640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f318806c8e0 0x7f318806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:19:59.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.932+0000 7f31a3fff640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f318806c8e0 0x7f318806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:19:59.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.932+0000 7f31a3fff640 10 monclient: get_auth_request con 0x7f318806c8e0 auth_method 0 2024-01-20T11:19:59.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.932+0000 7f31a3fff640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f318806c8e0 0x7f318806eda0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f31a40b6bc0 tx=0x7f319003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:19:59.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.932+0000 7f31a17fa640 10 client.5263.objecter ms_handle_connect 0x7f318806c8e0 2024-01-20T11:19:59.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.936+0000 7f31a17fa640 1 -- 172.21.15.110:0/3191157830 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f319800ce30 con 0x7f31a40a9d80 2024-01-20T11:19:59.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.936+0000 7f31a17fa640 10 client.5263.objecter ms_dispatch 0x7f31a4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:19:59.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.936+0000 7f31a17fa640 3 client.5263.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:19:59.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.936+0000 7f31a17fa640 3 client.5263.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:19:59.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.936+0000 7f31a17fa640 20 client.5263.objecter dump_active .. 0 homeless 2024-01-20T11:19:59.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.946+0000 7f31a17fa640 1 -- 172.21.15.110:0/3191157830 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f3198054cc0 con 0x7f31a40a9d80 2024-01-20T11:19:59.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.946+0000 7f31a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:19:59.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:19:59.946+0000 7f31a17fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.130+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f31a413b0c0 con 0x7f318806c8e0 2024-01-20T11:20:00.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.133+0000 7f31a17fa640 1 -- 172.21.15.110:0/3191157830 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63935 (secure 0 0 0) 0x7f3190031540 con 0x7f318806c8e0 2024-01-20T11:20:00.133 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:00.134 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:00.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 10 librados: watch_flush enter 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 10 librados: watch_flush exit 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 20 client.5263.objecter shutdown clearing up homeless session... 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 10 client.5263.objecter successfully canceled tick 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f318806c8e0 msgr2=0x7f318806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f318806c8e0 0x7f318806eda0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f31a40b6bc0 tx=0x7f319003c040 comp rx=0 tx=0).stop 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 10 monclient: shutdown 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a40a9d80 msgr2=0x7f31a4142580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a40a9d80 0x7f31a4142580 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f319800e870 tx=0x7f319800ed30 comp rx=0 tx=0).stop 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 shutdown_connections 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f318806c8e0 0x7f318806eda0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31a4145760 0x7f31a41dafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31a413f6e0 0x7f31a41daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 --2- 172.21.15.110:0/3191157830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a40a9d80 0x7f31a4142580 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 >> 172.21.15.110:0/3191157830 conn(0x7f31a40663b0 msgr2=0x7f31a41376f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:00.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 shutdown_connections 2024-01-20T11:20:00.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 -- 172.21.15.110:0/3191157830 wait complete. 2024-01-20T11:20:00.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.135+0000 7f31aaf1d640 1 librados: shutdown 2024-01-20T11:20:00.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~62s 2024-01-20T11:20:00.145 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-01-20T11:19:58.765168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":13.589},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.0619999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.119},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.2110000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":5.7240000000000002},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.693}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.7479999999999993},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":14.867000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.619},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":15.606}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7460000000000004},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":13.265000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313277,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.8979999999999997},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.0389999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.8419999999999996},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.519}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.782},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.6760000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.9569999999999999},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.359}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.483000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.3149999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.119999999999999},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.427}]}]},{"osd":3,"up_from":21,"seq":90194313278,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.5830000000000002},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.3130000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.6400000000000006},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.081}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":11.207000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":16.100999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.206},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.1}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":7.1360000000000001},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.879}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":17.378},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.083}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444092,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":10.474},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":7.3719999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.5389999999999997},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.09}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.874000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.8520000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.827},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.6470000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.506},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.173999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.678000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.16}]}]},{"osd":1,"up_from":17,"seq":73014444093,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.6699999999999999},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.9610000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.132},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.9030000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":11.708},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.1639999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":9.6859999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":10.254},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":14.242000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.962999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.992000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444093,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":11.095000000000001},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":5.0419999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.6890000000000001},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.476000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":16.576000000000001},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.911}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.637},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.0030000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.885},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.657}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":17.866},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":10.653}]}]},{"osd":6,"up_from":15,"seq":64424509501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":12.169},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.6280000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":10.831},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":14.772}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.7140000000000004},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.3630000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":9.3900000000000006},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.901}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.509},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.599}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.919},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.132999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:00.147 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:00.147 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:00.147 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:00.147 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:00.147 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:00.147 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:00.147 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:00.148 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:00.148 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:00.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.248+0000 7f72323c9640 10 monclient: get_monmap_and_config 2024-01-20T11:20:00.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.248+0000 7f72323c9640 10 monclient: build_initial_monmap 2024-01-20T11:20:00.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.248+0000 7f72323c9640 10 monclient: monmap: 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:00.249325+0000 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:00.249325+0000 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:00.249 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.249+0000 7f72323c9640 1 Processor -- start 2024-01-20T11:20:00.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.249+0000 7f72323c9640 1 -- start start 2024-01-20T11:20:00.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.249+0000 7f72323c9640 10 monclient: init 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.249+0000 7f72323c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.249+0000 7f72323c9640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.249+0000 7f72323c9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f722c0b0ba0 0x7f722c0ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 10 monclient(hunting): picked mon.noname-a con 0x7f722c0b0ba0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f722c0adc30 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f722c0ac4d0 0x7f722c13bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 10 monclient(hunting): picked mon.noname-c con 0x7f722c0ac4d0 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722c0ad530 con 0x7f722c0adc30 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722c0af840 con 0x7f722c0b0ba0 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722c0b0f80 con 0x7f722c0ac4d0 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f72323c9640 10 monclient(hunting): authenticate will time out at 1643.600231s 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f7230966640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f722c0ac4d0 0x7f722c13bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.250+0000 7f722bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f722c0b0ba0 0x7f722c0ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f7230966640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f722c0ac4d0 0x7f722c13bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:38516/0 (socket says 172.21.15.110:38516) 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f7230966640 1 -- 172.21.15.110:0/3237693544 learned_addr learned my addr 172.21.15.110:0/3237693544 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f7230966640 10 monclient(hunting): get_auth_request con 0x7f722c0ac4d0 auth_method 0 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f7230966640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f7230966640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f7230966640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f722bfff640 10 monclient(hunting): get_auth_request con 0x7f722c0b0ba0 auth_method 0 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f722bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f722bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f722bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f7230966640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f7230966640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f7230966640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f722bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f722bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.251+0000 7f722bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.253+0000 7f7230966640 10 monclient(hunting): handle_auth_done global_id 5266 payload 274 2024-01-20T11:20:00.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.253+0000 7f7230966640 1 -- 172.21.15.110:0/3237693544 >> v2:172.21.15.110:3300/0 conn(0x7f722c0b0ba0 msgr2=0x7f722c0ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:00.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.253+0000 7f7230966640 1 --2- 172.21.15.110:0/3237693544 >> v2:172.21.15.110:3300/0 conn(0x7f722c0b0ba0 0x7f722c0ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.253+0000 7f7230966640 1 -- 172.21.15.110:0/3237693544 >> v1:172.21.15.110:6789/0 conn(0x7f722c0adc30 legacy=0x7f722c0af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.253+0000 7f7230966640 10 monclient: _finish_hunting 0 2024-01-20T11:20:00.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.253+0000 7f7230966640 1 monclient: found mon.noname-c 2024-01-20T11:20:00.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.253+0000 7f7230966640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:00.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.253+0000 7f7230966640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.253+0000 7f7230966640 1 -- 172.21.15.110:0/3237693544 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c0ad810 con 0x7f722c0ac4d0 2024-01-20T11:20:00.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.254+0000 7f7230966640 1 --2- 172.21.15.110:0/3237693544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c0ac4d0 0x7f722c13bf30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f722c0ac980 tx=0x7f722402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83a470b833390fc3 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:00.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 1 -- 172.21.15.110:0/3237693544 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722402ca80 con 0x7f722c0ac4d0 2024-01-20T11:20:00.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 10 monclient: dump: 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 10 monclient: _finish_auth 0 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 5 monclient: authenticate success, global_id 5266 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 1 -- 172.21.15.110:0/3237693544 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722402cbe0 con 0x7f722c0ac4d0 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 4 set_mon_vals no callback set 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 1 -- 172.21.15.110:0/3237693544 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72240343f0 con 0x7f722c0ac4d0 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f722affd640 10 monclient: dump: 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 10 monclient: shutdown 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 1 -- 172.21.15.110:0/3237693544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c0ac4d0 msgr2=0x7f722c13bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 1 --2- 172.21.15.110:0/3237693544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c0ac4d0 0x7f722c13bf30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f722c0ac980 tx=0x7f722402bee0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 1 -- 172.21.15.110:0/3237693544 shutdown_connections 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 1 --2- 172.21.15.110:0/3237693544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c0ac4d0 0x7f722c13bf30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.258+0000 7f72323c9640 1 --2- 172.21.15.110:0/3237693544 >> v2:172.21.15.110:3300/0 conn(0x7f722c0b0ba0 0x7f722c0ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.259+0000 7f72323c9640 1 -- 172.21.15.110:0/3237693544 >> 172.21.15.110:0/3237693544 conn(0x7f722c0b3e60 msgr2=0x7f722c0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.259+0000 7f72323c9640 1 -- 172.21.15.110:0/3237693544 shutdown_connections 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.259+0000 7f72323c9640 1 -- 172.21.15.110:0/3237693544 wait complete. 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.259+0000 7f72323c9640 10 monclient: build_initial_monmap 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.259+0000 7f72323c9640 10 monclient: monmap: 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.259+0000 7f72323c9640 1 librados: starting msgr at 2024-01-20T11:20:00.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.259+0000 7f72323c9640 1 librados: starting objecter 2024-01-20T11:20:00.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 Processor -- start 2024-01-20T11:20:00.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 -- start start 2024-01-20T11:20:00.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 librados: setting wanted keys 2024-01-20T11:20:00.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 librados: calling monclient init 2024-01-20T11:20:00.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient: init 2024-01-20T11:20:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f722c0ac4d0 0x7f722c1413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient(hunting): picked mon.noname-a con 0x7f722c0ac4d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f722c0b0ba0 0x7f722c071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient(hunting): picked mon.noname-c con 0x7f722c0b0ba0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c1da930 0x7f722c1d7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient(hunting): picked mon.noname-b con 0x7f722c1da930 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c13cd60 con 0x7f722c0ac4d0 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c13cf60 con 0x7f722c1da930 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c1d7810 con 0x7f722c0b0ba0 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.260+0000 7f72323c9640 10 monclient(hunting): authenticate will time out at 1643.610940s 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f722c0ac4d0 0x7f722c1413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f7230966640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c1da930 0x7f722c1d7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f722c0b0ba0 0x7f722c071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f722c0ac4d0 0x7f722c1413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36184/0 (socket says 172.21.15.110:36184) 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722bfff640 1 -- 172.21.15.110:0/1763604883 learned_addr learned my addr 172.21.15.110:0/1763604883 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722bfff640 10 monclient(hunting): get_auth_request con 0x7f722c0ac4d0 auth_method 0 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f7230966640 10 monclient(hunting): get_auth_request con 0x7f722c1da930 auth_method 0 2024-01-20T11:20:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f7230966640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f7230966640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f7230966640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722b7fe640 10 monclient(hunting): get_auth_request con 0x7f722c0b0ba0 auth_method 0 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.261+0000 7f722b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.262+0000 7f7230966640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.262+0000 7f7230966640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.262+0000 7f7230966640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.262+0000 7f722bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.262+0000 7f722bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.262+0000 7f722bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.262+0000 7f722b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.262+0000 7f722b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.262+0000 7f722b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 10 monclient(hunting): handle_auth_done global_id 5269 payload 824 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 1 -- 172.21.15.110:0/1763604883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f722c0b0ba0 msgr2=0x7f722c071880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f722c0b0ba0 0x7f722c071880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 1 -- 172.21.15.110:0/1763604883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f722c0ac4d0 msgr2=0x7f722c1413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f722c0ac4d0 0x7f722c1413f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 10 monclient: _finish_hunting 0 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 1 monclient: found mon.noname-b 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.264+0000 7f7230966640 1 -- 172.21.15.110:0/1763604883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7224009b30 con 0x7f722c1da930 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.266+0000 7f7230966640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c1da930 0x7f722c1d7d10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f722c06c200 tx=0x7f7224004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.266+0000 7f7228ff9640 10 client.?.objecter ms_handle_connect 0x7f722c1da930 2024-01-20T11:20:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.266+0000 7f7228ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 1 -- 172.21.15.110:0/1763604883 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224009510 con 0x7f722c1da930 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 10 monclient: dump: 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 10 monclient: _finish_auth 0 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 1 -- 172.21.15.110:0/1763604883 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722403e450 con 0x7f722c1da930 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f72323c9640 5 monclient: authenticate success, global_id 5269 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f72323c9640 10 monclient: _renew_subs 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f72323c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722c1dade0 con 0x7f722c1da930 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 1 -- 172.21.15.110:0/1763604883 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722403e800 con 0x7f722c1da930 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f7228ff9640 10 monclient: dump: 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f722a7fc640 4 set_mon_vals no callback set 2024-01-20T11:20:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f72323c9640 10 client.5269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f72323c9640 10 monclient: _renew_subs 2024-01-20T11:20:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f72323c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c1d7430 con 0x7f722c1da930 2024-01-20T11:20:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.270+0000 7f72323c9640 1 librados: init done 2024-01-20T11:20:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.271+0000 7f72323c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.271+0000 7f72323c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.271+0000 7f72323c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.271+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722c075490 con 0x7f722c1da930 2024-01-20T11:20:00.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.290+0000 7f7228ff9640 1 -- 172.21.15.110:0/1763604883 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f722402cd50 con 0x7f722c1da930 2024-01-20T11:20:00.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.290+0000 7f7228ff9640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721806c8b0 0x7f721806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.290+0000 7f722bfff640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721806c8b0 0x7f721806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.291+0000 7f722bfff640 10 monclient: get_auth_request con 0x7f721806c8b0 auth_method 0 2024-01-20T11:20:00.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.291+0000 7f722bfff640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721806c8b0 0x7f721806ed70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f721c009b60 tx=0x7f721c009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:00.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.291+0000 7f7228ff9640 10 client.5269.objecter ms_handle_connect 0x7f721806c8b0 2024-01-20T11:20:00.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.292+0000 7f7228ff9640 1 -- 172.21.15.110:0/1763604883 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f722407b2b0 con 0x7f722c1da930 2024-01-20T11:20:00.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.292+0000 7f7228ff9640 10 client.5269.objecter ms_dispatch 0x7f722c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:00.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.292+0000 7f7228ff9640 3 client.5269.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:00.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.292+0000 7f7228ff9640 3 client.5269.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:00.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.292+0000 7f7228ff9640 20 client.5269.objecter dump_active .. 0 homeless 2024-01-20T11:20:00.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.518+0000 7f7228ff9640 1 -- 172.21.15.110:0/1763604883 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f72240775f0 con 0x7f722c1da930 2024-01-20T11:20:00.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.518+0000 7f7228ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:00.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.518+0000 7f7228ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:00.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.759+0000 7f72323c9640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:00.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.759+0000 7f72323c9640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:00.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.759+0000 7f72323c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.759+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f722c0aef30 con 0x7f722c1da930 2024-01-20T11:20:00.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.765+0000 7f7228ff9640 1 -- 172.21.15.110:0/1763604883 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f722407ac10 con 0x7f722c1da930 2024-01-20T11:20:00.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.765+0000 7f7228ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:00.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.765+0000 7f7228ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:00.765 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:00.765 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":335,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:00.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.766+0000 7f72323c9640 10 librados: watch_flush enter 2024-01-20T11:20:00.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.766+0000 7f72323c9640 10 librados: watch_flush exit 2024-01-20T11:20:00.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.766+0000 7f72323c9640 20 client.5269.objecter shutdown clearing up homeless session... 2024-01-20T11:20:00.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.766+0000 7f72323c9640 10 client.5269.objecter successfully canceled tick 2024-01-20T11:20:00.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721806c8b0 msgr2=0x7f721806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721806c8b0 0x7f721806ed70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f721c009b60 tx=0x7f721c009190 comp rx=0 tx=0).stop 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 10 monclient: shutdown 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c1da930 msgr2=0x7f722c1d7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c1da930 0x7f722c1d7d10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f722c06c200 tx=0x7f7224004a40 comp rx=0 tx=0).stop 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 shutdown_connections 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721806c8b0 0x7f721806ed70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f722c1da930 0x7f722c1d7d10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f722c0b0ba0 0x7f722c071880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 --2- 172.21.15.110:0/1763604883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f722c0ac4d0 0x7f722c1413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 >> 172.21.15.110:0/1763604883 conn(0x7f722c0b3e60 msgr2=0x7f722c13bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 shutdown_connections 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 -- 172.21.15.110:0/1763604883 wait complete. 2024-01-20T11:20:00.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.767+0000 7f72323c9640 1 librados: shutdown 2024-01-20T11:20:00.778 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:00.778 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:00.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.877+0000 7fb7dda64640 10 monclient: get_monmap_and_config 2024-01-20T11:20:00.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.877+0000 7fb7dda64640 10 monclient: build_initial_monmap 2024-01-20T11:20:00.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.877+0000 7fb7dda64640 10 monclient: monmap: 2024-01-20T11:20:00.877 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:00.877 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.877 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:00.878048+0000 2024-01-20T11:20:00.877 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:00.878048+0000 2024-01-20T11:20:00.878 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:00.878 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.878 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:00.878 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:00.878 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:00.878 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:00.878 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:00.878 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 1 Processor -- start 2024-01-20T11:20:00.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 1 -- start start 2024-01-20T11:20:00.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient: init 2024-01-20T11:20:00.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:00.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-01-20T11:20:00.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7d8066620 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb7d806e2a0 0x7fb7d81456a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient(hunting): picked mon.noname-a con 0x7fb7d806e2a0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d814c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient(hunting): picked mon.noname-b con 0x7fb7d8145be0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d806dfe0 con 0x7fb7d8066620 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d8066a00 con 0x7fb7d806e2a0 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80ae130 con 0x7fb7d8145be0 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.878+0000 7fb7dda64640 10 monclient(hunting): authenticate will time out at 1644.229009s 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d814c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d6ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb7d806e2a0 0x7fb7d81456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d814c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57948/0 (socket says 172.21.15.110:57948) 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d7fff640 1 -- 172.21.15.110:0/741465704 learned_addr learned my addr 172.21.15.110:0/741465704 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d7fff640 10 monclient(hunting): get_auth_request con 0x7fb7d8145be0 auth_method 0 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d6ffd640 10 monclient(hunting): get_auth_request con 0x7fb7d806e2a0 auth_method 0 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.879+0000 7fb7d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.880+0000 7fb7d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.880+0000 7fb7d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.880+0000 7fb7d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.880+0000 7fb7d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.880+0000 7fb7d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.880+0000 7fb7d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.882+0000 7fb7d7fff640 10 monclient(hunting): handle_auth_done global_id 4976 payload 274 2024-01-20T11:20:00.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.882+0000 7fb7d7fff640 1 -- 172.21.15.110:0/741465704 >> v2:172.21.15.110:3300/0 conn(0x7fb7d806e2a0 msgr2=0x7fb7d81456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:00.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.882+0000 7fb7d7fff640 1 --2- 172.21.15.110:0/741465704 >> v2:172.21.15.110:3300/0 conn(0x7fb7d806e2a0 0x7fb7d81456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.882+0000 7fb7d7fff640 1 -- 172.21.15.110:0/741465704 >> v1:172.21.15.110:6789/0 conn(0x7fb7d8066620 legacy=0x7fb7d813f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:00.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.882+0000 7fb7d7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:00.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.882+0000 7fb7d7fff640 1 monclient: found mon.noname-b 2024-01-20T11:20:00.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.882+0000 7fb7d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:00.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.882+0000 7fb7d7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:00.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.882+0000 7fb7d7fff640 1 -- 172.21.15.110:0/741465704 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d80ade00 con 0x7fb7d8145be0 2024-01-20T11:20:00.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.884+0000 7fb7d7fff640 1 --2- 172.21.15.110:0/741465704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d814c7c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009e50 tx=0x7fb7d002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe1b1af252a3b862 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:00.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 1 -- 172.21.15.110:0/741465704 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d002c9e0 con 0x7fb7d8145be0 2024-01-20T11:20:00.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:00.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 10 monclient: dump: 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:00.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 1 -- 172.21.15.110:0/741465704 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d002cb40 con 0x7fb7d8145be0 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 1 -- 172.21.15.110:0/741465704 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d00343f0 con 0x7fb7d8145be0 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7dda64640 5 monclient: authenticate success, global_id 4976 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7dda64640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7dda64640 4 set_mon_vals no callback set 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.887+0000 7fb7d67fc640 10 monclient: dump: 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:00.889 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.890 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:00.890 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:00.890 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:00.890 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 10 monclient: shutdown 2024-01-20T11:20:00.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 1 -- 172.21.15.110:0/741465704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 msgr2=0x7fb7d814c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 1 --2- 172.21.15.110:0/741465704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d814c7c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009e50 tx=0x7fb7d002be40 comp rx=0 tx=0).stop 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 1 -- 172.21.15.110:0/741465704 shutdown_connections 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 1 --2- 172.21.15.110:0/741465704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d814c7c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 1 --2- 172.21.15.110:0/741465704 >> v2:172.21.15.110:3300/0 conn(0x7fb7d806e2a0 0x7fb7d81456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 1 -- 172.21.15.110:0/741465704 >> 172.21.15.110:0/741465704 conn(0x7fb7d80b09e0 msgr2=0x7fb7d80b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 1 -- 172.21.15.110:0/741465704 shutdown_connections 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.888+0000 7fb7dda64640 1 -- 172.21.15.110:0/741465704 wait complete. 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.889+0000 7fb7dda64640 10 monclient: build_initial_monmap 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.889+0000 7fb7dda64640 10 monclient: monmap: 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.889+0000 7fb7dda64640 1 librados: starting msgr at 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.889+0000 7fb7dda64640 1 librados: starting objecter 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.889+0000 7fb7dda64640 1 Processor -- start 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 1 -- start start 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 1 librados: setting wanted keys 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 1 librados: calling monclient init 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient: init 2024-01-20T11:20:00.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7d8066620 0x7fb7d81375f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient(hunting): picked mon.noname-a con 0x7fb7d8066620 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient(hunting): picked mon.noname-c con 0x7fb7d8145be0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d81421e0 0x7fb7d8070240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient(hunting): picked mon.noname-b con 0x7fb7d81421e0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8144e70 con 0x7fb7d8066620 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8071910 con 0x7fb7d81421e0 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81450c0 con 0x7fb7d8145be0 2024-01-20T11:20:00.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.890+0000 7fb7dda64640 10 monclient(hunting): authenticate will time out at 1644.240965s 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7d8066620 0x7fb7d81375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d81421e0 0x7fb7d8070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57958/0 (socket says 172.21.15.110:57958) 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d6ffd640 1 -- 172.21.15.110:0/1264894994 learned_addr learned my addr 172.21.15.110:0/1264894994 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d6ffd640 10 monclient(hunting): get_auth_request con 0x7fb7d8145be0 auth_method 0 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d7fff640 10 monclient(hunting): get_auth_request con 0x7fb7d81421e0 auth_method 0 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d77fe640 10 monclient(hunting): get_auth_request con 0x7fb7d8066620 auth_method 0 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.891+0000 7fb7d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.892+0000 7fb7d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.892+0000 7fb7d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.892+0000 7fb7d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.892+0000 7fb7d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.892+0000 7fb7d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.892+0000 7fb7d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.892+0000 7fb7d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.892+0000 7fb7d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:00.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.892+0000 7fb7d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:00.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.894+0000 7fb7d7fff640 10 monclient(hunting): handle_auth_done global_id 5272 payload 824 2024-01-20T11:20:00.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.894+0000 7fb7d7fff640 1 -- 172.21.15.110:0/1264894994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 msgr2=0x7fb7d806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.894+0000 7fb7d7fff640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d806fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.895+0000 7fb7d7fff640 1 -- 172.21.15.110:0/1264894994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7d8066620 msgr2=0x7fb7d81375f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.895+0000 7fb7d7fff640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7d8066620 0x7fb7d81375f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.895+0000 7fb7d7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.895+0000 7fb7d7fff640 1 monclient: found mon.noname-b 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.895+0000 7fb7d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.895+0000 7fb7d7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.895+0000 7fb7d7fff640 1 -- 172.21.15.110:0/1264894994 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d0009b30 con 0x7fb7d81421e0 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.897+0000 7fb7d7fff640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d81421e0 0x7fb7d8070240 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0032000 tx=0x7fb7d0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.897+0000 7fb7bffff640 10 client.?.objecter ms_handle_connect 0x7fb7d81421e0 2024-01-20T11:20:00.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.897+0000 7fb7bffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:00.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 1 -- 172.21.15.110:0/1264894994 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d003e450 con 0x7fb7d81421e0 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 10 monclient: dump: 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 10 monclient: _finish_auth 0 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 1 -- 172.21.15.110:0/1264894994 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d003e5d0 con 0x7fb7d81421e0 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7bffff640 1 -- 172.21.15.110:0/1264894994 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d003e9c0 con 0x7fb7d81421e0 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7dda64640 5 monclient: authenticate success, global_id 5272 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7dda64640 10 monclient: _renew_subs 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7dda64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.900+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d8142c80 con 0x7fb7d81421e0 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.901+0000 7fb7bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.901+0000 7fb7bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.901+0000 7fb7bffff640 10 monclient: dump: 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:00.903 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.901+0000 7fb7d5ffb640 4 set_mon_vals no callback set 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.901+0000 7fb7dda64640 10 client.5272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.901+0000 7fb7dda64640 10 monclient: _renew_subs 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.901+0000 7fb7dda64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.901+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d8070960 con 0x7fb7d81421e0 2024-01-20T11:20:00.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.901+0000 7fb7dda64640 1 librados: init done 2024-01-20T11:20:00.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.902+0000 7fb7dda64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:00.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.902+0000 7fb7dda64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:00.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.902+0000 7fb7dda64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:00.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:00.902+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d8075490 con 0x7fb7d81421e0 2024-01-20T11:20:01.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.143+0000 7fb7bffff640 1 -- 172.21.15.110:0/1264894994 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb7d003eb20 con 0x7fb7d81421e0 2024-01-20T11:20:01.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.143+0000 7fb7bffff640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7c406c8b0 0x7fb7c406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:01.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.143+0000 7fb7d77fe640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7c406c8b0 0x7fb7c406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:01.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.143+0000 7fb7d77fe640 10 monclient: get_auth_request con 0x7fb7c406c8b0 auth_method 0 2024-01-20T11:20:01.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.144+0000 7fb7d77fe640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7c406c8b0 0x7fb7c406ed70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8004f60 tx=0x7fb7c8007e30 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:01.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.144+0000 7fb7bffff640 10 client.5272.objecter ms_handle_connect 0x7fb7c406c8b0 2024-01-20T11:20:01.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.146+0000 7fb7bffff640 1 -- 172.21.15.110:0/1264894994 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb7d007b3b0 con 0x7fb7d81421e0 2024-01-20T11:20:01.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.146+0000 7fb7bffff640 10 client.5272.objecter ms_dispatch 0x7fb7d8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:01.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.146+0000 7fb7bffff640 3 client.5272.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:01.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.146+0000 7fb7bffff640 3 client.5272.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:01.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.146+0000 7fb7bffff640 20 client.5272.objecter dump_active .. 0 homeless 2024-01-20T11:20:01.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.156+0000 7fb7bffff640 1 -- 172.21.15.110:0/1264894994 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb7d00776f0 con 0x7fb7d81421e0 2024-01-20T11:20:01.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.156+0000 7fb7bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:01.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.156+0000 7fb7bffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:01.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.334+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7d8069e70 con 0x7fb7c406c8b0 2024-01-20T11:20:01.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.337+0000 7fb7bffff640 1 -- 172.21.15.110:0/1264894994 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63935 (secure 0 0 0) 0x7fb7d8069e70 con 0x7fb7c406c8b0 2024-01-20T11:20:01.338 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:01.338 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:01.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 10 librados: watch_flush enter 2024-01-20T11:20:01.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 10 librados: watch_flush exit 2024-01-20T11:20:01.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 20 client.5272.objecter shutdown clearing up homeless session... 2024-01-20T11:20:01.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 10 client.5272.objecter successfully canceled tick 2024-01-20T11:20:01.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7c406c8b0 msgr2=0x7fb7c406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7c406c8b0 0x7fb7c406ed70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8004f60 tx=0x7fb7c8007e30 comp rx=0 tx=0).stop 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 10 monclient: shutdown 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d81421e0 msgr2=0x7fb7d8070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d81421e0 0x7fb7d8070240 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0032000 tx=0x7fb7d0004a40 comp rx=0 tx=0).stop 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 shutdown_connections 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7c406c8b0 0x7fb7c406ed70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d81421e0 0x7fb7d8070240 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7d8145be0 0x7fb7d806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 --2- 172.21.15.110:0/1264894994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7d8066620 0x7fb7d81375f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:01.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.339+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 >> 172.21.15.110:0/1264894994 conn(0x7fb7d80b09e0 msgr2=0x7fb7d813f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:01.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.340+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 shutdown_connections 2024-01-20T11:20:01.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.340+0000 7fb7dda64640 1 -- 172.21.15.110:0/1264894994 wait complete. 2024-01-20T11:20:01.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:01.340+0000 7fb7dda64640 1 librados: shutdown 2024-01-20T11:20:01.350 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-01-20T11:20:00.765532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002320"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":13.589},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.0619999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.119},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.2110000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":5.7240000000000002},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.693}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.7479999999999993},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":14.867000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.619},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":15.606}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7460000000000004},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":13.265000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313277,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.8979999999999997},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.0389999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.8419999999999996},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.519}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.782},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.6760000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.9569999999999999},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.359}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.483000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.3149999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":11.119999999999999},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.427}]}]},{"osd":3,"up_from":21,"seq":90194313278,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.5830000000000002},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.3130000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.6400000000000006},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.081}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":11.207000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":16.100999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":10.206},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.1}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":7.1360000000000001},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.879}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":17.378},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":14.083}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444092,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":10.474},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":7.3719999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.5389999999999997},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.09}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.874000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.8520000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.827},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":9.6470000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.506},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.173999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.678000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.16}]}]},{"osd":1,"up_from":17,"seq":73014444093,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.6699999999999999},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.9610000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":11.132},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.9030000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":11.708},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":6.1639999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":9.6859999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":10.254},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":14.242000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.962999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.992000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444093,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":11.095000000000001},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":5.0419999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.6890000000000001},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.476000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":16.576000000000001},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.911}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.637},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.0030000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.885},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.657}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":17.866},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":10.653}]}]},{"osd":6,"up_from":15,"seq":64424509501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":12.169},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.6280000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":10.831},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":14.772}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.7140000000000004},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.3630000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":9.3900000000000006},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.901}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.509},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.599}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.919},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.132999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:04.353 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:04.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.454+0000 7faf6f547640 10 monclient: get_monmap_and_config 2024-01-20T11:20:04.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.454+0000 7faf6f547640 10 monclient: build_initial_monmap 2024-01-20T11:20:04.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.454+0000 7faf6f547640 10 monclient: monmap: 2024-01-20T11:20:04.455 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:04.455 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:04.455781+0000 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:04.455781+0000 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.454+0000 7faf6f547640 1 Processor -- start 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.454+0000 7faf6f547640 1 -- start start 2024-01-20T11:20:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.454+0000 7faf6f547640 10 monclient: init 2024-01-20T11:20:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-01-20T11:20:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf6806bb00 0x7faf680b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 10 monclient(hunting): picked mon.noname-b con 0x7faf6806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf680af790 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7faf680b0900 0x7faf68145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 10 monclient(hunting): picked mon.noname-c con 0x7faf680b0900 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6806c220 con 0x7faf680af790 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6806bee0 con 0x7faf680b0900 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf680ada30 con 0x7faf6806bb00 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.455+0000 7faf6f547640 10 monclient(hunting): authenticate will time out at 1647.806776s 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6cae2640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7faf680af790 0x7faf68145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48630/0 (socket says 172.21.15.110:48630) 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6cae2640 1 -- 172.21.15.110:0/3327597161 learned_addr learned my addr 172.21.15.110:0/3327597161 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6d2e3640 1 --2- 172.21.15.110:0/3327597161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf6806bb00 0x7faf680b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6dae4640 1 --2- 172.21.15.110:0/3327597161 >> v2:172.21.15.134:3300/0 conn(0x7faf680b0900 0x7faf68145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6d2e3640 10 monclient(hunting): get_auth_request con 0x7faf6806bb00 auth_method 0 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6d2e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6d2e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6d2e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6dae4640 10 monclient(hunting): get_auth_request con 0x7faf680b0900 auth_method 0 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6dae4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6dae4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.456+0000 7faf6dae4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.457+0000 7faf6dae4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.457+0000 7faf6dae4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:04.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.457+0000 7faf6dae4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.458+0000 7faf5ffff640 1 -- 172.21.15.110:0/3327597161 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086985860 0 0) 0x7faf6806c220 con 0x7faf680af790 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.458+0000 7faf5ffff640 10 monclient(hunting): _negotiate 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.458+0000 7faf5ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.458+0000 7faf5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.458+0000 7faf5ffff640 10 monclient(hunting): my global_id is 5193 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.458+0000 7faf5ffff640 1 -- 172.21.15.110:0/3327597161 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf54003520 con 0x7faf680af790 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.459+0000 7faf6dae4640 10 monclient(hunting): handle_auth_done global_id 5275 payload 274 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.459+0000 7faf6dae4640 1 -- 172.21.15.110:0/3327597161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf6806bb00 msgr2=0x7faf680b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.459+0000 7faf6dae4640 1 --2- 172.21.15.110:0/3327597161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf6806bb00 0x7faf680b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.459+0000 7faf6dae4640 1 -- 172.21.15.110:0/3327597161 >> v1:172.21.15.110:6789/0 conn(0x7faf680af790 legacy=0x7faf68145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:04.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.459+0000 7faf6dae4640 10 monclient: _finish_hunting 0 2024-01-20T11:20:04.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.459+0000 7faf6dae4640 1 monclient: found mon.noname-c 2024-01-20T11:20:04.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.459+0000 7faf6dae4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:04.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.459+0000 7faf6dae4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:04.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.459+0000 7faf6dae4640 1 -- 172.21.15.110:0/3327597161 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf680add10 con 0x7faf680b0900 2024-01-20T11:20:04.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.461+0000 7faf6dae4640 1 --2- 172.21.15.110:0/3327597161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf680b0900 0x7faf68145e80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faf60009d90 tx=0x7faf6002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82fabaa9aa010364 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:04.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 1 -- 172.21.15.110:0/3327597161 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6002cbc0 con 0x7faf680b0900 2024-01-20T11:20:04.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:04.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:04.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 10 monclient: dump: 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 10 monclient: _finish_auth 0 2024-01-20T11:20:04.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 1 -- 172.21.15.110:0/3327597161 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6002cd20 con 0x7faf680b0900 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 1 -- 172.21.15.110:0/3327597161 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60034660 con 0x7faf680b0900 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf6f547640 5 monclient: authenticate success, global_id 5275 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf6f547640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf6f547640 4 set_mon_vals no callback set 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.465+0000 7faf5ffff640 10 monclient: dump: 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:04.468 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:04.469 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:04.469 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:04.469 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:04.469 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:04.469 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:04.469 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:04.469 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:04.469 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:04.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 10 monclient: shutdown 2024-01-20T11:20:04.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:04.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 1 -- 172.21.15.110:0/3327597161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf680b0900 msgr2=0x7faf68145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:04.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 1 --2- 172.21.15.110:0/3327597161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf680b0900 0x7faf68145e80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faf60009d90 tx=0x7faf6002be10 comp rx=0 tx=0).stop 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 1 -- 172.21.15.110:0/3327597161 shutdown_connections 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 1 --2- 172.21.15.110:0/3327597161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf680b0900 0x7faf68145e80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 1 --2- 172.21.15.110:0/3327597161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf6806bb00 0x7faf680b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 1 -- 172.21.15.110:0/3327597161 >> 172.21.15.110:0/3327597161 conn(0x7faf68066d60 msgr2=0x7faf680b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 1 -- 172.21.15.110:0/3327597161 shutdown_connections 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.466+0000 7faf6f547640 1 -- 172.21.15.110:0/3327597161 wait complete. 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.467+0000 7faf6f547640 10 monclient: build_initial_monmap 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.467+0000 7faf6f547640 10 monclient: monmap: 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:04.473 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.467+0000 7faf6f547640 1 librados: starting msgr at 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.467+0000 7faf6f547640 1 librados: starting objecter 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.467+0000 7faf6f547640 1 Processor -- start 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.467+0000 7faf6f547640 1 -- start start 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.467+0000 7faf6f547640 1 librados: setting wanted keys 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.467+0000 7faf6f547640 1 librados: calling monclient init 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.467+0000 7faf6f547640 10 monclient: init 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf6806bb00 0x7faf680b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient(hunting): picked mon.noname-b con 0x7faf6806bb00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf680af790 0x7faf681d6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient(hunting): picked mon.noname-a con 0x7faf680af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf680b0900 0x7faf68120340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:04.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient(hunting): picked mon.noname-c con 0x7faf680b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf681d7290 con 0x7faf680af790 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf681d7ca0 con 0x7faf6806bb00 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6811ed30 con 0x7faf680b0900 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6f547640 10 monclient(hunting): authenticate will time out at 1647.819476s 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6cae2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf680af790 0x7faf681d6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.468+0000 7faf6d2e3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf6806bb00 0x7faf680b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6dae4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf680b0900 0x7faf68120340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6cae2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf680af790 0x7faf681d6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36206/0 (socket says 172.21.15.110:36206) 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6cae2640 1 -- 172.21.15.110:0/4043706784 learned_addr learned my addr 172.21.15.110:0/4043706784 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6cae2640 10 monclient(hunting): get_auth_request con 0x7faf680af790 auth_method 0 2024-01-20T11:20:04.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6cae2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6cae2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6cae2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6d2e3640 10 monclient(hunting): get_auth_request con 0x7faf6806bb00 auth_method 0 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6d2e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6d2e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6d2e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6dae4640 10 monclient(hunting): get_auth_request con 0x7faf680b0900 auth_method 0 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6dae4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6dae4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6dae4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6cae2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6cae2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6cae2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6d2e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:04.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6d2e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:04.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6d2e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:04.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6dae4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:04.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6dae4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:04.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.469+0000 7faf6dae4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:04.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 10 monclient(hunting): handle_auth_done global_id 5278 payload 824 2024-01-20T11:20:04.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 1 -- 172.21.15.110:0/4043706784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf680b0900 msgr2=0x7faf68120340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:04.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf680b0900 0x7faf68120340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 1 -- 172.21.15.110:0/4043706784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf680af790 msgr2=0x7faf681d6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf680af790 0x7faf681d6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 10 monclient: _finish_hunting 0 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 1 monclient: found mon.noname-b 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.472+0000 7faf6d2e3640 1 -- 172.21.15.110:0/4043706784 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf60009a70 con 0x7faf6806bb00 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.475+0000 7faf6d2e3640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf6806bb00 0x7faf680b4f40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faf6400d5f0 tx=0x7faf6400dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.475+0000 7faf5dffb640 10 client.?.objecter ms_handle_connect 0x7faf6806bb00 2024-01-20T11:20:04.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.475+0000 7faf5dffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.478+0000 7faf5dffb640 1 -- 172.21.15.110:0/4043706784 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640125f0 con 0x7faf6806bb00 2024-01-20T11:20:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.478+0000 7faf5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.478+0000 7faf5dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.478+0000 7faf5dffb640 10 monclient: dump: 2024-01-20T11:20:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5dffb640 10 monclient: _finish_auth 0 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5dffb640 10 monclient: _check_auth_tickets 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5dffb640 1 -- 172.21.15.110:0/4043706784 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf64012750 con 0x7faf6806bb00 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf6f547640 5 monclient: authenticate success, global_id 5278 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf6f547640 10 monclient: _renew_subs 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf6f547640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf681208e0 con 0x7faf6806bb00 2024-01-20T11:20:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5dffb640 1 -- 172.21.15.110:0/4043706784 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64012b00 con 0x7faf6806bb00 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5dffb640 10 monclient: dump: 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:04.482 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf5f7fe640 4 set_mon_vals no callback set 2024-01-20T11:20:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf6f547640 10 client.5278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf6f547640 10 monclient: _renew_subs 2024-01-20T11:20:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf6f547640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6811f5c0 con 0x7faf6806bb00 2024-01-20T11:20:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.479+0000 7faf6f547640 1 librados: init done 2024-01-20T11:20:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.480+0000 7faf6f547640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.480+0000 7faf6f547640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.480+0000 7faf6f547640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.480+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf68075490 con 0x7faf6806bb00 2024-01-20T11:20:04.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.721+0000 7faf5dffb640 1 -- 172.21.15.110:0/4043706784 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7faf64012c60 con 0x7faf6806bb00 2024-01-20T11:20:04.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.721+0000 7faf5dffb640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf4c06c8b0 0x7faf4c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:04.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.722+0000 7faf6cae2640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf4c06c8b0 0x7faf4c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:04.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.722+0000 7faf6cae2640 10 monclient: get_auth_request con 0x7faf4c06c8b0 auth_method 0 2024-01-20T11:20:04.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.722+0000 7faf5dffb640 10 client.5278.objecter ms_handle_connect 0x7faf4c06c8b0 2024-01-20T11:20:04.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.722+0000 7faf6cae2640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf4c06c8b0 0x7faf4c06ed70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7faf68144060 tx=0x7faf58009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:04.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.724+0000 7faf5dffb640 1 -- 172.21.15.110:0/4043706784 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faf64059140 con 0x7faf6806bb00 2024-01-20T11:20:04.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.724+0000 7faf5dffb640 10 client.5278.objecter ms_dispatch 0x7faf68000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:04.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.724+0000 7faf5dffb640 3 client.5278.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:04.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.724+0000 7faf5dffb640 3 client.5278.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:04.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.724+0000 7faf5dffb640 20 client.5278.objecter dump_active .. 0 homeless 2024-01-20T11:20:04.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.735+0000 7faf5dffb640 1 -- 172.21.15.110:0/4043706784 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7faf64055480 con 0x7faf6806bb00 2024-01-20T11:20:04.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.735+0000 7faf5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:04.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.735+0000 7faf5dffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:04.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.907+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faf6806c910 con 0x7faf4c06c8b0 2024-01-20T11:20:04.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.910+0000 7faf5dffb640 1 -- 172.21.15.110:0/4043706784 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64091 (secure 0 0 0) 0x7faf5800f820 con 0x7faf4c06c8b0 2024-01-20T11:20:04.911 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:04.911 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:04.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 10 librados: watch_flush enter 2024-01-20T11:20:04.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 10 librados: watch_flush exit 2024-01-20T11:20:04.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 20 client.5278.objecter shutdown clearing up homeless session... 2024-01-20T11:20:04.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 10 client.5278.objecter successfully canceled tick 2024-01-20T11:20:04.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf4c06c8b0 msgr2=0x7faf4c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:04.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf4c06c8b0 0x7faf4c06ed70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7faf68144060 tx=0x7faf58009040 comp rx=0 tx=0).stop 2024-01-20T11:20:04.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 10 monclient: shutdown 2024-01-20T11:20:04.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf6806bb00 msgr2=0x7faf680b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf6806bb00 0x7faf680b4f40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faf6400d5f0 tx=0x7faf6400dab0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 shutdown_connections 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7faf4c06c8b0 0x7faf4c06ed70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf680b0900 0x7faf68120340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf680af790 0x7faf681d6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 --2- 172.21.15.110:0/4043706784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf6806bb00 0x7faf680b4f40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.911+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 >> 172.21.15.110:0/4043706784 conn(0x7faf68066d60 msgr2=0x7faf681428b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.912+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 shutdown_connections 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.912+0000 7faf6f547640 1 -- 172.21.15.110:0/4043706784 wait complete. 2024-01-20T11:20:04.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:04.912+0000 7faf6f547640 1 librados: shutdown 2024-01-20T11:20:04.923 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-01-20T11:20:04.766327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002217"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280576,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.7510000000000003},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":14.115}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.449999999999999},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":5.7910000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":15.288},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.587999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.8860000000000001},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":12.114000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.696999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":10.013999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.2319999999999993},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.120000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313278,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.7450000000000001},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.9619999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.2080000000000002},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.755000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.0549999999999997},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.226000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.444000000000001},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.484}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.6280000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":10.85}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.122},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.747999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313279,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":4.3200000000000003},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.8879999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7319999999999993},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.352}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":8.6950000000000003},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.717000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.219000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":13.526},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.638999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":17.420999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.081}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444093,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.2709999999999999},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":8.3559999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.984999999999999},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":19.446000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.151},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.612}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.253},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":16.096}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.641999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.808}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.430999999999999},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":14.721}]}]},{"osd":1,"up_from":17,"seq":73014444094,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.9120000000000008},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":13.863}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.066000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":14.817}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":13.071999999999999},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.0449999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":17.187999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.407999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.9060000000000006},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":16.091999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.988},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.966000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444094,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":5.2039999999999997},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.433}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.0460000000000003},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.2050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":12.630000000000001},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.744999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.897},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.5540000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":12.023999999999999},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.49}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.725},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":9.609}]}]},{"osd":6,"up_from":15,"seq":64424509502,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":10.923},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.8689999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":4.8319999999999999},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":13.234999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.8260000000000005},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.1809999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.84},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.558999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.994999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.608000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.561999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.276999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:04.925 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:05.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.022+0000 7f464493a640 10 monclient: get_monmap_and_config 2024-01-20T11:20:05.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.022+0000 7f464493a640 10 monclient: build_initial_monmap 2024-01-20T11:20:05.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.022+0000 7f464493a640 10 monclient: monmap: 2024-01-20T11:20:05.023 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:05.023 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.023 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:05.023929+0000 2024-01-20T11:20:05.023 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:05.023929+0000 2024-01-20T11:20:05.024 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:05.024 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.024 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:05.024 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:05.024 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:05.024 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:05.024 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:05.024 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 1 Processor -- start 2024-01-20T11:20:05.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 1 -- start start 2024-01-20T11:20:05.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient: init 2024-01-20T11:20:05.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:05.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-01-20T11:20:05.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4640066680 0x7f464013cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient(hunting): picked mon.noname-c con 0x7f4640066680 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f464006dcf0 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4640143980 0x7f464013d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient(hunting): picked mon.noname-a con 0x7f4640143980 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4640143e10 con 0x7f464006dcf0 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464013d980 con 0x7f4640143980 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464006e2f0 con 0x7f4640066680 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.023+0000 7f464493a640 10 monclient(hunting): authenticate will time out at 1648.374965s 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463ed9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4640143980 0x7f464013d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463e59c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4640066680 0x7f464013cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463ed9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4640143980 0x7f464013d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36220/0 (socket says 172.21.15.110:36220) 2024-01-20T11:20:05.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463ed9d640 1 -- 172.21.15.110:0/3202475153 learned_addr learned my addr 172.21.15.110:0/3202475153 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463ed9d640 10 monclient(hunting): get_auth_request con 0x7f4640143980 auth_method 0 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463e59c640 10 monclient(hunting): get_auth_request con 0x7f4640066680 auth_method 0 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.024+0000 7f463e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.025+0000 7f463e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.025+0000 7f463e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.025+0000 7f463e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.026+0000 7f463ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.026+0000 7f463ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.026+0000 7f463ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.027+0000 7f463e59c640 10 monclient(hunting): handle_auth_done global_id 5281 payload 274 2024-01-20T11:20:05.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.027+0000 7f463e59c640 1 -- 172.21.15.110:0/3202475153 >> v2:172.21.15.110:3300/0 conn(0x7f4640143980 msgr2=0x7f464013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:05.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.027+0000 7f463e59c640 1 --2- 172.21.15.110:0/3202475153 >> v2:172.21.15.110:3300/0 conn(0x7f4640143980 0x7f464013d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.027+0000 7f463e59c640 1 -- 172.21.15.110:0/3202475153 >> v1:172.21.15.110:6789/0 conn(0x7f464006dcf0 legacy=0x7f4640143270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.027+0000 7f463e59c640 10 monclient: _finish_hunting 0 2024-01-20T11:20:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.027+0000 7f463e59c640 1 monclient: found mon.noname-c 2024-01-20T11:20:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.027+0000 7f463e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.027+0000 7f463e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.027+0000 7f463e59c640 1 -- 172.21.15.110:0/3202475153 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4640072140 con 0x7f4640066680 2024-01-20T11:20:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.029+0000 7f463e59c640 1 --2- 172.21.15.110:0/3202475153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4640066680 0x7f464013cf00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4640066bf0 tx=0x7f463002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e19ff58b12e6d4c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.032+0000 7f463dd9b640 1 -- 172.21.15.110:0/3202475153 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630031830 con 0x7f4640066680 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.032+0000 7f463dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.032+0000 7f463dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.032+0000 7f463dd9b640 10 monclient: dump: 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:05.034 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.032+0000 7f463dd9b640 10 monclient: _finish_auth 0 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f463dd9b640 10 monclient: _check_auth_tickets 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f463dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f463dd9b640 1 -- 172.21.15.110:0/3202475153 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4630031990 con 0x7f4640066680 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 5 monclient: authenticate success, global_id 5281 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f463dd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f463dd9b640 1 -- 172.21.15.110:0/3202475153 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630031d20 con 0x7f4640066680 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 4 set_mon_vals no callback set 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f463dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f463dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f463dd9b640 10 monclient: dump: 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:05.035 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.036 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.036 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.036 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:05.036 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.036 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:05.036 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:05.036 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:05.036 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 10 monclient: shutdown 2024-01-20T11:20:05.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:05.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 1 -- 172.21.15.110:0/3202475153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4640066680 msgr2=0x7f464013cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 1 --2- 172.21.15.110:0/3202475153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4640066680 0x7f464013cf00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4640066bf0 tx=0x7f463002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 1 -- 172.21.15.110:0/3202475153 shutdown_connections 2024-01-20T11:20:05.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 1 --2- 172.21.15.110:0/3202475153 >> v2:172.21.15.110:3300/0 conn(0x7f4640143980 0x7f464013d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 1 --2- 172.21.15.110:0/3202475153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4640066680 0x7f464013cf00 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 1 -- 172.21.15.110:0/3202475153 >> 172.21.15.110:0/3202475153 conn(0x7f46400b09e0 msgr2=0x7f46400b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:05.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 1 -- 172.21.15.110:0/3202475153 shutdown_connections 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.033+0000 7f464493a640 1 -- 172.21.15.110:0/3202475153 wait complete. 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.034+0000 7f464493a640 10 monclient: build_initial_monmap 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.034+0000 7f464493a640 10 monclient: monmap: 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.034+0000 7f464493a640 1 librados: starting msgr at 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.034+0000 7f464493a640 1 librados: starting objecter 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.034+0000 7f464493a640 1 Processor -- start 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.034+0000 7f464493a640 1 -- start start 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.034+0000 7f464493a640 1 librados: setting wanted keys 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.034+0000 7f464493a640 1 librados: calling monclient init 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.034+0000 7f464493a640 10 monclient: init 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4640066680 0x7f4640138a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient(hunting): picked mon.noname-b con 0x7f4640066680 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f464006dcf0 0x7f46401da940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient(hunting): picked mon.noname-c con 0x7f464006dcf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4640143980 0x7f4640149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient(hunting): picked mon.noname-a con 0x7f4640143980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464006ee50 con 0x7f4640143980 2024-01-20T11:20:05.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464014a1e0 con 0x7f4640066680 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46400709a0 con 0x7f464006dcf0 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f464493a640 10 monclient(hunting): authenticate will time out at 1648.386718s 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.035+0000 7f463ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4640143980 0x7f4640149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f4637fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f464006dcf0 0x7f46401da940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463e59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4640066680 0x7f4640138a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4640143980 0x7f4640149ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36224/0 (socket says 172.21.15.110:36224) 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463ed9d640 1 -- 172.21.15.110:0/3156958810 learned_addr learned my addr 172.21.15.110:0/3156958810 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463ed9d640 10 monclient(hunting): get_auth_request con 0x7f4640143980 auth_method 0 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f4637fff640 10 monclient(hunting): get_auth_request con 0x7f464006dcf0 auth_method 0 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f4637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f4637fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f4637fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463e59c640 10 monclient(hunting): get_auth_request con 0x7f4640066680 auth_method 0 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f463e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f4637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.036+0000 7f4637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.037+0000 7f4637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 10 monclient(hunting): handle_auth_done global_id 5205 payload 824 2024-01-20T11:20:05.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 1 -- 172.21.15.110:0/3156958810 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f464006dcf0 msgr2=0x7f46401da940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f464006dcf0 0x7f46401da940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 1 -- 172.21.15.110:0/3156958810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4640066680 msgr2=0x7f4640138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4640066680 0x7f4640138a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:20:05.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 1 monclient: found mon.noname-a 2024-01-20T11:20:05.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:05.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:05.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.039+0000 7f463ed9d640 1 -- 172.21.15.110:0/3156958810 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4630009400 con 0x7f4640143980 2024-01-20T11:20:05.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.041+0000 7f463ed9d640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4640143980 0x7f4640149ca0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f463800efc0 tx=0x7f463800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:05.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.041+0000 7f4636ffd640 10 client.?.objecter ms_handle_connect 0x7f4640143980 2024-01-20T11:20:05.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.041+0000 7f4636ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 1 -- 172.21.15.110:0/3156958810 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638009b60 con 0x7f4640143980 2024-01-20T11:20:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 10 monclient: dump: 2024-01-20T11:20:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 1 -- 172.21.15.110:0/3156958810 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463800f040 con 0x7f4640143980 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 1 -- 172.21.15.110:0/3156958810 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638003990 con 0x7f4640143980 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f464493a640 5 monclient: authenticate success, global_id 5205 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f464493a640 10 monclient: _renew_subs 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f464493a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:05.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4640070c80 con 0x7f4640143980 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f463d59a640 4 set_mon_vals no callback set 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f4636ffd640 10 monclient: dump: 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:05.048 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:05.049 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f464493a640 10 client.5205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:05.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f464493a640 10 monclient: _renew_subs 2024-01-20T11:20:05.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f464493a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:05.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464006f450 con 0x7f4640143980 2024-01-20T11:20:05.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.045+0000 7f464493a640 1 librados: init done 2024-01-20T11:20:05.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.046+0000 7f464493a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:05.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.046+0000 7f464493a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:05.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.046+0000 7f464493a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:05.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.046+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464006ef90 con 0x7f4640143980 2024-01-20T11:20:05.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.285+0000 7f4636ffd640 1 -- 172.21.15.110:0/3156958810 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4638003af0 con 0x7f4640143980 2024-01-20T11:20:05.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.285+0000 7f4636ffd640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f462006c860 0x7f462006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.286+0000 7f463e59c640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f462006c860 0x7f462006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.286+0000 7f463e59c640 10 monclient: get_auth_request con 0x7f462006c860 auth_method 0 2024-01-20T11:20:05.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.286+0000 7f463e59c640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f462006c860 0x7f462006ed20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4630036000 tx=0x7f463003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:05.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.286+0000 7f4636ffd640 10 client.5205.objecter ms_handle_connect 0x7f462006c860 2024-01-20T11:20:05.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.310+0000 7f4636ffd640 1 -- 172.21.15.110:0/3156958810 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4638059360 con 0x7f4640143980 2024-01-20T11:20:05.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.310+0000 7f4636ffd640 10 client.5205.objecter ms_dispatch 0x7f4640000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:05.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.310+0000 7f4636ffd640 3 client.5205.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:05.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.310+0000 7f4636ffd640 3 client.5205.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:05.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.310+0000 7f4636ffd640 20 client.5205.objecter dump_active .. 0 homeless 2024-01-20T11:20:05.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.310+0000 7f4636ffd640 1 -- 172.21.15.110:0/3156958810 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f46380556a0 con 0x7f4640143980 2024-01-20T11:20:05.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.310+0000 7f4636ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:05.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.310+0000 7f4636ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:05.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.491+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f464006ef90 con 0x7f462006c860 2024-01-20T11:20:05.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.494+0000 7f4636ffd640 1 -- 172.21.15.110:0/3156958810 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64091 (secure 0 0 0) 0x7f464006ef90 con 0x7f462006c860 2024-01-20T11:20:05.495 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:05.495 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 10 librados: watch_flush enter 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 10 librados: watch_flush exit 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 20 client.5205.objecter shutdown clearing up homeless session... 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 10 client.5205.objecter successfully canceled tick 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f462006c860 msgr2=0x7f462006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f462006c860 0x7f462006ed20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4630036000 tx=0x7f463003c040 comp rx=0 tx=0).stop 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 10 monclient: shutdown 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4640143980 msgr2=0x7f4640149ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4640143980 0x7f4640149ca0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f463800efc0 tx=0x7f463800c490 comp rx=0 tx=0).stop 2024-01-20T11:20:05.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 shutdown_connections 2024-01-20T11:20:05.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f462006c860 0x7f462006ed20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4640143980 0x7f4640149ca0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f464006dcf0 0x7f46401da940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 --2- 172.21.15.110:0/3156958810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4640066680 0x7f4640138a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 >> 172.21.15.110:0/3156958810 conn(0x7f46400b09e0 msgr2=0x7f4640143230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:05.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 shutdown_connections 2024-01-20T11:20:05.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 -- 172.21.15.110:0/3156958810 wait complete. 2024-01-20T11:20:05.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.496+0000 7f464493a640 1 librados: shutdown 2024-01-20T11:20:05.507 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-01-20T11:20:04.766327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002217"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280576,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.7510000000000003},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":14.115}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.449999999999999},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":5.7910000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":15.288},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.587999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.8860000000000001},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":12.114000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.696999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":10.013999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.2319999999999993},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.120000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313278,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.7450000000000001},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.9619999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.2080000000000002},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.755000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.0549999999999997},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.226000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.444000000000001},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.484}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.6280000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":10.85}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.122},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.747999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313279,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":4.3200000000000003},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.8879999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7319999999999993},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.352}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":8.6950000000000003},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.717000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.219000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":13.526},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.638999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":17.420999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.081}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444093,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.2709999999999999},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":8.3559999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.984999999999999},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":19.446000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.151},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.612}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.253},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":16.096}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.641999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.808}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.430999999999999},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":14.721}]}]},{"osd":1,"up_from":17,"seq":73014444094,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.9120000000000008},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":13.863}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.066000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":14.817}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":13.071999999999999},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.0449999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":17.187999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.407999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.9060000000000006},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":16.091999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.988},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.966000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444094,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":5.2039999999999997},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.433}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.0460000000000003},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.2050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":12.630000000000001},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.744999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.897},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.5540000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":12.023999999999999},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.49}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.725},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":9.609}]}]},{"osd":6,"up_from":15,"seq":64424509502,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":10.923},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.8689999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":4.8319999999999999},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":13.234999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.8260000000000005},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.1809999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.84},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.558999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.994999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.608000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.561999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.276999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:05.509 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:05.509 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:05.509 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:05.509 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:05.509 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:05.510 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:05.510 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:05.510 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:05.510 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:05.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.607+0000 7f6d6baf7640 10 monclient: get_monmap_and_config 2024-01-20T11:20:05.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.607+0000 7f6d6baf7640 10 monclient: build_initial_monmap 2024-01-20T11:20:05.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.607+0000 7f6d6baf7640 10 monclient: monmap: 2024-01-20T11:20:05.608 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:05.608 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.608 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:05.608698+0000 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:05.608698+0000 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.607+0000 7f6d6baf7640 1 Processor -- start 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.607+0000 7f6d6baf7640 1 -- start start 2024-01-20T11:20:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.607+0000 7f6d6baf7640 10 monclient: init 2024-01-20T11:20:05.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:05.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-01-20T11:20:05.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6d640b4d10 0x7f6d640b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 10 monclient(hunting): picked mon.noname-a con 0x7f6d640b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:05.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b17c0 0x7f6d640b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 10 monclient(hunting): picked mon.noname-b con 0x7f6d640b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d64145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d640b2120 con 0x7f6d64145980 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d640b50f0 con 0x7f6d640b4d10 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d640aade0 con 0x7f6d640b17c0 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.608+0000 7f6d6baf7640 10 monclient(hunting): authenticate will time out at 1648.959659s 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69893640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6d640b4d10 0x7f6d640b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69092640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b17c0 0x7f6d640b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d6a094640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6d64145980 0x7f6d64145e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:48120/0 (socket says 172.21.15.110:48120) 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d6a094640 1 -- 172.21.15.110:0/1115136938 learned_addr learned my addr 172.21.15.110:0/1115136938 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69092640 10 monclient(hunting): get_auth_request con 0x7f6d640b17c0 auth_method 0 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69092640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69092640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69092640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69893640 10 monclient(hunting): get_auth_request con 0x7f6d640b4d10 auth_method 0 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69893640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69893640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69893640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69092640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69092640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69092640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69893640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69893640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.609+0000 7f6d69893640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d68891640 1 -- 172.21.15.110:0/1115136938 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314036125 0 0) 0x7f6d640b2120 con 0x7f6d64145980 2024-01-20T11:20:05.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d68891640 10 monclient(hunting): _negotiate 2024-01-20T11:20:05.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d68891640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d68891640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d68891640 10 monclient(hunting): my global_id is 5287 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d68891640 1 -- 172.21.15.110:0/1115136938 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d50003520 con 0x7f6d64145980 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d69893640 10 monclient(hunting): handle_auth_done global_id 5208 payload 274 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d69893640 1 -- 172.21.15.110:0/1115136938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b17c0 msgr2=0x7f6d640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d69893640 1 --2- 172.21.15.110:0/1115136938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b17c0 0x7f6d640b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d69893640 1 -- 172.21.15.110:0/1115136938 >> v1:172.21.15.134:6789/0 conn(0x7f6d64145980 legacy=0x7f6d64145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d69893640 10 monclient: _finish_hunting 0 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d69893640 1 monclient: found mon.noname-a 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d69893640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d69893640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.611+0000 7f6d69893640 1 -- 172.21.15.110:0/1115136938 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d64145460 con 0x7f6d640b4d10 2024-01-20T11:20:05.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.613+0000 7f6d69893640 1 --2- 172.21.15.110:0/1115136938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d640b4d10 0x7f6d640b3da0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097a0 tx=0x7f6d6002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d8dc42b168c031 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 1 -- 172.21.15.110:0/1115136938 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6002e070 con 0x7f6d640b4d10 2024-01-20T11:20:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 10 monclient: dump: 2024-01-20T11:20:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 10 monclient: _finish_auth 0 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 10 monclient: _check_auth_tickets 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 1 -- 172.21.15.110:0/1115136938 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d600315a0 con 0x7f6d640b4d10 2024-01-20T11:20:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d6baf7640 5 monclient: authenticate success, global_id 5208 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d6baf7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d6baf7640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d6baf7640 4 set_mon_vals no callback set 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 1 -- 172.21.15.110:0/1115136938 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60031930 con 0x7f6d640b4d10 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.616+0000 7f6d68891640 10 monclient: dump: 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 10 monclient: shutdown 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1115136938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d640b4d10 msgr2=0x7f6d640b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 1 --2- 172.21.15.110:0/1115136938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d640b4d10 0x7f6d640b3da0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097a0 tx=0x7f6d6002b840 comp rx=0 tx=0).stop 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1115136938 shutdown_connections 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 1 --2- 172.21.15.110:0/1115136938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b17c0 0x7f6d640b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 1 --2- 172.21.15.110:0/1115136938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d640b4d10 0x7f6d640b3da0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1115136938 >> 172.21.15.110:0/1115136938 conn(0x7f6d64066590 msgr2=0x7f6d640669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1115136938 shutdown_connections 2024-01-20T11:20:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.617+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1115136938 wait complete. 2024-01-20T11:20:05.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.618+0000 7f6d6baf7640 10 monclient: build_initial_monmap 2024-01-20T11:20:05.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.618+0000 7f6d6baf7640 10 monclient: monmap: 2024-01-20T11:20:05.624 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.618+0000 7f6d6baf7640 1 librados: starting msgr at 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.618+0000 7f6d6baf7640 1 librados: starting objecter 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.618+0000 7f6d6baf7640 1 Processor -- start 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.618+0000 7f6d6baf7640 1 -- start start 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.618+0000 7f6d6baf7640 1 librados: setting wanted keys 2024-01-20T11:20:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.618+0000 7f6d6baf7640 1 librados: calling monclient init 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.618+0000 7f6d6baf7640 10 monclient: init 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d640b17c0 0x7f6d6413ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient(hunting): picked mon.noname-b con 0x7f6d640b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b4d10 0x7f6d641d7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient(hunting): picked mon.noname-c con 0x7f6d640b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d64145980 0x7f6d641d7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient(hunting): picked mon.noname-a con 0x7f6d64145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641d7150 con 0x7f6d64145980 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d64148170 con 0x7f6d640b17c0 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:05.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d64146a10 con 0x7f6d640b4d10 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6baf7640 10 monclient(hunting): authenticate will time out at 1648.970419s 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d6a094640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d64145980 0x7f6d641d7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d69092640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b4d10 0x7f6d641d7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.619+0000 7f6d69893640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d640b17c0 0x7f6d6413ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d6a094640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d64145980 0x7f6d641d7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36238/0 (socket says 172.21.15.110:36238) 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d6a094640 1 -- 172.21.15.110:0/1284027733 learned_addr learned my addr 172.21.15.110:0/1284027733 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d6a094640 10 monclient(hunting): get_auth_request con 0x7f6d64145980 auth_method 0 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d6a094640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d6a094640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d6a094640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69893640 10 monclient(hunting): get_auth_request con 0x7f6d640b17c0 auth_method 0 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69893640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69893640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69893640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69092640 10 monclient(hunting): get_auth_request con 0x7f6d640b4d10 auth_method 0 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69092640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69092640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69092640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69893640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69893640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69893640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d6a094640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d6a094640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d6a094640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69092640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69092640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.620+0000 7f6d69092640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 10 monclient(hunting): handle_auth_done global_id 5290 payload 824 2024-01-20T11:20:05.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 1 -- 172.21.15.110:0/1284027733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b4d10 msgr2=0x7f6d641d7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b4d10 0x7f6d641d7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 1 -- 172.21.15.110:0/1284027733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d64145980 msgr2=0x7f6d641d7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:05.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d64145980 0x7f6d641d7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:05.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 10 monclient: _finish_hunting 0 2024-01-20T11:20:05.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 1 monclient: found mon.noname-b 2024-01-20T11:20:05.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:05.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:05.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.623+0000 7f6d69893640 1 -- 172.21.15.110:0/1284027733 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d60009400 con 0x7f6d640b17c0 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.625+0000 7f6d69893640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d640b17c0 0x7f6d6413ed10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6d60033000 tx=0x7f6d600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.625+0000 7f6d5a7fc640 10 client.?.objecter ms_handle_connect 0x7f6d640b17c0 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.625+0000 7f6d5a7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 1 -- 172.21.15.110:0/1284027733 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60032630 con 0x7f6d640b17c0 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 10 monclient: dump: 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 1 -- 172.21.15.110:0/1284027733 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d600327b0 con 0x7f6d640b17c0 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d6baf7640 5 monclient: authenticate success, global_id 5290 2024-01-20T11:20:05.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 1 -- 172.21.15.110:0/1284027733 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60032b60 con 0x7f6d640b17c0 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5a7fc640 10 monclient: dump: 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:05.633 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d6baf7640 10 monclient: _renew_subs 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d5bfff640 4 set_mon_vals no callback set 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d6baf7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6413f250 con 0x7f6d640b17c0 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d6baf7640 10 client.5290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d6baf7640 10 monclient: _renew_subs 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d6baf7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d64146cc0 con 0x7f6d640b17c0 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.628+0000 7f6d6baf7640 1 librados: init done 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.629+0000 7f6d6baf7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.629+0000 7f6d6baf7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.629+0000 7f6d6baf7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.629+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d64075490 con 0x7f6d640b17c0 2024-01-20T11:20:05.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.871+0000 7f6d5a7fc640 1 -- 172.21.15.110:0/1284027733 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6d6004e430 con 0x7f6d640b17c0 2024-01-20T11:20:05.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.871+0000 7f6d5a7fc640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d4806c8b0 0x7f6d4806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:05.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.872+0000 7f6d69092640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d4806c8b0 0x7f6d4806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:05.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.872+0000 7f6d69092640 10 monclient: get_auth_request con 0x7f6d4806c8b0 auth_method 0 2024-01-20T11:20:05.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.872+0000 7f6d69092640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d4806c8b0 0x7f6d4806ed70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6d6413f520 tx=0x7f6d54009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:05.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.872+0000 7f6d5a7fc640 10 client.5290.objecter ms_handle_connect 0x7f6d4806c8b0 2024-01-20T11:20:05.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.874+0000 7f6d5a7fc640 1 -- 172.21.15.110:0/1284027733 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6d6007a810 con 0x7f6d640b17c0 2024-01-20T11:20:05.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.874+0000 7f6d5a7fc640 10 client.5290.objecter ms_dispatch 0x7f6d64000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:05.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.874+0000 7f6d5a7fc640 3 client.5290.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:05.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.874+0000 7f6d5a7fc640 3 client.5290.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:05.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.874+0000 7f6d5a7fc640 20 client.5290.objecter dump_active .. 0 homeless 2024-01-20T11:20:05.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.884+0000 7f6d5a7fc640 1 -- 172.21.15.110:0/1284027733 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6d60076b50 con 0x7f6d640b17c0 2024-01-20T11:20:05.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.884+0000 7f6d5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:05.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:05.884+0000 7f6d5a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.121+0000 7f6d6baf7640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.121+0000 7f6d6baf7640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.121+0000 7f6d6baf7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.121+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f6d640b55c0 con 0x7f6d640b17c0 2024-01-20T11:20:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.127+0000 7f6d5a7fc640 1 -- 172.21.15.110:0/1284027733 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f6d6007a170 con 0x7f6d640b17c0 2024-01-20T11:20:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.127+0000 7f6d5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.127+0000 7f6d5a7fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:06.128 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:06.128 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":340,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.128+0000 7f6d6baf7640 10 librados: watch_flush enter 2024-01-20T11:20:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.128+0000 7f6d6baf7640 10 librados: watch_flush exit 2024-01-20T11:20:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.128+0000 7f6d6baf7640 20 client.5290.objecter shutdown clearing up homeless session... 2024-01-20T11:20:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.128+0000 7f6d6baf7640 10 client.5290.objecter successfully canceled tick 2024-01-20T11:20:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d4806c8b0 msgr2=0x7f6d4806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d4806c8b0 0x7f6d4806ed70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6d6413f520 tx=0x7f6d54009190 comp rx=0 tx=0).stop 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 10 monclient: shutdown 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d640b17c0 msgr2=0x7f6d6413ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d640b17c0 0x7f6d6413ed10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6d60033000 tx=0x7f6d600046c0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 shutdown_connections 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d4806c8b0 0x7f6d4806ed70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d64145980 0x7f6d641d7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d640b4d10 0x7f6d641d7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 --2- 172.21.15.110:0/1284027733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d640b17c0 0x7f6d6413ed10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 >> 172.21.15.110:0/1284027733 conn(0x7f6d64066590 msgr2=0x7f6d6413b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 shutdown_connections 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 -- 172.21.15.110:0/1284027733 wait complete. 2024-01-20T11:20:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.129+0000 7f6d6baf7640 1 librados: shutdown 2024-01-20T11:20:06.140 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:06.140 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.239+0000 7ffb80032640 10 monclient: get_monmap_and_config 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.239+0000 7ffb80032640 10 monclient: build_initial_monmap 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.239+0000 7ffb80032640 10 monclient: monmap: 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:06.240844+0000 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:06.240844+0000 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:06.240 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:06.241 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:06.241 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:06.241 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:06.241 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.239+0000 7ffb80032640 1 Processor -- start 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 1 -- start start 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient: init 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb7813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient(hunting): picked mon.noname-b con 0x7ffb780a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb7813fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffb780a9ad0 0x7ffb78142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient(hunting): picked mon.noname-a con 0x7ffb780a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb780b25c0 con 0x7ffb7813fa10 2024-01-20T11:20:06.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb780aaea0 con 0x7ffb780a9ad0 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb780ab0a0 con 0x7ffb780a96f0 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.240+0000 7ffb80032640 10 monclient(hunting): authenticate will time out at 1649.591780s 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7e5cf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffb780a9ad0 0x7ffb78142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7ddce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb7813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7d5cd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ffb7813fa10 0x7ffb781423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:48124/0 (socket says 172.21.15.110:48124) 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7d5cd640 1 -- 172.21.15.110:0/2044665085 learned_addr learned my addr 172.21.15.110:0/2044665085 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7ddce640 10 monclient(hunting): get_auth_request con 0x7ffb780a96f0 auth_method 0 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7ddce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7ddce640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7ddce640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7e5cf640 10 monclient(hunting): get_auth_request con 0x7ffb780a9ad0 auth_method 0 2024-01-20T11:20:06.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7e5cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:06.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7e5cf640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:06.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.241+0000 7ffb7e5cf640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:06.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.242+0000 7ffb7e5cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:06.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.242+0000 7ffb7e5cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:06.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.242+0000 7ffb7e5cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:06.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.242+0000 7ffb7ddce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:06.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.242+0000 7ffb7ddce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:06.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.242+0000 7ffb7ddce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:06.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7cdcc640 1 -- 172.21.15.110:0/2044665085 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193887794 0 0) 0x7ffb780b25c0 con 0x7ffb7813fa10 2024-01-20T11:20:06.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7cdcc640 10 monclient(hunting): _negotiate 2024-01-20T11:20:06.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7cdcc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:06.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7cdcc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:06.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7cdcc640 10 monclient(hunting): my global_id is 5293 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7cdcc640 1 -- 172.21.15.110:0/2044665085 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64003520 con 0x7ffb7813fa10 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7e5cf640 10 monclient(hunting): handle_auth_done global_id 5214 payload 274 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7e5cf640 1 -- 172.21.15.110:0/2044665085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 msgr2=0x7ffb7813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7e5cf640 1 --2- 172.21.15.110:0/2044665085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb7813f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7e5cf640 1 -- 172.21.15.110:0/2044665085 >> v1:172.21.15.134:6789/0 conn(0x7ffb7813fa10 legacy=0x7ffb781423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7e5cf640 10 monclient: _finish_hunting 0 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7e5cf640 1 monclient: found mon.noname-a 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7e5cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7e5cf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7e5cf640 1 -- 172.21.15.110:0/2044665085 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb780b3fb0 con 0x7ffb780a9ad0 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.243+0000 7ffb7ddce640 1 --2- 172.21.15.110:0/2044665085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb7813f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:20:06.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.245+0000 7ffb7e5cf640 1 --2- 172.21.15.110:0/2044665085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb780a9ad0 0x7ffb78142af0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009d90 tx=0x7ffb7002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc13ebceaf703703 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 1 -- 172.21.15.110:0/2044665085 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7002cbc0 con 0x7ffb780a9ad0 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 10 monclient: dump: 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:06.250 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 10 monclient: _finish_auth 0 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 1 -- 172.21.15.110:0/2044665085 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7002cd20 con 0x7ffb780a9ad0 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 1 -- 172.21.15.110:0/2044665085 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70034660 con 0x7ffb780a9ad0 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 5 monclient: authenticate success, global_id 5214 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 4 set_mon_vals no callback set 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb7cdcc640 10 monclient: dump: 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:06.251 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:06.252 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:06.252 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:06.252 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:06.252 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:06.252 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:06.252 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:06.252 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:06.252 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 10 monclient: shutdown 2024-01-20T11:20:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 1 -- 172.21.15.110:0/2044665085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb780a9ad0 msgr2=0x7ffb78142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 1 --2- 172.21.15.110:0/2044665085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb780a9ad0 0x7ffb78142af0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009d90 tx=0x7ffb7002be10 comp rx=0 tx=0).stop 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 1 -- 172.21.15.110:0/2044665085 shutdown_connections 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 1 --2- 172.21.15.110:0/2044665085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb780a9ad0 0x7ffb78142af0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 1 --2- 172.21.15.110:0/2044665085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb7813f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 1 -- 172.21.15.110:0/2044665085 >> 172.21.15.110:0/2044665085 conn(0x7ffb780b7bc0 msgr2=0x7ffb780b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 1 -- 172.21.15.110:0/2044665085 shutdown_connections 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.249+0000 7ffb80032640 1 -- 172.21.15.110:0/2044665085 wait complete. 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.250+0000 7ffb80032640 10 monclient: build_initial_monmap 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.250+0000 7ffb80032640 10 monclient: monmap: 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.250+0000 7ffb80032640 1 librados: starting msgr at 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.250+0000 7ffb80032640 1 librados: starting objecter 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 Processor -- start 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 -- start start 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 librados: setting wanted keys 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 librados: calling monclient init 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient: init 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:20:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb78134320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient(hunting): picked mon.noname-c con 0x7ffb780a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb780a9ad0 0x7ffb78137490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient(hunting): picked mon.noname-b con 0x7ffb780a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb781349f0 0x7ffb781d70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient(hunting): picked mon.noname-a con 0x7ffb781349f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb781d8040 con 0x7ffb781349f0 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb78136f70 con 0x7ffb780a9ad0 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb78134e70 con 0x7ffb780a96f0 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.251+0000 7ffb80032640 10 monclient(hunting): authenticate will time out at 1649.602778s 2024-01-20T11:20:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb781349f0 0x7ffb781d70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7ddce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb78134320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7d5cd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb780a9ad0 0x7ffb78137490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb781349f0 0x7ffb781d70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36262/0 (socket says 172.21.15.110:36262) 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 1 -- 172.21.15.110:0/2143790348 learned_addr learned my addr 172.21.15.110:0/2143790348 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 10 monclient(hunting): get_auth_request con 0x7ffb781349f0 auth_method 0 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7d5cd640 10 monclient(hunting): get_auth_request con 0x7ffb780a9ad0 auth_method 0 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7d5cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7d5cd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7d5cd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7ddce640 10 monclient(hunting): get_auth_request con 0x7ffb780a96f0 auth_method 0 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7ddce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7ddce640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7ddce640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7d5cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7d5cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7d5cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7e5cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7ddce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.252+0000 7ffb7ddce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.253+0000 7ffb7ddce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:06.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 10 monclient(hunting): handle_auth_done global_id 5217 payload 824 2024-01-20T11:20:06.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 1 -- 172.21.15.110:0/2143790348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 msgr2=0x7ffb78134320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:06.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb78134320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 1 -- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb780a9ad0 msgr2=0x7ffb78137490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb780a9ad0 0x7ffb78137490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 10 monclient: _finish_hunting 0 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 1 monclient: found mon.noname-a 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7e5cf640 1 -- 172.21.15.110:0/2143790348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb74009400 con 0x7ffb781349f0 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7d5cd640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb780a9ad0 0x7ffb78137490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.255+0000 7ffb7ddce640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb78134320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.257+0000 7ffb7e5cf640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb781349f0 0x7ffb781d70e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009d90 tx=0x7ffb70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.257+0000 7ffb6e7fc640 10 client.?.objecter ms_handle_connect 0x7ffb781349f0 2024-01-20T11:20:06.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.257+0000 7ffb6e7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 1 -- 172.21.15.110:0/2143790348 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70040650 con 0x7ffb781349f0 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 10 monclient: dump: 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:06.264 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 1 -- 172.21.15.110:0/2143790348 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7002ce90 con 0x7ffb781349f0 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 1 -- 172.21.15.110:0/2143790348 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700409f0 con 0x7ffb781349f0 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6e7fc640 10 monclient: dump: 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:06.265 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb80032640 5 monclient: authenticate success, global_id 5217 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb80032640 10 monclient: _renew_subs 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb6ffff640 4 set_mon_vals no callback set 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb80032640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb70009a70 con 0x7ffb781349f0 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb80032640 10 client.5217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb80032640 10 monclient: _renew_subs 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb80032640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb78135620 con 0x7ffb781349f0 2024-01-20T11:20:06.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.261+0000 7ffb80032640 1 librados: init done 2024-01-20T11:20:06.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.262+0000 7ffb80032640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:06.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.262+0000 7ffb80032640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:06.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.262+0000 7ffb80032640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:06.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.262+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb780aa4a0 con 0x7ffb781349f0 2024-01-20T11:20:06.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.280+0000 7ffb6e7fc640 1 -- 172.21.15.110:0/2143790348 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ffb70040b50 con 0x7ffb781349f0 2024-01-20T11:20:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.281+0000 7ffb6e7fc640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb5c06c8b0 0x7ffb5c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.281+0000 7ffb7ddce640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb5c06c8b0 0x7ffb5c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.281+0000 7ffb7ddce640 10 monclient: get_auth_request con 0x7ffb5c06c8b0 auth_method 0 2024-01-20T11:20:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.281+0000 7ffb6e7fc640 10 client.5217.objecter ms_handle_connect 0x7ffb5c06c8b0 2024-01-20T11:20:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.281+0000 7ffb7ddce640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb5c06c8b0 0x7ffb5c06ed70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffb74011380 tx=0x7ffb74009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:06.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.283+0000 7ffb6e7fc640 1 -- 172.21.15.110:0/2143790348 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffb7007b170 con 0x7ffb781349f0 2024-01-20T11:20:06.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.283+0000 7ffb6e7fc640 10 client.5217.objecter ms_dispatch 0x7ffb78000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.283+0000 7ffb6e7fc640 3 client.5217.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.283+0000 7ffb6e7fc640 3 client.5217.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.284+0000 7ffb6e7fc640 20 client.5217.objecter dump_active .. 0 homeless 2024-01-20T11:20:06.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~68s 2024-01-20T11:20:06.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.523+0000 7ffb6e7fc640 1 -- 172.21.15.110:0/2143790348 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ffb700774b0 con 0x7ffb781349f0 2024-01-20T11:20:06.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.523+0000 7ffb6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:06.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.523+0000 7ffb6e7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:06.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.712+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb78075490 con 0x7ffb5c06c8b0 2024-01-20T11:20:06.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.715+0000 7ffb6e7fc640 1 -- 172.21.15.110:0/2143790348 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64091 (secure 0 0 0) 0x7ffb78075490 con 0x7ffb5c06c8b0 2024-01-20T11:20:06.717 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:06.717 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 10 librados: watch_flush enter 2024-01-20T11:20:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 10 librados: watch_flush exit 2024-01-20T11:20:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 20 client.5217.objecter shutdown clearing up homeless session... 2024-01-20T11:20:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 10 client.5217.objecter successfully canceled tick 2024-01-20T11:20:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb5c06c8b0 msgr2=0x7ffb5c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb5c06c8b0 0x7ffb5c06ed70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffb74011380 tx=0x7ffb74009190 comp rx=0 tx=0).stop 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 10 monclient: shutdown 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb781349f0 msgr2=0x7ffb781d70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb781349f0 0x7ffb781d70e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009d90 tx=0x7ffb70004a40 comp rx=0 tx=0).stop 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 shutdown_connections 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb5c06c8b0 0x7ffb5c06ed70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb781349f0 0x7ffb781d70e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb780a9ad0 0x7ffb78137490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 --2- 172.21.15.110:0/2143790348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb780a96f0 0x7ffb78134320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.717+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 >> 172.21.15.110:0/2143790348 conn(0x7ffb780b7bc0 msgr2=0x7ffb7813c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.718+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 shutdown_connections 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.718+0000 7ffb80032640 1 -- 172.21.15.110:0/2143790348 wait complete. 2024-01-20T11:20:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:06.718+0000 7ffb80032640 1 librados: shutdown 2024-01-20T11:20:06.729 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-01-20T11:20:04.766327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002217"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280576,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.7510000000000003},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":14.115}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.449999999999999},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":5.7910000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":15.288},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.587999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.8860000000000001},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":12.114000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.696999999999999},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":10.013999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.2319999999999993},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.120000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313278,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.7450000000000001},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.9619999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.2080000000000002},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":13.755000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.0549999999999997},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.226000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":14.444000000000001},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.484}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.6280000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":10.85}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.122},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.747999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313279,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":4.3200000000000003},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.8879999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.7319999999999993},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.352}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":8.6950000000000003},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.717000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.219000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":13.526},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":15.638999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":17.420999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.081}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444093,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.2709999999999999},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":8.3559999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.984999999999999},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":19.446000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.151},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.612}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.253},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":16.096}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.641999999999999},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.808}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.430999999999999},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":14.721}]}]},{"osd":1,"up_from":17,"seq":73014444094,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.9120000000000008},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":13.863}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.066000000000001},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":14.817}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":13.071999999999999},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.0449999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":17.187999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.407999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.9060000000000006},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":16.091999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.988},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.966000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444094,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":5.2039999999999997},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":10.433}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.0460000000000003},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.2050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":12.630000000000001},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.744999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.897},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.5540000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":12.023999999999999},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.49}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.725},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":9.609}]}]},{"osd":6,"up_from":15,"seq":64424509502,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":10.923},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.8689999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":4.8319999999999999},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":13.234999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.8260000000000005},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.1809999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.84},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.558999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.994999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.608000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.561999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.276999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:09.731 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:09.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.835+0000 7fea601ce640 10 monclient: get_monmap_and_config 2024-01-20T11:20:09.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.835+0000 7fea601ce640 10 monclient: build_initial_monmap 2024-01-20T11:20:09.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.835+0000 7fea601ce640 10 monclient: monmap: 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:09.837468+0000 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:09.837468+0000 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:09.837 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:09.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.836+0000 7fea601ce640 1 Processor -- start 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.836+0000 7fea601ce640 1 -- start start 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.836+0000 7fea601ce640 10 monclient: init 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fea580b4d10 0x7fea580b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 10 monclient(hunting): picked mon.noname-a con 0x7fea580b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea580b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea58145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea580aade0 con 0x7fea58145980 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea58145460 con 0x7fea580b17c0 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea580b50f0 con 0x7fea580b4d10 2024-01-20T11:20:09.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:09.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.837+0000 7fea601ce640 10 monclient(hunting): authenticate will time out at 1653.188428s 2024-01-20T11:20:09.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.864+0000 7fea5d769640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fea580b17c0 0x7fea580b1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:48746/0 (socket says 172.21.15.110:48746) 2024-01-20T11:20:09.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.864+0000 7fea5d769640 1 -- 172.21.15.110:0/3961574179 learned_addr learned my addr 172.21.15.110:0/3961574179 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:09.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.876+0000 7fea5df6a640 1 --2- 172.21.15.110:0/3961574179 >> v2:172.21.15.110:3300/0 conn(0x7fea580b4d10 0x7fea580b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:09.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.878+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648775992 0 0) 0x7fea58145460 con 0x7fea580b17c0 2024-01-20T11:20:09.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.878+0000 7fea5cf68640 10 monclient(hunting): _negotiate 2024-01-20T11:20:09.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.878+0000 7fea5cf68640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:09.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.878+0000 7fea5cf68640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:09.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.878+0000 7fea5cf68640 10 monclient(hunting): my global_id is 5299 2024-01-20T11:20:09.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.878+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea44003520 con 0x7fea580b17c0 2024-01-20T11:20:09.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.879+0000 7fea5df6a640 10 monclient(hunting): get_auth_request con 0x7fea580b4d10 auth_method 0 2024-01-20T11:20:09.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.879+0000 7fea5df6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:09.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.879+0000 7fea5df6a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:09.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.879+0000 7fea5df6a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:09.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3276485796 0 0) 0x7fea44003520 con 0x7fea580b17c0 2024-01-20T11:20:09.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 >> v2:172.21.15.110:3300/0 conn(0x7fea580b4d10 msgr2=0x7fea580b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:09.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 --2- 172.21.15.110:0/3961574179 >> v2:172.21.15.110:3300/0 conn(0x7fea580b4d10 0x7fea580b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 >> v1:172.21.15.110:6789/0 conn(0x7fea58145980 legacy=0x7fea58145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient: _finish_hunting 0 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 monclient: found mon.noname-c-legacy 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.134:6789/0 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580b2660 con 0x7fea580b17c0 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient: _finish_auth 0 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient: _check_auth_tickets 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2748620985 0 0) 0x7fea54001670 con 0x7fea580b17c0 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-01-20T11:20:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient: dump: 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] but i'm connected to v1:172.21.15.134:6789/0, reconnecting 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 >> v1:172.21.15.134:6789/0 conn(0x7fea580b17c0 legacy=0x7fea580b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea440039c0 0x7fea44003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient(hunting): picked mon.a con 0x7fea440039c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440047c0 0x7fea44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient(hunting): picked mon.b con 0x7fea440047c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea440052a0 0x7fea4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient(hunting): picked mon.c con 0x7fea440052a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea580b50f0 con 0x7fea440039c0 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:09.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4400c1c0 con 0x7fea440047c0 2024-01-20T11:20:09.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:09.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4400c380 con 0x7fea440052a0 2024-01-20T11:20:09.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:09.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.882+0000 7fea5cf68640 10 monclient(hunting): _finish_auth 0 2024-01-20T11:20:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.883+0000 7fea5e76b640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea440052a0 0x7fea4400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.884+0000 7fea5e76b640 10 monclient(hunting): get_auth_request con 0x7fea440052a0 auth_method 0 2024-01-20T11:20:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.884+0000 7fea5e76b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.884+0000 7fea5e76b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.884+0000 7fea5e76b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:20:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.884+0000 7fea5e76b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.884+0000 7fea5e76b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.884+0000 7fea5e76b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:20:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.885+0000 7fea5d769640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440047c0 0x7fea44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.885+0000 7fea5d769640 10 monclient(hunting): get_auth_request con 0x7fea440047c0 auth_method 0 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.885+0000 7fea5d769640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.885+0000 7fea5d769640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.885+0000 7fea5d769640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.886+0000 7fea5d769640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.886+0000 7fea5d769640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.886+0000 7fea5d769640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.886+0000 7fea5e76b640 10 monclient(hunting): handle_auth_done global_id 5299 payload 293 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 1 -- 172.21.15.110:0/3961574179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440047c0 msgr2=0x7fea44004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440047c0 0x7fea44004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 1 -- 172.21.15.110:0/3961574179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea440039c0 msgr2=0x7fea44003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea440039c0 0x7fea44003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 10 monclient: _finish_hunting 0 2024-01-20T11:20:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 1 monclient: found mon.c 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 1 -- 172.21.15.110:0/3961574179 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea44003520 con 0x7fea440052a0 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 10 monclient: _finish_auth 0 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 10 monclient: _check_auth_tickets 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea601ce640 5 monclient: authenticate success, global_id 5299 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea601ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.887+0000 7fea5e76b640 1 -- 172.21.15.110:0/3961574179 reap_dead start 2024-01-20T11:20:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.889+0000 7fea5e76b640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea440052a0 0x7fea4400b980 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fea50001e50 tx=0x7fea440057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a21a4f10521bf97 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50003b70 con 0x7fea440052a0 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 10 monclient: dump: 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:09.894 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea50003cf0 con 0x7fea440052a0 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 4 set_mon_vals no callback set 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 1 -- 172.21.15.110:0/3961574179 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50005310 con 0x7fea440052a0 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea5cf68640 10 monclient: dump: 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:09.896 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 10 monclient: shutdown 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 1 -- 172.21.15.110:0/3961574179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea440052a0 msgr2=0x7fea4400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea440052a0 0x7fea4400b980 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fea50001e50 tx=0x7fea440057f0 comp rx=0 tx=0).stop 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 1 -- 172.21.15.110:0/3961574179 shutdown_connections 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 1 --2- 172.21.15.110:0/3961574179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea440052a0 0x7fea4400b980 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 1 -- 172.21.15.110:0/3961574179 >> 172.21.15.110:0/3961574179 conn(0x7fea58066590 msgr2=0x7fea580669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.893+0000 7fea601ce640 1 -- 172.21.15.110:0/3961574179 shutdown_connections 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.894+0000 7fea601ce640 1 -- 172.21.15.110:0/3961574179 wait complete. 2024-01-20T11:20:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.894+0000 7fea601ce640 10 monclient: build_initial_monmap 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.894+0000 7fea601ce640 10 monclient: monmap: 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.894+0000 7fea601ce640 1 librados: starting msgr at 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.894+0000 7fea601ce640 1 librados: starting objecter 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.894+0000 7fea601ce640 1 Processor -- start 2024-01-20T11:20:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 1 -- start start 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 1 librados: setting wanted keys 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 1 librados: calling monclient init 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient: init 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea440052a0 0x7fea58142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient(hunting): picked mon.noname-a con 0x7fea440052a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea581da6e0 0x7fea580acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient(hunting): picked mon.noname-b con 0x7fea581da6e0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea580adbf0 0x7fea580ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient(hunting): picked mon.noname-c con 0x7fea580adbf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea580ad490 con 0x7fea440052a0 2024-01-20T11:20:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea580af2d0 con 0x7fea581da6e0 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea580af5b0 con 0x7fea580adbf0 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea601ce640 10 monclient(hunting): authenticate will time out at 1653.246750s 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.895+0000 7fea5e76b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea580adbf0 0x7fea580ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5df6a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea440052a0 0x7fea58142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5d769640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea581da6e0 0x7fea580acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5e76b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea580adbf0 0x7fea580ae050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43674/0 (socket says 172.21.15.110:43674) 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5e76b640 1 -- 172.21.15.110:0/3049086845 learned_addr learned my addr 172.21.15.110:0/3049086845 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5e76b640 10 monclient(hunting): get_auth_request con 0x7fea580adbf0 auth_method 0 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5e76b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5e76b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5e76b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5df6a640 10 monclient(hunting): get_auth_request con 0x7fea440052a0 auth_method 0 2024-01-20T11:20:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5df6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5df6a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5df6a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5d769640 10 monclient(hunting): get_auth_request con 0x7fea581da6e0 auth_method 0 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5d769640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5d769640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5d769640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5e76b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5e76b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.896+0000 7fea5e76b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.897+0000 7fea5df6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.897+0000 7fea5df6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.897+0000 7fea5df6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.897+0000 7fea5d769640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.897+0000 7fea5d769640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.897+0000 7fea5d769640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 10 monclient(hunting): handle_auth_done global_id 5302 payload 824 2024-01-20T11:20:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 1 -- 172.21.15.110:0/3049086845 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea580adbf0 msgr2=0x7fea580ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea580adbf0 0x7fea580ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 1 -- 172.21.15.110:0/3049086845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea440052a0 msgr2=0x7fea58142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea440052a0 0x7fea58142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 10 monclient: _finish_hunting 0 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 1 monclient: found mon.noname-b 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.899+0000 7fea5d769640 1 -- 172.21.15.110:0/3049086845 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580ab290 con 0x7fea581da6e0 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.901+0000 7fea5d769640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea581da6e0 0x7fea580acf50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fea580b2320 tx=0x7fea54007720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.901+0000 7fea4e7fc640 10 client.?.objecter ms_handle_connect 0x7fea581da6e0 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.901+0000 7fea4e7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 1 -- 172.21.15.110:0/3049086845 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea540015d0 con 0x7fea581da6e0 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 10 monclient: dump: 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 1 -- 172.21.15.110:0/3049086845 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5400f940 con 0x7fea581da6e0 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 1 -- 172.21.15.110:0/3049086845 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5400fcf0 con 0x7fea581da6e0 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4e7fc640 10 monclient: dump: 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea601ce640 5 monclient: authenticate success, global_id 5302 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea601ce640 10 monclient: _renew_subs 2024-01-20T11:20:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea601ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:09.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea4ffff640 4 set_mon_vals no callback set 2024-01-20T11:20:09.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea580ab480 con 0x7fea581da6e0 2024-01-20T11:20:09.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea601ce640 10 client.5302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:09.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea601ce640 10 monclient: _renew_subs 2024-01-20T11:20:09.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea601ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:09.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea580ab940 con 0x7fea581da6e0 2024-01-20T11:20:09.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.905+0000 7fea601ce640 1 librados: init done 2024-01-20T11:20:09.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.906+0000 7fea601ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:09.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.906+0000 7fea601ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:09.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.906+0000 7fea601ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:09.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.906+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea58075490 con 0x7fea581da6e0 2024-01-20T11:20:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.924+0000 7fea4e7fc640 1 -- 172.21.15.110:0/3049086845 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fea5401d460 con 0x7fea581da6e0 2024-01-20T11:20:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.924+0000 7fea4e7fc640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea3c06c8b0 0x7fea3c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.925+0000 7fea5df6a640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea3c06c8b0 0x7fea3c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:09.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.925+0000 7fea5df6a640 10 monclient: get_auth_request con 0x7fea3c06c8b0 auth_method 0 2024-01-20T11:20:09.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.925+0000 7fea4e7fc640 10 client.5302.objecter ms_handle_connect 0x7fea3c06c8b0 2024-01-20T11:20:09.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.925+0000 7fea5df6a640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea3c06c8b0 0x7fea3c06ed70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fea48006a90 tx=0x7fea48008040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.927+0000 7fea4e7fc640 1 -- 172.21.15.110:0/3049086845 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fea54058c10 con 0x7fea581da6e0 2024-01-20T11:20:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.927+0000 7fea4e7fc640 10 client.5302.objecter ms_dispatch 0x7fea58000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.927+0000 7fea4e7fc640 3 client.5302.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.927+0000 7fea4e7fc640 3 client.5302.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:09.927+0000 7fea4e7fc640 20 client.5302.objecter dump_active .. 0 homeless 2024-01-20T11:20:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.158+0000 7fea4e7fc640 1 -- 172.21.15.110:0/3049086845 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fea54054f50 con 0x7fea581da6e0 2024-01-20T11:20:10.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.158+0000 7fea4e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:10.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.158+0000 7fea4e7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:10.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.326+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea580a9cf0 con 0x7fea3c06c8b0 2024-01-20T11:20:10.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.329+0000 7fea4e7fc640 1 -- 172.21.15.110:0/3049086845 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64032 (secure 0 0 0) 0x7fea48004030 con 0x7fea3c06c8b0 2024-01-20T11:20:10.330 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:10.331 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 10 librados: watch_flush enter 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 10 librados: watch_flush exit 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 20 client.5302.objecter shutdown clearing up homeless session... 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 10 client.5302.objecter successfully canceled tick 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea3c06c8b0 msgr2=0x7fea3c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea3c06c8b0 0x7fea3c06ed70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fea48006a90 tx=0x7fea48008040 comp rx=0 tx=0).stop 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 10 monclient: shutdown 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea581da6e0 msgr2=0x7fea580acf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea581da6e0 0x7fea580acf50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fea580b2320 tx=0x7fea54007720 comp rx=0 tx=0).stop 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 shutdown_connections 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea3c06c8b0 0x7fea3c06ed70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea580adbf0 0x7fea580ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea581da6e0 0x7fea580acf50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 --2- 172.21.15.110:0/3049086845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea440052a0 0x7fea58142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 >> 172.21.15.110:0/3049086845 conn(0x7fea58066590 msgr2=0x7fea580b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:10.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 shutdown_connections 2024-01-20T11:20:10.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 -- 172.21.15.110:0/3049086845 wait complete. 2024-01-20T11:20:10.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.331+0000 7fea601ce640 1 librados: shutdown 2024-01-20T11:20:10.342 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-01-20T11:20:08.766973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002145"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280577,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.1180000000000003},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.3900000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.7639999999999993},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.7230000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.212999999999999},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.022}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.75},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.237}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.975},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":14.558999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.329000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.318000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313279,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":4.524},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.7780000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.2110000000000003},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.4030000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.762},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.512}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.4180000000000001},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.477}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.127000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":14.778}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.749000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.212}]}]},{"osd":3,"up_from":21,"seq":90194313280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.693},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.23}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.4710000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.086}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.7530000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":11.305999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.945}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":15.090999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.782}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444094,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.9730000000000008},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":4.3230000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.1130000000000004},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.5709999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.784000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.3770000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.925000000000001},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.428000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.849},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.079000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.752000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":11.493}]}]},{"osd":1,"up_from":17,"seq":73014444095,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0970000000000004},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":5.1769999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.4290000000000003},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.2080000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":11.948},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.890000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":13.199},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":9.5760000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.8620000000000001},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":10.722}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.417999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.635}]}]},{"osd":0,"up_from":17,"seq":73014444095,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":5.0860000000000003},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":12.641}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.6929999999999996},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.8539999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":13.460000000000001},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":17.945}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.281000000000001},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.2789999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":10.461},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.646000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.696000000000002},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":9.2140000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509503,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":6.6470000000000002},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.3070000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.0730000000000004},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":12.166}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.2910000000000004},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.491}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.763},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.335000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.504},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.615}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.5210000000000008},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.228999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:10.344 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.444+0000 7f0ba0fd0640 10 monclient: get_monmap_and_config 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.444+0000 7f0ba0fd0640 10 monclient: build_initial_monmap 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.444+0000 7f0ba0fd0640 10 monclient: monmap: 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:10.445698+0000 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:10.445698+0000 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:10.446 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:10.446 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:10.446 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:10.446 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:10.446 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:10.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.444+0000 7f0ba0fd0640 1 Processor -- start 2024-01-20T11:20:10.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.444+0000 7f0ba0fd0640 1 -- start start 2024-01-20T11:20:10.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.444+0000 7f0ba0fd0640 10 monclient: init 2024-01-20T11:20:10.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:10.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b9c0b4d10 0x7f0b9c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f0b9c0b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0b9c0b17c0 0x7f0b9c0b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f0b9c0b17c0 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b9c145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c0b2150 con 0x7f0b9c145980 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c0b50f0 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c0aade0 con 0x7f0b9c0b17c0 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.445+0000 7f0ba0fd0640 10 monclient(hunting): authenticate will time out at 1653.796737s 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b9a59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b9c0b4d10 0x7f0b9c0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b93fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f0b9c0b17c0 0x7f0b9c0b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b9a59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b9c0b4d10 0x7f0b9c0b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:49062/0 (socket says 172.21.15.110:49062) 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b9a59c640 1 -- 172.21.15.110:0/2291340775 learned_addr learned my addr 172.21.15.110:0/2291340775 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b9a59c640 10 monclient(hunting): get_auth_request con 0x7f0b9c0b4d10 auth_method 0 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b9a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b9a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b9a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b93fff640 10 monclient(hunting): get_auth_request con 0x7f0b9c0b17c0 auth_method 0 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b93fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.446+0000 7f0b93fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.447+0000 7f0b9a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.447+0000 7f0b9a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.447+0000 7f0b9a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.447+0000 7f0b93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:10.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.447+0000 7f0b93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:10.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.447+0000 7f0b93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:10.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.448+0000 7f0b9a59c640 10 monclient(hunting): handle_auth_done global_id 5229 payload 274 2024-01-20T11:20:10.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.448+0000 7f0b9a59c640 1 -- 172.21.15.110:0/2291340775 >> v2:172.21.15.134:3300/0 conn(0x7f0b9c0b17c0 msgr2=0x7f0b9c0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:10.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.448+0000 7f0b9a59c640 1 --2- 172.21.15.110:0/2291340775 >> v2:172.21.15.134:3300/0 conn(0x7f0b9c0b17c0 0x7f0b9c0b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.448+0000 7f0b9a59c640 1 -- 172.21.15.110:0/2291340775 >> v1:172.21.15.134:6789/0 conn(0x7f0b9c145980 legacy=0x7f0b9c145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:10.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.448+0000 7f0b9a59c640 10 monclient: _finish_hunting 0 2024-01-20T11:20:10.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.448+0000 7f0b9a59c640 1 monclient: found mon.noname-a 2024-01-20T11:20:10.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.448+0000 7f0b9a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:10.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.448+0000 7f0b9a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:10.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.448+0000 7f0b9a59c640 1 -- 172.21.15.110:0/2291340775 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c145460 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.450+0000 7f0b9a59c640 1 --2- 172.21.15.110:0/2291340775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b9c0b4d10 0x7f0b9c0b3da0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009870 tx=0x7f0b8c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235b3b046792c2b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 1 -- 172.21.15.110:0/2291340775 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c031750 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 10 monclient: dump: 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:10.455 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 10 monclient: _finish_auth 0 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 5 monclient: authenticate success, global_id 5229 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 1 -- 172.21.15.110:0/2291340775 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c0318b0 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 4 set_mon_vals no callback set 2024-01-20T11:20:10.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 1 -- 172.21.15.110:0/2291340775 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c031c40 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0b99d9b640 10 monclient: dump: 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:10.457 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 10 monclient: shutdown 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/2291340775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b9c0b4d10 msgr2=0x7f0b9c0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 1 --2- 172.21.15.110:0/2291340775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b9c0b4d10 0x7f0b9c0b3da0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009870 tx=0x7f0b8c02b7f0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/2291340775 shutdown_connections 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 1 --2- 172.21.15.110:0/2291340775 >> v2:172.21.15.134:3300/0 conn(0x7f0b9c0b17c0 0x7f0b9c0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 1 --2- 172.21.15.110:0/2291340775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b9c0b4d10 0x7f0b9c0b3da0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.454+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/2291340775 >> 172.21.15.110:0/2291340775 conn(0x7f0b9c066590 msgr2=0x7f0b9c0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.455+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/2291340775 shutdown_connections 2024-01-20T11:20:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.455+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/2291340775 wait complete. 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.455+0000 7f0ba0fd0640 10 monclient: build_initial_monmap 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.455+0000 7f0ba0fd0640 10 monclient: monmap: 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 librados: starting msgr at 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 librados: starting objecter 2024-01-20T11:20:10.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 Processor -- start 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 -- start start 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 librados: setting wanted keys 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 librados: calling monclient init 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient: init 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b9c0b17c0 0x7f0b9c140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f0b9c0b17c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b9c0b4d10 0x7f0b9c135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f0b9c0b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b9c145980 0x7f0b9c136430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f0b9c145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:10.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c1370c0 con 0x7f0b9c0b17c0 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c136970 con 0x7f0b9c145980 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c145f90 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.456+0000 7f0ba0fd0640 10 monclient(hunting): authenticate will time out at 1653.807912s 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b9c0b4d10 0x7f0b9c135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9a59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b9c0b17c0 0x7f0b9c140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9ad9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b9c145980 0x7f0b9c136430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b9c0b4d10 0x7f0b9c135ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43690/0 (socket says 172.21.15.110:43690) 2024-01-20T11:20:10.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b93fff640 1 -- 172.21.15.110:0/40501991 learned_addr learned my addr 172.21.15.110:0/40501991 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b93fff640 10 monclient(hunting): get_auth_request con 0x7f0b9c0b4d10 auth_method 0 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b93fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b93fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9a59c640 10 monclient(hunting): get_auth_request con 0x7f0b9c0b17c0 auth_method 0 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9ad9d640 10 monclient(hunting): get_auth_request con 0x7f0b9c145980 auth_method 0 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.457+0000 7f0b9ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.458+0000 7f0b93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.458+0000 7f0b93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:10.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.458+0000 7f0b93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:10.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.458+0000 7f0b9a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:10.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.458+0000 7f0b9a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:10.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.458+0000 7f0b9a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:10.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.458+0000 7f0b9ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:10.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.458+0000 7f0b9ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:10.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.458+0000 7f0b9ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:10.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 10 monclient(hunting): handle_auth_done global_id 5006 payload 824 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 1 -- 172.21.15.110:0/40501991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b9c145980 msgr2=0x7f0b9c136430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b9c145980 0x7f0b9c136430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 1 -- 172.21.15.110:0/40501991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b9c0b17c0 msgr2=0x7f0b9c140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b9c0b17c0 0x7f0b9c140cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 1 monclient: found mon.noname-c 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.460+0000 7f0b93fff640 1 -- 172.21.15.110:0/40501991 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c0094d0 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.463+0000 7f0b93fff640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b9c0b4d10 0x7f0b9c135ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400e9e0 tx=0x7f0b8400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.463+0000 7f0b92ffd640 10 client.?.objecter ms_handle_connect 0x7f0b9c0b4d10 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.463+0000 7f0b92ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:10.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 1 -- 172.21.15.110:0/40501991 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84009670 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 10 monclient: dump: 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:10.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 1 -- 172.21.15.110:0/40501991 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8400c9d0 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 1 -- 172.21.15.110:0/40501991 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8400cd80 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0ba0fd0640 5 monclient: authenticate success, global_id 5006 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0ba0fd0640 10 monclient: _renew_subs 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0ba0fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c146270 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b9959a640 4 set_mon_vals no callback set 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0b92ffd640 10 monclient: dump: 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0ba0fd0640 10 client.5006.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0ba0fd0640 10 monclient: _renew_subs 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0ba0fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c140380 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.466+0000 7f0ba0fd0640 1 librados: init done 2024-01-20T11:20:10.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.467+0000 7f0ba0fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:10.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.467+0000 7f0ba0fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:10.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.467+0000 7f0ba0fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:10.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.467+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c0b29f0 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.486+0000 7f0b92ffd640 1 -- 172.21.15.110:0/40501991 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f0b84013470 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.487+0000 7f0b92ffd640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0b7c06c8b0 0x7f0b7c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:10.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.487+0000 7f0b9a59c640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0b7c06c8b0 0x7f0b7c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:10.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.487+0000 7f0b9a59c640 10 monclient: get_auth_request con 0x7f0b7c06c8b0 auth_method 0 2024-01-20T11:20:10.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.488+0000 7f0b9a59c640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0b7c06c8b0 0x7f0b7c06ed70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c035000 tx=0x7f0b8c008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:10.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.488+0000 7f0b92ffd640 10 client.5006.objecter ms_handle_connect 0x7f0b7c06c8b0 2024-01-20T11:20:10.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.730+0000 7f0b92ffd640 1 -- 172.21.15.110:0/40501991 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0b84058b90 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.730+0000 7f0b92ffd640 10 client.5006.objecter ms_dispatch 0x7f0b9c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:10.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.730+0000 7f0b92ffd640 3 client.5006.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:10.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.730+0000 7f0b92ffd640 3 client.5006.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:10.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.730+0000 7f0b92ffd640 20 client.5006.objecter dump_active .. 0 homeless 2024-01-20T11:20:10.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.730+0000 7f0b92ffd640 1 -- 172.21.15.110:0/40501991 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0b84054ed0 con 0x7f0b9c0b4d10 2024-01-20T11:20:10.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.730+0000 7f0b92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:10.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.731+0000 7f0b92ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:10.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.923+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b9c14c7f0 con 0x7f0b7c06c8b0 2024-01-20T11:20:10.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.925+0000 7f0b92ffd640 1 -- 172.21.15.110:0/40501991 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64032 (secure 0 0 0) 0x7f0b8c041070 con 0x7f0b7c06c8b0 2024-01-20T11:20:10.927 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:10.927 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 10 librados: watch_flush enter 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 10 librados: watch_flush exit 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 20 client.5006.objecter shutdown clearing up homeless session... 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 10 client.5006.objecter successfully canceled tick 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0b7c06c8b0 msgr2=0x7f0b7c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0b7c06c8b0 0x7f0b7c06ed70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c035000 tx=0x7f0b8c008030 comp rx=0 tx=0).stop 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 10 monclient: shutdown 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b9c0b4d10 msgr2=0x7f0b9c135ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b9c0b4d10 0x7f0b9c135ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400e9e0 tx=0x7f0b8400eea0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 shutdown_connections 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f0b7c06c8b0 0x7f0b7c06ed70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b9c145980 0x7f0b9c136430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b9c0b4d10 0x7f0b9c135ef0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 --2- 172.21.15.110:0/40501991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b9c0b17c0 0x7f0b9c140cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:10.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 >> 172.21.15.110:0/40501991 conn(0x7f0b9c066590 msgr2=0x7f0b9c0b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:10.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 shutdown_connections 2024-01-20T11:20:10.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 -- 172.21.15.110:0/40501991 wait complete. 2024-01-20T11:20:10.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:10.928+0000 7f0ba0fd0640 1 librados: shutdown 2024-01-20T11:20:10.941 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-01-20T11:20:10.767384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002216"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280577,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.1180000000000003},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.3900000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.7639999999999993},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.7230000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.212999999999999},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.022}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.75},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.237}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.975},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":14.558999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.329000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.318000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313279,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":4.524},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.7780000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.2110000000000003},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.4030000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.762},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.512}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.4180000000000001},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.477}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.127000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":14.778}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.749000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.212}]}]},{"osd":3,"up_from":21,"seq":90194313280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.693},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.23}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.4710000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.086}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.7530000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":11.305999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.945}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":15.090999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.782}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444094,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.9730000000000008},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":4.3230000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.1130000000000004},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.5709999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.784000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.3770000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.925000000000001},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.428000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.849},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.079000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.752000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":11.493}]}]},{"osd":1,"up_from":17,"seq":73014444095,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0970000000000004},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":5.1769999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.4290000000000003},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.2080000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":11.948},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.890000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":13.199},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":9.5760000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.8620000000000001},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":10.722}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.417999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.635}]}]},{"osd":0,"up_from":17,"seq":73014444095,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":5.0860000000000003},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":12.641}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.6929999999999996},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.8539999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":13.460000000000001},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":17.945}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.281000000000001},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.2789999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":10.461},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.646000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.696000000000002},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":9.2140000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509503,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":6.6470000000000002},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.3070000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.0730000000000004},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":12.166}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.2910000000000004},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.491}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.763},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.335000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.504},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.615}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.5210000000000008},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.228999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:10.942 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:10.942 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:10.942 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:10.942 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:10.943 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:10.943 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:10.943 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:10.943 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:10.943 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:11.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.043+0000 7f513809b640 10 monclient: get_monmap_and_config 2024-01-20T11:20:11.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.043+0000 7f513809b640 10 monclient: build_initial_monmap 2024-01-20T11:20:11.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.043+0000 7f513809b640 10 monclient: monmap: 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:11.045389+0000 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:11.045389+0000 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:11.045 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.044+0000 7f513809b640 1 Processor -- start 2024-01-20T11:20:11.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.044+0000 7f513809b640 1 -- start start 2024-01-20T11:20:11.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.044+0000 7f513809b640 10 monclient: init 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f513006b380 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f513006e430 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f513013d600 0x7f5130143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 10 monclient(hunting): picked mon.noname-c con 0x7f513013d600 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5130071de0 con 0x7f513006e430 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5130070ef0 con 0x7f513006b380 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5130144220 con 0x7f513013d600 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f513809b640 10 monclient(hunting): authenticate will time out at 1654.396486s 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f5135636640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f513006e430 0x7f51301435d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56310/0 (socket says 172.21.15.110:56310) 2024-01-20T11:20:11.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.045+0000 7f5135636640 1 -- 172.21.15.110:0/3153363637 learned_addr learned my addr 172.21.15.110:0/3153363637 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:11.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.046+0000 7f5136638640 1 --2- 172.21.15.110:0/3153363637 >> v2:172.21.15.134:3300/0 conn(0x7f513013d600 0x7f5130143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.046+0000 7f5136638640 10 monclient(hunting): get_auth_request con 0x7f513013d600 auth_method 0 2024-01-20T11:20:11.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.046+0000 7f5136638640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:11.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.046+0000 7f5136638640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.046+0000 7f5136638640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.046+0000 7f5136638640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:11.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.046+0000 7f5136638640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:11.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.046+0000 7f5136638640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5136638640 10 monclient(hunting): handle_auth_done global_id 5311 payload 274 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5134e35640 1 -- 172.21.15.110:0/3153363637 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954062962 0 0) 0x7f5130071de0 con 0x7f513006e430 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5136638640 1 -- 172.21.15.110:0/3153363637 >> v1:172.21.15.134:6789/0 conn(0x7f513006b380 legacy=0x7f513013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5136638640 1 -- 172.21.15.110:0/3153363637 >> v1:172.21.15.110:6789/0 conn(0x7f513006e430 legacy=0x7f51301435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5136638640 10 monclient: _finish_hunting 0 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5136638640 1 monclient: found mon.noname-c 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5136638640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5136638640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5136638640 1 -- 172.21.15.110:0/3153363637 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513006dc70 con 0x7f513013d600 2024-01-20T11:20:11.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.048+0000 7f5134e35640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:20:11.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.051+0000 7f5136638640 1 --2- 172.21.15.110:0/3153363637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 0x7f5130143ce0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5128009d90 tx=0x7f512802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c50646c02a080e16 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 1 -- 172.21.15.110:0/3153363637 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512802cbd0 con 0x7f513013d600 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 10 monclient: dump: 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:11.056 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 10 monclient: _finish_auth 0 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 10 monclient: _check_auth_tickets 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 1 -- 172.21.15.110:0/3153363637 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512802cd30 con 0x7f513013d600 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 5 monclient: authenticate success, global_id 5311 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 1 -- 172.21.15.110:0/3153363637 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512802cea0 con 0x7f513013d600 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f5134e35640 10 monclient: dump: 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.057 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.058 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:11.058 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.058 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:11.058 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:11.058 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:11.058 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:11.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 4 set_mon_vals no callback set 2024-01-20T11:20:11.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 10 monclient: shutdown 2024-01-20T11:20:11.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:11.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 1 -- 172.21.15.110:0/3153363637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 msgr2=0x7f5130143ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 1 --2- 172.21.15.110:0/3153363637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 0x7f5130143ce0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5128009d90 tx=0x7f512802be20 comp rx=0 tx=0).stop 2024-01-20T11:20:11.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 1 -- 172.21.15.110:0/3153363637 shutdown_connections 2024-01-20T11:20:11.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 1 --2- 172.21.15.110:0/3153363637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 0x7f5130143ce0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 1 -- 172.21.15.110:0/3153363637 >> 172.21.15.110:0/3153363637 conn(0x7f513006a5f0 msgr2=0x7f513006aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:11.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 1 -- 172.21.15.110:0/3153363637 shutdown_connections 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.055+0000 7f513809b640 1 -- 172.21.15.110:0/3153363637 wait complete. 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.056+0000 7f513809b640 10 monclient: build_initial_monmap 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.056+0000 7f513809b640 10 monclient: monmap: 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.056+0000 7f513809b640 1 librados: starting msgr at 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.056+0000 7f513809b640 1 librados: starting objecter 2024-01-20T11:20:11.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.056+0000 7f513809b640 1 Processor -- start 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.056+0000 7f513809b640 1 -- start start 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.056+0000 7f513809b640 1 librados: setting wanted keys 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.056+0000 7f513809b640 1 librados: calling monclient init 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.056+0000 7f513809b640 10 monclient: init 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f513006b380 0x7f5130148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient(hunting): picked mon.noname-c con 0x7f513006b380 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513006e430 0x7f513014a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient(hunting): picked mon.noname-a con 0x7f513006e430 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 0x7f5130070240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient(hunting): picked mon.noname-b con 0x7f513013d600 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:11.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513014a610 con 0x7f513006e430 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51301d7000 con 0x7f513013d600 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51301d72e0 con 0x7f513006b380 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f513809b640 10 monclient(hunting): authenticate will time out at 1654.408570s 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f5135636640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513006e430 0x7f513014a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f5136638640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 0x7f5130070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.057+0000 7f5135e37640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f513006b380 0x7f5130148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5136638640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 0x7f5130070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:36850/0 (socket says 172.21.15.110:36850) 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5136638640 1 -- 172.21.15.110:0/1860199596 learned_addr learned my addr 172.21.15.110:0/1860199596 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5136638640 10 monclient(hunting): get_auth_request con 0x7f513013d600 auth_method 0 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5136638640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5136638640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5136638640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135636640 10 monclient(hunting): get_auth_request con 0x7f513006e430 auth_method 0 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135636640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135636640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135636640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135e37640 10 monclient(hunting): get_auth_request con 0x7f513006b380 auth_method 0 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135e37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135e37640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135e37640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5136638640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5136638640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5136638640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135636640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135636640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135636640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135e37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135e37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:11.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.058+0000 7f5135e37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 10 monclient(hunting): handle_auth_done global_id 5314 payload 824 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 1 -- 172.21.15.110:0/1860199596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f513006b380 msgr2=0x7f5130148770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f513006b380 0x7f5130148770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 1 -- 172.21.15.110:0/1860199596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513006e430 msgr2=0x7f513014a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513006e430 0x7f513014a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 10 monclient: _finish_hunting 0 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 1 monclient: found mon.noname-b 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.061+0000 7f5136638640 1 -- 172.21.15.110:0/1860199596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5128009a70 con 0x7f513013d600 2024-01-20T11:20:11.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.063+0000 7f5136638640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 0x7f5130070240 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5128009d90 tx=0x7f5128004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.063+0000 7f51267fc640 10 client.?.objecter ms_handle_connect 0x7f513013d600 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.063+0000 7f51267fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 1 -- 172.21.15.110:0/1860199596 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512802cea0 con 0x7f513013d600 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 10 monclient: dump: 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:11.069 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 1 -- 172.21.15.110:0/1860199596 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512803e630 con 0x7f513013d600 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 1 -- 172.21.15.110:0/1860199596 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512803e9e0 con 0x7f513013d600 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51267fc640 10 monclient: dump: 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.070 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f513809b640 5 monclient: authenticate success, global_id 5314 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f513809b640 10 monclient: _renew_subs 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f513809b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5130070780 con 0x7f513013d600 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f51277fe640 4 set_mon_vals no callback set 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f513809b640 10 client.5314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f513809b640 10 monclient: _renew_subs 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f513809b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51301d79e0 con 0x7f513013d600 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.066+0000 7f513809b640 1 librados: init done 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.067+0000 7f513809b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.067+0000 7f513809b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:11.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.067+0000 7f513809b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.067+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5130149e00 con 0x7f513013d600 2024-01-20T11:20:11.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.085+0000 7f51267fc640 1 -- 172.21.15.110:0/1860199596 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f512803b050 con 0x7f513013d600 2024-01-20T11:20:11.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.085+0000 7f51267fc640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f511406c8e0 0x7f511406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.086+0000 7f5135e37640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f511406c8e0 0x7f511406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.086+0000 7f5135e37640 10 monclient: get_auth_request con 0x7f511406c8e0 auth_method 0 2024-01-20T11:20:11.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.086+0000 7f5135e37640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f511406c8e0 0x7f511406eda0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f512c0017f0 tx=0x7f512c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:11.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.086+0000 7f51267fc640 10 client.5314.objecter ms_handle_connect 0x7f511406c8e0 2024-01-20T11:20:11.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.089+0000 7f51267fc640 1 -- 172.21.15.110:0/1860199596 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f512807a310 con 0x7f513013d600 2024-01-20T11:20:11.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.089+0000 7f51267fc640 10 client.5314.objecter ms_dispatch 0x7f5130000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:11.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.089+0000 7f51267fc640 3 client.5314.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:11.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.089+0000 7f51267fc640 3 client.5314.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:11.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.089+0000 7f51267fc640 20 client.5314.objecter dump_active .. 0 homeless 2024-01-20T11:20:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.321+0000 7f51267fc640 1 -- 172.21.15.110:0/1860199596 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5128076650 con 0x7f513013d600 2024-01-20T11:20:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.321+0000 7f51267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.321+0000 7f51267fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:11.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.561+0000 7f513809b640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:11.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.561+0000 7f513809b640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:11.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.561+0000 7f513809b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.561+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5130075490 con 0x7f513013d600 2024-01-20T11:20:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.568+0000 7f51267fc640 1 -- 172.21.15.110:0/1860199596 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f5128079c70 con 0x7f513013d600 2024-01-20T11:20:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.568+0000 7f51267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:11.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.568+0000 7f51267fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:11.569 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:11.569 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":346,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 10 librados: watch_flush enter 2024-01-20T11:20:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 10 librados: watch_flush exit 2024-01-20T11:20:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 20 client.5314.objecter shutdown clearing up homeless session... 2024-01-20T11:20:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 10 client.5314.objecter successfully canceled tick 2024-01-20T11:20:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f511406c8e0 msgr2=0x7f511406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f511406c8e0 0x7f511406eda0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f512c0017f0 tx=0x7f512c009040 comp rx=0 tx=0).stop 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 10 monclient: shutdown 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 msgr2=0x7f5130070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 0x7f5130070240 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5128009d90 tx=0x7f5128004a40 comp rx=0 tx=0).stop 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 shutdown_connections 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f511406c8e0 0x7f511406eda0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f513013d600 0x7f5130070240 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513006e430 0x7f513014a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 --2- 172.21.15.110:0/1860199596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f513006b380 0x7f5130148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.569+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 >> 172.21.15.110:0/1860199596 conn(0x7f513006a5f0 msgr2=0x7f513013ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.570+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 shutdown_connections 2024-01-20T11:20:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.570+0000 7f513809b640 1 -- 172.21.15.110:0/1860199596 wait complete. 2024-01-20T11:20:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.570+0000 7f513809b640 1 librados: shutdown 2024-01-20T11:20:11.582 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:11.582 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:11.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.679+0000 7f4c3231d640 10 monclient: get_monmap_and_config 2024-01-20T11:20:11.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.679+0000 7f4c3231d640 10 monclient: build_initial_monmap 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.679+0000 7f4c3231d640 10 monclient: monmap: 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:11.681442+0000 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:11.681442+0000 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:11.681 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.680+0000 7f4c3231d640 1 Processor -- start 2024-01-20T11:20:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.680+0000 7f4c3231d640 1 -- start start 2024-01-20T11:20:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.680+0000 7f4c3231d640 10 monclient: init 2024-01-20T11:20:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:11.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4c2c06f920 0x7f4c2c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c06f920 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c2c13d230 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 0x7f4c2c13d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c144060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0715b0 con 0x7f4c2c13d230 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c071890 con 0x7f4c2c06f920 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c071b70 con 0x7f4c2c144060 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:11.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c3231d640 10 monclient(hunting): authenticate will time out at 1655.032535s 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.681+0000 7f4c308ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 0x7f4c2c13d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c2b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4c2c13d230 0x7f4c2c143950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56318/0 (socket says 172.21.15.110:56318) 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c2b7fe640 1 -- 172.21.15.110:0/3909614985 learned_addr learned my addr 172.21.15.110:0/3909614985 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c2bfff640 1 --2- 172.21.15.110:0/3909614985 >> v2:172.21.15.134:3300/0 conn(0x7f4c2c06f920 0x7f4c2c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c308ba640 10 monclient(hunting): get_auth_request con 0x7f4c2c144060 auth_method 0 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c308ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c308ba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c308ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c2bfff640 10 monclient(hunting): get_auth_request con 0x7f4c2c06f920 auth_method 0 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c2bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c308ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c308ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:11.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.682+0000 7f4c308ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:11.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.683+0000 7f4c2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:11.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.683+0000 7f4c2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:11.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.683+0000 7f4c2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:11.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2affd640 1 -- 172.21.15.110:0/3909614985 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241197127 0 0) 0x7f4c2c0715b0 con 0x7f4c2c13d230 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2affd640 10 monclient(hunting): _negotiate 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2affd640 10 monclient(hunting): my global_id is 5241 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2affd640 1 -- 172.21.15.110:0/3909614985 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c14003520 con 0x7f4c2c13d230 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2bfff640 10 monclient(hunting): handle_auth_done global_id 5317 payload 274 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2bfff640 1 -- 172.21.15.110:0/3909614985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 msgr2=0x7f4c2c13d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2bfff640 1 --2- 172.21.15.110:0/3909614985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 0x7f4c2c13d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2bfff640 1 -- 172.21.15.110:0/3909614985 >> v1:172.21.15.110:6789/0 conn(0x7f4c2c13d230 legacy=0x7f4c2c143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:11.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2bfff640 1 monclient: found mon.noname-c 2024-01-20T11:20:11.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:11.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.684+0000 7f4c2bfff640 1 -- 172.21.15.110:0/3909614985 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c143430 con 0x7f4c2c06f920 2024-01-20T11:20:11.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.686+0000 7f4c2bfff640 1 --2- 172.21.15.110:0/3909614985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c06f920 0x7f4c2c06fd00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0097a0 tx=0x7f4c1c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4053ee318b612a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 1 -- 172.21.15.110:0/3909614985 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c02e070 con 0x7f4c2c06f920 2024-01-20T11:20:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 10 monclient: dump: 2024-01-20T11:20:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 10 monclient: _finish_auth 0 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c3231d640 5 monclient: authenticate success, global_id 5317 2024-01-20T11:20:11.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c3231d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:11.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 1 -- 172.21.15.110:0/3909614985 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1c0315a0 con 0x7f4c2c06f920 2024-01-20T11:20:11.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:11.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c3231d640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:11.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c3231d640 4 set_mon_vals no callback set 2024-01-20T11:20:11.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 1 -- 172.21.15.110:0/3909614985 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c031930 con 0x7f4c2c06f920 2024-01-20T11:20:11.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:11.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:11.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c2affd640 10 monclient: dump: 2024-01-20T11:20:11.695 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:11.695 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.695 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:11.697 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:11.697 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:11.697 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c3231d640 10 monclient: shutdown 2024-01-20T11:20:11.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c3231d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:11.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c3231d640 1 -- 172.21.15.110:0/3909614985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c06f920 msgr2=0x7f4c2c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.690+0000 7f4c3231d640 1 --2- 172.21.15.110:0/3909614985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c06f920 0x7f4c2c06fd00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0097a0 tx=0x7f4c1c02b840 comp rx=0 tx=0).stop 2024-01-20T11:20:11.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.691+0000 7f4c3231d640 1 -- 172.21.15.110:0/3909614985 shutdown_connections 2024-01-20T11:20:11.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.691+0000 7f4c3231d640 1 --2- 172.21.15.110:0/3909614985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 0x7f4c2c13d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.691+0000 7f4c3231d640 1 --2- 172.21.15.110:0/3909614985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c06f920 0x7f4c2c06fd00 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.691+0000 7f4c3231d640 1 -- 172.21.15.110:0/3909614985 >> 172.21.15.110:0/3909614985 conn(0x7f4c2c066d60 msgr2=0x7f4c2c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:11.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.691+0000 7f4c3231d640 1 -- 172.21.15.110:0/3909614985 shutdown_connections 2024-01-20T11:20:11.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.691+0000 7f4c3231d640 1 -- 172.21.15.110:0/3909614985 wait complete. 2024-01-20T11:20:11.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.691+0000 7f4c3231d640 10 monclient: build_initial_monmap 2024-01-20T11:20:11.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.691+0000 7f4c3231d640 10 monclient: monmap: 2024-01-20T11:20:11.699 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:11.699 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:11.700 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:11.700 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:11.700 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:11.700 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.700 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:11.700 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:11.701 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:11.701 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 librados: starting msgr at 2024-01-20T11:20:11.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 librados: starting objecter 2024-01-20T11:20:11.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 Processor -- start 2024-01-20T11:20:11.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 -- start start 2024-01-20T11:20:11.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 librados: setting wanted keys 2024-01-20T11:20:11.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 librados: calling monclient init 2024-01-20T11:20:11.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient: init 2024-01-20T11:20:11.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:11.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06f920 0x7f4c2c137ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c06f920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c13d230 0x7f4c2c13dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c13d230 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 0x7f4c2c13e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c144060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c138da0 con 0x7f4c2c06f920 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c138ff0 con 0x7f4c2c13d230 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:11.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c1392d0 con 0x7f4c2c144060 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.692+0000 7f4c3231d640 10 monclient(hunting): authenticate will time out at 1655.044051s 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c308ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 0x7f4c2c13e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06f920 0x7f4c2c137ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c13d230 0x7f4c2c13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c13d230 0x7f4c2c13dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:36860/0 (socket says 172.21.15.110:36860) 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2b7fe640 1 -- 172.21.15.110:0/665225673 learned_addr learned my addr 172.21.15.110:0/665225673 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2b7fe640 10 monclient(hunting): get_auth_request con 0x7f4c2c13d230 auth_method 0 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c308ba640 10 monclient(hunting): get_auth_request con 0x7f4c2c144060 auth_method 0 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c308ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:11.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c308ba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c308ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2bfff640 10 monclient(hunting): get_auth_request con 0x7f4c2c06f920 auth_method 0 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.693+0000 7f4c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.694+0000 7f4c2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.694+0000 7f4c2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.694+0000 7f4c2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.694+0000 7f4c308ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.694+0000 7f4c308ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.694+0000 7f4c308ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.694+0000 7f4c2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.694+0000 7f4c2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.694+0000 7f4c2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:11.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.696+0000 7f4c2b7fe640 10 monclient(hunting): handle_auth_done global_id 5320 payload 824 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.696+0000 7f4c2b7fe640 1 -- 172.21.15.110:0/665225673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 msgr2=0x7f4c2c13e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.696+0000 7f4c2b7fe640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 0x7f4c2c13e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.696+0000 7f4c2b7fe640 1 -- 172.21.15.110:0/665225673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06f920 msgr2=0x7f4c2c137ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.696+0000 7f4c2b7fe640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06f920 0x7f4c2c137ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.696+0000 7f4c2b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.696+0000 7f4c2b7fe640 1 monclient: found mon.noname-b 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.697+0000 7f4c2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.697+0000 7f4c2b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.697+0000 7f4c2b7fe640 1 -- 172.21.15.110:0/665225673 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c009400 con 0x7f4c2c13d230 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.699+0000 7f4c2b7fe640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c13d230 0x7f4c2c13dc70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000c700 tx=0x7f4c2000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.699+0000 7f4c28ff9640 10 client.?.objecter ms_handle_connect 0x7f4c2c13d230 2024-01-20T11:20:11.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.699+0000 7f4c28ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:11.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 1 -- 172.21.15.110:0/665225673 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20007270 con 0x7f4c2c13d230 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 10 monclient: dump: 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 10 monclient: _finish_auth 0 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 1 -- 172.21.15.110:0/665225673 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c200073f0 con 0x7f4c2c13d230 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c3231d640 5 monclient: authenticate success, global_id 5320 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c3231d640 10 monclient: _renew_subs 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c3231d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c1409a0 con 0x7f4c2c13d230 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 1 -- 172.21.15.110:0/665225673 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20001d40 con 0x7f4c2c13d230 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c28ff9640 10 monclient: dump: 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c2a7fc640 4 set_mon_vals no callback set 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c3231d640 10 client.5320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c3231d640 10 monclient: _renew_subs 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c3231d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c137ff0 con 0x7f4c2c13d230 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.704+0000 7f4c3231d640 1 librados: init done 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.705+0000 7f4c3231d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.705+0000 7f4c3231d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.705+0000 7f4c3231d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.706+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c1407a0 con 0x7f4c2c13d230 2024-01-20T11:20:11.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.944+0000 7f4c28ff9640 1 -- 172.21.15.110:0/665225673 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4c2000a5e0 con 0x7f4c2c13d230 2024-01-20T11:20:11.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.944+0000 7f4c28ff9640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c1806c8e0 0x7f4c1806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:11.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.945+0000 7f4c2bfff640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c1806c8e0 0x7f4c1806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:11.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.945+0000 7f4c2bfff640 10 monclient: get_auth_request con 0x7f4c1806c8e0 auth_method 0 2024-01-20T11:20:11.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.945+0000 7f4c2bfff640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c1806c8e0 0x7f4c1806eda0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c036000 tx=0x7f4c1c03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:11.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.945+0000 7f4c28ff9640 10 client.5320.objecter ms_handle_connect 0x7f4c1806c8e0 2024-01-20T11:20:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.948+0000 7f4c28ff9640 1 -- 172.21.15.110:0/665225673 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4c20007d50 con 0x7f4c2c13d230 2024-01-20T11:20:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.948+0000 7f4c28ff9640 10 client.5320.objecter ms_dispatch 0x7f4c2c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.948+0000 7f4c28ff9640 3 client.5320.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.948+0000 7f4c28ff9640 3 client.5320.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.948+0000 7f4c28ff9640 20 client.5320.objecter dump_active .. 0 homeless 2024-01-20T11:20:11.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.958+0000 7f4c28ff9640 1 -- 172.21.15.110:0/665225673 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4c20053f00 con 0x7f4c2c13d230 2024-01-20T11:20:11.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.958+0000 7f4c28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:11.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:11.958+0000 7f4c28ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.150+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c2c0b1720 con 0x7f4c1806c8e0 2024-01-20T11:20:12.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.154+0000 7f4c28ff9640 1 -- 172.21.15.110:0/665225673 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64032 (secure 0 0 0) 0x7f4c2c0b1720 con 0x7f4c1806c8e0 2024-01-20T11:20:12.155 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:12.155 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 10 librados: watch_flush enter 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 10 librados: watch_flush exit 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 20 client.5320.objecter shutdown clearing up homeless session... 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 10 client.5320.objecter successfully canceled tick 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c1806c8e0 msgr2=0x7f4c1806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c1806c8e0 0x7f4c1806eda0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c036000 tx=0x7f4c1c03d040 comp rx=0 tx=0).stop 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 10 monclient: shutdown 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c13d230 msgr2=0x7f4c2c13dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c13d230 0x7f4c2c13dc70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000c700 tx=0x7f4c2000cbc0 comp rx=0 tx=0).stop 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 shutdown_connections 2024-01-20T11:20:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c1806c8e0 0x7f4c1806eda0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c2c144060 0x7f4c2c13e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c2c13d230 0x7f4c2c13dc70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 --2- 172.21.15.110:0/665225673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c2c06f920 0x7f4c2c137ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 >> 172.21.15.110:0/665225673 conn(0x7f4c2c066d60 msgr2=0x7f4c2c073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 shutdown_connections 2024-01-20T11:20:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 -- 172.21.15.110:0/665225673 wait complete. 2024-01-20T11:20:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:12.155+0000 7f4c3231d640 1 librados: shutdown 2024-01-20T11:20:12.167 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-01-20T11:20:10.767384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002216"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280577,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.1180000000000003},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.3900000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.7639999999999993},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.7230000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":12.212999999999999},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.022}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.75},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.237}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.975},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":14.558999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.329000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.318000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313279,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":4.524},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.7780000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.2110000000000003},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.4030000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.762},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.512}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.4180000000000001},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.477}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.127000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":14.778}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.749000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.212}]}]},{"osd":3,"up_from":21,"seq":90194313280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.693},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.23}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.4710000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.086}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.7530000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":11.305999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.945}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":15.090999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.782}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444094,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.9730000000000008},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":4.3230000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.1130000000000004},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.5709999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":10.784000000000001},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.3770000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.925000000000001},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.428000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":15.849},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.079000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.752000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":11.493}]}]},{"osd":1,"up_from":17,"seq":73014444095,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0970000000000004},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":5.1769999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.4290000000000003},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.2080000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":11.948},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":13.890000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":13.199},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":9.5760000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.8620000000000001},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":10.722}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.417999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.635}]}]},{"osd":0,"up_from":17,"seq":73014444095,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":5.0860000000000003},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":12.641}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.6929999999999996},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":9.8539999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":13.460000000000001},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":17.945}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.281000000000001},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.2789999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":10.461},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.646000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.696000000000002},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":9.2140000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509503,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":6.6470000000000002},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.3070000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.0730000000000004},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":12.166}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":8.2910000000000004},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":10.491}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.763},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.335000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.504},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.615}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.5210000000000008},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.228999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:12.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~75s 2024-01-20T11:20:15.170 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.274+0000 7f01d6aec640 10 monclient: get_monmap_and_config 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.274+0000 7f01d6aec640 10 monclient: build_initial_monmap 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.274+0000 7f01d6aec640 10 monclient: monmap: 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:15.275764+0000 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:15.275764+0000 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:15.275 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:15.276 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.274+0000 7f01d6aec640 1 Processor -- start 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.274+0000 7f01d6aec640 1 -- start start 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.274+0000 7f01d6aec640 10 monclient: init 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01d00b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01d00b1ed0 0x7f01d00b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 10 monclient(hunting): picked mon.noname-a con 0x7f01d00b1ed0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01d0145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d00aade0 con 0x7f01d00b4d10 2024-01-20T11:20:15.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d0145460 con 0x7f01d0145980 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d00b50f0 con 0x7f01d00b1ed0 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.275+0000 7f01d6aec640 10 monclient(hunting): authenticate will time out at 1658.626708s 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.276+0000 7f01cffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01d00b1ed0 0x7f01d00b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.276+0000 7f01d5089640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f01d0145980 0x7f01d0145e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:48790/0 (socket says 172.21.15.110:48790) 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.276+0000 7f01d5089640 1 -- 172.21.15.110:0/1215698807 learned_addr learned my addr 172.21.15.110:0/1215698807 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.276+0000 7f01cffff640 10 monclient(hunting): get_auth_request con 0x7f01d00b1ed0 auth_method 0 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.276+0000 7f01cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.276+0000 7f01cffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.276+0000 7f01cffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.277+0000 7f01cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:15.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.277+0000 7f01cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:15.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.277+0000 7f01cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:15.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.278+0000 7f01cf7fe640 1 -- 172.21.15.110:0/1215698807 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950478383 0 0) 0x7f01d0145460 con 0x7f01d0145980 2024-01-20T11:20:15.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.278+0000 7f01cf7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:20:15.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.278+0000 7f01cf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.278+0000 7f01cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.278+0000 7f01cf7fe640 10 monclient(hunting): my global_id is 5323 2024-01-20T11:20:15.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.278+0000 7f01cf7fe640 1 -- 172.21.15.110:0/1215698807 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c0003520 con 0x7f01d0145980 2024-01-20T11:20:15.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.279+0000 7f01cffff640 10 monclient(hunting): handle_auth_done global_id 5247 payload 274 2024-01-20T11:20:15.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.279+0000 7f01cffff640 1 -- 172.21.15.110:0/1215698807 >> v1:172.21.15.134:6789/0 conn(0x7f01d0145980 legacy=0x7f01d0145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:15.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.279+0000 7f01cffff640 1 -- 172.21.15.110:0/1215698807 >> v1:172.21.15.110:6789/0 conn(0x7f01d00b4d10 legacy=0x7f01d00b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:15.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.279+0000 7f01cffff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:15.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.279+0000 7f01cffff640 1 monclient: found mon.noname-a 2024-01-20T11:20:15.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.279+0000 7f01cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:15.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.279+0000 7f01cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:15.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.279+0000 7f01cffff640 1 -- 172.21.15.110:0/1215698807 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d00b2660 con 0x7f01d00b1ed0 2024-01-20T11:20:15.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.282+0000 7f01cffff640 1 --2- 172.21.15.110:0/1215698807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 0x7f01d00b3da0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0097a0 tx=0x7f01bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45cf2d67322022fa server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:15.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 1 -- 172.21.15.110:0/1215698807 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc02e070 con 0x7f01d00b1ed0 2024-01-20T11:20:15.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:15.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:15.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 10 monclient: dump: 2024-01-20T11:20:15.291 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:15.291 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 1 -- 172.21.15.110:0/1215698807 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01bc0315a0 con 0x7f01d00b1ed0 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:15.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 1 -- 172.21.15.110:0/1215698807 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc031930 con 0x7f01d00b1ed0 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01cf7fe640 10 monclient: dump: 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01d6aec640 5 monclient: authenticate success, global_id 5247 2024-01-20T11:20:15.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01d6aec640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:15.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.285+0000 7f01d6aec640 4 set_mon_vals no callback set 2024-01-20T11:20:15.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.286+0000 7f01d6aec640 10 monclient: shutdown 2024-01-20T11:20:15.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.286+0000 7f01d6aec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:15.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.286+0000 7f01d6aec640 1 -- 172.21.15.110:0/1215698807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 msgr2=0x7f01d00b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:15.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.286+0000 7f01d6aec640 1 --2- 172.21.15.110:0/1215698807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 0x7f01d00b3da0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0097a0 tx=0x7f01bc02b840 comp rx=0 tx=0).stop 2024-01-20T11:20:15.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.286+0000 7f01d6aec640 1 -- 172.21.15.110:0/1215698807 shutdown_connections 2024-01-20T11:20:15.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.286+0000 7f01d6aec640 1 --2- 172.21.15.110:0/1215698807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 0x7f01d00b3da0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.286+0000 7f01d6aec640 1 -- 172.21.15.110:0/1215698807 >> 172.21.15.110:0/1215698807 conn(0x7f01d0066590 msgr2=0x7f01d00669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:15.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.286+0000 7f01d6aec640 1 -- 172.21.15.110:0/1215698807 shutdown_connections 2024-01-20T11:20:15.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.286+0000 7f01d6aec640 1 -- 172.21.15.110:0/1215698807 wait complete. 2024-01-20T11:20:15.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.287+0000 7f01d6aec640 10 monclient: build_initial_monmap 2024-01-20T11:20:15.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.287+0000 7f01d6aec640 10 monclient: monmap: 2024-01-20T11:20:15.295 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:15.296 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:15.296 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:15.296 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:15.296 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:15.296 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.296 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:15.297 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:15.297 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:15.297 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.287+0000 7f01d6aec640 1 librados: starting msgr at 2024-01-20T11:20:15.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.287+0000 7f01d6aec640 1 librados: starting objecter 2024-01-20T11:20:15.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.287+0000 7f01d6aec640 1 Processor -- start 2024-01-20T11:20:15.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.287+0000 7f01d6aec640 1 -- start start 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.287+0000 7f01d6aec640 1 librados: setting wanted keys 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.287+0000 7f01d6aec640 1 librados: calling monclient init 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.287+0000 7f01d6aec640 10 monclient: init 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 0x7f01d013d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient(hunting): picked mon.noname-a con 0x7f01d00b1ed0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01d00b4d10 0x7f01d013dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient(hunting): picked mon.noname-c con 0x7f01d00b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01d0145980 0x7f01d013e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient(hunting): picked mon.noname-b con 0x7f01d0145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:15.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d013d7e0 con 0x7f01d00b1ed0 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d013c420 con 0x7f01d0145980 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d013bf40 con 0x7f01d00b4d10 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d6aec640 10 monclient(hunting): authenticate will time out at 1658.639555s 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.288+0000 7f01d4888640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 0x7f01d013d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d5089640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01d0145980 0x7f01d013e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01cffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01d00b4d10 0x7f01d013dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d4888640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 0x7f01d013d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:49128/0 (socket says 172.21.15.110:49128) 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d4888640 1 -- 172.21.15.110:0/291531119 learned_addr learned my addr 172.21.15.110:0/291531119 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d4888640 10 monclient(hunting): get_auth_request con 0x7f01d00b1ed0 auth_method 0 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d4888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d4888640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d4888640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d5089640 10 monclient(hunting): get_auth_request con 0x7f01d0145980 auth_method 0 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d5089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d5089640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d5089640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01cffff640 10 monclient(hunting): get_auth_request con 0x7f01d00b4d10 auth_method 0 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01cffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01cffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d4888640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d4888640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d4888640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d5089640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d5089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:15.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.289+0000 7f01d5089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:15.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.290+0000 7f01cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:15.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.290+0000 7f01cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:15.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.290+0000 7f01cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:15.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 10 monclient(hunting): handle_auth_done global_id 5326 payload 824 2024-01-20T11:20:15.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 1 -- 172.21.15.110:0/291531119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01d00b4d10 msgr2=0x7f01d013dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01d00b4d10 0x7f01d013dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 1 -- 172.21.15.110:0/291531119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 msgr2=0x7f01d013d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 0x7f01d013d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 10 monclient: _finish_hunting 0 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 1 monclient: found mon.noname-b 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.292+0000 7f01d5089640 1 -- 172.21.15.110:0/291531119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc009400 con 0x7f01d0145980 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.294+0000 7f01d5089640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01d0145980 0x7f01d013e150 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f01c800caa0 tx=0x7f01c800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.295+0000 7f01cd7fa640 10 client.?.objecter ms_handle_connect 0x7f01d0145980 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.295+0000 7f01cd7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 1 -- 172.21.15.110:0/291531119 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c800aae0 con 0x7f01d0145980 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 10 monclient: dump: 2024-01-20T11:20:15.303 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 1 -- 172.21.15.110:0/291531119 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c800ac40 con 0x7f01d0145980 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:15.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 1 -- 172.21.15.110:0/291531119 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8012400 con 0x7f01d0145980 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01cd7fa640 10 monclient: dump: 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01d6aec640 5 monclient: authenticate success, global_id 5326 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01d6aec640 10 monclient: _renew_subs 2024-01-20T11:20:15.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01d6aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01ceffd640 4 set_mon_vals no callback set 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d00ab780 con 0x7f01d0145980 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01d6aec640 10 client.5326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01d6aec640 10 monclient: _renew_subs 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01d6aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d013e690 con 0x7f01d0145980 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.298+0000 7f01d6aec640 1 librados: init done 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.299+0000 7f01d6aec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.299+0000 7f01d6aec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.299+0000 7f01d6aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:15.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.299+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d013e8e0 con 0x7f01d0145980 2024-01-20T11:20:15.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.534+0000 7f01cd7fa640 1 -- 172.21.15.110:0/291531119 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f01c8007070 con 0x7f01d0145980 2024-01-20T11:20:15.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.534+0000 7f01cd7fa640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f01b406c860 0x7f01b406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.535+0000 7f01d4888640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f01b406c860 0x7f01b406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.535+0000 7f01d4888640 10 monclient: get_auth_request con 0x7f01b406c860 auth_method 0 2024-01-20T11:20:15.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.535+0000 7f01d4888640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f01b406c860 0x7f01b406ed20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005020 tx=0x7f01c4007ef0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:15.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.535+0000 7f01cd7fa640 10 client.5326.objecter ms_handle_connect 0x7f01b406c860 2024-01-20T11:20:15.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.540+0000 7f01cd7fa640 1 -- 172.21.15.110:0/291531119 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f01c80584a0 con 0x7f01d0145980 2024-01-20T11:20:15.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.540+0000 7f01cd7fa640 10 client.5326.objecter ms_dispatch 0x7f01d0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:15.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.540+0000 7f01cd7fa640 3 client.5326.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:15.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.540+0000 7f01cd7fa640 3 client.5326.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:15.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.541+0000 7f01cd7fa640 20 client.5326.objecter dump_active .. 0 homeless 2024-01-20T11:20:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.551+0000 7f01cd7fa640 1 -- 172.21.15.110:0/291531119 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f01c80547e0 con 0x7f01d0145980 2024-01-20T11:20:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.551+0000 7f01cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.551+0000 7f01cd7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:15.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.715+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01d013c100 con 0x7f01b406c860 2024-01-20T11:20:15.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.718+0000 7f01cd7fa640 1 -- 172.21.15.110:0/291531119 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64040 (secure 0 0 0) 0x7f01d013c100 con 0x7f01b406c860 2024-01-20T11:20:15.719 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:15.720 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:15.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 10 librados: watch_flush enter 2024-01-20T11:20:15.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 10 librados: watch_flush exit 2024-01-20T11:20:15.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 20 client.5326.objecter shutdown clearing up homeless session... 2024-01-20T11:20:15.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 10 client.5326.objecter successfully canceled tick 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f01b406c860 msgr2=0x7f01b406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f01b406c860 0x7f01b406ed20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005020 tx=0x7f01c4007ef0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 10 monclient: shutdown 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01d0145980 msgr2=0x7f01d013e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01d0145980 0x7f01d013e150 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f01c800caa0 tx=0x7f01c800cf60 comp rx=0 tx=0).stop 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 shutdown_connections 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f01b406c860 0x7f01b406ed20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01d0145980 0x7f01d013e150 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01d00b4d10 0x7f01d013dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 --2- 172.21.15.110:0/291531119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01d00b1ed0 0x7f01d013d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 >> 172.21.15.110:0/291531119 conn(0x7f01d0066590 msgr2=0x7f01d0143520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 shutdown_connections 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 -- 172.21.15.110:0/291531119 wait complete. 2024-01-20T11:20:15.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.720+0000 7f01d6aec640 1 librados: shutdown 2024-01-20T11:20:15.732 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-01-20T11:20:14.768345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002454"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.3630000000000004},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.2359999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.894},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.9770000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.130000000000001},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.538}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.25},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":14.968999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.782},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":12.154999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.609},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.643999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313280,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.896000000000001},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.848000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.5919999999999996},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.8209999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.8540000000000001},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":5.7720000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.5269999999999992},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.000999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.2279999999999998},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":13.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.239000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.153}]}]},{"osd":3,"up_from":21,"seq":90194313281,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.693},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.23}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.4710000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.086}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.7530000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":11.305999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.945}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":15.090999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.782}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444095,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.3900000000000006},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.2439999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.5579999999999998},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.9619999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.574},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.254}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.220000000000001},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.847}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.8170000000000002},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":15.689}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.798999999999999},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":13.42}]}]},{"osd":1,"up_from":17,"seq":73014444096,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.2910000000000004},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.2779999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.488},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.5359999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":13.526999999999999},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.8689999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":15.321},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":14.667}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.7880000000000003},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":11.061999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.840999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.741}]}]},{"osd":0,"up_from":17,"seq":73014444096,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.8090000000000002},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":5.6879999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.1390000000000002},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.3879999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":9.3729999999999993},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.513999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.948},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.84}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.156000000000001},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.337}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.085000000000001},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":12.539}]}]},{"osd":6,"up_from":15,"seq":64424509504,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":11.565},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.5939999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.3940000000000001},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":12.425000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.292999999999999},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.4619999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":6.4119999999999999},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.65}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.2759999999999998},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.032}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.007999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.952}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:15.733 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:15.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.830+0000 7fe9d2007640 10 monclient: get_monmap_and_config 2024-01-20T11:20:15.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.830+0000 7fe9d2007640 10 monclient: build_initial_monmap 2024-01-20T11:20:15.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.830+0000 7fe9d2007640 10 monclient: monmap: 2024-01-20T11:20:15.831 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:15.831 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.831 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:15.832329+0000 2024-01-20T11:20:15.831 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:15.832329+0000 2024-01-20T11:20:15.831 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:15.832 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.832 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:15.832 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:15.832 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:15.832 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:15.832 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:15.832 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.831+0000 7fe9d2007640 1 Processor -- start 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.831+0000 7fe9d2007640 1 -- start start 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.831+0000 7fe9d2007640 10 monclient: init 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.831+0000 7fe9d2007640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe9cc0b0c00 0x7fe9cc0accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 10 monclient(hunting): picked mon.noname-c con 0x7fe9cc0b0c00 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 10 monclient(hunting): picked mon.noname-b con 0x7fe9cc13d0e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:15.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9cc143200 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc06dcc0 con 0x7fe9cc143200 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9cc0ad450 con 0x7fe9cc0b0c00 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc0ad8e0 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9d2007640 10 monclient(hunting): authenticate will time out at 1659.183256s 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9caffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.832+0000 7fe9cb7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe9cc0b0c00 0x7fe9cc0accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9cb7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe9cc0b0c00 0x7fe9cc0accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:36868/0 (socket says 172.21.15.110:36868) 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9caffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43736/0 (socket says 172.21.15.110:43736) 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9cb7fe640 1 -- 172.21.15.110:0/2341133485 learned_addr learned my addr 172.21.15.110:0/2341133485 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9cb7fe640 10 monclient(hunting): get_auth_request con 0x7fe9cc0b0c00 auth_method 0 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9caffd640 10 monclient(hunting): get_auth_request con 0x7fe9cc13d0e0 auth_method 0 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9caffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9caffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:15.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.833+0000 7fe9cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:15.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.835+0000 7fe9caffd640 10 monclient(hunting): handle_auth_done global_id 5021 payload 274 2024-01-20T11:20:15.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.835+0000 7fe9caffd640 1 -- 172.21.15.110:0/2341133485 >> v2:172.21.15.134:3300/0 conn(0x7fe9cc0b0c00 msgr2=0x7fe9cc0accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:15.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.835+0000 7fe9caffd640 1 --2- 172.21.15.110:0/2341133485 >> v2:172.21.15.134:3300/0 conn(0x7fe9cc0b0c00 0x7fe9cc0accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.835+0000 7fe9caffd640 1 -- 172.21.15.110:0/2341133485 >> v1:172.21.15.134:6789/0 conn(0x7fe9cc143200 legacy=0x7fe9cc143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:15.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.835+0000 7fe9caffd640 10 monclient: _finish_hunting 0 2024-01-20T11:20:15.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.835+0000 7fe9caffd640 1 monclient: found mon.noname-b 2024-01-20T11:20:15.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.835+0000 7fe9caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:15.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.835+0000 7fe9caffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:15.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.835+0000 7fe9caffd640 1 -- 172.21.15.110:0/2341133485 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc143da0 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.836+0000 7fe9caffd640 1 --2- 172.21.15.110:0/2341133485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13d520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00097a0 tx=0x7fe9c002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc6fc53f2a415156 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 1 -- 172.21.15.110:0/2341133485 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c002f050 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 10 monclient: dump: 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.842 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 1 -- 172.21.15.110:0/2341133485 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c0031700 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 1 -- 172.21.15.110:0/2341133485 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0031a90 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9d2007640 5 monclient: authenticate success, global_id 5021 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9d2007640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9d2007640 4 set_mon_vals no callback set 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.840+0000 7fe9ca7fc640 10 monclient: dump: 2024-01-20T11:20:15.843 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 10 monclient: shutdown 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 1 -- 172.21.15.110:0/2341133485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 msgr2=0x7fe9cc13d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 1 --2- 172.21.15.110:0/2341133485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13d520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00097a0 tx=0x7fe9c002b7a0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 1 -- 172.21.15.110:0/2341133485 shutdown_connections 2024-01-20T11:20:15.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 1 --2- 172.21.15.110:0/2341133485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13d520 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 1 --2- 172.21.15.110:0/2341133485 >> v2:172.21.15.134:3300/0 conn(0x7fe9cc0b0c00 0x7fe9cc0accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 1 -- 172.21.15.110:0/2341133485 >> 172.21.15.110:0/2341133485 conn(0x7fe9cc0b3e60 msgr2=0x7fe9cc0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:15.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 1 -- 172.21.15.110:0/2341133485 shutdown_connections 2024-01-20T11:20:15.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.841+0000 7fe9d2007640 1 -- 172.21.15.110:0/2341133485 wait complete. 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.842+0000 7fe9d2007640 10 monclient: build_initial_monmap 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.842+0000 7fe9d2007640 10 monclient: monmap: 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:15.848 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.842+0000 7fe9d2007640 1 librados: starting msgr at 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.842+0000 7fe9d2007640 1 librados: starting objecter 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.842+0000 7fe9d2007640 1 Processor -- start 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.842+0000 7fe9d2007640 1 -- start start 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.842+0000 7fe9d2007640 1 librados: setting wanted keys 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.842+0000 7fe9d2007640 1 librados: calling monclient init 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.842+0000 7fe9d2007640 10 monclient: init 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9cc0b0c00 0x7fe9cc142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient(hunting): picked mon.noname-a con 0x7fe9cc0b0c00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient(hunting): picked mon.noname-c con 0x7fe9cc13d0e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9cc143200 0x7fe9cc13fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient(hunting): picked mon.noname-b con 0x7fe9cc143200 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc13f810 con 0x7fe9cc0b0c00 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc13ea30 con 0x7fe9cc143200 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc13eca0 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9d2007640 10 monclient(hunting): authenticate will time out at 1659.194402s 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9caffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9cb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9cc0b0c00 0x7fe9cc142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9cbfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9cc143200 0x7fe9cc13fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.843+0000 7fe9caffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43740/0 (socket says 172.21.15.110:43740) 2024-01-20T11:20:15.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9caffd640 1 -- 172.21.15.110:0/3701822320 learned_addr learned my addr 172.21.15.110:0/3701822320 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9caffd640 10 monclient(hunting): get_auth_request con 0x7fe9cc13d0e0 auth_method 0 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9caffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9caffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cb7fe640 10 monclient(hunting): get_auth_request con 0x7fe9cc0b0c00 auth_method 0 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cbfff640 10 monclient(hunting): get_auth_request con 0x7fe9cc143200 auth_method 0 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.844+0000 7fe9cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 10 monclient(hunting): handle_auth_done global_id 5024 payload 824 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 1 -- 172.21.15.110:0/3701822320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9cc143200 msgr2=0x7fe9cc13fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9cc143200 0x7fe9cc13fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 1 -- 172.21.15.110:0/3701822320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9cc0b0c00 msgr2=0x7fe9cc142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9cc0b0c00 0x7fe9cc142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 10 monclient: _finish_hunting 0 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 1 monclient: found mon.noname-c 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:20:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.847+0000 7fe9caffd640 1 -- 172.21.15.110:0/3701822320 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0009400 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.849+0000 7fe9caffd640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13db00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009890 tx=0x7fe9c00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.849+0000 7fe9b3fff640 10 client.?.objecter ms_handle_connect 0x7fe9cc13d0e0 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.849+0000 7fe9b3fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 1 -- 172.21.15.110:0/3701822320 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0033be0 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 10 monclient: dump: 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:15.854 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 10 monclient: _finish_auth 0 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 1 -- 172.21.15.110:0/3701822320 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c0033d60 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 1 -- 172.21.15.110:0/3701822320 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c003d650 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9b3fff640 10 monclient: dump: 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:15.855 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9d2007640 5 monclient: authenticate success, global_id 5024 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9d2007640 10 monclient: _renew_subs 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9d2007640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc13f480 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9c9ffb640 4 set_mon_vals no callback set 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9d2007640 10 client.5024.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9d2007640 10 monclient: _renew_subs 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9d2007640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc1403c0 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.852+0000 7fe9d2007640 1 librados: init done 2024-01-20T11:20:15.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.854+0000 7fe9d2007640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:15.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.854+0000 7fe9d2007640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:15.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.854+0000 7fe9d2007640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:15.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.854+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc13f9c0 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.871+0000 7fe9b3fff640 1 -- 172.21.15.110:0/3701822320 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe9c003a070 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.871+0000 7fe9b3fff640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe9b806c860 0x7fe9b806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:15.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.872+0000 7fe9cb7fe640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe9b806c860 0x7fe9b806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:15.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.872+0000 7fe9cb7fe640 10 monclient: get_auth_request con 0x7fe9b806c860 auth_method 0 2024-01-20T11:20:15.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.872+0000 7fe9b3fff640 10 client.5024.objecter ms_handle_connect 0x7fe9b806c860 2024-01-20T11:20:15.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.872+0000 7fe9cb7fe640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe9b806c860 0x7fe9b806ed20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009bd0 tx=0x7fe9bc009240 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:15.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.874+0000 7fe9b3fff640 1 -- 172.21.15.110:0/3701822320 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe9c007a1b0 con 0x7fe9cc13d0e0 2024-01-20T11:20:15.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.874+0000 7fe9b3fff640 10 client.5024.objecter ms_dispatch 0x7fe9cc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:15.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.874+0000 7fe9b3fff640 3 client.5024.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:15.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.874+0000 7fe9b3fff640 3 client.5024.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:15.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:15.874+0000 7fe9b3fff640 20 client.5024.objecter dump_active .. 0 homeless 2024-01-20T11:20:16.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.103+0000 7fe9b3fff640 1 -- 172.21.15.110:0/3701822320 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe9c00764f0 con 0x7fe9cc13d0e0 2024-01-20T11:20:16.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.104+0000 7fe9b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:16.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.104+0000 7fe9b3fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:16.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.280+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe9cc0adcf0 con 0x7fe9b806c860 2024-01-20T11:20:16.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.283+0000 7fe9b3fff640 1 -- 172.21.15.110:0/3701822320 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64040 (secure 0 0 0) 0x7fe9bc011af0 con 0x7fe9b806c860 2024-01-20T11:20:16.284 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:16.285 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:16.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 10 librados: watch_flush enter 2024-01-20T11:20:16.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 10 librados: watch_flush exit 2024-01-20T11:20:16.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 20 client.5024.objecter shutdown clearing up homeless session... 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 10 client.5024.objecter successfully canceled tick 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe9b806c860 msgr2=0x7fe9b806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe9b806c860 0x7fe9b806ed20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009bd0 tx=0x7fe9bc009240 comp rx=0 tx=0).stop 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 10 monclient: shutdown 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 msgr2=0x7fe9cc13db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13db00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009890 tx=0x7fe9c00046c0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 shutdown_connections 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe9b806c860 0x7fe9b806ed20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9cc143200 0x7fe9cc13fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9cc13d0e0 0x7fe9cc13db00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 --2- 172.21.15.110:0/3701822320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9cc0b0c00 0x7fe9cc142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 >> 172.21.15.110:0/3701822320 conn(0x7fe9cc0b3e60 msgr2=0x7fe9cc139f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:16.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 shutdown_connections 2024-01-20T11:20:16.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 -- 172.21.15.110:0/3701822320 wait complete. 2024-01-20T11:20:16.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.285+0000 7fe9d2007640 1 librados: shutdown 2024-01-20T11:20:16.296 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-01-20T11:20:14.768345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002454"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.3630000000000004},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.2359999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.894},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.9770000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.130000000000001},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.538}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.25},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":14.968999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.782},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":12.154999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.609},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.643999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313280,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.896000000000001},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":11.848000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.5919999999999996},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.8209999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.8540000000000001},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":5.7720000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.5269999999999992},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.000999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.2279999999999998},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":13.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.239000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.153}]}]},{"osd":3,"up_from":21,"seq":90194313281,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.693},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.23}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.4710000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.086}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.7530000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":11.305999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.945}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":15.090999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.782}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444095,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.3900000000000006},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.2439999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.5579999999999998},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.9619999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.574},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.254}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.220000000000001},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.847}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.8170000000000002},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":15.689}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.798999999999999},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":13.42}]}]},{"osd":1,"up_from":17,"seq":73014444096,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.2910000000000004},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.2779999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.488},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.5359999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":13.526999999999999},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.8689999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":15.321},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":14.667}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.7880000000000003},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":11.061999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.840999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.741}]}]},{"osd":0,"up_from":17,"seq":73014444096,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.8090000000000002},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":5.6879999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.1390000000000002},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.3879999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":9.3729999999999993},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.513999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.948},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.84}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.156000000000001},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.337}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.085000000000001},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":12.539}]}]},{"osd":6,"up_from":15,"seq":64424509504,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":11.565},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.5939999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.3940000000000001},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":12.425000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.292999999999999},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.4619999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":6.4119999999999999},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.65}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":9.2759999999999998},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.032}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.007999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.952}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:16.298 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:16.298 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:16.298 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:16.298 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:16.298 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:16.298 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:16.298 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:16.299 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:16.299 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:16.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.395+0000 7f7f308e5640 10 monclient: get_monmap_and_config 2024-01-20T11:20:16.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.395+0000 7f7f308e5640 10 monclient: build_initial_monmap 2024-01-20T11:20:16.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.395+0000 7f7f308e5640 10 monclient: monmap: 2024-01-20T11:20:16.396 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:16.396 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:16.396 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:16.396895+0000 2024-01-20T11:20:16.396 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:16.396895+0000 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.395+0000 7f7f308e5640 1 Processor -- start 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.395+0000 7f7f308e5640 1 -- start start 2024-01-20T11:20:16.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.395+0000 7f7f308e5640 10 monclient: init 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7f2c0b4d10 0x7f7f2c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 10 monclient(hunting): picked mon.noname-c con 0x7f7f2c0b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f2c0b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 10 monclient(hunting): picked mon.noname-b con 0x7f7f2c145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c146370 con 0x7f7f2c0b17c0 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c0b50f0 con 0x7f7f2c0b4d10 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0aade0 con 0x7f7f2c145980 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:16.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.396+0000 7f7f308e5640 10 monclient(hunting): authenticate will time out at 1659.747848s 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2a59c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7f2c0b4d10 0x7f7f2c0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2ad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2a59c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7f2c0b4d10 0x7f7f2c0b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:36898/0 (socket says 172.21.15.110:36898) 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2a59c640 1 -- 172.21.15.110:0/2325477596 learned_addr learned my addr 172.21.15.110:0/2325477596 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2a59c640 10 monclient(hunting): get_auth_request con 0x7f7f2c0b4d10 auth_method 0 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2ad9d640 10 monclient(hunting): get_auth_request con 0x7f7f2c145980 auth_method 0 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.397+0000 7f7f2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.398+0000 7f7f2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.398+0000 7f7f2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:16.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.398+0000 7f7f2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:16.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.398+0000 7f7f2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:16.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.398+0000 7f7f2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:16.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.398+0000 7f7f2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:16.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.399+0000 7f7f2ad9d640 10 monclient(hunting): handle_auth_done global_id 5027 payload 274 2024-01-20T11:20:16.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.399+0000 7f7f2ad9d640 1 -- 172.21.15.110:0/2325477596 >> v2:172.21.15.134:3300/0 conn(0x7f7f2c0b4d10 msgr2=0x7f7f2c0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:16.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.399+0000 7f7f2ad9d640 1 --2- 172.21.15.110:0/2325477596 >> v2:172.21.15.134:3300/0 conn(0x7f7f2c0b4d10 0x7f7f2c0b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.399+0000 7f7f2ad9d640 1 -- 172.21.15.110:0/2325477596 >> v1:172.21.15.134:6789/0 conn(0x7f7f2c0b17c0 legacy=0x7f7f2c0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:16.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.399+0000 7f7f2ad9d640 10 monclient: _finish_hunting 0 2024-01-20T11:20:16.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.399+0000 7f7f2ad9d640 1 monclient: found mon.noname-b 2024-01-20T11:20:16.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.399+0000 7f7f2ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:16.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.399+0000 7f7f2ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:16.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.399+0000 7f7f2ad9d640 1 -- 172.21.15.110:0/2325477596 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c145460 con 0x7f7f2c145980 2024-01-20T11:20:16.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.401+0000 7f7f2ad9d640 1 --2- 172.21.15.110:0/2325477596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c145e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0097a0 tx=0x7f7f1c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2a53c6191ac0cb5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 1 -- 172.21.15.110:0/2325477596 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c02f050 con 0x7f7f2c145980 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 10 monclient: dump: 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:16.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 10 monclient: _finish_auth 0 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 10 monclient: _check_auth_tickets 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 5 monclient: authenticate success, global_id 5027 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 1 -- 172.21.15.110:0/2325477596 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c031700 con 0x7f7f2c145980 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 1 -- 172.21.15.110:0/2325477596 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c031a90 con 0x7f7f2c145980 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 4 set_mon_vals no callback set 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f2959a640 10 monclient: dump: 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:16.408 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 10 monclient: shutdown 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 1 -- 172.21.15.110:0/2325477596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 msgr2=0x7f7f2c145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 1 --2- 172.21.15.110:0/2325477596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c145e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0097a0 tx=0x7f7f1c02b7a0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 1 -- 172.21.15.110:0/2325477596 shutdown_connections 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 1 --2- 172.21.15.110:0/2325477596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c145e30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 1 --2- 172.21.15.110:0/2325477596 >> v2:172.21.15.134:3300/0 conn(0x7f7f2c0b4d10 0x7f7f2c0b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.405+0000 7f7f308e5640 1 -- 172.21.15.110:0/2325477596 >> 172.21.15.110:0/2325477596 conn(0x7f7f2c066590 msgr2=0x7f7f2c0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.406+0000 7f7f308e5640 1 -- 172.21.15.110:0/2325477596 shutdown_connections 2024-01-20T11:20:16.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.406+0000 7f7f308e5640 1 -- 172.21.15.110:0/2325477596 wait complete. 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.406+0000 7f7f308e5640 10 monclient: build_initial_monmap 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.406+0000 7f7f308e5640 10 monclient: monmap: 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 librados: starting msgr at 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 librados: starting objecter 2024-01-20T11:20:16.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 Processor -- start 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 -- start start 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 librados: setting wanted keys 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 librados: calling monclient init 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient: init 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f2c0b17c0 0x7f7f2c147130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient(hunting): picked mon.noname-b con 0x7f7f2c0b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f2c0b4d10 0x7f7f2c13e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient(hunting): picked mon.noname-a con 0x7f7f2c0b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c141750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient(hunting): picked mon.noname-c con 0x7f7f2c145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c141c90 con 0x7f7f2c0b4d10 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c1467c0 con 0x7f7f2c0b17c0 2024-01-20T11:20:16.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c146a30 con 0x7f7f2c145980 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.407+0000 7f7f308e5640 10 monclient(hunting): authenticate will time out at 1659.759018s 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f29d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f2c0b4d10 0x7f7f2c13e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2ad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2ad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c141750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43760/0 (socket says 172.21.15.110:43760) 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2ad9d640 1 -- 172.21.15.110:0/1967299776 learned_addr learned my addr 172.21.15.110:0/1967299776 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2ad9d640 10 monclient(hunting): get_auth_request con 0x7f7f2c145980 auth_method 0 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2a59c640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f2c0b17c0 0x7f7f2c147130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f29d9b640 10 monclient(hunting): get_auth_request con 0x7f7f2c0b4d10 auth_method 0 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f29d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:16.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2a59c640 10 monclient(hunting): get_auth_request con 0x7f7f2c0b17c0 auth_method 0 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.408+0000 7f7f2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.409+0000 7f7f2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.409+0000 7f7f2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.409+0000 7f7f2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.409+0000 7f7f29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.409+0000 7f7f29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.409+0000 7f7f29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.409+0000 7f7f2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.409+0000 7f7f2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.409+0000 7f7f2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 10 monclient(hunting): handle_auth_done global_id 5030 payload 824 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 1 -- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f2c0b17c0 msgr2=0x7f7f2c147130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:16.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f2c0b17c0 0x7f7f2c147130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 1 -- 172.21.15.110:0/1967299776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f2c0b4d10 msgr2=0x7f7f2c13e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f2c0b4d10 0x7f7f2c13e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 10 monclient: _finish_hunting 0 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 1 monclient: found mon.noname-c 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.412+0000 7f7f2ad9d640 1 -- 172.21.15.110:0/1967299776 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c009400 con 0x7f7f2c145980 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.414+0000 7f7f2ad9d640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c141750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0098c0 tx=0x7f7f1c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.414+0000 7f7f1affd640 10 client.?.objecter ms_handle_connect 0x7f7f2c145980 2024-01-20T11:20:16.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.414+0000 7f7f1affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:16.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 1 -- 172.21.15.110:0/1967299776 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c006e10 con 0x7f7f2c145980 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 10 monclient: dump: 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 10 monclient: _finish_auth 0 2024-01-20T11:20:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 1 -- 172.21.15.110:0/1967299776 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c0339b0 con 0x7f7f2c145980 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 1 -- 172.21.15.110:0/1967299776 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c033d40 con 0x7f7f2c145980 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f1affd640 10 monclient: dump: 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f308e5640 5 monclient: authenticate success, global_id 5030 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f308e5640 10 monclient: _renew_subs 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f28d99640 4 set_mon_vals no callback set 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f308e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c13fea0 con 0x7f7f2c145980 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f308e5640 10 client.5030.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f308e5640 10 monclient: _renew_subs 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f308e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c1405a0 con 0x7f7f2c145980 2024-01-20T11:20:16.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.417+0000 7f7f308e5640 1 librados: init done 2024-01-20T11:20:16.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.418+0000 7f7f308e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:16.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.418+0000 7f7f308e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:16.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.418+0000 7f7f308e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:16.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.418+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c13fd00 con 0x7f7f2c145980 2024-01-20T11:20:16.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.663+0000 7f7f1affd640 1 -- 172.21.15.110:0/1967299776 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7f1c03a070 con 0x7f7f2c145980 2024-01-20T11:20:16.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.664+0000 7f7f1affd640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7f0c06c8b0 0x7f7f0c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:16.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.664+0000 7f7f2a59c640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7f0c06c8b0 0x7f7f0c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:16.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.664+0000 7f7f2a59c640 10 monclient: get_auth_request con 0x7f7f0c06c8b0 auth_method 0 2024-01-20T11:20:16.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.664+0000 7f7f2a59c640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7f0c06c8b0 0x7f7f0c06ed70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009bd0 tx=0x7f7f24009240 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:16.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.664+0000 7f7f1affd640 10 client.5030.objecter ms_handle_connect 0x7f7f0c06c8b0 2024-01-20T11:20:16.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.676+0000 7f7f1affd640 1 -- 172.21.15.110:0/1967299776 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7f1c07a070 con 0x7f7f2c145980 2024-01-20T11:20:16.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.676+0000 7f7f1affd640 10 client.5030.objecter ms_dispatch 0x7f7f2c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:16.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.676+0000 7f7f1affd640 3 client.5030.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:16.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.676+0000 7f7f1affd640 3 client.5030.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:16.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.676+0000 7f7f1affd640 20 client.5030.objecter dump_active .. 0 homeless 2024-01-20T11:20:16.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.676+0000 7f7f1affd640 1 -- 172.21.15.110:0/1967299776 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7f1c0763b0 con 0x7f7f2c145980 2024-01-20T11:20:16.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.676+0000 7f7f1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:16.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.676+0000 7f7f1affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:16.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.923+0000 7f7f308e5640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:16.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.923+0000 7f7f308e5640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:16.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.923+0000 7f7f308e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:16.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.923+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7f2c13fd00 con 0x7f7f2c145980 2024-01-20T11:20:16.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.928+0000 7f7f1affd640 1 -- 172.21.15.110:0/1967299776 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f7f2c13fd00 con 0x7f7f2c145980 2024-01-20T11:20:16.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.928+0000 7f7f1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:16.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.928+0000 7f7f1affd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:16.932 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:16.932 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":351,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 10 librados: watch_flush enter 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 10 librados: watch_flush exit 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 20 client.5030.objecter shutdown clearing up homeless session... 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 10 client.5030.objecter successfully canceled tick 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7f0c06c8b0 msgr2=0x7f7f0c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7f0c06c8b0 0x7f7f0c06ed70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009bd0 tx=0x7f7f24009240 comp rx=0 tx=0).stop 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 10 monclient: shutdown 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 msgr2=0x7f7f2c141750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c141750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0098c0 tx=0x7f7f1c0046c0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 shutdown_connections 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7f0c06c8b0 0x7f7f0c06ed70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f2c145980 0x7f7f2c141750 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f2c0b4d10 0x7f7f2c13e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 --2- 172.21.15.110:0/1967299776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f2c0b17c0 0x7f7f2c147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:16.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 >> 172.21.15.110:0/1967299776 conn(0x7f7f2c066590 msgr2=0x7f7f2c144bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:16.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 shutdown_connections 2024-01-20T11:20:16.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 -- 172.21.15.110:0/1967299776 wait complete. 2024-01-20T11:20:16.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:16.931+0000 7f7f308e5640 1 librados: shutdown 2024-01-20T11:20:16.944 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:16.944 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.042+0000 7fcb42f7d640 10 monclient: get_monmap_and_config 2024-01-20T11:20:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.042+0000 7fcb42f7d640 10 monclient: build_initial_monmap 2024-01-20T11:20:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.042+0000 7fcb42f7d640 10 monclient: monmap: 2024-01-20T11:20:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:17.044051+0000 2024-01-20T11:20:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:17.044051+0000 2024-01-20T11:20:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:17.044 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 1 Processor -- start 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 1 -- start start 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 10 monclient: init 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fcb3c06a860 0x7fcb3c13f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb3c06a860 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb3c06ac40 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb3c13f990 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3c0afd30 con 0x7fcb3c13f990 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3c0b01c0 con 0x7fcb3c06ac40 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3c0723e0 con 0x7fcb3c06a860 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.043+0000 7fcb42f7d640 10 monclient(hunting): authenticate will time out at 1660.395031s 2024-01-20T11:20:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.044+0000 7fcb4151a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcb3c13f990 0x7fcb3c14c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56342/0 (socket says 172.21.15.110:56342) 2024-01-20T11:20:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.044+0000 7fcb4151a640 1 -- 172.21.15.110:0/373296819 learned_addr learned my addr 172.21.15.110:0/373296819 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.044+0000 7fcb40d19640 1 --2- 172.21.15.110:0/373296819 >> v2:172.21.15.134:3300/0 conn(0x7fcb3c06a860 0x7fcb3c13f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.045+0000 7fcb40d19640 10 monclient(hunting): get_auth_request con 0x7fcb3c06a860 auth_method 0 2024-01-20T11:20:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.045+0000 7fcb40d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.045+0000 7fcb40d19640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.045+0000 7fcb40d19640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.045+0000 7fcb40d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.045+0000 7fcb40d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.045+0000 7fcb40d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.046+0000 7fcb3b7fe640 1 -- 172.21.15.110:0/373296819 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135740486 0 0) 0x7fcb3c0afd30 con 0x7fcb3c13f990 2024-01-20T11:20:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.046+0000 7fcb3b7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:20:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.046+0000 7fcb3b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.046+0000 7fcb3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.046+0000 7fcb3b7fe640 10 monclient(hunting): my global_id is 5259 2024-01-20T11:20:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.046+0000 7fcb3b7fe640 1 -- 172.21.15.110:0/373296819 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb2c003520 con 0x7fcb3c13f990 2024-01-20T11:20:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.047+0000 7fcb40d19640 10 monclient(hunting): handle_auth_done global_id 5341 payload 274 2024-01-20T11:20:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.047+0000 7fcb40d19640 1 -- 172.21.15.110:0/373296819 >> v1:172.21.15.134:6789/0 conn(0x7fcb3c06ac40 legacy=0x7fcb3c1455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.047+0000 7fcb40d19640 1 -- 172.21.15.110:0/373296819 >> v1:172.21.15.110:6789/0 conn(0x7fcb3c13f990 legacy=0x7fcb3c14c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.047+0000 7fcb40d19640 10 monclient: _finish_hunting 0 2024-01-20T11:20:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.047+0000 7fcb40d19640 1 monclient: found mon.noname-c 2024-01-20T11:20:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.047+0000 7fcb40d19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.047+0000 7fcb40d19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.047+0000 7fcb40d19640 1 -- 172.21.15.110:0/373296819 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c072520 con 0x7fcb3c06a860 2024-01-20T11:20:17.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.049+0000 7fcb40d19640 1 --2- 172.21.15.110:0/373296819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c06a860 0x7fcb3c13f450 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcb300097a0 tx=0x7fcb3002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=936d48a109c3c44e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 1 -- 172.21.15.110:0/373296819 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3002e070 con 0x7fcb3c06a860 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 10 monclient: dump: 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb42f7d640 5 monclient: authenticate success, global_id 5341 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 1 -- 172.21.15.110:0/373296819 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb300315a0 con 0x7fcb3c06a860 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb42f7d640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 1 -- 172.21.15.110:0/373296819 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb30031930 con 0x7fcb3c06a860 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb42f7d640 4 set_mon_vals no callback set 2024-01-20T11:20:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.054+0000 7fcb3b7fe640 10 monclient: dump: 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:17.057 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 10 monclient: shutdown 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 1 -- 172.21.15.110:0/373296819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c06a860 msgr2=0x7fcb3c13f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 1 --2- 172.21.15.110:0/373296819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c06a860 0x7fcb3c13f450 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcb300097a0 tx=0x7fcb3002b840 comp rx=0 tx=0).stop 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 1 -- 172.21.15.110:0/373296819 shutdown_connections 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 1 --2- 172.21.15.110:0/373296819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c06a860 0x7fcb3c13f450 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 1 -- 172.21.15.110:0/373296819 >> 172.21.15.110:0/373296819 conn(0x7fcb3c0b47d0 msgr2=0x7fcb3c0b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 1 -- 172.21.15.110:0/373296819 shutdown_connections 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 1 -- 172.21.15.110:0/373296819 wait complete. 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 10 monclient: build_initial_monmap 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.055+0000 7fcb42f7d640 10 monclient: monmap: 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.056+0000 7fcb42f7d640 1 librados: starting msgr at 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.056+0000 7fcb42f7d640 1 librados: starting objecter 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.056+0000 7fcb42f7d640 1 Processor -- start 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.056+0000 7fcb42f7d640 1 -- start start 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.056+0000 7fcb42f7d640 1 librados: setting wanted keys 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.056+0000 7fcb42f7d640 1 librados: calling monclient init 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.056+0000 7fcb42f7d640 10 monclient: init 2024-01-20T11:20:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c06a860 0x7fcb3c1428c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb3c06a860 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb3c06ac40 0x7fcb3c13a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb3c06ac40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c13f990 0x7fcb3c13a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb3c13f990 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c138ca0 con 0x7fcb3c06a860 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c1420d0 con 0x7fcb3c13f990 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c140f50 con 0x7fcb3c06ac40 2024-01-20T11:20:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb42f7d640 10 monclient(hunting): authenticate will time out at 1660.408276s 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb40d19640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c06a860 0x7fcb3c1428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb4151a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c13f990 0x7fcb3c13a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb3bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb3c06ac40 0x7fcb3c13a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb40d19640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c06a860 0x7fcb3c1428c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:49174/0 (socket says 172.21.15.110:49174) 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb40d19640 1 -- 172.21.15.110:0/4134590783 learned_addr learned my addr 172.21.15.110:0/4134590783 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb40d19640 10 monclient(hunting): get_auth_request con 0x7fcb3c06a860 auth_method 0 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb40d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb40d19640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.057+0000 7fcb40d19640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb4151a640 10 monclient(hunting): get_auth_request con 0x7fcb3c13f990 auth_method 0 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb4151a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb4151a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb4151a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb3bfff640 10 monclient(hunting): get_auth_request con 0x7fcb3c06ac40 auth_method 0 2024-01-20T11:20:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb3bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb4151a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb4151a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb4151a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb40d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb40d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb40d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.058+0000 7fcb3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 10 monclient(hunting): handle_auth_done global_id 5344 payload 824 2024-01-20T11:20:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 1 -- 172.21.15.110:0/4134590783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb3c06ac40 msgr2=0x7fcb3c13a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb3c06ac40 0x7fcb3c13a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 1 -- 172.21.15.110:0/4134590783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c06a860 msgr2=0x7fcb3c1428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c06a860 0x7fcb3c1428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 10 monclient: _finish_hunting 0 2024-01-20T11:20:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 1 monclient: found mon.noname-b 2024-01-20T11:20:17.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:17.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:17.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.061+0000 7fcb4151a640 1 -- 172.21.15.110:0/4134590783 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30009400 con 0x7fcb3c13f990 2024-01-20T11:20:17.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.063+0000 7fcb4151a640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c13f990 0x7fcb3c13a750 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400caa0 tx=0x7fcb3400cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:17.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.063+0000 7fcb397fa640 10 client.?.objecter ms_handle_connect 0x7fcb3c13f990 2024-01-20T11:20:17.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.063+0000 7fcb397fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 1 -- 172.21.15.110:0/4134590783 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3400ab40 con 0x7fcb3c13f990 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 10 monclient: dump: 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 1 -- 172.21.15.110:0/4134590783 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3400aca0 con 0x7fcb3c13f990 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 1 -- 172.21.15.110:0/4134590783 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34012600 con 0x7fcb3c13f990 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb42f7d640 5 monclient: authenticate success, global_id 5344 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb42f7d640 10 monclient: _renew_subs 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb42f7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3c142e00 con 0x7fcb3c13f990 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb3affd640 4 set_mon_vals no callback set 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.066+0000 7fcb397fa640 10 monclient: dump: 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.067+0000 7fcb42f7d640 10 client.5344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.067+0000 7fcb42f7d640 10 monclient: _renew_subs 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.067+0000 7fcb42f7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.067+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c13ae70 con 0x7fcb3c13f990 2024-01-20T11:20:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.067+0000 7fcb42f7d640 1 librados: init done 2024-01-20T11:20:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.067+0000 7fcb42f7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.067+0000 7fcb42f7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.067+0000 7fcb42f7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.067+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3c075490 con 0x7fcb3c13f990 2024-01-20T11:20:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.086+0000 7fcb397fa640 1 -- 172.21.15.110:0/4134590783 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fcb34007070 con 0x7fcb3c13f990 2024-01-20T11:20:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.086+0000 7fcb397fa640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb2006c8e0 0x7fcb2006eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.086+0000 7fcb40d19640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb2006c8e0 0x7fcb2006eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:17.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.086+0000 7fcb40d19640 10 monclient: get_auth_request con 0x7fcb2006c8e0 auth_method 0 2024-01-20T11:20:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.087+0000 7fcb40d19640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb2006c8e0 0x7fcb2006eda0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb3002b880 tx=0x7fcb3003d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:17.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.087+0000 7fcb397fa640 10 client.5344.objecter ms_handle_connect 0x7fcb2006c8e0 2024-01-20T11:20:17.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.091+0000 7fcb397fa640 1 -- 172.21.15.110:0/4134590783 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcb34058520 con 0x7fcb3c13f990 2024-01-20T11:20:17.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.091+0000 7fcb397fa640 10 client.5344.objecter ms_dispatch 0x7fcb3c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:17.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.091+0000 7fcb397fa640 3 client.5344.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:17.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.091+0000 7fcb397fa640 3 client.5344.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:17.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.091+0000 7fcb397fa640 20 client.5344.objecter dump_active .. 0 homeless 2024-01-20T11:20:17.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.323+0000 7fcb397fa640 1 -- 172.21.15.110:0/4134590783 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fcb34054860 con 0x7fcb3c13f990 2024-01-20T11:20:17.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.323+0000 7fcb397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:17.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.323+0000 7fcb397fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:17.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.514+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb3c0afa00 con 0x7fcb2006c8e0 2024-01-20T11:20:17.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.517+0000 7fcb397fa640 1 -- 172.21.15.110:0/4134590783 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64006 (secure 0 0 0) 0x7fcb30031a80 con 0x7fcb2006c8e0 2024-01-20T11:20:17.519 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:17.519 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:17.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 10 librados: watch_flush enter 2024-01-20T11:20:17.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 10 librados: watch_flush exit 2024-01-20T11:20:17.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 20 client.5344.objecter shutdown clearing up homeless session... 2024-01-20T11:20:17.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 10 client.5344.objecter successfully canceled tick 2024-01-20T11:20:17.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb2006c8e0 msgr2=0x7fcb2006eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb2006c8e0 0x7fcb2006eda0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb3002b880 tx=0x7fcb3003d040 comp rx=0 tx=0).stop 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 10 monclient: shutdown 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c13f990 msgr2=0x7fcb3c13a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.519+0000 7fcb42f7d640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c13f990 0x7fcb3c13a750 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400caa0 tx=0x7fcb3400cea0 comp rx=0 tx=0).stop 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.520+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 shutdown_connections 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.520+0000 7fcb42f7d640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcb2006c8e0 0x7fcb2006eda0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.520+0000 7fcb42f7d640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb3c13f990 0x7fcb3c13a750 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.520+0000 7fcb42f7d640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb3c06ac40 0x7fcb3c13a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.520+0000 7fcb42f7d640 1 --2- 172.21.15.110:0/4134590783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb3c06a860 0x7fcb3c1428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.520+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 >> 172.21.15.110:0/4134590783 conn(0x7fcb3c0b47d0 msgr2=0x7fcb3c0b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.520+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 shutdown_connections 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.520+0000 7fcb42f7d640 1 -- 172.21.15.110:0/4134590783 wait complete. 2024-01-20T11:20:17.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:17.520+0000 7fcb42f7d640 1 librados: shutdown 2024-01-20T11:20:17.532 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-01-20T11:20:16.768760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002432"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.3630000000000004},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.2359999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.894},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.9770000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.130000000000001},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.538}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.25},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":14.968999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.782},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":12.154999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.609},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.643999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313281,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.8890000000000002},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.5679999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.9199999999999999},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.186}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.804},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":7.9809999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.5270000000000001},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.648}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.438000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.2669999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.526999999999999},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.300000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313281,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.693},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.23}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.4710000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.086}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.7530000000000001},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":16.048999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":11.305999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.945}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":15.090999999999999},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.782}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444095,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.3900000000000006},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":6.2439999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.5579999999999998},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.9619999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":11.574},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.254}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.220000000000001},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.847}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.8170000000000002},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":15.689}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.798999999999999},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":13.42}]}]},{"osd":1,"up_from":17,"seq":73014444096,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.2910000000000004},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":6.2779999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":10.488},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.5359999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":13.526999999999999},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.8689999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":15.321},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":14.667}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.7880000000000003},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":11.061999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.840999999999999},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.741}]}]},{"osd":0,"up_from":17,"seq":73014444096,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.8090000000000002},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":5.6879999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.1390000000000002},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.3879999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":9.3729999999999993},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.513999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.948},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.84}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.156000000000001},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":15.337}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.085000000000001},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":12.539}]}]},{"osd":6,"up_from":15,"seq":64424509505,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.2210000000000001},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.6969999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.8819999999999997},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":13.387}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.378},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.4219999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":17.021000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.513}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.252000000000001},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.182}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.852},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.263999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:18.715 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:09.739 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:18.763 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:06:09.769 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:18.764 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:09.789 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:18.766 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:06:09.772 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:18.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~81s 2024-01-20T11:20:18.782 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:06:09.788 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:18.782 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:09.807 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:18.810 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:09.835 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:18.816 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:09.840 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:18.831 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:09.855 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:20.534 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:20.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.632+0000 7f9a6acd4640 10 monclient: get_monmap_and_config 2024-01-20T11:20:20.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.632+0000 7f9a6acd4640 10 monclient: build_initial_monmap 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.632+0000 7f9a6acd4640 10 monclient: monmap: 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:20.634568+0000 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:20.634568+0000 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:20.634 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:20.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.633+0000 7f9a6acd4640 1 Processor -- start 2024-01-20T11:20:20.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.633+0000 7f9a6acd4640 1 -- start start 2024-01-20T11:20:20.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.633+0000 7f9a6acd4640 10 monclient: init 2024-01-20T11:20:20.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a64071c60 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9a640ac540 0x7f9a640ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 10 monclient(hunting): picked mon.noname-c con 0x7f9a640ac540 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a640b0930 0x7f9a64149510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 10 monclient(hunting): picked mon.noname-b con 0x7f9a640b0930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a640ad610 con 0x7f9a64071c60 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a64072040 con 0x7f9a640ac540 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640aba00 con 0x7f9a640b0930 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a6acd4640 10 monclient(hunting): authenticate will time out at 1663.985570s 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.634+0000 7f9a69271640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a640b0930 0x7f9a64149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:20.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a63fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9a640ac540 0x7f9a640ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:20.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a69271640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a640b0930 0x7f9a64149510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:42414/0 (socket says 172.21.15.110:42414) 2024-01-20T11:20:20.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a69271640 1 -- 172.21.15.110:0/3200125803 learned_addr learned my addr 172.21.15.110:0/3200125803 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:20.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a69271640 10 monclient(hunting): get_auth_request con 0x7f9a640b0930 auth_method 0 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a69271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a69271640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a69271640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a63fff640 10 monclient(hunting): get_auth_request con 0x7f9a640ac540 auth_method 0 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a63fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a69271640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a69271640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.635+0000 7f9a69271640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.636+0000 7f9a63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.636+0000 7f9a63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:20.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.636+0000 7f9a63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:20.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.637+0000 7f9a69271640 10 monclient(hunting): handle_auth_done global_id 5036 payload 274 2024-01-20T11:20:20.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.637+0000 7f9a69271640 1 -- 172.21.15.110:0/3200125803 >> v2:172.21.15.134:3300/0 conn(0x7f9a640ac540 msgr2=0x7f9a640ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:20.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.637+0000 7f9a69271640 1 --2- 172.21.15.110:0/3200125803 >> v2:172.21.15.134:3300/0 conn(0x7f9a640ac540 0x7f9a640ab4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:20.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.637+0000 7f9a69271640 1 -- 172.21.15.110:0/3200125803 >> v1:172.21.15.134:6789/0 conn(0x7f9a64071c60 legacy=0x7f9a64142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:20.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.637+0000 7f9a69271640 10 monclient: _finish_hunting 0 2024-01-20T11:20:20.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.637+0000 7f9a69271640 1 monclient: found mon.noname-b 2024-01-20T11:20:20.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.637+0000 7f9a69271640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:20.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.637+0000 7f9a69271640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:20.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.637+0000 7f9a69271640 1 -- 172.21.15.110:0/3200125803 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640721b0 con 0x7f9a640b0930 2024-01-20T11:20:20.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.639+0000 7f9a69271640 1 --2- 172.21.15.110:0/3200125803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a640b0930 0x7f9a64149510 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009e50 tx=0x7f9a5c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4291bf55d30b5a18 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:20.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 1 -- 172.21.15.110:0/3200125803 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c02ca70 con 0x7f9a640b0930 2024-01-20T11:20:20.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:20.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 10 monclient: dump: 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 1 -- 172.21.15.110:0/3200125803 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c02cbd0 con 0x7f9a640b0930 2024-01-20T11:20:20.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 1 -- 172.21.15.110:0/3200125803 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c0343f0 con 0x7f9a640b0930 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a6acd4640 5 monclient: authenticate success, global_id 5036 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a6acd4640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a6acd4640 4 set_mon_vals no callback set 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.643+0000 7f9a637fe640 10 monclient: dump: 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:20.647 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:20.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 10 monclient: shutdown 2024-01-20T11:20:20.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 1 -- 172.21.15.110:0/3200125803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a640b0930 msgr2=0x7f9a64149510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 1 --2- 172.21.15.110:0/3200125803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a640b0930 0x7f9a64149510 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009e50 tx=0x7f9a5c02bed0 comp rx=0 tx=0).stop 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 1 -- 172.21.15.110:0/3200125803 shutdown_connections 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 1 --2- 172.21.15.110:0/3200125803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a640b0930 0x7f9a64149510 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 1 --2- 172.21.15.110:0/3200125803 >> v2:172.21.15.134:3300/0 conn(0x7f9a640ac540 0x7f9a640ab4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 1 -- 172.21.15.110:0/3200125803 >> 172.21.15.110:0/3200125803 conn(0x7f9a640b3e60 msgr2=0x7f9a640b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 1 -- 172.21.15.110:0/3200125803 shutdown_connections 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.644+0000 7f9a6acd4640 1 -- 172.21.15.110:0/3200125803 wait complete. 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.645+0000 7f9a6acd4640 10 monclient: build_initial_monmap 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.645+0000 7f9a6acd4640 10 monclient: monmap: 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:20.650 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.645+0000 7f9a6acd4640 1 librados: starting msgr at 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.645+0000 7f9a6acd4640 1 librados: starting objecter 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.645+0000 7f9a6acd4640 1 Processor -- start 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.645+0000 7f9a6acd4640 1 -- start start 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.645+0000 7f9a6acd4640 1 librados: setting wanted keys 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.645+0000 7f9a6acd4640 1 librados: calling monclient init 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.645+0000 7f9a6acd4640 10 monclient: init 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a64071c60 0x7f9a6414ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:20.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient(hunting): picked mon.noname-c con 0x7f9a64071c60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a640ac540 0x7f9a6413ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient(hunting): picked mon.noname-b con 0x7f9a640ac540 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a640b0930 0x7f9a641318b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient(hunting): picked mon.noname-a con 0x7f9a640b0930 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6413d120 con 0x7f9a640b0930 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6413d690 con 0x7f9a640ac540 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64141980 con 0x7f9a64071c60 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a6acd4640 10 monclient(hunting): authenticate will time out at 1663.997370s 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a69271640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a640b0930 0x7f9a641318b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.646+0000 7f9a63fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a640ac540 0x7f9a6413ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:20.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a63fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a640ac540 0x7f9a6413ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:59490/0 (socket says 172.21.15.110:59490) 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a63fff640 1 -- 172.21.15.110:0/111735227 learned_addr learned my addr 172.21.15.110:0/111735227 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a63fff640 10 monclient(hunting): get_auth_request con 0x7f9a640ac540 auth_method 0 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a63fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a69271640 10 monclient(hunting): get_auth_request con 0x7f9a640b0930 auth_method 0 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a69271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a69271640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a69271640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a69271640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a69271640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.647+0000 7f9a69271640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.648+0000 7f9a68a70640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a64071c60 0x7f9a6414ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:20.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.649+0000 7f9a68a70640 10 monclient(hunting): get_auth_request con 0x7f9a64071c60 auth_method 0 2024-01-20T11:20:20.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.649+0000 7f9a68a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:20.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.649+0000 7f9a68a70640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:20.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.649+0000 7f9a68a70640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.649+0000 7f9a68a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.649+0000 7f9a68a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.649+0000 7f9a68a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 10 monclient(hunting): handle_auth_done global_id 5350 payload 824 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 1 -- 172.21.15.110:0/111735227 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a64071c60 msgr2=0x7f9a6414ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a64071c60 0x7f9a6414ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 1 -- 172.21.15.110:0/111735227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a640b0930 msgr2=0x7f9a641318b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a640b0930 0x7f9a641318b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 1 monclient: found mon.noname-b 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:20.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:20.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.650+0000 7f9a63fff640 1 -- 172.21.15.110:0/111735227 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c009b30 con 0x7f9a640ac540 2024-01-20T11:20:20.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.652+0000 7f9a63fff640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a640ac540 0x7f9a6413ca00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800e9b0 tx=0x7f9a5800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:20.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.653+0000 7f9a617fa640 10 client.?.objecter ms_handle_connect 0x7f9a640ac540 2024-01-20T11:20:20.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.653+0000 7f9a617fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 1 -- 172.21.15.110:0/111735227 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58009670 con 0x7f9a640ac540 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 10 monclient: dump: 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:20.658 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 1 -- 172.21.15.110:0/111735227 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5800c9a0 con 0x7f9a640ac540 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a6acd4640 5 monclient: authenticate success, global_id 5350 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a6acd4640 10 monclient: _renew_subs 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a6acd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a64141bd0 con 0x7f9a640ac540 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 1 -- 172.21.15.110:0/111735227 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5800cd50 con 0x7f9a640ac540 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:20.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a617fa640 10 monclient: dump: 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a62ffd640 4 set_mon_vals no callback set 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a6acd4640 10 client.5350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a6acd4640 10 monclient: _renew_subs 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a6acd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:20.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a64131fd0 con 0x7f9a640ac540 2024-01-20T11:20:20.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.656+0000 7f9a6acd4640 1 librados: init done 2024-01-20T11:20:20.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.657+0000 7f9a6acd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:20.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.657+0000 7f9a6acd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:20.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.657+0000 7f9a6acd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:20.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.657+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a64075490 con 0x7f9a640ac540 2024-01-20T11:20:20.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.899+0000 7f9a617fa640 1 -- 172.21.15.110:0/111735227 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9a58013470 con 0x7f9a640ac540 2024-01-20T11:20:20.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.900+0000 7f9a617fa640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9a5006c860 0x7f9a5006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:20.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.900+0000 7f9a68a70640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9a5006c860 0x7f9a5006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:20.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.900+0000 7f9a68a70640 10 monclient: get_auth_request con 0x7f9a5006c860 auth_method 0 2024-01-20T11:20:20.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.900+0000 7f9a617fa640 10 client.5350.objecter ms_handle_connect 0x7f9a5006c860 2024-01-20T11:20:20.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.900+0000 7f9a68a70640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9a5006c860 0x7f9a5006ed20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9a540017f0 tx=0x7f9a54009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:20.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.904+0000 7f9a617fa640 1 -- 172.21.15.110:0/111735227 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9a58058c80 con 0x7f9a640ac540 2024-01-20T11:20:20.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.904+0000 7f9a617fa640 10 client.5350.objecter ms_dispatch 0x7f9a64000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:20.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.904+0000 7f9a617fa640 3 client.5350.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:20.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.904+0000 7f9a617fa640 3 client.5350.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:20.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.904+0000 7f9a617fa640 20 client.5350.objecter dump_active .. 0 homeless 2024-01-20T11:20:20.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.916+0000 7f9a617fa640 1 -- 172.21.15.110:0/111735227 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9a58054fc0 con 0x7f9a640ac540 2024-01-20T11:20:20.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.916+0000 7f9a617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:20.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:20.916+0000 7f9a617fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:21.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.089+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a64072340 con 0x7f9a5006c860 2024-01-20T11:20:21.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.093+0000 7f9a617fa640 1 -- 172.21.15.110:0/111735227 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64018 (secure 0 0 0) 0x7f9a54001440 con 0x7f9a5006c860 2024-01-20T11:20:21.095 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:21.095 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:21.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 10 librados: watch_flush enter 2024-01-20T11:20:21.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 10 librados: watch_flush exit 2024-01-20T11:20:21.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 20 client.5350.objecter shutdown clearing up homeless session... 2024-01-20T11:20:21.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 10 client.5350.objecter successfully canceled tick 2024-01-20T11:20:21.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9a5006c860 msgr2=0x7f9a5006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9a5006c860 0x7f9a5006ed20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9a540017f0 tx=0x7f9a54009040 comp rx=0 tx=0).stop 2024-01-20T11:20:21.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 10 monclient: shutdown 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a640ac540 msgr2=0x7f9a6413ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a640ac540 0x7f9a6413ca00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800e9b0 tx=0x7f9a5800ee70 comp rx=0 tx=0).stop 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 shutdown_connections 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9a5006c860 0x7f9a5006ed20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a640b0930 0x7f9a641318b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a640ac540 0x7f9a6413ca00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 --2- 172.21.15.110:0/111735227 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9a64071c60 0x7f9a6414ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 >> 172.21.15.110:0/111735227 conn(0x7f9a640b3e60 msgr2=0x7f9a640a9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 shutdown_connections 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.095+0000 7f9a6acd4640 1 -- 172.21.15.110:0/111735227 wait complete. 2024-01-20T11:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.096+0000 7f9a6acd4640 1 librados: shutdown 2024-01-20T11:20:21.107 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-01-20T11:20:20.769485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002511"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280579,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.8170000000000002},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":11.154}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.976},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":17.707999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.176},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.146000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.7810000000000006},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.054}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.709},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":11.976000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7680000000000007},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.370000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313281,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.8890000000000002},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.5679999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.9199999999999999},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.186}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.804},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":7.9809999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.5270000000000001},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.648}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.438000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.2669999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.526999999999999},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.300000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313282,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.5099999999999998},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.3220000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.994},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.366}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.7770000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.736000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.327},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":8.8290000000000006},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":14.372999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.872},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.84}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444096,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.7320000000000002},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":7.375}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.4059999999999997},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.882999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.856},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.3010000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.927},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.162000000000001},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.9130000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.515000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.664000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444097,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.6420000000000003},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":7.1109999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.159000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":14.214},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.7590000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":8.4199999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.852}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.6110000000000007},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.926}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.194000000000001},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.250999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444097,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.7069999999999999},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":9.0150000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":4.9989999999999997},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.223000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":12.449},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.6400000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.888999999999999},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.96}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.733000000000001},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":10.456}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.407},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":13.096}]}]},{"osd":6,"up_from":15,"seq":64424509505,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.2210000000000001},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.6969999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.8819999999999997},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":13.387}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.378},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.4219999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":17.021000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.513}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.252000000000001},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.182}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.852},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.263999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:21.109 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:21.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.208+0000 7fea9df2c640 10 monclient: get_monmap_and_config 2024-01-20T11:20:21.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.208+0000 7fea9df2c640 10 monclient: build_initial_monmap 2024-01-20T11:20:21.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.208+0000 7fea9df2c640 10 monclient: monmap: 2024-01-20T11:20:21.209 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:21.210442+0000 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:21.210442+0000 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:21.210 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.209+0000 7fea9df2c640 1 Processor -- start 2024-01-20T11:20:21.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.209+0000 7fea9df2c640 1 -- start start 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.209+0000 7fea9df2c640 10 monclient: init 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fea980a96f0 0x7fea9813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient(hunting): picked mon.noname-a con 0x7fea980a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea9813fa10 0x7fea981423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient(hunting): picked mon.noname-b con 0x7fea9813fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fea980a9ad0 0x7fea98142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient(hunting): picked mon.noname-c con 0x7fea980a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea980aaea0 con 0x7fea980a96f0 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea980b25c0 con 0x7fea980a9ad0 2024-01-20T11:20:21.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea980ab0a0 con 0x7fea9813fa10 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea9df2c640 10 monclient(hunting): authenticate will time out at 1664.561521s 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea977fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fea980a96f0 0x7fea9813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.210+0000 7fea96ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea9813fa10 0x7fea981423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea977fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fea980a96f0 0x7fea9813f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:55502/0 (socket says 172.21.15.110:55502) 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea977fe640 1 -- 172.21.15.110:0/2183523075 learned_addr learned my addr 172.21.15.110:0/2183523075 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea96ffd640 10 monclient(hunting): get_auth_request con 0x7fea9813fa10 auth_method 0 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea96ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea97fff640 1 --2- 172.21.15.110:0/2183523075 >> v2:172.21.15.134:3300/0 conn(0x7fea980a9ad0 0x7fea98142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea977fe640 10 monclient(hunting): get_auth_request con 0x7fea980a96f0 auth_method 0 2024-01-20T11:20:21.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea977fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea977fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea97fff640 10 monclient(hunting): get_auth_request con 0x7fea980a9ad0 auth_method 0 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea97fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea97fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.211+0000 7fea96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.212+0000 7fea97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.212+0000 7fea97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.212+0000 7fea97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 10 monclient(hunting): handle_auth_done global_id 5268 payload 274 2024-01-20T11:20:21.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 1 -- 172.21.15.110:0/2183523075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea9813fa10 msgr2=0x7fea981423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:21.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 1 --2- 172.21.15.110:0/2183523075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea9813fa10 0x7fea981423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 1 -- 172.21.15.110:0/2183523075 >> v2:172.21.15.134:3300/0 conn(0x7fea980a9ad0 msgr2=0x7fea98142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:21.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 1 --2- 172.21.15.110:0/2183523075 >> v2:172.21.15.134:3300/0 conn(0x7fea980a9ad0 0x7fea98142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:21.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 1 monclient: found mon.noname-a 2024-01-20T11:20:21.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:21.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:21.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.213+0000 7fea977fe640 1 -- 172.21.15.110:0/2183523075 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea980b3fb0 con 0x7fea980a96f0 2024-01-20T11:20:21.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.215+0000 7fea977fe640 1 --2- 172.21.15.110:0/2183523075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea980a96f0 0x7fea9813f4d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea980749e0 tx=0x7fea8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac70a6c17de7f681 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.218+0000 7fea967fc640 1 -- 172.21.15.110:0/2183523075 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea880317c0 con 0x7fea980a96f0 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.218+0000 7fea967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.218+0000 7fea967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.218+0000 7fea967fc640 10 monclient: dump: 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:21.220 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea967fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea967fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea967fc640 1 -- 172.21.15.110:0/2183523075 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea88031920 con 0x7fea980a96f0 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea967fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea967fc640 1 -- 172.21.15.110:0/2183523075 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88031cb0 con 0x7fea980a96f0 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 5 monclient: authenticate success, global_id 5268 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 4 set_mon_vals no callback set 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:21.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea967fc640 10 monclient: dump: 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:21.222 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 10 monclient: shutdown 2024-01-20T11:20:21.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:21.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 1 -- 172.21.15.110:0/2183523075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea980a96f0 msgr2=0x7fea9813f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 1 --2- 172.21.15.110:0/2183523075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea980a96f0 0x7fea9813f4d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea980749e0 tx=0x7fea8802b860 comp rx=0 tx=0).stop 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 1 -- 172.21.15.110:0/2183523075 shutdown_connections 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 1 --2- 172.21.15.110:0/2183523075 >> v2:172.21.15.134:3300/0 conn(0x7fea980a9ad0 0x7fea98142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 1 --2- 172.21.15.110:0/2183523075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea9813fa10 0x7fea981423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 1 --2- 172.21.15.110:0/2183523075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea980a96f0 0x7fea9813f4d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 1 -- 172.21.15.110:0/2183523075 >> 172.21.15.110:0/2183523075 conn(0x7fea980b7bc0 msgr2=0x7fea980b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 1 -- 172.21.15.110:0/2183523075 shutdown_connections 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.219+0000 7fea9df2c640 1 -- 172.21.15.110:0/2183523075 wait complete. 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.220+0000 7fea9df2c640 10 monclient: build_initial_monmap 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.220+0000 7fea9df2c640 10 monclient: monmap: 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.220+0000 7fea9df2c640 1 librados: starting msgr at 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.220+0000 7fea9df2c640 1 librados: starting objecter 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.220+0000 7fea9df2c640 1 Processor -- start 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 1 -- start start 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 1 librados: setting wanted keys 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 1 librados: calling monclient init 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient: init 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:20:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea980a96f0 0x7fea9813e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient(hunting): picked mon.noname-c con 0x7fea980a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea980a9ad0 0x7fea981d7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient(hunting): picked mon.noname-b con 0x7fea980a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981da600 0x7fea981daab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient(hunting): picked mon.noname-a con 0x7fea981da600 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98137bb0 con 0x7fea981da600 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea981d79d0 con 0x7fea980a9ad0 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea981d7640 con 0x7fea980a96f0 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.221+0000 7fea9df2c640 10 monclient(hunting): authenticate will time out at 1664.572889s 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea977fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea980a96f0 0x7fea9813e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea97fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981da600 0x7fea981daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea96ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea980a9ad0 0x7fea981d7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea977fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea980a96f0 0x7fea9813e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:42454/0 (socket says 172.21.15.110:42454) 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea977fe640 1 -- 172.21.15.110:0/1922369536 learned_addr learned my addr 172.21.15.110:0/1922369536 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea977fe640 10 monclient(hunting): get_auth_request con 0x7fea980a96f0 auth_method 0 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea977fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea977fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea96ffd640 10 monclient(hunting): get_auth_request con 0x7fea980a9ad0 auth_method 0 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea96ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea97fff640 10 monclient(hunting): get_auth_request con 0x7fea981da600 auth_method 0 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea97fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.222+0000 7fea97fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.223+0000 7fea96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.223+0000 7fea96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.223+0000 7fea96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.223+0000 7fea97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.223+0000 7fea97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.223+0000 7fea97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.223+0000 7fea977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.223+0000 7fea977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.223+0000 7fea977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 10 monclient(hunting): handle_auth_done global_id 5356 payload 824 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 1 -- 172.21.15.110:0/1922369536 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea980a96f0 msgr2=0x7fea9813e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea980a96f0 0x7fea9813e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 1 -- 172.21.15.110:0/1922369536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981da600 msgr2=0x7fea981daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981da600 0x7fea981daab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 1 monclient: found mon.noname-b 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.225+0000 7fea96ffd640 1 -- 172.21.15.110:0/1922369536 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea880094d0 con 0x7fea980a9ad0 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.227+0000 7fea96ffd640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea980a9ad0 0x7fea981d7100 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fea980aac70 tx=0x7fea8c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.227+0000 7fea7ffff640 10 client.?.objecter ms_handle_connect 0x7fea980a9ad0 2024-01-20T11:20:21.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.227+0000 7fea7ffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea7ffff640 1 -- 172.21.15.110:0/1922369536 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c00c700 con 0x7fea980a9ad0 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea7ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea7ffff640 10 monclient: dump: 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea7ffff640 10 monclient: _finish_auth 0 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea7ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:21.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea9df2c640 5 monclient: authenticate success, global_id 5356 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea9df2c640 10 monclient: _renew_subs 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea9df2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea981d7e60 con 0x7fea980a9ad0 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea9df2c640 10 client.5356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea9df2c640 10 monclient: _renew_subs 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea9df2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea981daff0 con 0x7fea980a9ad0 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.229+0000 7fea9df2c640 1 librados: init done 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.230+0000 7fea9df2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.230+0000 7fea9df2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.230+0000 7fea9df2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:21.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.230+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea98075490 con 0x7fea980a9ad0 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.232+0000 7fea7ffff640 1 -- 172.21.15.110:0/1922369536 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c005990 con 0x7fea980a9ad0 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.232+0000 7fea7ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.232+0000 7fea7ffff640 1 -- 172.21.15.110:0/1922369536 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c005d00 con 0x7fea980a9ad0 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.232+0000 7fea7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.232+0000 7fea7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.232+0000 7fea7ffff640 10 monclient: dump: 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:21.235 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:21.236 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:21.236 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.232+0000 7fea95ffb640 4 set_mon_vals no callback set 2024-01-20T11:20:21.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.471+0000 7fea7ffff640 1 -- 172.21.15.110:0/1922369536 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fea8c00c860 con 0x7fea980a9ad0 2024-01-20T11:20:21.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.471+0000 7fea7ffff640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea8406c860 0x7fea8406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.471+0000 7fea977fe640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea8406c860 0x7fea8406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.471+0000 7fea977fe640 10 monclient: get_auth_request con 0x7fea8406c860 auth_method 0 2024-01-20T11:20:21.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.472+0000 7fea977fe640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea8406c860 0x7fea8406ed20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fea88009870 tx=0x7fea8803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:21.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.472+0000 7fea7ffff640 10 client.5356.objecter ms_handle_connect 0x7fea8406c860 2024-01-20T11:20:21.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.474+0000 7fea7ffff640 1 -- 172.21.15.110:0/1922369536 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fea8c058900 con 0x7fea980a9ad0 2024-01-20T11:20:21.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.474+0000 7fea7ffff640 10 client.5356.objecter ms_dispatch 0x7fea98000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:21.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.474+0000 7fea7ffff640 3 client.5356.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:21.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.474+0000 7fea7ffff640 3 client.5356.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:21.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.474+0000 7fea7ffff640 20 client.5356.objecter dump_active .. 0 homeless 2024-01-20T11:20:21.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.484+0000 7fea7ffff640 1 -- 172.21.15.110:0/1922369536 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fea8c054c40 con 0x7fea980a9ad0 2024-01-20T11:20:21.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.484+0000 7fea7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:21.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.484+0000 7fea7ffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:21.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.654+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea980b4400 con 0x7fea8406c860 2024-01-20T11:20:21.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.658+0000 7fea7ffff640 1 -- 172.21.15.110:0/1922369536 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64018 (secure 0 0 0) 0x7fea88031b10 con 0x7fea8406c860 2024-01-20T11:20:21.658 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:21.659 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:21.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 10 librados: watch_flush enter 2024-01-20T11:20:21.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 10 librados: watch_flush exit 2024-01-20T11:20:21.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 20 client.5356.objecter shutdown clearing up homeless session... 2024-01-20T11:20:21.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 10 client.5356.objecter successfully canceled tick 2024-01-20T11:20:21.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea8406c860 msgr2=0x7fea8406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea8406c860 0x7fea8406ed20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fea88009870 tx=0x7fea8803d040 comp rx=0 tx=0).stop 2024-01-20T11:20:21.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 10 monclient: shutdown 2024-01-20T11:20:21.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:21.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea980a9ad0 msgr2=0x7fea981d7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea980a9ad0 0x7fea981d7100 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fea980aac70 tx=0x7fea8c00ecd0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 shutdown_connections 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fea8406c860 0x7fea8406ed20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981da600 0x7fea981daab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea980a9ad0 0x7fea981d7100 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 --2- 172.21.15.110:0/1922369536 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea980a96f0 0x7fea9813e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 >> 172.21.15.110:0/1922369536 conn(0x7fea980b7bc0 msgr2=0x7fea981423e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 shutdown_connections 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 -- 172.21.15.110:0/1922369536 wait complete. 2024-01-20T11:20:21.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.659+0000 7fea9df2c640 1 librados: shutdown 2024-01-20T11:20:21.671 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-01-20T11:20:20.769485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002511"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280579,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.8170000000000002},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":11.154}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.976},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":17.707999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.176},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.146000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.7810000000000006},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.054}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.709},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":11.976000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7680000000000007},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.370000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313281,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.8890000000000002},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":8.5679999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.9199999999999999},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.186}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.804},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":7.9809999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.5270000000000001},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.648}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.438000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.2669999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.526999999999999},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.300000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313282,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.5099999999999998},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.3220000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.994},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.366}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.7770000000000001},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":11.736000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.327},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":8.8290000000000006},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":14.372999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":16.872},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.84}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444096,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.7320000000000002},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":7.375}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.4059999999999997},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.882999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.856},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.3010000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.927},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.162000000000001},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.9130000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.515000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.664000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444097,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.6420000000000003},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":7.1109999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.159000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":14.214},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.7590000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":8.4199999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.852}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.6110000000000007},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.926}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.194000000000001},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.250999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444097,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.7069999999999999},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":9.0150000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":4.9989999999999997},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.223000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":12.449},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.6400000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.888999999999999},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.96}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.733000000000001},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":10.456}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.407},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":13.096}]}]},{"osd":6,"up_from":15,"seq":64424509505,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":6.8559999999999999,"5min":9.343,"15min":10.968},"min":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.664,"5min":106.059,"15min":106.059},"last":5.2210000000000001},{"interface":"front","average":{"1min":8.2609999999999992,"5min":10.849,"15min":13.597},"min":{"1min":4.8250000000000002,"5min":3.96,"15min":3.96},"max":{"1min":14.807,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.6969999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9510000000000005,"5min":18.295000000000002,"15min":26.472999999999999},"min":{"1min":5.0629999999999997,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":16.399000000000001,"5min":189.37700000000001,"15min":189.37700000000001},"last":8.8819999999999997},{"interface":"front","average":{"1min":10.151,"5min":19.34,"15min":27.602},"min":{"1min":5.7599999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.751000000000001,"5min":190.304,"15min":190.304},"last":13.387}]},{"osd":2,"last update":"Sat Jan 20 11:19:19 2024","interfaces":[{"interface":"back","average":{"1min":11.041,"5min":12.308,"15min":14.199999999999999},"min":{"1min":6.3079999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":15.805999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":10.378},{"interface":"front","average":{"1min":10.598000000000001,"5min":13.395,"15min":15.243},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":16.323,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.4219999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":17.021000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.513}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.252000000000001},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.182}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.852},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":16.263999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:21.672 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:21.673 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:21.673 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:21.673 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:21.673 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:21.673 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:21.673 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:21.673 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:21.673 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:21.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.770+0000 7f35562e5640 10 monclient: get_monmap_and_config 2024-01-20T11:20:21.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.771+0000 7f35562e5640 10 monclient: build_initial_monmap 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.771+0000 7f35562e5640 10 monclient: monmap: 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:21.772611+0000 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:21.772611+0000 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:21.772 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.771+0000 7f35562e5640 1 Processor -- start 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.771+0000 7f35562e5640 1 -- start start 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.771+0000 7f35562e5640 10 monclient: init 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35500b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35500b1ed0 0x7f35500b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 10 monclient(hunting): picked mon.noname-b con 0x7f35500b1ed0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3550145980 0x7f3550145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 10 monclient(hunting): picked mon.noname-c con 0x7f3550145980 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3550146370 con 0x7f35500b4d10 2024-01-20T11:20:21.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35500b50f0 con 0x7f3550145980 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35500aade0 con 0x7f35500b1ed0 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.772+0000 7f35562e5640 10 monclient(hunting): authenticate will time out at 1665.123651s 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f354f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35500b1ed0 0x7f35500b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f3554882640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f3550145980 0x7f3550145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f354ffff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f35500b4d10 0x7f35500b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55126/0 (socket says 172.21.15.110:55126) 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f354ffff640 1 -- 172.21.15.110:0/1684062534 learned_addr learned my addr 172.21.15.110:0/1684062534 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f354f7fe640 10 monclient(hunting): get_auth_request con 0x7f35500b1ed0 auth_method 0 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f354f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f354f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f354f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f3554882640 10 monclient(hunting): get_auth_request con 0x7f3550145980 auth_method 0 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f3554882640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f3554882640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f3554882640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f3554882640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f3554882640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.773+0000 7f3554882640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.774+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.774+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.774+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.775+0000 7f3554882640 10 monclient(hunting): handle_auth_done global_id 5359 payload 274 2024-01-20T11:20:21.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.775+0000 7f3554882640 1 -- 172.21.15.110:0/1684062534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35500b1ed0 msgr2=0x7f35500b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:21.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.775+0000 7f3554882640 1 --2- 172.21.15.110:0/1684062534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35500b1ed0 0x7f35500b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.775+0000 7f3554882640 1 -- 172.21.15.110:0/1684062534 >> v1:172.21.15.110:6789/0 conn(0x7f35500b4d10 legacy=0x7f35500b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.775+0000 7f3554882640 10 monclient: _finish_hunting 0 2024-01-20T11:20:21.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.775+0000 7f3554882640 1 monclient: found mon.noname-c 2024-01-20T11:20:21.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.775+0000 7f3554882640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:21.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.775+0000 7f3554882640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:21.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.775+0000 7f3554882640 1 -- 172.21.15.110:0/1684062534 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3550145460 con 0x7f3550145980 2024-01-20T11:20:21.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.777+0000 7f3554882640 1 --2- 172.21.15.110:0/1684062534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3550145980 0x7f3550145e30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3548009d90 tx=0x7f354802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=917a6b9c32764fdc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:21.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 1 -- 172.21.15.110:0/1684062534 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354802cbc0 con 0x7f3550145980 2024-01-20T11:20:21.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:21.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 10 monclient: dump: 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 10 monclient: _finish_auth 0 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:21.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 1 -- 172.21.15.110:0/1684062534 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354802cd20 con 0x7f3550145980 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 5 monclient: authenticate success, global_id 5359 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 4 set_mon_vals no callback set 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 1 -- 172.21.15.110:0/1684062534 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548034660 con 0x7f3550145980 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f354effd640 10 monclient: dump: 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 10 monclient: shutdown 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 1 -- 172.21.15.110:0/1684062534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3550145980 msgr2=0x7f3550145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 1 --2- 172.21.15.110:0/1684062534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3550145980 0x7f3550145e30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3548009d90 tx=0x7f354802be10 comp rx=0 tx=0).stop 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 1 -- 172.21.15.110:0/1684062534 shutdown_connections 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 1 --2- 172.21.15.110:0/1684062534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3550145980 0x7f3550145e30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 1 --2- 172.21.15.110:0/1684062534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35500b1ed0 0x7f35500b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 1 -- 172.21.15.110:0/1684062534 >> 172.21.15.110:0/1684062534 conn(0x7f3550066590 msgr2=0x7f35500669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:21.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 1 -- 172.21.15.110:0/1684062534 shutdown_connections 2024-01-20T11:20:21.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.781+0000 7f35562e5640 1 -- 172.21.15.110:0/1684062534 wait complete. 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.782+0000 7f35562e5640 10 monclient: build_initial_monmap 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.782+0000 7f35562e5640 10 monclient: monmap: 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:21.789 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.782+0000 7f35562e5640 1 librados: starting msgr at 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.782+0000 7f35562e5640 1 librados: starting objecter 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.782+0000 7f35562e5640 1 Processor -- start 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 1 -- start start 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 1 librados: setting wanted keys 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 1 librados: calling monclient init 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient: init 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35500b1ed0 0x7f3550142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient(hunting): picked mon.noname-b con 0x7f35500b1ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35500b4d10 0x7f3550140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient(hunting): picked mon.noname-a con 0x7f35500b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3550145980 0x7f35501405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient(hunting): picked mon.noname-c con 0x7f3550145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3550140ae0 con 0x7f35500b4d10 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3550147a30 con 0x7f35500b1ed0 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3550146590 con 0x7f3550145980 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.783+0000 7f35562e5640 10 monclient(hunting): authenticate will time out at 1665.134859s 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35500b4d10 0x7f3550140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f3554882640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3550145980 0x7f35501405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35500b1ed0 0x7f3550142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f3554882640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3550145980 0x7f35501405a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:42466/0 (socket says 172.21.15.110:42466) 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f3554882640 1 -- 172.21.15.110:0/189930881 learned_addr learned my addr 172.21.15.110:0/189930881 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f3554882640 10 monclient(hunting): get_auth_request con 0x7f3550145980 auth_method 0 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f3554882640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f3554882640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f3554882640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354f7fe640 10 monclient(hunting): get_auth_request con 0x7f35500b4d10 auth_method 0 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354ffff640 10 monclient(hunting): get_auth_request con 0x7f35500b1ed0 auth_method 0 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.784+0000 7f354ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.785+0000 7f354ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.785+0000 7f354ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.785+0000 7f354ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.785+0000 7f3554882640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.785+0000 7f3554882640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.785+0000 7f3554882640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.785+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.785+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.785+0000 7f354f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 10 monclient(hunting): handle_auth_done global_id 5362 payload 824 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 1 -- 172.21.15.110:0/189930881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3550145980 msgr2=0x7f35501405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3550145980 0x7f35501405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 1 -- 172.21.15.110:0/189930881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35500b4d10 msgr2=0x7f3550140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35500b4d10 0x7f3550140060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 1 monclient: found mon.noname-b 2024-01-20T11:20:21.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:21.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:21.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.787+0000 7f354ffff640 1 -- 172.21.15.110:0/189930881 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3548009a70 con 0x7f35500b1ed0 2024-01-20T11:20:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.789+0000 7f354ffff640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35500b1ed0 0x7f3550142bb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f354400c370 tx=0x7f354400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.789+0000 7f354cff9640 10 client.?.objecter ms_handle_connect 0x7f35500b1ed0 2024-01-20T11:20:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.789+0000 7f354cff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 1 -- 172.21.15.110:0/189930881 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544005ce0 con 0x7f35500b1ed0 2024-01-20T11:20:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 10 monclient: dump: 2024-01-20T11:20:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 10 monclient: _finish_auth 0 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 10 monclient: _check_auth_tickets 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 1 -- 172.21.15.110:0/189930881 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3544005e40 con 0x7f35500b1ed0 2024-01-20T11:20:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 1 -- 172.21.15.110:0/189930881 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544007310 con 0x7f35500b1ed0 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f35562e5640 5 monclient: authenticate success, global_id 5362 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f35562e5640 10 monclient: _renew_subs 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f35562e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3550147c60 con 0x7f35500b1ed0 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354cff9640 10 monclient: dump: 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f354e7fc640 4 set_mon_vals no callback set 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f35562e5640 10 client.5362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f35562e5640 10 monclient: _renew_subs 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f35562e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3550148220 con 0x7f35500b1ed0 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.793+0000 7f35562e5640 1 librados: init done 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.794+0000 7f35562e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.794+0000 7f35562e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.794+0000 7f35562e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.794+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3550143150 con 0x7f35500b1ed0 2024-01-20T11:20:21.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.812+0000 7f354cff9640 1 -- 172.21.15.110:0/189930881 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3544007470 con 0x7f35500b1ed0 2024-01-20T11:20:21.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.812+0000 7f354cff9640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f353c06c860 0x7f353c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:21.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.812+0000 7f354f7fe640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f353c06c860 0x7f353c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:21.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.812+0000 7f354f7fe640 10 monclient: get_auth_request con 0x7f353c06c860 auth_method 0 2024-01-20T11:20:21.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.813+0000 7f354f7fe640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f353c06c860 0x7f353c06ed20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f35501478b0 tx=0x7f3540009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:21.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.813+0000 7f354cff9640 10 client.5362.objecter ms_handle_connect 0x7f353c06c860 2024-01-20T11:20:21.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.814+0000 7f354cff9640 1 -- 172.21.15.110:0/189930881 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3544007720 con 0x7f35500b1ed0 2024-01-20T11:20:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.814+0000 7f354cff9640 10 client.5362.objecter ms_dispatch 0x7f3550000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.814+0000 7f354cff9640 3 client.5362.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.814+0000 7f354cff9640 3 client.5362.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:21.814+0000 7f354cff9640 20 client.5362.objecter dump_active .. 0 homeless 2024-01-20T11:20:22.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.041+0000 7f354cff9640 1 -- 172.21.15.110:0/189930881 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f35440541d0 con 0x7f35500b1ed0 2024-01-20T11:20:22.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.041+0000 7f354cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:22.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.041+0000 7f354cff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:22.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.284+0000 7f35562e5640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:22.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.284+0000 7f35562e5640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:22.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.285+0000 7f35562e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:22.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.285+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3550075490 con 0x7f35500b1ed0 2024-01-20T11:20:22.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.290+0000 7f354cff9640 1 -- 172.21.15.110:0/189930881 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f3544017080 con 0x7f35500b1ed0 2024-01-20T11:20:22.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.290+0000 7f354cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:22.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.290+0000 7f354cff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:22.291 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:22.291 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":356,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 10 librados: watch_flush enter 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 10 librados: watch_flush exit 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 20 client.5362.objecter shutdown clearing up homeless session... 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 10 client.5362.objecter successfully canceled tick 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f353c06c860 msgr2=0x7f353c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f353c06c860 0x7f353c06ed20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f35501478b0 tx=0x7f3540009190 comp rx=0 tx=0).stop 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 10 monclient: shutdown 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35500b1ed0 msgr2=0x7f3550142bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35500b1ed0 0x7f3550142bb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f354400c370 tx=0x7f354400c830 comp rx=0 tx=0).stop 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 shutdown_connections 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f353c06c860 0x7f353c06ed20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3550145980 0x7f35501405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35500b4d10 0x7f3550140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 --2- 172.21.15.110:0/189930881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35500b1ed0 0x7f3550142bb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 >> 172.21.15.110:0/189930881 conn(0x7f3550066590 msgr2=0x7f35500b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:22.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 shutdown_connections 2024-01-20T11:20:22.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 -- 172.21.15.110:0/189930881 wait complete. 2024-01-20T11:20:22.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.292+0000 7f35562e5640 1 librados: shutdown 2024-01-20T11:20:22.304 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:22.304 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:22.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.405+0000 7fc5d99a4640 10 monclient: get_monmap_and_config 2024-01-20T11:20:22.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.405+0000 7fc5d99a4640 10 monclient: build_initial_monmap 2024-01-20T11:20:22.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.405+0000 7fc5d99a4640 10 monclient: monmap: 2024-01-20T11:20:22.406 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:22.406 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:22.406 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:22.407218+0000 2024-01-20T11:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:22.407218+0000 2024-01-20T11:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:22.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 1 Processor -- start 2024-01-20T11:20:22.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 1 -- start start 2024-01-20T11:20:22.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 10 monclient: init 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc5d40b4d10 0x7fc5d40b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 10 monclient(hunting): picked mon.noname-a con 0x7fc5d40b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5d40b17c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5d4145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d40aade0 con 0x7fc5d40b17c0 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d4145460 con 0x7fc5d4145980 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d40b50f0 con 0x7fc5d40b4d10 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.406+0000 7fc5d99a4640 10 monclient(hunting): authenticate will time out at 1665.758152s 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.407+0000 7fc5d2ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc5d40b4d10 0x7fc5d40b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.407+0000 7fc5d37fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc5d4145980 0x7fc5d4145e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:57402/0 (socket says 172.21.15.110:57402) 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.407+0000 7fc5d37fe640 1 -- 172.21.15.110:0/2980433221 learned_addr learned my addr 172.21.15.110:0/2980433221 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.407+0000 7fc5d2ffd640 10 monclient(hunting): get_auth_request con 0x7fc5d40b4d10 auth_method 0 2024-01-20T11:20:22.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.407+0000 7fc5d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:22.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.407+0000 7fc5d2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:22.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.407+0000 7fc5d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:22.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.409+0000 7fc5d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:22.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.409+0000 7fc5d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:22.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.409+0000 7fc5d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:22.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d27fc640 1 -- 172.21.15.110:0/2980433221 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290862487 0 0) 0x7fc5d4145460 con 0x7fc5d4145980 2024-01-20T11:20:22.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d27fc640 10 monclient(hunting): _negotiate 2024-01-20T11:20:22.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:22.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:22.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d27fc640 10 monclient(hunting): my global_id is 5365 2024-01-20T11:20:22.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d27fc640 1 -- 172.21.15.110:0/2980433221 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c0003520 con 0x7fc5d4145980 2024-01-20T11:20:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d2ffd640 10 monclient(hunting): handle_auth_done global_id 5280 payload 274 2024-01-20T11:20:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d2ffd640 1 -- 172.21.15.110:0/2980433221 >> v1:172.21.15.134:6789/0 conn(0x7fc5d4145980 legacy=0x7fc5d4145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d2ffd640 1 -- 172.21.15.110:0/2980433221 >> v1:172.21.15.110:6789/0 conn(0x7fc5d40b17c0 legacy=0x7fc5d40b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d2ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:20:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d2ffd640 1 monclient: found mon.noname-a 2024-01-20T11:20:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:22.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.410+0000 7fc5d2ffd640 1 -- 172.21.15.110:0/2980433221 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d40b2660 con 0x7fc5d40b4d10 2024-01-20T11:20:22.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.414+0000 7fc5d2ffd640 1 --2- 172.21.15.110:0/2980433221 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b4d10 0x7fc5d40b3da0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40097a0 tx=0x7fc5c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be3d144c3f13b9e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 1 -- 172.21.15.110:0/2980433221 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4031830 con 0x7fc5d40b4d10 2024-01-20T11:20:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 10 monclient: dump: 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 1 -- 172.21.15.110:0/2980433221 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c4031990 con 0x7fc5d40b4d10 2024-01-20T11:20:22.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 1 -- 172.21.15.110:0/2980433221 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4031d20 con 0x7fc5d40b4d10 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d27fc640 10 monclient: dump: 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d99a4640 5 monclient: authenticate success, global_id 5280 2024-01-20T11:20:22.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d99a4640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:22.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.418+0000 7fc5d99a4640 4 set_mon_vals no callback set 2024-01-20T11:20:22.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.419+0000 7fc5d99a4640 10 monclient: shutdown 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.419+0000 7fc5d99a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.419+0000 7fc5d99a4640 1 -- 172.21.15.110:0/2980433221 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b4d10 msgr2=0x7fc5d40b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.419+0000 7fc5d99a4640 1 --2- 172.21.15.110:0/2980433221 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b4d10 0x7fc5d40b3da0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40097a0 tx=0x7fc5c402b8d0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.419+0000 7fc5d99a4640 1 -- 172.21.15.110:0/2980433221 shutdown_connections 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.419+0000 7fc5d99a4640 1 --2- 172.21.15.110:0/2980433221 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b4d10 0x7fc5d40b3da0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.419+0000 7fc5d99a4640 1 -- 172.21.15.110:0/2980433221 >> 172.21.15.110:0/2980433221 conn(0x7fc5d4066590 msgr2=0x7fc5d40669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.419+0000 7fc5d99a4640 1 -- 172.21.15.110:0/2980433221 shutdown_connections 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.419+0000 7fc5d99a4640 1 -- 172.21.15.110:0/2980433221 wait complete. 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.420+0000 7fc5d99a4640 10 monclient: build_initial_monmap 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.420+0000 7fc5d99a4640 10 monclient: monmap: 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:22.425 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.420+0000 7fc5d99a4640 1 librados: starting msgr at 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.420+0000 7fc5d99a4640 1 librados: starting objecter 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.420+0000 7fc5d99a4640 1 Processor -- start 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.420+0000 7fc5d99a4640 1 -- start start 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.420+0000 7fc5d99a4640 1 librados: setting wanted keys 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.420+0000 7fc5d99a4640 1 librados: calling monclient init 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.420+0000 7fc5d99a4640 10 monclient: init 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b17c0 0x7fc5d413cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:22.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient(hunting): picked mon.noname-a con 0x7fc5d40b17c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5d40b4d10 0x7fc5d413d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient(hunting): picked mon.noname-c con 0x7fc5d40b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5d4145980 0x7fc5d413afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient(hunting): picked mon.noname-b con 0x7fc5d4145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d413e200 con 0x7fc5d40b17c0 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d413da40 con 0x7fc5d4145980 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d413e690 con 0x7fc5d40b4d10 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d99a4640 10 monclient(hunting): authenticate will time out at 1665.772577s 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5d2ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b17c0 0x7fc5d413cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.421+0000 7fc5cbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5d40b4d10 0x7fc5d413d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d2ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b17c0 0x7fc5d413cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:55530/0 (socket says 172.21.15.110:55530) 2024-01-20T11:20:22.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d37fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5d4145980 0x7fc5d413afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d2ffd640 1 -- 172.21.15.110:0/1933606824 learned_addr learned my addr 172.21.15.110:0/1933606824 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d2ffd640 10 monclient(hunting): get_auth_request con 0x7fc5d40b17c0 auth_method 0 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5cbfff640 10 monclient(hunting): get_auth_request con 0x7fc5d40b4d10 auth_method 0 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d37fe640 10 monclient(hunting): get_auth_request con 0x7fc5d4145980 auth_method 0 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:22.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:22.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:22.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:22.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:22.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:22.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.422+0000 7fc5cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:22.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.423+0000 7fc5d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:22.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.423+0000 7fc5d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:22.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.423+0000 7fc5d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 10 monclient(hunting): handle_auth_done global_id 5368 payload 824 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 1 -- 172.21.15.110:0/1933606824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5d40b4d10 msgr2=0x7fc5d413d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5d40b4d10 0x7fc5d413d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 1 -- 172.21.15.110:0/1933606824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b17c0 msgr2=0x7fc5d413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b17c0 0x7fc5d413cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 1 monclient: found mon.noname-b 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d37fe640 1 -- 172.21.15.110:0/1933606824 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c4009400 con 0x7fc5d4145980 2024-01-20T11:20:22.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.425+0000 7fc5d2ffd640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b17c0 0x7fc5d413cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:20:22.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.427+0000 7fc5d37fe640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5d4145980 0x7fc5d413afc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00caa0 tx=0x7fc5cc00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:22.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.427+0000 7fc5d0ff9640 10 client.?.objecter ms_handle_connect 0x7fc5d4145980 2024-01-20T11:20:22.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.427+0000 7fc5d0ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:22.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 1 -- 172.21.15.110:0/1933606824 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc00aae0 con 0x7fc5d4145980 2024-01-20T11:20:22.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:22.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:22.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 10 monclient: dump: 2024-01-20T11:20:22.432 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:22.432 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:22.432 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 10 monclient: _finish_auth 0 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 1 -- 172.21.15.110:0/1933606824 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc00ac40 con 0x7fc5d4145980 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 1 -- 172.21.15.110:0/1933606824 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc012400 con 0x7fc5d4145980 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d99a4640 5 monclient: authenticate success, global_id 5368 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d99a4640 10 monclient: _renew_subs 2024-01-20T11:20:22.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d99a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d413dce0 con 0x7fc5d4145980 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d1ffb640 4 set_mon_vals no callback set 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d0ff9640 10 monclient: dump: 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d99a4640 10 client.5368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:22.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d99a4640 10 monclient: _renew_subs 2024-01-20T11:20:22.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d99a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:22.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d413b6e0 con 0x7fc5d4145980 2024-01-20T11:20:22.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.431+0000 7fc5d99a4640 1 librados: init done 2024-01-20T11:20:22.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.432+0000 7fc5d99a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:22.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.432+0000 7fc5d99a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:22.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.432+0000 7fc5d99a4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:22.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.432+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d413e370 con 0x7fc5d4145980 2024-01-20T11:20:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.450+0000 7fc5d0ff9640 1 -- 172.21.15.110:0/1933606824 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc5cc007070 con 0x7fc5d4145980 2024-01-20T11:20:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.451+0000 7fc5d0ff9640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b406c860 0x7fc5b406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.451+0000 7fc5d2ffd640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b406c860 0x7fc5b406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.451+0000 7fc5d2ffd640 10 monclient: get_auth_request con 0x7fc5b406c860 auth_method 0 2024-01-20T11:20:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.451+0000 7fc5d0ff9640 10 client.5368.objecter ms_handle_connect 0x7fc5b406c860 2024-01-20T11:20:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.451+0000 7fc5d2ffd640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b406c860 0x7fc5b406ed20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc5c402b910 tx=0x7fc5c403c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:22.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.671+0000 7fc5d0ff9640 1 -- 172.21.15.110:0/1933606824 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc5cc058450 con 0x7fc5d4145980 2024-01-20T11:20:22.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.671+0000 7fc5d0ff9640 10 client.5368.objecter ms_dispatch 0x7fc5d4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:22.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.671+0000 7fc5d0ff9640 3 client.5368.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:22.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.671+0000 7fc5d0ff9640 3 client.5368.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:22.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.671+0000 7fc5d0ff9640 20 client.5368.objecter dump_active .. 0 homeless 2024-01-20T11:20:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.682+0000 7fc5d0ff9640 1 -- 172.21.15.110:0/1933606824 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc5cc054790 con 0x7fc5d4145980 2024-01-20T11:20:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.682+0000 7fc5d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:22.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.682+0000 7fc5d0ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:22.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.853+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5d40a9cf0 con 0x7fc5b406c860 2024-01-20T11:20:22.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.856+0000 7fc5d0ff9640 1 -- 172.21.15.110:0/1933606824 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64079 (secure 0 0 0) 0x7fc5c403a420 con 0x7fc5b406c860 2024-01-20T11:20:22.857 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:22.857 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 10 librados: watch_flush enter 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 10 librados: watch_flush exit 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 20 client.5368.objecter shutdown clearing up homeless session... 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 10 client.5368.objecter successfully canceled tick 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b406c860 msgr2=0x7fc5b406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b406c860 0x7fc5b406ed20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc5c402b910 tx=0x7fc5c403c040 comp rx=0 tx=0).stop 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 10 monclient: shutdown 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5d4145980 msgr2=0x7fc5d413afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:22.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5d4145980 0x7fc5d413afc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00caa0 tx=0x7fc5cc00cf60 comp rx=0 tx=0).stop 2024-01-20T11:20:22.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 shutdown_connections 2024-01-20T11:20:22.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc5b406c860 0x7fc5b406ed20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5d4145980 0x7fc5d413afc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5d40b4d10 0x7fc5d413d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 --2- 172.21.15.110:0/1933606824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5d40b17c0 0x7fc5d413cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:22.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 >> 172.21.15.110:0/1933606824 conn(0x7fc5d4066590 msgr2=0x7fc5d413a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:22.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 shutdown_connections 2024-01-20T11:20:22.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.857+0000 7fc5d99a4640 1 -- 172.21.15.110:0/1933606824 wait complete. 2024-01-20T11:20:22.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:22.858+0000 7fc5d99a4640 1 librados: shutdown 2024-01-20T11:20:22.869 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-01-20T11:20:22.769919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002534"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280579,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.8170000000000002},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":11.154}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.976},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":17.707999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.176},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.146000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.7810000000000006},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":15.054}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.709},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":11.976000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.7680000000000007},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":16.370000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313282,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":11.94},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.8129999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.1449999999999996},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":7.4199999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.492000000000001},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.917999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.1170000000000009},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.4209999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.02},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":13.167999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.191000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":10.997}]}]},{"osd":3,"up_from":21,"seq":90194313283,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":9.4410000000000007},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6630000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.6870000000000003},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.869}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":11.315},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":8.5630000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":15.872999999999999},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.108000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":13.994},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.927}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":14.413},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.279999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444096,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.0820000000000007,"5min":9.1769999999999996,"15min":9.9969999999999999},"min":{"1min":4.3079999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":12.111000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.7320000000000002},{"interface":"front","average":{"1min":8.2439999999999998,"5min":8.6440000000000001,"15min":8.6259999999999994},"min":{"1min":3.9630000000000001,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":14.657999999999999,"5min":16.163,"15min":16.163},"last":7.375}]},{"osd":1,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":8.3369999999999997,"5min":10.922000000000001,"15min":13.167999999999999},"min":{"1min":4.7720000000000002,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.331,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.4059999999999997},{"interface":"front","average":{"1min":11.217000000000001,"5min":13.571,"15min":15.741},"min":{"1min":7.218,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":15.714,"5min":32.414999999999999,"15min":32.414999999999999},"last":13.882999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.856},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.3010000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.927},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:19:18 2024","interfaces":[{"interface":"back","average":{"1min":13.218,"5min":14.569000000000001,"15min":15.573},"min":{"1min":9.0980000000000008,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":17.364999999999998,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.162000000000001},{"interface":"front","average":{"1min":11.19,"5min":10.773999999999999,"15min":10.890000000000001},"min":{"1min":6.7220000000000004,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":16.632000000000001,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.9130000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":10.515000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":16.664000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444097,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":8.4130000000000003,"5min":9.0009999999999994,"15min":9.4930000000000003},"min":{"1min":5.0170000000000003,"5min":4.6379999999999999,"15min":4.6379999999999999},"max":{"1min":14.347,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.6420000000000003},{"interface":"front","average":{"1min":8.2720000000000002,"5min":9.3610000000000007,"15min":10.183999999999999},"min":{"1min":5.8959999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":16.992999999999999,"5min":30.439,"15min":30.439},"last":7.1109999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.125999999999999,"5min":12.128,"15min":13.004},"min":{"1min":6.891,"5min":6.351,"15min":6.351},"max":{"1min":12.380000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.047000000000001,"5min":12.087999999999999,"15min":13.095000000000001},"min":{"1min":7.1699999999999999,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.843999999999999,"5min":23.981999999999999,"15min":23.981999999999999},"last":12.159000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":14.214},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.7590000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":8.4199999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.852}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:19:17 2024","interfaces":[{"interface":"back","average":{"1min":10.198,"5min":11.512,"15min":12.026},"min":{"1min":5.8780000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.209,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.6110000000000007},{"interface":"front","average":{"1min":13.682,"5min":14.634,"15min":15.398},"min":{"1min":8.4250000000000007,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.094999999999999,"5min":31.634,"15min":31.634},"last":12.926}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.194000000000001},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.250999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444097,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.7069999999999999},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":9.0150000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":4.9989999999999997},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.223000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":12.449},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.6400000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.888999999999999},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.96}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.733000000000001},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":10.456}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.407},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":13.096}]}]},{"osd":6,"up_from":15,"seq":64424509506,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":6.8689999999999998},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":10.583}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.0060000000000002},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":12.363}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.625},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.5810000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":9.1780000000000008},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.760999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.039999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":9.9689999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.544},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.074}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:24.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~87s 2024-01-20T11:20:25.871 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.972+0000 7f2c1759e640 10 monclient: get_monmap_and_config 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.972+0000 7f2c1759e640 10 monclient: build_initial_monmap 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.972+0000 7f2c1759e640 10 monclient: monmap: 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:25.973744+0000 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:25.973744+0000 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:25.974 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.972+0000 7f2c1759e640 1 Processor -- start 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.972+0000 7f2c1759e640 1 -- start start 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.972+0000 7f2c1759e640 10 monclient: init 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18072660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 10 monclient(hunting): picked mon.noname-b con 0x7f2c1806cd80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2c1806bfe0 0x7f2c1813ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 10 monclient(hunting): picked mon.noname-a con 0x7f2c1806bfe0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:25.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c1813f450 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c180ae790 con 0x7f2c1813f450 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1806d160 con 0x7f2c1806bfe0 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1806d380 con 0x7f2c1806cd80 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.973+0000 7f2c1759e640 10 monclient(hunting): authenticate will time out at 1669.324767s 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c1659c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18072660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c15d9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2c1806bfe0 0x7f2c1813ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c1659c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18072660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:42476/0 (socket says 172.21.15.110:42476) 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c1659c640 1 -- 172.21.15.110:0/1197809437 learned_addr learned my addr 172.21.15.110:0/1197809437 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c1659c640 10 monclient(hunting): get_auth_request con 0x7f2c1806cd80 auth_method 0 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c1659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c1659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:25.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c15d9b640 10 monclient(hunting): get_auth_request con 0x7f2c1806bfe0 auth_method 0 2024-01-20T11:20:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c15d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.974+0000 7f2c15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.975+0000 7f2c1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.975+0000 7f2c1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.975+0000 7f2c1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.975+0000 7f2c15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.975+0000 7f2c15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.975+0000 7f2c15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.976+0000 7f2c1659c640 10 monclient(hunting): handle_auth_done global_id 5057 payload 274 2024-01-20T11:20:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.976+0000 7f2c1659c640 1 -- 172.21.15.110:0/1197809437 >> v2:172.21.15.110:3300/0 conn(0x7f2c1806bfe0 msgr2=0x7f2c1813ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.976+0000 7f2c1659c640 1 --2- 172.21.15.110:0/1197809437 >> v2:172.21.15.110:3300/0 conn(0x7f2c1806bfe0 0x7f2c1813ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.976+0000 7f2c1659c640 1 -- 172.21.15.110:0/1197809437 >> v1:172.21.15.110:6789/0 conn(0x7f2c1813f450 legacy=0x7f2c180aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.976+0000 7f2c1659c640 10 monclient: _finish_hunting 0 2024-01-20T11:20:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.976+0000 7f2c1659c640 1 monclient: found mon.noname-b 2024-01-20T11:20:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.976+0000 7f2c1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.976+0000 7f2c1659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.976+0000 7f2c1659c640 1 -- 172.21.15.110:0/1197809437 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1813f880 con 0x7f2c1806cd80 2024-01-20T11:20:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.978+0000 7f2c1659c640 1 --2- 172.21.15.110:0/1197809437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18072660 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009870 tx=0x7f2c0802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=215fcbf4ab0fb7b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 1 -- 172.21.15.110:0/1197809437 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08031640 con 0x7f2c1806cd80 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 10 monclient: dump: 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:25.983 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 10 monclient: _finish_auth 0 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 10 monclient: _check_auth_tickets 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 1 -- 172.21.15.110:0/1197809437 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c080317a0 con 0x7f2c1806cd80 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1759e640 5 monclient: authenticate success, global_id 5057 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 1 -- 172.21.15.110:0/1197809437 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08031b30 con 0x7f2c1806cd80 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1759e640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1759e640 4 set_mon_vals no callback set 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1559a640 10 monclient: dump: 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:25.985 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:25.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1759e640 10 monclient: shutdown 2024-01-20T11:20:25.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1759e640 1 -- 172.21.15.110:0/1197809437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 msgr2=0x7f2c18072660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1759e640 1 --2- 172.21.15.110:0/1197809437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18072660 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009870 tx=0x7f2c0802b8f0 comp rx=0 tx=0).stop 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.982+0000 7f2c1759e640 1 -- 172.21.15.110:0/1197809437 shutdown_connections 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.983+0000 7f2c1759e640 1 --2- 172.21.15.110:0/1197809437 >> v2:172.21.15.110:3300/0 conn(0x7f2c1806bfe0 0x7f2c1813ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.983+0000 7f2c1759e640 1 --2- 172.21.15.110:0/1197809437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18072660 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.983+0000 7f2c1759e640 1 -- 172.21.15.110:0/1197809437 >> 172.21.15.110:0/1197809437 conn(0x7f2c180b59c0 msgr2=0x7f2c180b5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.983+0000 7f2c1759e640 1 -- 172.21.15.110:0/1197809437 shutdown_connections 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.983+0000 7f2c1759e640 1 -- 172.21.15.110:0/1197809437 wait complete. 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.983+0000 7f2c1759e640 10 monclient: build_initial_monmap 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.983+0000 7f2c1759e640 10 monclient: monmap: 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 librados: starting msgr at 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 librados: starting objecter 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 Processor -- start 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 -- start start 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 librados: setting wanted keys 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 librados: calling monclient init 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient: init 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c1806bfe0 0x7f2c180b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient(hunting): picked mon.noname-a con 0x7f2c1806bfe0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18131230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient(hunting): picked mon.noname-c con 0x7f2c1806cd80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c1813f450 0x7f2c181d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient(hunting): picked mon.noname-b con 0x7f2c1813f450 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1806f650 con 0x7f2c1806bfe0 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c181d7530 con 0x7f2c1813f450 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c181d7810 con 0x7f2c1806cd80 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.984+0000 7f2c1759e640 10 monclient(hunting): authenticate will time out at 1669.336090s 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c15d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18131230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c1659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c1806bfe0 0x7f2c180b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c16d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c1813f450 0x7f2c181d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:25.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c15d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18131230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:42490/0 (socket says 172.21.15.110:42490) 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c15d9b640 1 -- 172.21.15.110:0/2804227934 learned_addr learned my addr 172.21.15.110:0/2804227934 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c15d9b640 10 monclient(hunting): get_auth_request con 0x7f2c1806cd80 auth_method 0 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c15d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c1659c640 10 monclient(hunting): get_auth_request con 0x7f2c1806bfe0 auth_method 0 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c1659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c1659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.985+0000 7f2c16d9d640 10 monclient(hunting): get_auth_request con 0x7f2c1813f450 auth_method 0 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c16d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:25.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:25.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:25.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:25.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:25.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:25.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:25.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.986+0000 7f2c16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 10 monclient(hunting): handle_auth_done global_id 5060 payload 824 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 1 -- 172.21.15.110:0/2804227934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c1813f450 msgr2=0x7f2c181d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c1813f450 0x7f2c181d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 1 -- 172.21.15.110:0/2804227934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c1806bfe0 msgr2=0x7f2c180b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c1806bfe0 0x7f2c180b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 1 monclient: found mon.noname-c 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.988+0000 7f2c15d9b640 1 -- 172.21.15.110:0/2804227934 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080094d0 con 0x7f2c1806cd80 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.990+0000 7f2c15d9b640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18131230 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00e9b0 tx=0x7f2c0c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:25.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.990+0000 7f2c06ffd640 10 client.?.objecter ms_handle_connect 0x7f2c1806cd80 2024-01-20T11:20:25.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.990+0000 7f2c06ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:25.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 1 -- 172.21.15.110:0/2804227934 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c009670 con 0x7f2c1806cd80 2024-01-20T11:20:25.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 10 monclient: dump: 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:25.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 1 -- 172.21.15.110:0/2804227934 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c00c9a0 con 0x7f2c1806cd80 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 1 -- 172.21.15.110:0/2804227934 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c00cd30 con 0x7f2c1806cd80 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c1759e640 5 monclient: authenticate success, global_id 5060 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c1759e640 10 monclient: _renew_subs 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c1759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1806f8b0 con 0x7f2c1806cd80 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c14d99640 4 set_mon_vals no callback set 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c06ffd640 10 monclient: dump: 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:25.997 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c1759e640 10 client.5060.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c1759e640 10 monclient: _renew_subs 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c1759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c18132db0 con 0x7f2c1806cd80 2024-01-20T11:20:25.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.994+0000 7f2c1759e640 1 librados: init done 2024-01-20T11:20:25.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.995+0000 7f2c1759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:25.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.995+0000 7f2c1759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:25.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.995+0000 7f2c1759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:25.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:25.995+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c180ad9e0 con 0x7f2c1806cd80 2024-01-20T11:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.012+0000 7f2c06ffd640 1 -- 172.21.15.110:0/2804227934 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2c0c013470 con 0x7f2c1806cd80 2024-01-20T11:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.013+0000 7f2c06ffd640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8b0 0x7f2bf806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.013+0000 7f2c1659c640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8b0 0x7f2bf806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.013+0000 7f2c1659c640 10 monclient: get_auth_request con 0x7f2bf806c8b0 auth_method 0 2024-01-20T11:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.013+0000 7f2c1659c640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8b0 0x7f2bf806ed70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2c08036000 tx=0x7f2c0803c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.013+0000 7f2c06ffd640 10 client.5060.objecter ms_handle_connect 0x7f2bf806c8b0 2024-01-20T11:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.015+0000 7f2c06ffd640 1 -- 172.21.15.110:0/2804227934 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2c0c058b40 con 0x7f2c1806cd80 2024-01-20T11:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.015+0000 7f2c06ffd640 10 client.5060.objecter ms_dispatch 0x7f2c18000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.015+0000 7f2c06ffd640 3 client.5060.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.015+0000 7f2c06ffd640 3 client.5060.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.015+0000 7f2c06ffd640 20 client.5060.objecter dump_active .. 0 homeless 2024-01-20T11:20:26.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.245+0000 7f2c06ffd640 1 -- 172.21.15.110:0/2804227934 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2c0c054e80 con 0x7f2c1806cd80 2024-01-20T11:20:26.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.245+0000 7f2c06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:26.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.245+0000 7f2c06ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:26.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.427+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c18075490 con 0x7f2bf806c8b0 2024-01-20T11:20:26.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.430+0000 7f2c06ffd640 1 -- 172.21.15.110:0/2804227934 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64044 (secure 0 0 0) 0x7f2c18075490 con 0x7f2bf806c8b0 2024-01-20T11:20:26.431 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:26.431 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:26.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.431+0000 7f2c1759e640 10 librados: watch_flush enter 2024-01-20T11:20:26.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.431+0000 7f2c1759e640 10 librados: watch_flush exit 2024-01-20T11:20:26.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.431+0000 7f2c1759e640 20 client.5060.objecter shutdown clearing up homeless session... 2024-01-20T11:20:26.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.431+0000 7f2c1759e640 10 client.5060.objecter successfully canceled tick 2024-01-20T11:20:26.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.431+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8b0 msgr2=0x7f2bf806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:26.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.431+0000 7f2c1759e640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8b0 0x7f2bf806ed70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2c08036000 tx=0x7f2c0803c040 comp rx=0 tx=0).stop 2024-01-20T11:20:26.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 10 monclient: shutdown 2024-01-20T11:20:26.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:26.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 msgr2=0x7f2c18131230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18131230 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00e9b0 tx=0x7f2c0c00ee70 comp rx=0 tx=0).stop 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 shutdown_connections 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8b0 0x7f2bf806ed70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c1813f450 0x7f2c181d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c1806cd80 0x7f2c18131230 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 --2- 172.21.15.110:0/2804227934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c1806bfe0 0x7f2c180b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 >> 172.21.15.110:0/2804227934 conn(0x7f2c180b59c0 msgr2=0x7f2c180b5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 shutdown_connections 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 -- 172.21.15.110:0/2804227934 wait complete. 2024-01-20T11:20:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.432+0000 7f2c1759e640 1 librados: shutdown 2024-01-20T11:20:26.443 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-01-20T11:20:24.770364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002560"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280580,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.5810000000000004},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.1219999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.8520000000000003},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.4080000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":5.6879999999999997},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.417999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.5830000000000002},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":11.128}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.73},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":13.103999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.803000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":12.374000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313282,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":11.94},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.8129999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.1449999999999996},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":7.4199999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.492000000000001},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.917999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.1170000000000009},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.4209999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.02},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":13.167999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.191000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":10.997}]}]},{"osd":3,"up_from":21,"seq":90194313283,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":9.4410000000000007},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6630000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.6870000000000003},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.869}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":11.315},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":8.5630000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":15.872999999999999},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.108000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":13.994},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.927}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":14.413},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.279999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444097,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.29},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":4.4770000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.513999999999999},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.9559999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.32},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.944000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.782999999999999},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.347}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":9.6289999999999996},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.629}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.127000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":15.948}]}]},{"osd":1,"up_from":17,"seq":73014444098,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.605},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":15.962999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":15.167999999999999},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.5579999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":9.9469999999999992},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.3399999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":6.8109999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":10.932}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.1360000000000001},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":13.063000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.238},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.343}]}]},{"osd":0,"up_from":17,"seq":73014444098,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.0229999999999997},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":6.6680000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.5759999999999996},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.095000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":12.849},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.092000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":8.4920000000000009},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":14.199}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.475},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":13.837}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.57},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":16.960000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509506,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":6.8689999999999998},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":10.583}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.0060000000000002},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":12.363}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.625},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.5810000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":9.1780000000000008},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.760999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.039999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":9.9689999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":14.544},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.074}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:26.444 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:26.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.544+0000 7f5149b47640 10 monclient: get_monmap_and_config 2024-01-20T11:20:26.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.544+0000 7f5149b47640 10 monclient: build_initial_monmap 2024-01-20T11:20:26.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.544+0000 7f5149b47640 10 monclient: monmap: 2024-01-20T11:20:26.545 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:26.545 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:26.545 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:26.546201+0000 2024-01-20T11:20:26.545 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:26.546201+0000 2024-01-20T11:20:26.545 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:26.545 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:26.546 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:26.546 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:26.546 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:26.546 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:26.546 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:26.546 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:26.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.545+0000 7f5149b47640 1 Processor -- start 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.545+0000 7f5149b47640 1 -- start start 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.545+0000 7f5149b47640 10 monclient: init 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f514406a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f51440af730 0x7f51441455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 10 monclient(hunting): picked mon.noname-a con 0x7f51440af730 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5144145af0 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51440afc50 con 0x7f5144145af0 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51440b0170 con 0x7f514406a860 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514406e450 con 0x7f51440af730 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5149b47640 10 monclient(hunting): authenticate will time out at 1669.897323s 2024-01-20T11:20:26.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f5142ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f51440af730 0x7f51441455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:26.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f51437fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f514406a860 0x7f514413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:57404/0 (socket says 172.21.15.110:57404) 2024-01-20T11:20:26.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.546+0000 7f51437fe640 1 -- 172.21.15.110:0/105029977 learned_addr learned my addr 172.21.15.110:0/105029977 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:26.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.547+0000 7f5142ffd640 10 monclient(hunting): get_auth_request con 0x7f51440af730 auth_method 0 2024-01-20T11:20:26.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.547+0000 7f5142ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:26.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.547+0000 7f5142ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:26.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.547+0000 7f5142ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:26.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.547+0000 7f5142ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:26.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.547+0000 7f5142ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:26.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.547+0000 7f5142ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:26.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.549+0000 7f5142ffd640 10 monclient(hunting): handle_auth_done global_id 5295 payload 274 2024-01-20T11:20:26.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.549+0000 7f5142ffd640 1 -- 172.21.15.110:0/105029977 >> v1:172.21.15.134:6789/0 conn(0x7f514406a860 legacy=0x7f514413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:26.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.549+0000 7f5142ffd640 1 -- 172.21.15.110:0/105029977 >> v1:172.21.15.110:6789/0 conn(0x7f5144145af0 legacy=0x7f514414c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:26.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.549+0000 7f5142ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:20:26.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.549+0000 7f5142ffd640 1 monclient: found mon.noname-a 2024-01-20T11:20:26.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.549+0000 7f5142ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:26.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.549+0000 7f5142ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:26.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.549+0000 7f5142ffd640 1 -- 172.21.15.110:0/105029977 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514413fb60 con 0x7f51440af730 2024-01-20T11:20:26.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.554+0000 7f5142ffd640 1 --2- 172.21.15.110:0/105029977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51440af730 0x7f51441455b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f51380097a0 tx=0x7f513802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb1296c536be151 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:26.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 1 -- 172.21.15.110:0/105029977 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513802e070 con 0x7f51440af730 2024-01-20T11:20:26.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 10 monclient: dump: 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:26.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 1 -- 172.21.15.110:0/105029977 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51380315a0 con 0x7f51440af730 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 1 -- 172.21.15.110:0/105029977 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138031930 con 0x7f51440af730 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f51427fc640 10 monclient: dump: 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:26.560 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:26.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f5149b47640 5 monclient: authenticate success, global_id 5295 2024-01-20T11:20:26.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f5149b47640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:26.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.557+0000 7f5149b47640 4 set_mon_vals no callback set 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.558+0000 7f5149b47640 10 monclient: shutdown 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.558+0000 7f5149b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.558+0000 7f5149b47640 1 -- 172.21.15.110:0/105029977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51440af730 msgr2=0x7f51441455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.558+0000 7f5149b47640 1 --2- 172.21.15.110:0/105029977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51440af730 0x7f51441455b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f51380097a0 tx=0x7f513802b840 comp rx=0 tx=0).stop 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.558+0000 7f5149b47640 1 -- 172.21.15.110:0/105029977 shutdown_connections 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.558+0000 7f5149b47640 1 --2- 172.21.15.110:0/105029977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51440af730 0x7f51441455b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.558+0000 7f5149b47640 1 -- 172.21.15.110:0/105029977 >> 172.21.15.110:0/105029977 conn(0x7f51440b47d0 msgr2=0x7f51440b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.558+0000 7f5149b47640 1 -- 172.21.15.110:0/105029977 shutdown_connections 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.558+0000 7f5149b47640 1 -- 172.21.15.110:0/105029977 wait complete. 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.559+0000 7f5149b47640 10 monclient: build_initial_monmap 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.559+0000 7f5149b47640 10 monclient: monmap: 2024-01-20T11:20:26.564 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.559+0000 7f5149b47640 1 librados: starting msgr at 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.559+0000 7f5149b47640 1 librados: starting objecter 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.559+0000 7f5149b47640 1 Processor -- start 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 1 -- start start 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 1 librados: setting wanted keys 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 1 librados: calling monclient init 2024-01-20T11:20:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient: init 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f514406a860 0x7f514406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient(hunting): picked mon.noname-a con 0x7f514406a860 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51440af730 0x7f5144070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient(hunting): picked mon.noname-c con 0x7f51440af730 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5144145af0 0x7f51441d7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient(hunting): picked mon.noname-b con 0x7f5144145af0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5144071360 con 0x7f514406a860 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5144070ed0 con 0x7f5144145af0 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5144071af0 con 0x7f51440af730 2024-01-20T11:20:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.560+0000 7f5149b47640 10 monclient(hunting): authenticate will time out at 1669.911802s 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f514406a860 0x7f514406ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5142ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51440af730 0x7f5144070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5143fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5144145af0 0x7f51441d7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f514406a860 0x7f514406ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:55564/0 (socket says 172.21.15.110:55564) 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 1 -- 172.21.15.110:0/214772602 learned_addr learned my addr 172.21.15.110:0/214772602 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 10 monclient(hunting): get_auth_request con 0x7f514406a860 auth_method 0 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5142ffd640 10 monclient(hunting): get_auth_request con 0x7f51440af730 auth_method 0 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5142ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5142ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5142ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5143fff640 10 monclient(hunting): get_auth_request con 0x7f5144145af0 auth_method 0 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5143fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5143fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f5143fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.561+0000 7f51437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.562+0000 7f5142ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.562+0000 7f5142ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.562+0000 7f5142ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.562+0000 7f5143fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.562+0000 7f5143fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:26.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.562+0000 7f5143fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 10 monclient(hunting): handle_auth_done global_id 5301 payload 824 2024-01-20T11:20:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 1 -- 172.21.15.110:0/214772602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51440af730 msgr2=0x7f5144070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51440af730 0x7f5144070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 1 -- 172.21.15.110:0/214772602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5144145af0 msgr2=0x7f51441d7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5144145af0 0x7f51441d7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 1 monclient: found mon.noname-a 2024-01-20T11:20:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.564+0000 7f51437fe640 1 -- 172.21.15.110:0/214772602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5138009400 con 0x7f514406a860 2024-01-20T11:20:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.567+0000 7f51437fe640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f514406a860 0x7f514406ff30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f513400c4b0 tx=0x7f513400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.567+0000 7f512bfff640 10 client.?.objecter ms_handle_connect 0x7f514406a860 2024-01-20T11:20:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.567+0000 7f512bfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 1 -- 172.21.15.110:0/214772602 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134005ce0 con 0x7f514406a860 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 10 monclient: dump: 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:26.572 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 10 monclient: _finish_auth 0 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 1 -- 172.21.15.110:0/214772602 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5134005e40 con 0x7f514406a860 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5149b47640 5 monclient: authenticate success, global_id 5301 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5149b47640 10 monclient: _renew_subs 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5149b47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5144140030 con 0x7f514406a860 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 1 -- 172.21.15.110:0/214772602 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134007270 con 0x7f514406a860 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f512bfff640 10 monclient: dump: 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5141ffb640 4 set_mon_vals no callback set 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5149b47640 10 client.5301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5149b47640 10 monclient: _renew_subs 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5149b47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51441d7c40 con 0x7f514406a860 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.570+0000 7f5149b47640 1 librados: init done 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.571+0000 7f5149b47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.571+0000 7f5149b47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.571+0000 7f5149b47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.571+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5144075490 con 0x7f514406a860 2024-01-20T11:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.811+0000 7f512bfff640 1 -- 172.21.15.110:0/214772602 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f51340073d0 con 0x7f514406a860 2024-01-20T11:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.811+0000 7f512bfff640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f513006c8b0 0x7f513006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.811+0000 7f5142ffd640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f513006c8b0 0x7f513006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.811+0000 7f5142ffd640 10 monclient: get_auth_request con 0x7f513006c8b0 auth_method 0 2024-01-20T11:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.812+0000 7f5142ffd640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f513006c8b0 0x7f513006ed70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5138036000 tx=0x7f513803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.812+0000 7f512bfff640 10 client.5301.objecter ms_handle_connect 0x7f513006c8b0 2024-01-20T11:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.828+0000 7f512bfff640 1 -- 172.21.15.110:0/214772602 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5134057c30 con 0x7f514406a860 2024-01-20T11:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.828+0000 7f512bfff640 10 client.5301.objecter ms_dispatch 0x7f5144000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.828+0000 7f512bfff640 3 client.5301.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.828+0000 7f512bfff640 3 client.5301.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.828+0000 7f512bfff640 20 client.5301.objecter dump_active .. 0 homeless 2024-01-20T11:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.828+0000 7f512bfff640 1 -- 172.21.15.110:0/214772602 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5134053f70 con 0x7f514406a860 2024-01-20T11:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.828+0000 7f512bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.828+0000 7f512bfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:26.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:26.996+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51440723e0 con 0x7f513006c8b0 2024-01-20T11:20:27.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.000+0000 7f512bfff640 1 -- 172.21.15.110:0/214772602 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64057 (secure 0 0 0) 0x7f5138031b40 con 0x7f513006c8b0 2024-01-20T11:20:27.001 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:27.001 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 10 librados: watch_flush enter 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 10 librados: watch_flush exit 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 20 client.5301.objecter shutdown clearing up homeless session... 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 10 client.5301.objecter successfully canceled tick 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f513006c8b0 msgr2=0x7f513006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f513006c8b0 0x7f513006ed70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5138036000 tx=0x7f513803d040 comp rx=0 tx=0).stop 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 10 monclient: shutdown 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f514406a860 msgr2=0x7f514406ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f514406a860 0x7f514406ff30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f513400c4b0 tx=0x7f513400c970 comp rx=0 tx=0).stop 2024-01-20T11:20:27.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 shutdown_connections 2024-01-20T11:20:27.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f513006c8b0 0x7f513006ed70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5144145af0 0x7f51441d7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51440af730 0x7f5144070820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 --2- 172.21.15.110:0/214772602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f514406a860 0x7f514406ff30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 >> 172.21.15.110:0/214772602 conn(0x7f51440b47d0 msgr2=0x7f51440b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:27.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 shutdown_connections 2024-01-20T11:20:27.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 -- 172.21.15.110:0/214772602 wait complete. 2024-01-20T11:20:27.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.001+0000 7f5149b47640 1 librados: shutdown 2024-01-20T11:20:27.012 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-01-20T11:20:26.770787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002442"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280580,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.5810000000000004},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.1219999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.8520000000000003},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.4080000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":5.6879999999999997},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.417999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.5830000000000002},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":11.128}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.73},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":13.103999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.803000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":12.374000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313283,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.6689999999999996},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":15.167}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.3230000000000004},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.4719999999999995},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":5.9119999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.0869999999999997},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.300000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.965},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.8070000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.976000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.577}]}]},{"osd":3,"up_from":21,"seq":90194313283,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":9.4410000000000007},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6630000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.6870000000000003},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.869}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":11.315},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":8.5630000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":15.872999999999999},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.108000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":13.994},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.927}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":14.413},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.279999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444097,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.29},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":4.4770000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.513999999999999},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.9559999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.32},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.944000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.782999999999999},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.347}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":9.6289999999999996},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.629}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.127000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":15.948}]}]},{"osd":1,"up_from":17,"seq":73014444098,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.605},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":15.962999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":15.167999999999999},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.5579999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":9.9469999999999992},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.3399999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":6.8109999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":10.932}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.1360000000000001},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":13.063000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.238},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.343}]}]},{"osd":0,"up_from":17,"seq":73014444098,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.0229999999999997},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":6.6680000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.5759999999999996},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.095000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":12.849},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.092000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":8.4920000000000009},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":14.199}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.475},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":13.837}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.57},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":16.960000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":9.8819999999999997},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.1840000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.6980000000000004},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.629}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":6.976},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.484}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.297000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":8.7240000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.933999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.686}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.3109999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.747999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:27.014 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:27.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:27.014 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:27.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:27.014 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:27.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:27.014 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:27.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:27.015 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:27.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.111+0000 7f69253dc640 10 monclient: get_monmap_and_config 2024-01-20T11:20:27.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.111+0000 7f69253dc640 10 monclient: build_initial_monmap 2024-01-20T11:20:27.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.111+0000 7f69253dc640 10 monclient: monmap: 2024-01-20T11:20:27.112 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:27.112 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.112 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:27.112977+0000 2024-01-20T11:20:27.112 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:27.112977+0000 2024-01-20T11:20:27.113 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:27.113 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.113 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:27.113 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:27.113 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:27.113 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:27.113 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:27.113 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 1 Processor -- start 2024-01-20T11:20:27.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 1 -- start start 2024-01-20T11:20:27.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 10 monclient: init 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69200b0c00 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69200accf0 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6920143910 0x7f692014b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 10 monclient(hunting): picked mon.noname-c con 0x7f6920143910 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200ad450 con 0x7f69200accf0 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200ad8e0 con 0x7f69200b0c00 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69200ad130 con 0x7f6920143910 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.112+0000 7f69253dc640 10 monclient(hunting): authenticate will time out at 1670.463909s 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.113+0000 7f6917fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f69200accf0 0x7f6920143200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55180/0 (socket says 172.21.15.110:55180) 2024-01-20T11:20:27.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.113+0000 7f6917fff640 1 -- 172.21.15.110:0/3378046591 learned_addr learned my addr 172.21.15.110:0/3378046591 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:27.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.113+0000 7f691f7fe640 1 --2- 172.21.15.110:0/3378046591 >> v2:172.21.15.134:3300/0 conn(0x7f6920143910 0x7f692014b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.113+0000 7f691f7fe640 10 monclient(hunting): get_auth_request con 0x7f6920143910 auth_method 0 2024-01-20T11:20:27.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.113+0000 7f691f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:27.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.113+0000 7f691f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.113+0000 7f691f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.114+0000 7f691f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.114+0000 7f691f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.114+0000 7f691f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.115+0000 7f691e7fc640 1 -- 172.21.15.110:0/3378046591 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358999656 0 0) 0x7f69200ad450 con 0x7f69200accf0 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.115+0000 7f691e7fc640 10 monclient(hunting): _negotiate 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.115+0000 7f691e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.115+0000 7f691e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.115+0000 7f691e7fc640 10 monclient(hunting): my global_id is 5304 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.115+0000 7f691e7fc640 1 -- 172.21.15.110:0/3378046591 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690c003520 con 0x7f69200accf0 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.116+0000 7f691f7fe640 10 monclient(hunting): handle_auth_done global_id 5380 payload 274 2024-01-20T11:20:27.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.116+0000 7f691f7fe640 1 -- 172.21.15.110:0/3378046591 >> v1:172.21.15.134:6789/0 conn(0x7f69200b0c00 legacy=0x7f692013d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:27.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.116+0000 7f691f7fe640 1 -- 172.21.15.110:0/3378046591 >> v1:172.21.15.110:6789/0 conn(0x7f69200accf0 legacy=0x7f6920143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.116+0000 7f691f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:27.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.116+0000 7f691f7fe640 1 monclient: found mon.noname-c 2024-01-20T11:20:27.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.116+0000 7f691f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:27.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.116+0000 7f691f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.116+0000 7f691f7fe640 1 -- 172.21.15.110:0/3378046591 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6920143d70 con 0x7f6920143910 2024-01-20T11:20:27.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.119+0000 7f691f7fe640 1 --2- 172.21.15.110:0/3378046591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6920143910 0x7f692014b560 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6918009d90 tx=0x7f691802bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2044a6ebf54366f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:27.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 1 -- 172.21.15.110:0/3378046591 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691802cb40 con 0x7f6920143910 2024-01-20T11:20:27.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:27.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 10 monclient: dump: 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:27.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 1 -- 172.21.15.110:0/3378046591 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691802cca0 con 0x7f6920143910 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 1 -- 172.21.15.110:0/3378046591 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918034660 con 0x7f6920143910 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f691e7fc640 10 monclient: dump: 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:27.125 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:27.126 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 5 monclient: authenticate success, global_id 5380 2024-01-20T11:20:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:27.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 4 set_mon_vals no callback set 2024-01-20T11:20:27.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 10 monclient: shutdown 2024-01-20T11:20:27.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:27.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 1 -- 172.21.15.110:0/3378046591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6920143910 msgr2=0x7f692014b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 1 --2- 172.21.15.110:0/3378046591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6920143910 0x7f692014b560 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6918009d90 tx=0x7f691802bd90 comp rx=0 tx=0).stop 2024-01-20T11:20:27.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 1 -- 172.21.15.110:0/3378046591 shutdown_connections 2024-01-20T11:20:27.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 1 --2- 172.21.15.110:0/3378046591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6920143910 0x7f692014b560 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 1 -- 172.21.15.110:0/3378046591 >> 172.21.15.110:0/3378046591 conn(0x7f69200b3e60 msgr2=0x7f69200b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:27.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 1 -- 172.21.15.110:0/3378046591 shutdown_connections 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.123+0000 7f69253dc640 1 -- 172.21.15.110:0/3378046591 wait complete. 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.124+0000 7f69253dc640 10 monclient: build_initial_monmap 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.124+0000 7f69253dc640 10 monclient: monmap: 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.124+0000 7f69253dc640 1 librados: starting msgr at 2024-01-20T11:20:27.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.124+0000 7f69253dc640 1 librados: starting objecter 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.124+0000 7f69253dc640 1 Processor -- start 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.124+0000 7f69253dc640 1 -- start start 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.124+0000 7f69253dc640 1 librados: setting wanted keys 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.124+0000 7f69253dc640 1 librados: calling monclient init 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.124+0000 7f69253dc640 10 monclient: init 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69200accf0 0x7f6920149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient(hunting): picked mon.noname-b con 0x7f69200accf0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69200b0c00 0x7f69200b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient(hunting): picked mon.noname-a con 0x7f69200b0c00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6920143910 0x7f69200b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient(hunting): picked mon.noname-c con 0x7f6920143910 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:27.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6920139680 con 0x7f69200b0c00 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6920148d40 con 0x7f69200accf0 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6920149020 con 0x7f6920143910 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f69253dc640 10 monclient(hunting): authenticate will time out at 1670.476409s 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f6917fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69200b0c00 0x7f69200b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f691effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69200accf0 0x7f6920149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.125+0000 7f691f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6920143910 0x7f69200b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f6917fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69200b0c00 0x7f69200b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:55572/0 (socket says 172.21.15.110:55572) 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f6917fff640 1 -- 172.21.15.110:0/1455831938 learned_addr learned my addr 172.21.15.110:0/1455831938 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f6917fff640 10 monclient(hunting): get_auth_request con 0x7f69200b0c00 auth_method 0 2024-01-20T11:20:27.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f6917fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f6917fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f6917fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691effd640 10 monclient(hunting): get_auth_request con 0x7f69200accf0 auth_method 0 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691f7fe640 10 monclient(hunting): get_auth_request con 0x7f6920143910 auth_method 0 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:27.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:27.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:27.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:27.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:27.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.126+0000 7f691f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 10 monclient(hunting): handle_auth_done global_id 5383 payload 824 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 1 -- 172.21.15.110:0/1455831938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6920143910 msgr2=0x7f69200b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6920143910 0x7f69200b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 1 -- 172.21.15.110:0/1455831938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69200b0c00 msgr2=0x7f69200b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69200b0c00 0x7f69200b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 10 monclient: _finish_hunting 0 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 1 monclient: found mon.noname-b 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.129+0000 7f691effd640 1 -- 172.21.15.110:0/1455831938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6918009a70 con 0x7f69200accf0 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.131+0000 7f691effd640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69200accf0 0x7f6920149a90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f691000c410 tx=0x7f691000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.131+0000 7f69177fe640 10 client.?.objecter ms_handle_connect 0x7f69200accf0 2024-01-20T11:20:27.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.131+0000 7f69177fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 1 -- 172.21.15.110:0/1455831938 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69100035f0 con 0x7f69200accf0 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 10 monclient: dump: 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.137 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 1 -- 172.21.15.110:0/1455831938 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6910003770 con 0x7f69200accf0 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 1 -- 172.21.15.110:0/1455831938 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910003b60 con 0x7f69200accf0 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69253dc640 5 monclient: authenticate success, global_id 5383 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69253dc640 10 monclient: _renew_subs 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69253dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692013d970 con 0x7f69200accf0 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:27.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69177fe640 10 monclient: dump: 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f691dffb640 4 set_mon_vals no callback set 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69253dc640 10 client.5383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:27.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69253dc640 10 monclient: _renew_subs 2024-01-20T11:20:27.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69253dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69201493b0 con 0x7f69200accf0 2024-01-20T11:20:27.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.135+0000 7f69253dc640 1 librados: init done 2024-01-20T11:20:27.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.136+0000 7f69253dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:27.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.136+0000 7f69253dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:27.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.136+0000 7f69253dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.136+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6920075490 con 0x7f69200accf0 2024-01-20T11:20:27.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.372+0000 7f69177fe640 1 -- 172.21.15.110:0/1455831938 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6910003cc0 con 0x7f69200accf0 2024-01-20T11:20:27.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.372+0000 7f69177fe640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f690006c860 0x7f690006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.373+0000 7f6917fff640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f690006c860 0x7f690006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.373+0000 7f6917fff640 10 monclient: get_auth_request con 0x7f690006c860 auth_method 0 2024-01-20T11:20:27.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.373+0000 7f6917fff640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f690006c860 0x7f690006ed20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6920148bf0 tx=0x7f6908009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:27.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.373+0000 7f69177fe640 10 client.5383.objecter ms_handle_connect 0x7f690006c860 2024-01-20T11:20:27.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.378+0000 7f69177fe640 1 -- 172.21.15.110:0/1455831938 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6910057d20 con 0x7f69200accf0 2024-01-20T11:20:27.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.378+0000 7f69177fe640 10 client.5383.objecter ms_dispatch 0x7f6920000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:27.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.378+0000 7f69177fe640 3 client.5383.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:27.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.378+0000 7f69177fe640 3 client.5383.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:27.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.378+0000 7f69177fe640 20 client.5383.objecter dump_active .. 0 homeless 2024-01-20T11:20:27.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.406+0000 7f69177fe640 1 -- 172.21.15.110:0/1455831938 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6910054060 con 0x7f69200accf0 2024-01-20T11:20:27.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.406+0000 7f69177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:27.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.406+0000 7f69177fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:27.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.651+0000 7f69253dc640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:27.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.651+0000 7f69253dc640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:27.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.651+0000 7f69253dc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.651+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f6920069970 con 0x7f69200accf0 2024-01-20T11:20:27.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.656+0000 7f69177fe640 1 -- 172.21.15.110:0/1455831938 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f6910017020 con 0x7f69200accf0 2024-01-20T11:20:27.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.656+0000 7f69177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:27.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.656+0000 7f69177fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:27.658 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:27.658 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":362,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 10 librados: watch_flush enter 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 10 librados: watch_flush exit 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 20 client.5383.objecter shutdown clearing up homeless session... 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 10 client.5383.objecter successfully canceled tick 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f690006c860 msgr2=0x7f690006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f690006c860 0x7f690006ed20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6920148bf0 tx=0x7f6908009040 comp rx=0 tx=0).stop 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 10 monclient: shutdown 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69200accf0 msgr2=0x7f6920149a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69200accf0 0x7f6920149a90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f691000c410 tx=0x7f691000c810 comp rx=0 tx=0).stop 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 shutdown_connections 2024-01-20T11:20:27.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f690006c860 0x7f690006ed20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6920143910 0x7f69200b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69200b0c00 0x7f69200b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 --2- 172.21.15.110:0/1455831938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69200accf0 0x7f6920149a90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 >> 172.21.15.110:0/1455831938 conn(0x7f69200b3e60 msgr2=0x7f692013c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:27.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 shutdown_connections 2024-01-20T11:20:27.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 -- 172.21.15.110:0/1455831938 wait complete. 2024-01-20T11:20:27.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.658+0000 7f69253dc640 1 librados: shutdown 2024-01-20T11:20:27.670 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:27.670 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:27.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.769+0000 7efdd6b3c640 10 monclient: get_monmap_and_config 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.769+0000 7efdd6b3c640 10 monclient: build_initial_monmap 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.769+0000 7efdd6b3c640 10 monclient: monmap: 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:27.770774+0000 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:27.770774+0000 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:27.770 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.769+0000 7efdd6b3c640 1 Processor -- start 2024-01-20T11:20:27.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.769+0000 7efdd6b3c640 1 -- start start 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.769+0000 7efdd6b3c640 10 monclient: init 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efdd00a96f0 0x7efdd013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 10 monclient(hunting): picked mon.noname-a con 0x7efdd00a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efdd013fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 10 monclient(hunting): picked mon.noname-b con 0x7efdd00a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00b25c0 con 0x7efdd013fa10 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd00aaea0 con 0x7efdd00a96f0 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd00ab0a0 con 0x7efdd00a9ad0 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:27.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.770+0000 7efdd6b3c640 10 monclient(hunting): authenticate will time out at 1671.121818s 2024-01-20T11:20:27.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd48d8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efdd00a96f0 0x7efdd013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd50d9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd50d9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:42524/0 (socket says 172.21.15.110:42524) 2024-01-20T11:20:27.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd48d8640 10 monclient(hunting): get_auth_request con 0x7efdd00a96f0 auth_method 0 2024-01-20T11:20:27.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd50d9640 1 -- 172.21.15.110:0/460055203 learned_addr learned my addr 172.21.15.110:0/460055203 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd48d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd48d8640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd48d8640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd50d9640 10 monclient(hunting): get_auth_request con 0x7efdd00a9ad0 auth_method 0 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd50d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd50d9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.771+0000 7efdd50d9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.772+0000 7efdd50d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.772+0000 7efdd50d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.772+0000 7efdd50d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.772+0000 7efdd48d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.772+0000 7efdd48d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:27.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.772+0000 7efdd48d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.773+0000 7efdd50d9640 10 monclient(hunting): handle_auth_done global_id 5069 payload 274 2024-01-20T11:20:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.773+0000 7efdd50d9640 1 -- 172.21.15.110:0/460055203 >> v2:172.21.15.110:3300/0 conn(0x7efdd00a96f0 msgr2=0x7efdd013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.773+0000 7efdd50d9640 1 --2- 172.21.15.110:0/460055203 >> v2:172.21.15.110:3300/0 conn(0x7efdd00a96f0 0x7efdd013f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.773+0000 7efdd50d9640 1 -- 172.21.15.110:0/460055203 >> v1:172.21.15.110:6789/0 conn(0x7efdd013fa10 legacy=0x7efdd01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.773+0000 7efdd50d9640 10 monclient: _finish_hunting 0 2024-01-20T11:20:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.773+0000 7efdd50d9640 1 monclient: found mon.noname-b 2024-01-20T11:20:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.773+0000 7efdd50d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.773+0000 7efdd50d9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.773+0000 7efdd50d9640 1 -- 172.21.15.110:0/460055203 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd00b3fb0 con 0x7efdd00a9ad0 2024-01-20T11:20:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.775+0000 7efdd50d9640 1 --2- 172.21.15.110:0/460055203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0142af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009e50 tx=0x7efdc802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cce5d590d6c37fc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:27.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 1 -- 172.21.15.110:0/460055203 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc802ca70 con 0x7efdd00a9ad0 2024-01-20T11:20:27.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:27.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:27.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 10 monclient: dump: 2024-01-20T11:20:27.780 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:27.780 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.780 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.780 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 1 -- 172.21.15.110:0/460055203 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc802cbd0 con 0x7efdd00a9ad0 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 1 -- 172.21.15.110:0/460055203 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc80343f0 con 0x7efdd00a9ad0 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdd6b3c640 5 monclient: authenticate success, global_id 5069 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdd6b3c640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdd6b3c640 4 set_mon_vals no callback set 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.779+0000 7efdcf7fe640 10 monclient: dump: 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:27.782 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 10 monclient: shutdown 2024-01-20T11:20:27.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:27.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 1 -- 172.21.15.110:0/460055203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 msgr2=0x7efdd0142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 1 --2- 172.21.15.110:0/460055203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0142af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009e50 tx=0x7efdc802bed0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 1 -- 172.21.15.110:0/460055203 shutdown_connections 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 1 --2- 172.21.15.110:0/460055203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0142af0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 1 --2- 172.21.15.110:0/460055203 >> v2:172.21.15.110:3300/0 conn(0x7efdd00a96f0 0x7efdd013f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 1 -- 172.21.15.110:0/460055203 >> 172.21.15.110:0/460055203 conn(0x7efdd00b7bc0 msgr2=0x7efdd00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 1 -- 172.21.15.110:0/460055203 shutdown_connections 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.780+0000 7efdd6b3c640 1 -- 172.21.15.110:0/460055203 wait complete. 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.781+0000 7efdd6b3c640 10 monclient: build_initial_monmap 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.781+0000 7efdd6b3c640 10 monclient: monmap: 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.781+0000 7efdd6b3c640 1 librados: starting msgr at 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.781+0000 7efdd6b3c640 1 librados: starting objecter 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.781+0000 7efdd6b3c640 1 Processor -- start 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.781+0000 7efdd6b3c640 1 -- start start 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.781+0000 7efdd6b3c640 1 librados: setting wanted keys 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.781+0000 7efdd6b3c640 1 librados: calling monclient init 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.781+0000 7efdd6b3c640 10 monclient: init 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd00a96f0 0x7efdd0133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient(hunting): picked mon.noname-b con 0x7efdd00a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient(hunting): picked mon.noname-c con 0x7efdd00a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd01419e0 0x7efdd0138310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient(hunting): picked mon.noname-a con 0x7efdd01419e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd01321b0 con 0x7efdd01419e0 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0138850 con 0x7efdd00a96f0 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0138a10 con 0x7efdd00a9ad0 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdd6b3c640 10 monclient(hunting): authenticate will time out at 1671.133672s 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.782+0000 7efdcffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd50d9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd01419e0 0x7efdd0138310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd48d8640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd00a96f0 0x7efdd0133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdcffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0141180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:42532/0 (socket says 172.21.15.110:42532) 2024-01-20T11:20:27.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdcffff640 1 -- 172.21.15.110:0/3341992630 learned_addr learned my addr 172.21.15.110:0/3341992630 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdcffff640 10 monclient(hunting): get_auth_request con 0x7efdd00a9ad0 auth_method 0 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdcffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdcffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd50d9640 10 monclient(hunting): get_auth_request con 0x7efdd01419e0 auth_method 0 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd50d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd50d9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd50d9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd48d8640 10 monclient(hunting): get_auth_request con 0x7efdd00a96f0 auth_method 0 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd48d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd48d8640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd48d8640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:27.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd48d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:27.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd48d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:27.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd48d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:27.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd50d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:27.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.783+0000 7efdd50d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:27.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.784+0000 7efdd50d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:27.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 10 monclient(hunting): handle_auth_done global_id 5386 payload 824 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 1 -- 172.21.15.110:0/3341992630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 msgr2=0x7efdd0141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 1 -- 172.21.15.110:0/3341992630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd01419e0 msgr2=0x7efdd0138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd01419e0 0x7efdd0138310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 10 monclient: _finish_hunting 0 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 1 monclient: found mon.noname-b 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.786+0000 7efdd48d8640 1 -- 172.21.15.110:0/3341992630 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8009b30 con 0x7efdd00a96f0 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.788+0000 7efdd48d8640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd00a96f0 0x7efdd0133060 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efdc400e850 tx=0x7efdc400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.788+0000 7efdcd7fa640 10 client.?.objecter ms_handle_connect 0x7efdd00a96f0 2024-01-20T11:20:27.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.788+0000 7efdcd7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:27.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 1 -- 172.21.15.110:0/3341992630 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc40095b0 con 0x7efdd00a96f0 2024-01-20T11:20:27.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:27.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 10 monclient: dump: 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdd6b3c640 5 monclient: authenticate success, global_id 5386 2024-01-20T11:20:27.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 1 -- 172.21.15.110:0/3341992630 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc400c940 con 0x7efdd00a96f0 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdd6b3c640 10 monclient: _renew_subs 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdd6b3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd013c8b0 con 0x7efdd00a96f0 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 1 -- 172.21.15.110:0/3341992630 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc400ccd0 con 0x7efdd00a96f0 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdcd7fa640 10 monclient: dump: 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:27.795 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:27.796 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:27.796 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:27.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdd6b3c640 10 client.5386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:27.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdd6b3c640 10 monclient: _renew_subs 2024-01-20T11:20:27.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdceffd640 4 set_mon_vals no callback set 2024-01-20T11:20:27.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdd6b3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd012d880 con 0x7efdd00a96f0 2024-01-20T11:20:27.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.792+0000 7efdd6b3c640 1 librados: init done 2024-01-20T11:20:27.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.793+0000 7efdd6b3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:27.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.793+0000 7efdd6b3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:27.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.793+0000 7efdd6b3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:27.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:27.793+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd0075490 con 0x7efdd00a96f0 2024-01-20T11:20:28.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.030+0000 7efdcd7fa640 1 -- 172.21.15.110:0/3341992630 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7efdc4013470 con 0x7efdd00a96f0 2024-01-20T11:20:28.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.030+0000 7efdcd7fa640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdb406c8b0 0x7efdb406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:28.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.030+0000 7efdcffff640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdb406c8b0 0x7efdb406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:28.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.030+0000 7efdcffff640 10 monclient: get_auth_request con 0x7efdb406c8b0 auth_method 0 2024-01-20T11:20:28.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.031+0000 7efdcd7fa640 10 client.5386.objecter ms_handle_connect 0x7efdb406c8b0 2024-01-20T11:20:28.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.031+0000 7efdcffff640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdb406c8b0 0x7efdb406ed70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efdd01348c0 tx=0x7efdbc009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:28.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.034+0000 7efdcd7fa640 1 -- 172.21.15.110:0/3341992630 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efdc400ce30 con 0x7efdd00a96f0 2024-01-20T11:20:28.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.034+0000 7efdcd7fa640 10 client.5386.objecter ms_dispatch 0x7efdd0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:28.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.034+0000 7efdcd7fa640 3 client.5386.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:28.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.034+0000 7efdcd7fa640 3 client.5386.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:28.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.034+0000 7efdcd7fa640 20 client.5386.objecter dump_active .. 0 homeless 2024-01-20T11:20:28.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.043+0000 7efdcd7fa640 1 -- 172.21.15.110:0/3341992630 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7efdc4054e70 con 0x7efdd00a96f0 2024-01-20T11:20:28.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.043+0000 7efdcd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:28.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.043+0000 7efdcd7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:28.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.213+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efdd00b62d0 con 0x7efdb406c8b0 2024-01-20T11:20:28.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.216+0000 7efdcd7fa640 1 -- 172.21.15.110:0/3341992630 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64057 (secure 0 0 0) 0x7efdd00b62d0 con 0x7efdb406c8b0 2024-01-20T11:20:28.217 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:28.217 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:28.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 10 librados: watch_flush enter 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 10 librados: watch_flush exit 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 20 client.5386.objecter shutdown clearing up homeless session... 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 10 client.5386.objecter successfully canceled tick 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdb406c8b0 msgr2=0x7efdb406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdb406c8b0 0x7efdb406ed70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efdd01348c0 tx=0x7efdbc009040 comp rx=0 tx=0).stop 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 10 monclient: shutdown 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd00a96f0 msgr2=0x7efdd0133060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd00a96f0 0x7efdd0133060 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efdc400e850 tx=0x7efdc400ed10 comp rx=0 tx=0).stop 2024-01-20T11:20:28.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 shutdown_connections 2024-01-20T11:20:28.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdb406c8b0 0x7efdb406ed70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:28.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd01419e0 0x7efdd0138310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:28.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd00a9ad0 0x7efdd0141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:28.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 --2- 172.21.15.110:0/3341992630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd00a96f0 0x7efdd0133060 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:28.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 >> 172.21.15.110:0/3341992630 conn(0x7efdd00b7bc0 msgr2=0x7efdd0137310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:28.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 shutdown_connections 2024-01-20T11:20:28.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 -- 172.21.15.110:0/3341992630 wait complete. 2024-01-20T11:20:28.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:28.218+0000 7efdd6b3c640 1 librados: shutdown 2024-01-20T11:20:28.229 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-01-20T11:20:26.770787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002442"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280580,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.5810000000000004},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":8.1219999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.8520000000000003},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.4080000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":5.6879999999999997},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.417999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":9.5830000000000002},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":11.128}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.73},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":13.103999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":13.803000000000001},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":12.374000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313283,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.6689999999999996},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":15.167}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.3230000000000004},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.4719999999999995},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":5.9119999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.0869999999999997},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.300000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.965},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.8070000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.976000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.577}]}]},{"osd":3,"up_from":21,"seq":90194313283,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":9.4410000000000007},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.6630000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.6870000000000003},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.869}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":11.315},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":8.5630000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":15.872999999999999},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.108000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":13.994},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":11.927}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":14.413},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.279999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444097,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.29},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":4.4770000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.513999999999999},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":8.9559999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.779999999999999,"5min":14.702999999999999,"15min":16.745000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":18.375,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.32},{"interface":"front","average":{"1min":10.356,"5min":12.359999999999999,"15min":14.147},"min":{"1min":5.7110000000000003,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.262,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.944000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":12.782999999999999},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.347}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":9.6289999999999996},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":10.629}]},{"osd":7,"last update":"Sat Jan 20 11:19:25 2024","interfaces":[{"interface":"back","average":{"1min":12.269,"5min":12.926,"15min":13.536},"min":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":18.106000000000002,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.127000000000001},{"interface":"front","average":{"1min":15.177,"5min":16.265000000000001,"15min":17.481999999999999},"min":{"1min":10.791,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":20.079000000000001,"5min":28.148,"15min":28.148},"last":15.948}]}]},{"osd":1,"up_from":17,"seq":73014444098,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":11.605},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":15.962999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":15.167999999999999},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":7.5579999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":12.303000000000001,"5min":18.931999999999999,"15min":24.331},"min":{"1min":8.7159999999999993,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":15.266999999999999,"5min":259.483,"15min":259.483},"last":9.9469999999999992},{"interface":"front","average":{"1min":9.4299999999999997,"5min":14.621,"15min":19.300000000000001},"min":{"1min":5.8899999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":14.593999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.3399999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":6.8109999999999999},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":10.932}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.1360000000000001},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":13.063000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:22 2024","interfaces":[{"interface":"back","average":{"1min":11.532,"5min":20.306999999999999,"15min":28.286999999999999},"min":{"1min":7.5960000000000001,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":15.813000000000001,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.238},{"interface":"front","average":{"1min":15.066000000000001,"5min":22.111999999999998,"15min":29.061},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.370000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.343}]}]},{"osd":0,"up_from":17,"seq":73014444098,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":11.137,"15min":13.965},"min":{"1min":4.3250000000000002,"5min":4.1159999999999997,"15min":4.1159999999999997},"max":{"1min":14.74,"5min":114.843,"15min":114.843},"last":6.0229999999999997},{"interface":"front","average":{"1min":8.7170000000000005,"5min":11.837999999999999,"15min":14.823},"min":{"1min":4.3040000000000003,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":15.5,"5min":121.044,"15min":121.044},"last":6.6680000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":8.0090000000000003,"5min":17.452000000000002,"15min":27.155999999999999},"min":{"1min":4.5880000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":13.349,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.5759999999999996},{"interface":"front","average":{"1min":10.551,"5min":19.966999999999999,"15min":28.991},"min":{"1min":7.2240000000000002,"5min":6.1849999999999996,"15min":6.1849999999999996},"max":{"1min":19.094999999999999,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.095000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":12.849},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.092000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":8.4920000000000009},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":14.199}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:23 2024","interfaces":[{"interface":"back","average":{"1min":13.209,"5min":22.693000000000001,"15min":31.66},"min":{"1min":8.7390000000000008,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":17.779,"5min":375.733,"15min":375.733},"last":13.475},{"interface":"front","average":{"1min":13.815,"5min":21.702000000000002,"15min":29.559999999999999},"min":{"1min":9.7129999999999992,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.405000000000001,"5min":374.654,"15min":374.654},"last":13.837}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.57},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":16.960000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":9.8819999999999997},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.1840000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.6980000000000004},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.629}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":6.976},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.484}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.297000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":8.7240000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.933999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.686}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.3109999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.747999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:31.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~93s 2024-01-20T11:20:31.231 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:31.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.333+0000 7f2e12d41640 10 monclient: get_monmap_and_config 2024-01-20T11:20:31.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.333+0000 7f2e12d41640 10 monclient: build_initial_monmap 2024-01-20T11:20:31.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.333+0000 7f2e12d41640 10 monclient: monmap: 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:31.334323+0000 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:31.334323+0000 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:31.334 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.334+0000 7f2e12d41640 1 Processor -- start 2024-01-20T11:20:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.334+0000 7f2e12d41640 1 -- start start 2024-01-20T11:20:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.334+0000 7f2e12d41640 10 monclient: init 2024-01-20T11:20:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.334+0000 7f2e12d41640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-01-20T11:20:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e0c06a860 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2e0c06ac40 0x7f2e0c1455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 10 monclient(hunting): picked mon.noname-c con 0x7f2e0c06ac40 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c0af730 0x7f2e0c145af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 10 monclient(hunting): picked mon.noname-b con 0x7f2e0c0af730 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c0afd30 con 0x7f2e0c06a860 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e0c146030 con 0x7f2e0c06ac40 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c0b01c0 con 0x7f2e0c0af730 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e12d41640 10 monclient(hunting): authenticate will time out at 1674.685247s 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e112de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c0af730 0x7f2e0c145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.335+0000 7f2e0bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2e0c06ac40 0x7f2e0c1455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e112de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c0af730 0x7f2e0c145af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45740/0 (socket says 172.21.15.110:45740) 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e112de640 1 -- 172.21.15.110:0/3418529673 learned_addr learned my addr 172.21.15.110:0/3418529673 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e112de640 10 monclient(hunting): get_auth_request con 0x7f2e0c0af730 auth_method 0 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e112de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e112de640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e112de640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e0bfff640 10 monclient(hunting): get_auth_request con 0x7f2e0c06ac40 auth_method 0 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e0bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e112de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e112de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e112de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.336+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.338+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_done global_id 5389 payload 274 2024-01-20T11:20:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.338+0000 7f2e0bfff640 1 -- 172.21.15.110:0/3418529673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c0af730 msgr2=0x7f2e0c145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.338+0000 7f2e0bfff640 1 --2- 172.21.15.110:0/3418529673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c0af730 0x7f2e0c145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.338+0000 7f2e0bfff640 1 -- 172.21.15.110:0/3418529673 >> v1:172.21.15.110:6789/0 conn(0x7f2e0c06a860 legacy=0x7f2e0c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.338+0000 7f2e0bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.338+0000 7f2e0bfff640 1 monclient: found mon.noname-c 2024-01-20T11:20:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.338+0000 7f2e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.338+0000 7f2e0bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.338+0000 7f2e0bfff640 1 -- 172.21.15.110:0/3418529673 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c13fb60 con 0x7f2e0c06ac40 2024-01-20T11:20:31.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.340+0000 7f2e0bfff640 1 --2- 172.21.15.110:0/3418529673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e0c06ac40 0x7f2e0c1455b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2df80097a0 tx=0x7f2df802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c5c40eec7e6255d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 1 -- 172.21.15.110:0/3418529673 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df802e070 con 0x7f2e0c06ac40 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 10 monclient: dump: 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.345 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 1 -- 172.21.15.110:0/3418529673 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df80315a0 con 0x7f2e0c06ac40 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 1 -- 172.21.15.110:0/3418529673 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df8031930 con 0x7f2e0c06ac40 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e12d41640 5 monclient: authenticate success, global_id 5389 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e12d41640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e12d41640 4 set_mon_vals no callback set 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:31.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.344+0000 7f2e0b7fe640 10 monclient: dump: 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 10 monclient: shutdown 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 1 -- 172.21.15.110:0/3418529673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e0c06ac40 msgr2=0x7f2e0c1455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 1 --2- 172.21.15.110:0/3418529673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e0c06ac40 0x7f2e0c1455b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2df80097a0 tx=0x7f2df802b840 comp rx=0 tx=0).stop 2024-01-20T11:20:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 1 -- 172.21.15.110:0/3418529673 shutdown_connections 2024-01-20T11:20:31.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 1 --2- 172.21.15.110:0/3418529673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c0af730 0x7f2e0c145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 1 --2- 172.21.15.110:0/3418529673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e0c06ac40 0x7f2e0c1455b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 1 -- 172.21.15.110:0/3418529673 >> 172.21.15.110:0/3418529673 conn(0x7f2e0c0b47d0 msgr2=0x7f2e0c0b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:31.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 1 -- 172.21.15.110:0/3418529673 shutdown_connections 2024-01-20T11:20:31.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 1 -- 172.21.15.110:0/3418529673 wait complete. 2024-01-20T11:20:31.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 10 monclient: build_initial_monmap 2024-01-20T11:20:31.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.345+0000 7f2e12d41640 10 monclient: monmap: 2024-01-20T11:20:31.351 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:31.351 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:31.351 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:31.351 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 librados: starting msgr at 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 librados: starting objecter 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 Processor -- start 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 -- start start 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 librados: setting wanted keys 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 librados: calling monclient init 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient: init 2024-01-20T11:20:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e0c06a860 0x7f2e0c1378e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient(hunting): picked mon.noname-a con 0x7f2e0c06a860 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c06ac40 0x7f2e0c138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient(hunting): picked mon.noname-c con 0x7f2e0c06ac40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e0c0af730 0x7f2e0c1402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient(hunting): picked mon.noname-b con 0x7f2e0c0af730 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c1374e0 con 0x7f2e0c06a860 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c137290 con 0x7f2e0c0af730 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c1421c0 con 0x7f2e0c06ac40 2024-01-20T11:20:31.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.346+0000 7f2e12d41640 10 monclient(hunting): authenticate will time out at 1674.697098s 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e0bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c06ac40 0x7f2e0c138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e10add640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e0c06a860 0x7f2e0c1378e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e112de640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e0c0af730 0x7f2e0c1402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e0bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c06ac40 0x7f2e0c138220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45746/0 (socket says 172.21.15.110:45746) 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e0bfff640 1 -- 172.21.15.110:0/4199627628 learned_addr learned my addr 172.21.15.110:0/4199627628 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e0bfff640 10 monclient(hunting): get_auth_request con 0x7f2e0c06ac40 auth_method 0 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e0bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e10add640 10 monclient(hunting): get_auth_request con 0x7f2e0c06a860 auth_method 0 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e10add640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e10add640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e10add640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e112de640 10 monclient(hunting): get_auth_request con 0x7f2e0c0af730 auth_method 0 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e112de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e112de640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.347+0000 7f2e112de640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.348+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.348+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.348+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.348+0000 7f2e10add640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.348+0000 7f2e10add640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.348+0000 7f2e10add640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.348+0000 7f2e112de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.348+0000 7f2e112de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.348+0000 7f2e112de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_done global_id 5078 payload 824 2024-01-20T11:20:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 1 -- 172.21.15.110:0/4199627628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e0c0af730 msgr2=0x7f2e0c1402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e0c0af730 0x7f2e0c1402c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 1 -- 172.21.15.110:0/4199627628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e0c06a860 msgr2=0x7f2e0c1378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e0c06a860 0x7f2e0c1378e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:31.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 1 monclient: found mon.noname-c 2024-01-20T11:20:31.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:31.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:20:31.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.351+0000 7f2e0bfff640 1 -- 172.21.15.110:0/4199627628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df8009400 con 0x7f2e0c06ac40 2024-01-20T11:20:31.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.353+0000 7f2e0bfff640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c06ac40 0x7f2e0c138220 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2df8034040 tx=0x7f2df80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:31.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.353+0000 7f2e097fa640 10 client.?.objecter ms_handle_connect 0x7f2e0c06ac40 2024-01-20T11:20:31.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.353+0000 7f2e097fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:31.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 1 -- 172.21.15.110:0/4199627628 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df8032840 con 0x7f2e0c06ac40 2024-01-20T11:20:31.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 10 monclient: dump: 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:31.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 1 -- 172.21.15.110:0/4199627628 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df80329c0 con 0x7f2e0c06ac40 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 1 -- 172.21.15.110:0/4199627628 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df8032d70 con 0x7f2e0c06ac40 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e12d41640 5 monclient: authenticate success, global_id 5078 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e12d41640 10 monclient: _renew_subs 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e12d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0c139330 con 0x7f2e0c06ac40 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e0affd640 4 set_mon_vals no callback set 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e097fa640 10 monclient: dump: 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.359 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e12d41640 10 client.5078.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e12d41640 10 monclient: _renew_subs 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e12d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0c13fe30 con 0x7f2e0c06ac40 2024-01-20T11:20:31.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.357+0000 7f2e12d41640 1 librados: init done 2024-01-20T11:20:31.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.358+0000 7f2e12d41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:31.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.358+0000 7f2e12d41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:31.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.358+0000 7f2e12d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:31.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.358+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0c137f40 con 0x7f2e0c06ac40 2024-01-20T11:20:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.598+0000 7f2e097fa640 1 -- 172.21.15.110:0/4199627628 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2df804d3f0 con 0x7f2e0c06ac40 2024-01-20T11:20:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.598+0000 7f2e097fa640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2df006c8b0 0x7f2df006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.599+0000 7f2e10add640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2df006c8b0 0x7f2df006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.599+0000 7f2e10add640 10 monclient: get_auth_request con 0x7f2df006c8b0 auth_method 0 2024-01-20T11:20:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.599+0000 7f2e10add640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2df006c8b0 0x7f2df006ed70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2e00001780 tx=0x7f2e00009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.599+0000 7f2e097fa640 10 client.5078.objecter ms_handle_connect 0x7f2df006c8b0 2024-01-20T11:20:31.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.611+0000 7f2e097fa640 1 -- 172.21.15.110:0/4199627628 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2df807a060 con 0x7f2e0c06ac40 2024-01-20T11:20:31.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.611+0000 7f2e097fa640 10 client.5078.objecter ms_dispatch 0x7f2e0c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:31.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.611+0000 7f2e097fa640 3 client.5078.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:31.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.611+0000 7f2e097fa640 3 client.5078.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:31.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.611+0000 7f2e097fa640 20 client.5078.objecter dump_active .. 0 homeless 2024-01-20T11:20:31.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.611+0000 7f2e097fa640 1 -- 172.21.15.110:0/4199627628 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2df80763a0 con 0x7f2e0c06ac40 2024-01-20T11:20:31.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.611+0000 7f2e097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:31.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.611+0000 7f2e097fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.784+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e0c0b0f40 con 0x7f2df006c8b0 2024-01-20T11:20:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.787+0000 7f2e097fa640 1 -- 172.21.15.110:0/4199627628 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63988 (secure 0 0 0) 0x7f2e0c0b0f40 con 0x7f2df006c8b0 2024-01-20T11:20:31.787 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 10 librados: watch_flush enter 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 10 librados: watch_flush exit 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 20 client.5078.objecter shutdown clearing up homeless session... 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 10 client.5078.objecter successfully canceled tick 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2df006c8b0 msgr2=0x7f2df006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2df006c8b0 0x7f2df006ed70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2e00001780 tx=0x7f2e00009040 comp rx=0 tx=0).stop 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 10 monclient: shutdown 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c06ac40 msgr2=0x7f2e0c138220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c06ac40 0x7f2e0c138220 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2df8034040 tx=0x7f2df80046c0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 shutdown_connections 2024-01-20T11:20:31.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2df006c8b0 0x7f2df006ed70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e0c0af730 0x7f2e0c1402c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e0c06ac40 0x7f2e0c138220 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 --2- 172.21.15.110:0/4199627628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e0c06a860 0x7f2e0c1378e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 >> 172.21.15.110:0/4199627628 conn(0x7f2e0c0b47d0 msgr2=0x7f2e0c0adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:31.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 shutdown_connections 2024-01-20T11:20:31.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 -- 172.21.15.110:0/4199627628 wait complete. 2024-01-20T11:20:31.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.789+0000 7f2e12d41640 1 librados: shutdown 2024-01-20T11:20:31.800 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-01-20T11:20:30.771580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002455"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280581,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.556},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.678000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.4260000000000002},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.119999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.704000000000001},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":7.5960000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.5259999999999998},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":11.454000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.528},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":9.4879999999999995}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.081},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":15.339}]}]},{"osd":7,"up_from":21,"seq":90194313283,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.6689999999999996},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":15.167}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.3230000000000004},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.4719999999999995},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":5.9119999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.0869999999999997},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.300000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.965},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.8070000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.976000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.577}]}]},{"osd":3,"up_from":21,"seq":90194313284,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.2089999999999996},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.9710000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":6.6059999999999999},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.784000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":11.84},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":8.7129999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.4610000000000003},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.577999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":12.510999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":10.106999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":13.702},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.376999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444098,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.411},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":5.9889999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.2129999999999992},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.9180000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.6799999999999997},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.521000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":11.247},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.303000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.1470000000000002},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.8320000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.098000000000001},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":11.839}]}]},{"osd":1,"up_from":17,"seq":73014444099,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.2359999999999998},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":9.1359999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.297000000000001},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":14.974}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":16.401},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.7270000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":9.7729999999999997},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":17.196000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.012},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":10.459}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.999000000000001},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.977}]}]},{"osd":0,"up_from":17,"seq":73014444099,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":8.2149999999999999},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":9.1549999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.2889999999999997},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":16.462}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":15.670999999999999},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":6.609}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.534000000000001},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.8919999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":13.628},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":12.192}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.552},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":10.433999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":9.8819999999999997},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.1840000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.6980000000000004},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.629}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":6.976},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.484}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.297000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":8.7240000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.933999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.686}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.3109999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.747999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:31.802 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:31.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.905+0000 7f30ec37c640 10 monclient: get_monmap_and_config 2024-01-20T11:20:31.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.905+0000 7f30ec37c640 10 monclient: build_initial_monmap 2024-01-20T11:20:31.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.905+0000 7f30ec37c640 10 monclient: monmap: 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:31.906387+0000 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:31.906387+0000 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:31.906 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.906+0000 7f30ec37c640 1 Processor -- start 2024-01-20T11:20:31.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.906+0000 7f30ec37c640 1 -- start start 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.906+0000 7f30ec37c640 10 monclient: init 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30e40b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f30e40b1ed0 0x7f30e40b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 10 monclient(hunting): picked mon.noname-a con 0x7f30e40b1ed0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e4145980 0x7f30e4145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 10 monclient(hunting): picked mon.noname-b con 0x7f30e4145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e4146370 con 0x7f30e40b4d10 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e40b50f0 con 0x7f30e40b1ed0 2024-01-20T11:20:31.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e40aade0 con 0x7f30e4145980 2024-01-20T11:20:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.907+0000 7f30ec37c640 10 monclient(hunting): authenticate will time out at 1675.257347s 2024-01-20T11:20:31.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.925+0000 7f30e9917640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f30e40b1ed0 0x7f30e40b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.926+0000 7f30e9917640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f30e40b1ed0 0x7f30e40b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40278/0 (socket says 172.21.15.110:40278) 2024-01-20T11:20:31.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.926+0000 7f30e9917640 1 -- 172.21.15.110:0/2259132454 learned_addr learned my addr 172.21.15.110:0/2259132454 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:31.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.926+0000 7f30e9917640 10 monclient(hunting): get_auth_request con 0x7f30e40b1ed0 auth_method 0 2024-01-20T11:20:31.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.926+0000 7f30e9917640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:31.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.926+0000 7f30e9917640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:31.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.926+0000 7f30e9917640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:31.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.927+0000 7f30e9917640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:31.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.927+0000 7f30e9917640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:31.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.927+0000 7f30e9917640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:31.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.929+0000 7f30e9917640 10 monclient(hunting): handle_auth_done global_id 5322 payload 274 2024-01-20T11:20:31.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.929+0000 7f30e9917640 1 -- 172.21.15.110:0/2259132454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e4145980 msgr2=0x7f30e4145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:31.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.929+0000 7f30e9917640 1 --2- 172.21.15.110:0/2259132454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e4145980 0x7f30e4145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.929+0000 7f30e9917640 1 -- 172.21.15.110:0/2259132454 >> v1:172.21.15.110:6789/0 conn(0x7f30e40b4d10 legacy=0x7f30e40b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.929+0000 7f30e9917640 10 monclient: _finish_hunting 0 2024-01-20T11:20:31.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.929+0000 7f30e9917640 1 monclient: found mon.noname-a 2024-01-20T11:20:31.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.929+0000 7f30e9917640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:31.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.929+0000 7f30e9917640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:31.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.929+0000 7f30e9917640 1 -- 172.21.15.110:0/2259132454 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e4145460 con 0x7f30e40b1ed0 2024-01-20T11:20:31.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.933+0000 7f30e9917640 1 --2- 172.21.15.110:0/2259132454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30e40b1ed0 0x7f30e40b3da0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009870 tx=0x7f30d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fc37170d1906c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 1 -- 172.21.15.110:0/2259132454 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d4031640 con 0x7f30e40b1ed0 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 10 monclient: dump: 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 10 monclient: _finish_auth 0 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 10 monclient: _check_auth_tickets 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 1 -- 172.21.15.110:0/2259132454 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d40317a0 con 0x7f30e40b1ed0 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 1 -- 172.21.15.110:0/2259132454 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d4031b30 con 0x7f30e40b1ed0 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30ec37c640 5 monclient: authenticate success, global_id 5322 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30ec37c640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30ec37c640 4 set_mon_vals no callback set 2024-01-20T11:20:31.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30e9116640 10 monclient: dump: 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:31.941 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30ec37c640 10 monclient: shutdown 2024-01-20T11:20:31.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30ec37c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:31.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30ec37c640 1 -- 172.21.15.110:0/2259132454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30e40b1ed0 msgr2=0x7f30e40b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.939+0000 7f30ec37c640 1 --2- 172.21.15.110:0/2259132454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30e40b1ed0 0x7f30e40b3da0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009870 tx=0x7f30d402b8f0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.940+0000 7f30ec37c640 1 -- 172.21.15.110:0/2259132454 shutdown_connections 2024-01-20T11:20:31.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.940+0000 7f30ec37c640 1 --2- 172.21.15.110:0/2259132454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e4145980 0x7f30e4145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.940+0000 7f30ec37c640 1 --2- 172.21.15.110:0/2259132454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30e40b1ed0 0x7f30e40b3da0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.940+0000 7f30ec37c640 1 -- 172.21.15.110:0/2259132454 >> 172.21.15.110:0/2259132454 conn(0x7f30e4066590 msgr2=0x7f30e40669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:31.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.940+0000 7f30ec37c640 1 -- 172.21.15.110:0/2259132454 shutdown_connections 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.940+0000 7f30ec37c640 1 -- 172.21.15.110:0/2259132454 wait complete. 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.940+0000 7f30ec37c640 10 monclient: build_initial_monmap 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.940+0000 7f30ec37c640 10 monclient: monmap: 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:31.946 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 librados: starting msgr at 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 librados: starting objecter 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 Processor -- start 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 -- start start 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 librados: setting wanted keys 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 librados: calling monclient init 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient: init 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e40b1ed0 0x7f30e41392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient(hunting): picked mon.noname-c con 0x7f30e40b1ed0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e40b4d10 0x7f30e413da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient(hunting): picked mon.noname-b con 0x7f30e40b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:31.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30e4145980 0x7f30e413dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient(hunting): picked mon.noname-a con 0x7f30e4145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e413f150 con 0x7f30e4145980 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e413f430 con 0x7f30e40b4d10 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e413f710 con 0x7f30e40b1ed0 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.941+0000 7f30ec37c640 10 monclient(hunting): authenticate will time out at 1675.291873s 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30ea118640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e40b1ed0 0x7f30e41392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30ea118640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e40b1ed0 0x7f30e41392d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45764/0 (socket says 172.21.15.110:45764) 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30ea118640 1 -- 172.21.15.110:0/3262264133 learned_addr learned my addr 172.21.15.110:0/3262264133 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:31.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30ea118640 10 monclient(hunting): get_auth_request con 0x7f30e40b1ed0 auth_method 0 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30ea118640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30ea118640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30ea118640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30e9917640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e40b4d10 0x7f30e413da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30ea919640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30e4145980 0x7f30e413dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30e9917640 10 monclient(hunting): get_auth_request con 0x7f30e40b4d10 auth_method 0 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30e9917640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30e9917640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.942+0000 7f30e9917640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea118640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:31.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea118640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea118640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea919640 10 monclient(hunting): get_auth_request con 0x7f30e4145980 auth_method 0 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea919640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea919640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea919640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30e9917640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30e9917640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30e9917640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea919640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea919640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:31.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.943+0000 7f30ea919640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:31.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 10 monclient(hunting): handle_auth_done global_id 5395 payload 824 2024-01-20T11:20:31.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 1 -- 172.21.15.110:0/3262264133 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e40b1ed0 msgr2=0x7f30e41392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e40b1ed0 0x7f30e41392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 1 -- 172.21.15.110:0/3262264133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30e4145980 msgr2=0x7f30e413dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:31.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30e4145980 0x7f30e413dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:31.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 10 monclient: _finish_hunting 0 2024-01-20T11:20:31.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 1 monclient: found mon.noname-b 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.946+0000 7f30e9917640 1 -- 172.21.15.110:0/3262264133 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d40094d0 con 0x7f30e40b4d10 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.948+0000 7f30e9917640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e40b4d10 0x7f30e413da80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f30d4033000 tx=0x7f30d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.948+0000 7f30daffd640 10 client.?.objecter ms_handle_connect 0x7f30e40b4d10 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.948+0000 7f30daffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 1 -- 172.21.15.110:0/3262264133 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d403ac60 con 0x7f30e40b4d10 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 10 monclient: dump: 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:31.954 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 1 -- 172.21.15.110:0/3262264133 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d403adc0 con 0x7f30e40b4d10 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30ec37c640 5 monclient: authenticate success, global_id 5395 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30ec37c640 10 monclient: _renew_subs 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30ec37c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e4136ce0 con 0x7f30e40b4d10 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 1 -- 172.21.15.110:0/3262264133 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d4044630 con 0x7f30e40b4d10 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:31.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30daffd640 10 monclient: dump: 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30e8915640 4 set_mon_vals no callback set 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30ec37c640 10 client.5395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30ec37c640 10 monclient: _renew_subs 2024-01-20T11:20:31.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30ec37c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:31.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e4138960 con 0x7f30e40b4d10 2024-01-20T11:20:31.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.952+0000 7f30ec37c640 1 librados: init done 2024-01-20T11:20:31.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.953+0000 7f30ec37c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:31.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.953+0000 7f30ec37c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:31.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.953+0000 7f30ec37c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:31.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:31.953+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e4075490 con 0x7f30e40b4d10 2024-01-20T11:20:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.194+0000 7f30daffd640 1 -- 172.21.15.110:0/3262264133 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f30d402f050 con 0x7f30e40b4d10 2024-01-20T11:20:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.194+0000 7f30daffd640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f30c806c8b0 0x7f30c806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.194+0000 7f30ea118640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f30c806c8b0 0x7f30c806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.194+0000 7f30ea118640 10 monclient: get_auth_request con 0x7f30c806c8b0 auth_method 0 2024-01-20T11:20:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.195+0000 7f30daffd640 10 client.5395.objecter ms_handle_connect 0x7f30c806c8b0 2024-01-20T11:20:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.195+0000 7f30ea118640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f30c806c8b0 0x7f30c806ed70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f30e0001840 tx=0x7f30e0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:32.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.197+0000 7f30daffd640 1 -- 172.21.15.110:0/3262264133 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f30d407a6e0 con 0x7f30e40b4d10 2024-01-20T11:20:32.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.197+0000 7f30daffd640 10 client.5395.objecter ms_dispatch 0x7f30e4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:32.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.197+0000 7f30daffd640 3 client.5395.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:32.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.197+0000 7f30daffd640 3 client.5395.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:32.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.197+0000 7f30daffd640 20 client.5395.objecter dump_active .. 0 homeless 2024-01-20T11:20:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.208+0000 7f30daffd640 1 -- 172.21.15.110:0/3262264133 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f30d4076a20 con 0x7f30e40b4d10 2024-01-20T11:20:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.208+0000 7f30daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.208+0000 7f30daffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.380+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f30e414c7f0 con 0x7f30c806c8b0 2024-01-20T11:20:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.383+0000 7f30daffd640 1 -- 172.21.15.110:0/3262264133 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63988 (secure 0 0 0) 0x7f30e00014e0 con 0x7f30c806c8b0 2024-01-20T11:20:32.384 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 10 librados: watch_flush enter 2024-01-20T11:20:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 10 librados: watch_flush exit 2024-01-20T11:20:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 20 client.5395.objecter shutdown clearing up homeless session... 2024-01-20T11:20:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 10 client.5395.objecter successfully canceled tick 2024-01-20T11:20:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f30c806c8b0 msgr2=0x7f30c806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f30c806c8b0 0x7f30c806ed70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f30e0001840 tx=0x7f30e0009040 comp rx=0 tx=0).stop 2024-01-20T11:20:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 10 monclient: shutdown 2024-01-20T11:20:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e40b4d10 msgr2=0x7f30e413da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e40b4d10 0x7f30e413da80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f30d4033000 tx=0x7f30d40046c0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 shutdown_connections 2024-01-20T11:20:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f30c806c8b0 0x7f30c806ed70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30e4145980 0x7f30e413dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e40b4d10 0x7f30e413da80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 --2- 172.21.15.110:0/3262264133 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30e40b1ed0 0x7f30e41392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 >> 172.21.15.110:0/3262264133 conn(0x7f30e4066590 msgr2=0x7f30e4143860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 shutdown_connections 2024-01-20T11:20:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 -- 172.21.15.110:0/3262264133 wait complete. 2024-01-20T11:20:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.385+0000 7f30ec37c640 1 librados: shutdown 2024-01-20T11:20:32.396 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-01-20T11:20:30.771580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002455"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280581,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.556},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.678000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.4260000000000002},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.119999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.704000000000001},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":7.5960000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.5259999999999998},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":11.454000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.528},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":9.4879999999999995}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.081},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":15.339}]}]},{"osd":7,"up_from":21,"seq":90194313283,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":8.766,"15min":9.8439999999999994},"min":{"1min":5.0819999999999999,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":9.8759999999999994,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.6689999999999996},{"interface":"front","average":{"1min":9.2390000000000008,"5min":11.247,"15min":13.121},"min":{"1min":4.8140000000000001,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.512,"5min":76.718999999999994,"15min":76.718999999999994},"last":15.167}]},{"osd":1,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":9.7940000000000005,"15min":10.366},"min":{"1min":5.7140000000000004,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":14.778,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.3230000000000004},{"interface":"front","average":{"1min":11.031000000000001,"5min":12.076000000000001,"15min":13.022},"min":{"1min":8.1270000000000007,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":16.920000000000002,"5min":43.774000000000001,"15min":43.774000000000001},"last":11.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":11.118,"5min":14.406000000000001,"15min":16.466000000000001},"min":{"1min":7.0049999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":15.215,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.4719999999999995},{"interface":"front","average":{"1min":10.516,"5min":13.734999999999999,"15min":16.018999999999998},"min":{"1min":6.3310000000000004,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":16.513000000000002,"5min":78.548000000000002,"15min":78.548000000000002},"last":5.9119999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":10.657,"5min":13.214,"15min":14.715999999999999},"min":{"1min":6.298,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.388000000000002,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.0869999999999997},{"interface":"front","average":{"1min":13.032999999999999,"5min":15.616,"15min":16.827999999999999},"min":{"1min":9.968,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":15.944000000000001,"5min":49.829999999999998,"15min":49.829999999999998},"last":12.300000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.965},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":9.8070000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.401,"5min":16.376999999999999,"15min":18.300999999999998},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.183,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.976000000000001},{"interface":"front","average":{"1min":13.795,"5min":17.007999999999999,"15min":18.675000000000001},"min":{"1min":8.0419999999999998,"5min":8.0419999999999998,"15min":8.0419999999999998},"max":{"1min":18.315000000000001,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.577}]}]},{"osd":3,"up_from":21,"seq":90194313284,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.2089999999999996},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.9710000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":6.6059999999999999},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":14.784000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":11.84},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":8.7129999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.4610000000000003},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.577999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":12.510999999999999},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":10.106999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":13.702},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.376999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444098,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.411},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":5.9889999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.2129999999999992},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.9180000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.6799999999999997},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.521000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":11.247},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.303000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.1470000000000002},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.8320000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.098000000000001},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":11.839}]}]},{"osd":1,"up_from":17,"seq":73014444099,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.2359999999999998},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":9.1359999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.297000000000001},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":14.974}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":16.401},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.7270000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":9.7729999999999997},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":17.196000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.012},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":10.459}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.999000000000001},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.977}]}]},{"osd":0,"up_from":17,"seq":73014444099,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":8.2149999999999999},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":9.1549999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.2889999999999997},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":16.462}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":15.670999999999999},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":6.609}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.534000000000001},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.8919999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":13.628},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":12.192}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.552},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":10.433999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":9.8819999999999997},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.1840000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":7.6980000000000004},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.629}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":6.976},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":15.484}]},{"osd":3,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":12.535,"5min":14.175000000000001,"15min":15.582000000000001},"min":{"1min":6.7039999999999997,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":17.195,"5min":81.373999999999995,"15min":81.373999999999995},"last":13.297000000000001},{"interface":"front","average":{"1min":13.114000000000001,"5min":14.337999999999999,"15min":15.898},"min":{"1min":6.8700000000000001,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":18.533000000000001,"5min":84.524000000000001,"15min":84.524000000000001},"last":8.7240000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.933999999999999},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.686}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":15.445,"15min":16.317},"min":{"1min":9.9510000000000005,"5min":9.7590000000000003,"15min":9.7590000000000003},"max":{"1min":18.468,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.3109999999999999},{"interface":"front","average":{"1min":12.379,"5min":14.714,"15min":15.718999999999999},"min":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"max":{"1min":19.515000000000001,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.747999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:32.398 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:32.398 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:32.399 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:32.399 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:32.399 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:32.399 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:32.399 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:32.399 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:32.400 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.498+0000 7f3998561640 10 monclient: get_monmap_and_config 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.498+0000 7f3998561640 10 monclient: build_initial_monmap 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.498+0000 7f3998561640 10 monclient: monmap: 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:32.499373+0000 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:32.499373+0000 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:32.500 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:32.500 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:32.500 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:32.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.499+0000 7f3998561640 1 Processor -- start 2024-01-20T11:20:32.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.499+0000 7f3998561640 1 -- start start 2024-01-20T11:20:32.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.499+0000 7f3998561640 10 monclient: init 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f399006a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f39900af730 0x7f39901455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 10 monclient(hunting): picked mon.noname-c con 0x7f39900af730 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3990145af0 0x7f399014c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 10 monclient(hunting): picked mon.noname-b con 0x7f3990145af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900afc50 con 0x7f399006a860 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f399006e450 con 0x7f39900af730 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900b0170 con 0x7f3990145af0 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:32.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3998561640 10 monclient(hunting): authenticate will time out at 1675.850322s 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3996afe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3990145af0 0x7f399014c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.500+0000 7f3995afc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f39900af730 0x7f39901455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3996afe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3990145af0 0x7f399014c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45768/0 (socket says 172.21.15.110:45768) 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3996afe640 1 -- 172.21.15.110:0/2126512743 learned_addr learned my addr 172.21.15.110:0/2126512743 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3996afe640 10 monclient(hunting): get_auth_request con 0x7f3990145af0 auth_method 0 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3996afe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3996afe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3996afe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3995afc640 10 monclient(hunting): get_auth_request con 0x7f39900af730 auth_method 0 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3995afc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3995afc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3995afc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3996afe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:32.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3996afe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:32.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3996afe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:32.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3995afc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:32.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3995afc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:32.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.501+0000 7f3995afc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:32.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.503+0000 7f3996afe640 10 monclient(hunting): handle_auth_done global_id 5084 payload 274 2024-01-20T11:20:32.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.503+0000 7f3996afe640 1 -- 172.21.15.110:0/2126512743 >> v2:172.21.15.134:3300/0 conn(0x7f39900af730 msgr2=0x7f39901455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:32.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.503+0000 7f3996afe640 1 --2- 172.21.15.110:0/2126512743 >> v2:172.21.15.134:3300/0 conn(0x7f39900af730 0x7f39901455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.503+0000 7f3996afe640 1 -- 172.21.15.110:0/2126512743 >> v1:172.21.15.134:6789/0 conn(0x7f399006a860 legacy=0x7f399013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:32.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.503+0000 7f3996afe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:32.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.503+0000 7f3996afe640 1 monclient: found mon.noname-b 2024-01-20T11:20:32.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.503+0000 7f3996afe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:32.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.503+0000 7f3996afe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:32.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.503+0000 7f3996afe640 1 -- 172.21.15.110:0/2126512743 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3990145fa0 con 0x7f3990145af0 2024-01-20T11:20:32.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.504+0000 7f3996afe640 1 --2- 172.21.15.110:0/2126512743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3990145af0 0x7f399014c940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f398c009e50 tx=0x7f398c02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e593a44fe9c62cc0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:32.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 1 -- 172.21.15.110:0/2126512743 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c02c950 con 0x7f3990145af0 2024-01-20T11:20:32.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:32.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:32.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 10 monclient: dump: 2024-01-20T11:20:32.508 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:32.508 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 10 monclient: _finish_auth 0 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 10 monclient: _check_auth_tickets 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 1 -- 172.21.15.110:0/2126512743 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c02cab0 con 0x7f3990145af0 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 1 -- 172.21.15.110:0/2126512743 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c02ce40 con 0x7f3990145af0 2024-01-20T11:20:32.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f39952fb640 10 monclient: dump: 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f3998561640 5 monclient: authenticate success, global_id 5084 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f3998561640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:32.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.508+0000 7f3998561640 4 set_mon_vals no callback set 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 10 monclient: shutdown 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 1 -- 172.21.15.110:0/2126512743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3990145af0 msgr2=0x7f399014c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 1 --2- 172.21.15.110:0/2126512743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3990145af0 0x7f399014c940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f398c009e50 tx=0x7f398c02bdb0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 1 -- 172.21.15.110:0/2126512743 shutdown_connections 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 1 --2- 172.21.15.110:0/2126512743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3990145af0 0x7f399014c940 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 1 --2- 172.21.15.110:0/2126512743 >> v2:172.21.15.134:3300/0 conn(0x7f39900af730 0x7f39901455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 1 -- 172.21.15.110:0/2126512743 >> 172.21.15.110:0/2126512743 conn(0x7f39900b47d0 msgr2=0x7f39900b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 1 -- 172.21.15.110:0/2126512743 shutdown_connections 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 1 -- 172.21.15.110:0/2126512743 wait complete. 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.509+0000 7f3998561640 10 monclient: build_initial_monmap 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient: monmap: 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 librados: starting msgr at 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 librados: starting objecter 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 Processor -- start 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 -- start start 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 librados: setting wanted keys 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 librados: calling monclient init 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient: init 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:32.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f399006a860 0x7f39901430a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient(hunting): picked mon.noname-c con 0x7f399006a860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39900af730 0x7f39901d7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient(hunting): picked mon.noname-a con 0x7f39900af730 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3990145af0 0x7f39901d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient(hunting): picked mon.noname-b con 0x7f3990145af0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39901d81f0 con 0x7f39900af730 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39901d72e0 con 0x7f3990145af0 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3990144cc0 con 0x7f399006a860 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.510+0000 7f3998561640 10 monclient(hunting): authenticate will time out at 1675.861126s 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3995afc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39900af730 0x7f39901d7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f39962fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f399006a860 0x7f39901430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3996afe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3990145af0 0x7f39901d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f39962fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f399006a860 0x7f39901430a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45774/0 (socket says 172.21.15.110:45774) 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f39962fd640 1 -- 172.21.15.110:0/79718111 learned_addr learned my addr 172.21.15.110:0/79718111 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f39962fd640 10 monclient(hunting): get_auth_request con 0x7f399006a860 auth_method 0 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f39962fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f39962fd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f39962fd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3996afe640 10 monclient(hunting): get_auth_request con 0x7f3990145af0 auth_method 0 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3996afe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3996afe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3996afe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3995afc640 10 monclient(hunting): get_auth_request con 0x7f39900af730 auth_method 0 2024-01-20T11:20:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3995afc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3995afc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.511+0000 7f3995afc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.512+0000 7f39962fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.512+0000 7f39962fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.512+0000 7f39962fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.512+0000 7f3996afe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.512+0000 7f3996afe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.512+0000 7f3996afe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.512+0000 7f3995afc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.512+0000 7f3995afc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.512+0000 7f3995afc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:32.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 10 monclient(hunting): handle_auth_done global_id 5401 payload 824 2024-01-20T11:20:32.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 1 -- 172.21.15.110:0/79718111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f399006a860 msgr2=0x7f39901430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:32.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f399006a860 0x7f39901430a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 1 -- 172.21.15.110:0/79718111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39900af730 msgr2=0x7f39901d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:32.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39900af730 0x7f39901d7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:32.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:32.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 1 monclient: found mon.noname-b 2024-01-20T11:20:32.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:32.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.514+0000 7f3996afe640 1 -- 172.21.15.110:0/79718111 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c009b30 con 0x7f3990145af0 2024-01-20T11:20:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.516+0000 7f3996afe640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3990145af0 0x7f39901d7cb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f398c032000 tx=0x7f398c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.516+0000 7f3986ffd640 10 client.?.objecter ms_handle_connect 0x7f3990145af0 2024-01-20T11:20:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.516+0000 7f3986ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 1 -- 172.21.15.110:0/79718111 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c040670 con 0x7f3990145af0 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 10 monclient: dump: 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 1 -- 172.21.15.110:0/79718111 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c040810 con 0x7f3990145af0 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3998561640 5 monclient: authenticate success, global_id 5401 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3998561640 10 monclient: _renew_subs 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3998561640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3990140370 con 0x7f3990145af0 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 1 -- 172.21.15.110:0/79718111 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c040c00 con 0x7f3990145af0 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3986ffd640 10 monclient: dump: 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3994afa640 4 set_mon_vals no callback set 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3998561640 10 client.5401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3998561640 10 monclient: _renew_subs 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3998561640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3990143ca0 con 0x7f3990145af0 2024-01-20T11:20:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.520+0000 7f3998561640 1 librados: init done 2024-01-20T11:20:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.521+0000 7f3998561640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.521+0000 7f3998561640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.521+0000 7f3998561640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.521+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39901d7470 con 0x7f3990145af0 2024-01-20T11:20:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.540+0000 7f3986ffd640 1 -- 172.21.15.110:0/79718111 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f398c053480 con 0x7f3990145af0 2024-01-20T11:20:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.540+0000 7f3986ffd640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f397406c8e0 0x7f397406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.541+0000 7f39962fd640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f397406c8e0 0x7f397406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.541+0000 7f39962fd640 10 monclient: get_auth_request con 0x7f397406c8e0 auth_method 0 2024-01-20T11:20:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.541+0000 7f39962fd640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f397406c8e0 0x7f397406eda0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f39880099a0 tx=0x7f39880090e0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.541+0000 7f3986ffd640 10 client.5401.objecter ms_handle_connect 0x7f397406c8e0 2024-01-20T11:20:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.543+0000 7f3986ffd640 1 -- 172.21.15.110:0/79718111 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f398c07bbe0 con 0x7f3990145af0 2024-01-20T11:20:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.543+0000 7f3986ffd640 10 client.5401.objecter ms_dispatch 0x7f3990000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.543+0000 7f3986ffd640 3 client.5401.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.543+0000 7f3986ffd640 3 client.5401.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.543+0000 7f3986ffd640 20 client.5401.objecter dump_active .. 0 homeless 2024-01-20T11:20:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.773+0000 7f3986ffd640 1 -- 172.21.15.110:0/79718111 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f398c077f20 con 0x7f3990145af0 2024-01-20T11:20:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.773+0000 7f3986ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:32.773+0000 7f3986ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:33.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.025+0000 7f3998561640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:33.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.025+0000 7f3998561640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:33.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.025+0000 7f3998561640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:33.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.025+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3990075490 con 0x7f3990145af0 2024-01-20T11:20:33.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.044+0000 7f3986ffd640 1 -- 172.21.15.110:0/79718111 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f398c07b540 con 0x7f3990145af0 2024-01-20T11:20:33.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.044+0000 7f3986ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:33.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.044+0000 7f3986ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:33.044 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:33.044 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":367,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 10 librados: watch_flush enter 2024-01-20T11:20:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 10 librados: watch_flush exit 2024-01-20T11:20:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 20 client.5401.objecter shutdown clearing up homeless session... 2024-01-20T11:20:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 10 client.5401.objecter successfully canceled tick 2024-01-20T11:20:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f397406c8e0 msgr2=0x7f397406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f397406c8e0 0x7f397406eda0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f39880099a0 tx=0x7f39880090e0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 10 monclient: shutdown 2024-01-20T11:20:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3990145af0 msgr2=0x7f39901d7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3990145af0 0x7f39901d7cb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f398c032000 tx=0x7f398c004a40 comp rx=0 tx=0).stop 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 shutdown_connections 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f397406c8e0 0x7f397406eda0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3990145af0 0x7f39901d7cb0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39900af730 0x7f39901d7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 --2- 172.21.15.110:0/79718111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f399006a860 0x7f39901430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 >> 172.21.15.110:0/79718111 conn(0x7f39900b47d0 msgr2=0x7f39900b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 shutdown_connections 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 -- 172.21.15.110:0/79718111 wait complete. 2024-01-20T11:20:33.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.046+0000 7f3998561640 1 librados: shutdown 2024-01-20T11:20:33.058 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:33.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:33.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.156+0000 7f8ba98c9640 10 monclient: get_monmap_and_config 2024-01-20T11:20:33.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.156+0000 7f8ba98c9640 10 monclient: build_initial_monmap 2024-01-20T11:20:33.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.156+0000 7f8ba98c9640 10 monclient: monmap: 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:33.157323+0000 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:33.157323+0000 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:33.157 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:33.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.157+0000 7f8ba98c9640 1 Processor -- start 2024-01-20T11:20:33.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.157+0000 7f8ba98c9640 1 -- start start 2024-01-20T11:20:33.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.157+0000 7f8ba98c9640 10 monclient: init 2024-01-20T11:20:33.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.157+0000 7f8ba98c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:33.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.157+0000 7f8ba98c9640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-01-20T11:20:33.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.157+0000 7f8ba98c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ba40b4d10 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:33.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.157+0000 7f8ba98c9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ba40b1ed0 0x7f8ba40b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:33.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.157+0000 7f8ba98c9640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba40b1ed0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba98c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ba4145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba98c9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba40aade0 con 0x7f8ba4145980 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba98c9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba4145460 con 0x7f8ba40b4d10 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba98c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba98c9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba40b50f0 con 0x7f8ba40b1ed0 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba98c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba98c9640 10 monclient(hunting): authenticate will time out at 1676.508245s 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba27fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ba40b1ed0 0x7f8ba40b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba2ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8ba40b4d10 0x7f8ba40b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:53846/0 (socket says 172.21.15.110:53846) 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.158+0000 7f8ba2ffd640 1 -- 172.21.15.110:0/3294067194 learned_addr learned my addr 172.21.15.110:0/3294067194 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.159+0000 7f8ba27fc640 10 monclient(hunting): get_auth_request con 0x7f8ba40b1ed0 auth_method 0 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.159+0000 7f8ba27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.159+0000 7f8ba27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:33.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.159+0000 7f8ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:33.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.160+0000 7f8ba27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.160+0000 7f8ba27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.160+0000 7f8ba27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.160+0000 7f8ba1ffb640 1 -- 172.21.15.110:0/3294067194 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186456905 0 0) 0x7f8ba4145460 con 0x7f8ba40b4d10 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.161+0000 7f8ba1ffb640 10 monclient(hunting): _negotiate 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.161+0000 7f8ba1ffb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.161+0000 7f8ba1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.161+0000 7f8ba1ffb640 10 monclient(hunting): my global_id is 5404 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.161+0000 7f8ba1ffb640 1 -- 172.21.15.110:0/3294067194 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8c003520 con 0x7f8ba40b4d10 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.161+0000 7f8ba27fc640 10 monclient(hunting): handle_auth_done global_id 5331 payload 274 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.161+0000 7f8ba27fc640 1 -- 172.21.15.110:0/3294067194 >> v1:172.21.15.134:6789/0 conn(0x7f8ba40b4d10 legacy=0x7f8ba40b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.161+0000 7f8ba27fc640 1 -- 172.21.15.110:0/3294067194 >> v1:172.21.15.110:6789/0 conn(0x7f8ba4145980 legacy=0x7f8ba4145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.161+0000 7f8ba27fc640 10 monclient: _finish_hunting 0 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.162+0000 7f8ba27fc640 1 monclient: found mon.noname-a 2024-01-20T11:20:33.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.162+0000 7f8ba27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:33.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.162+0000 7f8ba27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:33.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.162+0000 7f8ba27fc640 1 -- 172.21.15.110:0/3294067194 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40b2660 con 0x7f8ba40b1ed0 2024-01-20T11:20:33.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.164+0000 7f8ba27fc640 1 --2- 172.21.15.110:0/3294067194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 0x7f8ba40b3da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8b980097a0 tx=0x7f8b9802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7639991ea6029fd4 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 1 -- 172.21.15.110:0/3294067194 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b98031830 con 0x7f8ba40b1ed0 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 10 monclient: dump: 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:33.169 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 10 monclient: _finish_auth 0 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 1 -- 172.21.15.110:0/3294067194 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b98031990 con 0x7f8ba40b1ed0 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 5 monclient: authenticate success, global_id 5331 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 4 set_mon_vals no callback set 2024-01-20T11:20:33.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 1 -- 172.21.15.110:0/3294067194 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b98031d20 con 0x7f8ba40b1ed0 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba1ffb640 10 monclient: dump: 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:33.171 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 10 monclient: shutdown 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 1 -- 172.21.15.110:0/3294067194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 msgr2=0x7f8ba40b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 1 --2- 172.21.15.110:0/3294067194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 0x7f8ba40b3da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8b980097a0 tx=0x7f8b9802b8d0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 1 -- 172.21.15.110:0/3294067194 shutdown_connections 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 1 --2- 172.21.15.110:0/3294067194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 0x7f8ba40b3da0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.169+0000 7f8ba98c9640 1 -- 172.21.15.110:0/3294067194 >> 172.21.15.110:0/3294067194 conn(0x7f8ba4066590 msgr2=0x7f8ba40669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.170+0000 7f8ba98c9640 1 -- 172.21.15.110:0/3294067194 shutdown_connections 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.170+0000 7f8ba98c9640 1 -- 172.21.15.110:0/3294067194 wait complete. 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.170+0000 7f8ba98c9640 10 monclient: build_initial_monmap 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.170+0000 7f8ba98c9640 10 monclient: monmap: 2024-01-20T11:20:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.170+0000 7f8ba98c9640 1 librados: starting msgr at 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.170+0000 7f8ba98c9640 1 librados: starting objecter 2024-01-20T11:20:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.170+0000 7f8ba98c9640 1 Processor -- start 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 1 -- start start 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 1 librados: setting wanted keys 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 1 librados: calling monclient init 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient: init 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 0x7f8ba40ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba40b1ed0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba40b4d10 0x7f8ba40ab8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba40b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ba4145980 0x7f8ba40ac350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba4145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba40ad9e0 con 0x7f8ba40b1ed0 2024-01-20T11:20:33.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba40adcc0 con 0x7f8ba40b4d10 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba40ac890 con 0x7f8ba4145980 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.171+0000 7f8ba98c9640 10 monclient(hunting): authenticate will time out at 1676.521895s 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ba4145980 0x7f8ba40ac350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba40b4d10 0x7f8ba40ab8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba2ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 0x7f8ba40ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba2ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 0x7f8ba40ae0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40318/0 (socket says 172.21.15.110:40318) 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba2ffd640 1 -- 172.21.15.110:0/4159668265 learned_addr learned my addr 172.21.15.110:0/4159668265 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba2ffd640 10 monclient(hunting): get_auth_request con 0x7f8ba40b1ed0 auth_method 0 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:33.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba27fc640 10 monclient(hunting): get_auth_request con 0x7f8ba40b4d10 auth_method 0 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request con 0x7f8ba4145980 auth_method 0 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.172+0000 7f8ba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.173+0000 7f8ba27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.173+0000 7f8ba27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.173+0000 7f8ba27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.173+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.173+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.173+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.173+0000 7f8ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.173+0000 7f8ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.173+0000 7f8ba2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 10 monclient(hunting): handle_auth_done global_id 5407 payload 824 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 1 -- 172.21.15.110:0/4159668265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ba4145980 msgr2=0x7f8ba40ac350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ba4145980 0x7f8ba40ac350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 1 -- 172.21.15.110:0/4159668265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 msgr2=0x7f8ba40ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 0x7f8ba40ae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 10 monclient: _finish_hunting 0 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 1 monclient: found mon.noname-b 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.175+0000 7f8ba27fc640 1 -- 172.21.15.110:0/4159668265 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98009400 con 0x7f8ba40b4d10 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.177+0000 7f8ba27fc640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba40b4d10 0x7f8ba40ab8d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8b98033000 tx=0x7f8b980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.177+0000 7f8b8bfff640 10 client.?.objecter ms_handle_connect 0x7f8ba40b4d10 2024-01-20T11:20:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.177+0000 7f8b8bfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 1 -- 172.21.15.110:0/4159668265 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9803a6f0 con 0x7f8ba40b4d10 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 10 monclient: dump: 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:33.183 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 10 monclient: _finish_auth 0 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 1 -- 172.21.15.110:0/4159668265 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9803a870 con 0x7f8ba40b4d10 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 1 -- 172.21.15.110:0/4159668265 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9803ac20 con 0x7f8ba40b4d10 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8b8bfff640 10 monclient: dump: 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:33.184 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba98c9640 5 monclient: authenticate success, global_id 5407 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba98c9640 10 monclient: _renew_subs 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba98c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba40acab0 con 0x7f8ba40b4d10 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba17fa640 4 set_mon_vals no callback set 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba98c9640 10 client.5407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba98c9640 10 monclient: _renew_subs 2024-01-20T11:20:33.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba98c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba4137e30 con 0x7f8ba40b4d10 2024-01-20T11:20:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.181+0000 7f8ba98c9640 1 librados: init done 2024-01-20T11:20:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.182+0000 7f8ba98c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.182+0000 7f8ba98c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.182+0000 7f8ba98c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.182+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba40adb20 con 0x7f8ba40b4d10 2024-01-20T11:20:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.201+0000 7f8b8bfff640 1 -- 172.21.15.110:0/4159668265 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8b9804c430 con 0x7f8ba40b4d10 2024-01-20T11:20:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.201+0000 7f8b8bfff640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8b9006c8b0 0x7f8b9006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.201+0000 7f8ba2ffd640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8b9006c8b0 0x7f8b9006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.201+0000 7f8ba2ffd640 10 monclient: get_auth_request con 0x7f8b9006c8b0 auth_method 0 2024-01-20T11:20:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.202+0000 7f8ba2ffd640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8b9006c8b0 0x7f8b9006ed70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c001560 tx=0x7f8b9c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.202+0000 7f8b8bfff640 10 client.5407.objecter ms_handle_connect 0x7f8b9006c8b0 2024-01-20T11:20:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.203+0000 7f8b8bfff640 1 -- 172.21.15.110:0/4159668265 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8b9807a1a0 con 0x7f8ba40b4d10 2024-01-20T11:20:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.203+0000 7f8b8bfff640 10 client.5407.objecter ms_dispatch 0x7f8ba4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.203+0000 7f8b8bfff640 3 client.5407.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.203+0000 7f8b8bfff640 3 client.5407.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.203+0000 7f8b8bfff640 20 client.5407.objecter dump_active .. 0 homeless 2024-01-20T11:20:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.452+0000 7f8b8bfff640 1 -- 172.21.15.110:0/4159668265 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8b980764e0 con 0x7f8ba40b4d10 2024-01-20T11:20:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.452+0000 7f8b8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.452+0000 7f8b8bfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:33.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.620+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ba40a9cf0 con 0x7f8b9006c8b0 2024-01-20T11:20:33.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.622+0000 7f8b8bfff640 1 -- 172.21.15.110:0/4159668265 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63918 (secure 0 0 0) 0x7f8b9c002060 con 0x7f8b9006c8b0 2024-01-20T11:20:33.623 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:33.623 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 10 librados: watch_flush enter 2024-01-20T11:20:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 10 librados: watch_flush exit 2024-01-20T11:20:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 20 client.5407.objecter shutdown clearing up homeless session... 2024-01-20T11:20:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 10 client.5407.objecter successfully canceled tick 2024-01-20T11:20:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8b9006c8b0 msgr2=0x7f8b9006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8b9006c8b0 0x7f8b9006ed70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c001560 tx=0x7f8b9c009040 comp rx=0 tx=0).stop 2024-01-20T11:20:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 10 monclient: shutdown 2024-01-20T11:20:33.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba40b4d10 msgr2=0x7f8ba40ab8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba40b4d10 0x7f8ba40ab8d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8b98033000 tx=0x7f8b980046c0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 shutdown_connections 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8b9006c8b0 0x7f8b9006ed70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ba4145980 0x7f8ba40ac350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba40b4d10 0x7f8ba40ab8d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 --2- 172.21.15.110:0/4159668265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba40b1ed0 0x7f8ba40ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.624+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 >> 172.21.15.110:0/4159668265 conn(0x7f8ba4066590 msgr2=0x7f8ba4145e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.625+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 shutdown_connections 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.625+0000 7f8ba98c9640 1 -- 172.21.15.110:0/4159668265 wait complete. 2024-01-20T11:20:33.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:33.625+0000 7f8ba98c9640 1 librados: shutdown 2024-01-20T11:20:33.635 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-01-20T11:20:32.772000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002515"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280581,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":7.1600000000000001,"5min":10.837999999999999,"15min":13.852},"min":{"1min":5.3259999999999996,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":11.455,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.556},{"interface":"front","average":{"1min":8.6750000000000007,"5min":11.579000000000001,"15min":15.037000000000001},"min":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":13.545999999999999,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.678000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":9.0269999999999992,"5min":13.44,"15min":16.718},"min":{"1min":4.9640000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.255000000000001,"5min":98.700000000000003,"15min":98.700000000000003},"last":8.4260000000000002},{"interface":"front","average":{"1min":9.8849999999999998,"5min":13.300000000000001,"15min":15.804},"min":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":14.574999999999999,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.119999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":10.895,"5min":14.766999999999999,"15min":17.803000000000001},"min":{"1min":7.8869999999999996,"5min":7.3819999999999997,"15min":7.3819999999999997},"max":{"1min":14.577,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.704000000000001},{"interface":"front","average":{"1min":11.564,"5min":15.077,"15min":17.175999999999998},"min":{"1min":7.968,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.108000000000001,"5min":105.43899999999999,"15min":105.43899999999999},"last":7.5960000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":11.786,"5min":16.126000000000001,"15min":19.981000000000002},"min":{"1min":6.7759999999999998,"5min":6.7240000000000002,"15min":6.7240000000000002},"max":{"1min":15.340999999999999,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.5259999999999998},{"interface":"front","average":{"1min":12.701000000000001,"5min":18.155999999999999,"15min":21.785},"min":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.388000000000002,"5min":100.633,"15min":100.633},"last":11.454000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321999999999999,"5min":18.341000000000001,"15min":21.454999999999998},"min":{"1min":9.6769999999999996,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":15.904,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.528},{"interface":"front","average":{"1min":11.775,"5min":16.882999999999999,"15min":20.137},"min":{"1min":6.5250000000000004,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.018999999999998,"5min":107.962,"15min":107.962},"last":9.4879999999999995}]},{"osd":7,"last update":"Sat Jan 20 11:19:29 2024","interfaces":[{"interface":"back","average":{"1min":12.295,"5min":15.765000000000001,"15min":18.776},"min":{"1min":6.931,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":16.808,"5min":96.911000000000001,"15min":96.911000000000001},"last":10.081},{"interface":"front","average":{"1min":14.852,"5min":19.527000000000001,"15min":22.373000000000001},"min":{"1min":10.699,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":17.954999999999998,"5min":129.828,"15min":129.828},"last":15.339}]}]},{"osd":7,"up_from":21,"seq":90194313284,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":12.958},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.0010000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.4489999999999998},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":7.883}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":14.396000000000001},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.298999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.634},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.402999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.657,"5min":15.317,"15min":17.263000000000002},"min":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":17.327999999999999,"5min":91.667000000000002,"15min":91.667000000000002},"last":15.111000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":14.957000000000001,"15min":17.196999999999999},"min":{"1min":7.6139999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"max":{"1min":18.236999999999998,"5min":92.613,"15min":92.613},"last":10.954000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":13.670999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.792999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313285,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.2119999999999997},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.4690000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.0500000000000007},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.528}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.734},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.336}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.523},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.194000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":8.4770000000000003},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":12.686999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":10.516},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":17}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444098,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.411},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":5.9889999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.2129999999999992},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.9180000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.6799999999999997},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.521000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":11.247},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.303000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":8.1470000000000002},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":9.8320000000000007}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":13.098000000000001},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":11.839}]}]},{"osd":1,"up_from":17,"seq":73014444099,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.2359999999999998},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":9.1359999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.297000000000001},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":14.974}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":16.401},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.7270000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":9.7729999999999997},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":17.196000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.012},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":10.459}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.999000000000001},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.977}]}]},{"osd":0,"up_from":17,"seq":73014444099,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":8.2149999999999999},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":9.1549999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.2889999999999997},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":16.462}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":15.670999999999999},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":6.609}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.534000000000001},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.8919999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":13.628},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":12.192}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.552},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":10.433999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":11.534000000000001},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":8.5839999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.7329999999999997},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":12.976000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":12.198},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.9779999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.798},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.662000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.269,"5min":15.138999999999999,"15min":16.692},"min":{"1min":8.0790000000000006,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.756,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.975},{"interface":"front","average":{"1min":12.814,"5min":14.727,"15min":16.582999999999998},"min":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":18.061,"5min":72.670000000000002,"15min":72.670000000000002},"last":9.6489999999999991}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":10.397},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":7.149}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:36.637 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:36.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.737+0000 7f7beac83640 10 monclient: get_monmap_and_config 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.737+0000 7f7beac83640 10 monclient: build_initial_monmap 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.738+0000 7f7beac83640 10 monclient: monmap: 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:36.738590+0000 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:36.738590+0000 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:36.738 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:36.739 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.738+0000 7f7beac83640 1 Processor -- start 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.738+0000 7f7beac83640 1 -- start start 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.738+0000 7f7beac83640 10 monclient: init 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7be40b0c00 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be40accf0 0x7f7be4143200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 10 monclient(hunting): picked mon.noname-b con 0x7f7be40accf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7be4143740 0x7f7be414a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 10 monclient(hunting): picked mon.noname-c con 0x7f7be4143740 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be40ad450 con 0x7f7be40b0c00 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be40ad8e0 con 0x7f7be4143740 2024-01-20T11:20:36.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be40ad130 con 0x7f7be40accf0 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.739+0000 7f7beac83640 10 monclient(hunting): authenticate will time out at 1680.089638s 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be40accf0 0x7f7be4143200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be9220640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7be4143740 0x7f7be414a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be40accf0 0x7f7be4143200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45792/0 (socket says 172.21.15.110:45792) 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 1 -- 172.21.15.110:0/1098018060 learned_addr learned my addr 172.21.15.110:0/1098018060 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 10 monclient(hunting): get_auth_request con 0x7f7be40accf0 auth_method 0 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be9220640 10 monclient(hunting): get_auth_request con 0x7f7be4143740 auth_method 0 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be9220640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be9220640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be9220640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:36.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.740+0000 7f7be3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.741+0000 7f7be9220640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.741+0000 7f7be9220640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.741+0000 7f7be9220640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.742+0000 7f7be3fff640 10 monclient(hunting): handle_auth_done global_id 5093 payload 274 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.742+0000 7f7be3fff640 1 -- 172.21.15.110:0/1098018060 >> v2:172.21.15.134:3300/0 conn(0x7f7be4143740 msgr2=0x7f7be414a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.742+0000 7f7be3fff640 1 --2- 172.21.15.110:0/1098018060 >> v2:172.21.15.134:3300/0 conn(0x7f7be4143740 0x7f7be414a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.742+0000 7f7be3fff640 1 -- 172.21.15.110:0/1098018060 >> v1:172.21.15.134:6789/0 conn(0x7f7be40b0c00 legacy=0x7f7be413d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:36.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.742+0000 7f7be3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:36.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.742+0000 7f7be3fff640 1 monclient: found mon.noname-b 2024-01-20T11:20:36.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.742+0000 7f7be3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:36.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.742+0000 7f7be3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:36.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.742+0000 7f7be3fff640 1 -- 172.21.15.110:0/1098018060 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be40aa300 con 0x7f7be40accf0 2024-01-20T11:20:36.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.744+0000 7f7be3fff640 1 --2- 172.21.15.110:0/1098018060 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be40accf0 0x7f7be4143200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009870 tx=0x7f7bd802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecdbdbfff9255d18 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:36.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 1 -- 172.21.15.110:0/1098018060 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd80317c0 con 0x7f7be40accf0 2024-01-20T11:20:36.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:36.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:36.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 10 monclient: dump: 2024-01-20T11:20:36.748 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:36.748 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:36.748 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 1 -- 172.21.15.110:0/1098018060 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd8031920 con 0x7f7be40accf0 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 1 -- 172.21.15.110:0/1098018060 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8031cb0 con 0x7f7be40accf0 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:36.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7be37fe640 10 monclient: dump: 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7beac83640 5 monclient: authenticate success, global_id 5093 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7beac83640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:36.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.748+0000 7f7beac83640 4 set_mon_vals no callback set 2024-01-20T11:20:36.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 10 monclient: shutdown 2024-01-20T11:20:36.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 1 -- 172.21.15.110:0/1098018060 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be40accf0 msgr2=0x7f7be4143200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 1 --2- 172.21.15.110:0/1098018060 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be40accf0 0x7f7be4143200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009870 tx=0x7f7bd802b860 comp rx=0 tx=0).stop 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 1 -- 172.21.15.110:0/1098018060 shutdown_connections 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 1 --2- 172.21.15.110:0/1098018060 >> v2:172.21.15.134:3300/0 conn(0x7f7be4143740 0x7f7be414a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 1 --2- 172.21.15.110:0/1098018060 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be40accf0 0x7f7be4143200 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 1 -- 172.21.15.110:0/1098018060 >> 172.21.15.110:0/1098018060 conn(0x7f7be40b3e60 msgr2=0x7f7be40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 1 -- 172.21.15.110:0/1098018060 shutdown_connections 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 1 -- 172.21.15.110:0/1098018060 wait complete. 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 10 monclient: build_initial_monmap 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.749+0000 7f7beac83640 10 monclient: monmap: 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.750+0000 7f7beac83640 1 librados: starting msgr at 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.750+0000 7f7beac83640 1 librados: starting objecter 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.750+0000 7f7beac83640 1 Processor -- start 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.750+0000 7f7beac83640 1 -- start start 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.750+0000 7f7beac83640 1 librados: setting wanted keys 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.750+0000 7f7beac83640 1 librados: calling monclient init 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.750+0000 7f7beac83640 10 monclient: init 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7be40accf0 0x7f7be41c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient(hunting): picked mon.noname-b con 0x7f7be40accf0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7be40b0c00 0x7f7be41cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient(hunting): picked mon.noname-a con 0x7f7be40b0c00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be4143740 0x7f7be41cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient(hunting): picked mon.noname-c con 0x7f7be4143740 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be406e620 con 0x7f7be40b0c00 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be406e8d0 con 0x7f7be40accf0 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be406ebb0 con 0x7f7be4143740 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7beac83640 10 monclient(hunting): authenticate will time out at 1680.101358s 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7be9220640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be4143740 0x7f7be41cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7be8a1f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7be40accf0 0x7f7be41c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7be3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7be40b0c00 0x7f7be41cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7be9220640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be4143740 0x7f7be41cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45802/0 (socket says 172.21.15.110:45802) 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7be9220640 1 -- 172.21.15.110:0/1531216135 learned_addr learned my addr 172.21.15.110:0/1531216135 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7be9220640 10 monclient(hunting): get_auth_request con 0x7f7be4143740 auth_method 0 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7be9220640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7be9220640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.751+0000 7f7be9220640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be8a1f640 10 monclient(hunting): get_auth_request con 0x7f7be40accf0 auth_method 0 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be8a1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be8a1f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be8a1f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be3fff640 10 monclient(hunting): get_auth_request con 0x7f7be40b0c00 auth_method 0 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be8a1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be8a1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be8a1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be9220640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be9220640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be9220640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.752+0000 7f7be3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 10 monclient(hunting): handle_auth_done global_id 5413 payload 824 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 1 -- 172.21.15.110:0/1531216135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be4143740 msgr2=0x7f7be41cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be4143740 0x7f7be41cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 1 -- 172.21.15.110:0/1531216135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7be40b0c00 msgr2=0x7f7be41cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7be40b0c00 0x7f7be41cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 10 monclient: _finish_hunting 0 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 1 monclient: found mon.noname-b 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:36.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.755+0000 7f7be8a1f640 1 -- 172.21.15.110:0/1531216135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd80094d0 con 0x7f7be40accf0 2024-01-20T11:20:36.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.757+0000 7f7be8a1f640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7be40accf0 0x7f7be41c9000 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400c340 tx=0x7f7bd400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:36.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.757+0000 7f7be17fa640 10 client.?.objecter ms_handle_connect 0x7f7be40accf0 2024-01-20T11:20:36.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.757+0000 7f7be17fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 1 -- 172.21.15.110:0/1531216135 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd40052d0 con 0x7f7be40accf0 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 10 monclient: dump: 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 1 -- 172.21.15.110:0/1531216135 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd4005450 con 0x7f7be40accf0 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 1 -- 172.21.15.110:0/1531216135 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4007260 con 0x7f7be40accf0 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7beac83640 5 monclient: authenticate success, global_id 5413 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7beac83640 10 monclient: _renew_subs 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7beac83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be41cbf40 con 0x7f7be40accf0 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be2ffd640 4 set_mon_vals no callback set 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7be17fa640 10 monclient: dump: 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7beac83640 10 client.5413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7beac83640 10 monclient: _renew_subs 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7beac83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be41cc9a0 con 0x7f7be40accf0 2024-01-20T11:20:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.761+0000 7f7beac83640 1 librados: init done 2024-01-20T11:20:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.762+0000 7f7beac83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.762+0000 7f7beac83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.762+0000 7f7beac83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:36.762+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be41cc180 con 0x7f7be40accf0 2024-01-20T11:20:37.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.002+0000 7f7be17fa640 1 -- 172.21.15.110:0/1531216135 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7bd40073c0 con 0x7f7be40accf0 2024-01-20T11:20:37.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.002+0000 7f7be17fa640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7bd006c8b0 0x7f7bd006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.003+0000 7f7be3fff640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7bd006c8b0 0x7f7bd006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.003+0000 7f7be3fff640 10 monclient: get_auth_request con 0x7f7bd006c8b0 auth_method 0 2024-01-20T11:20:37.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.003+0000 7f7be3fff640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7bd006c8b0 0x7f7bd006ed70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8036000 tx=0x7f7bd803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:37.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.003+0000 7f7be17fa640 10 client.5413.objecter ms_handle_connect 0x7f7bd006c8b0 2024-01-20T11:20:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.005+0000 7f7be17fa640 1 -- 172.21.15.110:0/1531216135 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7bd40578f0 con 0x7f7be40accf0 2024-01-20T11:20:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.005+0000 7f7be17fa640 10 client.5413.objecter ms_dispatch 0x7f7be4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.005+0000 7f7be17fa640 3 client.5413.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.005+0000 7f7be17fa640 3 client.5413.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.005+0000 7f7be17fa640 20 client.5413.objecter dump_active .. 0 homeless 2024-01-20T11:20:37.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.015+0000 7f7be17fa640 1 -- 172.21.15.110:0/1531216135 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7bd4053c30 con 0x7f7be40accf0 2024-01-20T11:20:37.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.015+0000 7f7be17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:37.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.015+0000 7f7be17fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:37.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.182+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7be4143bd0 con 0x7f7bd006c8b0 2024-01-20T11:20:37.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.186+0000 7f7be17fa640 1 -- 172.21.15.110:0/1531216135 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63866 (secure 0 0 0) 0x7f7bd8031b10 con 0x7f7bd006c8b0 2024-01-20T11:20:37.186 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:37.187 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:37.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 10 librados: watch_flush enter 2024-01-20T11:20:37.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 10 librados: watch_flush exit 2024-01-20T11:20:37.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 20 client.5413.objecter shutdown clearing up homeless session... 2024-01-20T11:20:37.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 10 client.5413.objecter successfully canceled tick 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7bd006c8b0 msgr2=0x7f7bd006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7bd006c8b0 0x7f7bd006ed70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8036000 tx=0x7f7bd803d040 comp rx=0 tx=0).stop 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 10 monclient: shutdown 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7be40accf0 msgr2=0x7f7be41c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7be40accf0 0x7f7be41c9000 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400c340 tx=0x7f7bd400c800 comp rx=0 tx=0).stop 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 shutdown_connections 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7bd006c8b0 0x7f7bd006ed70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7be4143740 0x7f7be41cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7be40b0c00 0x7f7be41cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 --2- 172.21.15.110:0/1531216135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7be40accf0 0x7f7be41c9000 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 >> 172.21.15.110:0/1531216135 conn(0x7f7be40b3e60 msgr2=0x7f7be40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 shutdown_connections 2024-01-20T11:20:37.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.188+0000 7f7beac83640 1 -- 172.21.15.110:0/1531216135 wait complete. 2024-01-20T11:20:37.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.189+0000 7f7beac83640 1 librados: shutdown 2024-01-20T11:20:37.199 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-01-20T11:20:36.772764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002400"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.0430000000000001},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.1100000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":6.8099999999999996},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.664}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":8.077},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.769}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.6370000000000005},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":12.483000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.154},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":14.558}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.2149999999999999},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":13.055}]}]},{"osd":7,"up_from":21,"seq":90194313285,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":10.974},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":10.342000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.284000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.4600000000000009},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.7919999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.643000000000001},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.147}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":13.757999999999999},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":7.8760000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":9.6890000000000001},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.409000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313285,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.2119999999999997},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.4690000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.0500000000000007},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.528}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.734},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.336}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.523},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.194000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":8.4770000000000003},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":12.686999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":10.516},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":17}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444099,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.9279999999999999},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":5.2430000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.452},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.544}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.7759999999999998},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.365}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":11.946999999999999},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.426}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":17.018000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.3579999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.8689999999999998},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":15.342000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444100,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":14.042},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":15.800000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":5.6920000000000002},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.4930000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":11.754},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.0920000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":6.9160000000000004},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.157}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.6790000000000003},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":12.081}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.372999999999999},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.907}]}]},{"osd":0,"up_from":17,"seq":73014444100,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":11.259},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":7.032}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.3029999999999999},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.333}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":14.734999999999999},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.6189999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":8.9610000000000003},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.2669999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":15.926},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":12.307}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.000999999999999},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":11.927}]}]},{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":11.6},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.6369999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.3869999999999996},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.242000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.6150000000000002},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.0410000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.102},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.24}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.005000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.3559999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.949},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":7.4889999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:37.201 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:37.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.300+0000 7ff612390640 10 monclient: get_monmap_and_config 2024-01-20T11:20:37.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.300+0000 7ff612390640 10 monclient: build_initial_monmap 2024-01-20T11:20:37.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.300+0000 7ff612390640 10 monclient: monmap: 2024-01-20T11:20:37.300 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:37.300 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.300 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:37.300920+0000 2024-01-20T11:20:37.301 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:37.300920+0000 2024-01-20T11:20:37.301 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:37.301 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.301 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:37.301 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:37.301 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:37.301 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:37.301 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:37.301 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 1 Processor -- start 2024-01-20T11:20:37.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 1 -- start start 2024-01-20T11:20:37.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient: init 2024-01-20T11:20:37.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:37.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-01-20T11:20:37.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a96f0 0x7ff60c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient(hunting): picked mon.noname-b con 0x7ff60c0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff60c13fa10 0x7ff60c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient(hunting): picked mon.noname-a con 0x7ff60c13fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff60c0a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60c0b25c0 con 0x7ff60c0a9ad0 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c0aaea0 con 0x7ff60c13fa10 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0ab0a0 con 0x7ff60c0a96f0 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.301+0000 7ff612390640 10 monclient(hunting): authenticate will time out at 1680.651975s 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.302+0000 7ff60bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a96f0 0x7ff60c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.302+0000 7ff60bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a96f0 0x7ff60c13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:45816/0 (socket says 172.21.15.110:45816) 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.302+0000 7ff60bfff640 1 -- 172.21.15.110:0/917363812 learned_addr learned my addr 172.21.15.110:0/917363812 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:37.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.302+0000 7ff60bfff640 10 monclient(hunting): get_auth_request con 0x7ff60c0a96f0 auth_method 0 2024-01-20T11:20:37.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.302+0000 7ff60bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.302+0000 7ff60bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.302+0000 7ff60bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.303+0000 7ff60bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.303+0000 7ff60bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.303+0000 7ff60bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.303+0000 7ff60b7fe640 1 --2- 172.21.15.110:0/917363812 >> v2:172.21.15.110:3300/0 conn(0x7ff60c13fa10 0x7ff60c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.304+0000 7ff60b7fe640 10 monclient(hunting): get_auth_request con 0x7ff60c13fa10 auth_method 0 2024-01-20T11:20:37.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.304+0000 7ff60b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.304+0000 7ff60b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.304+0000 7ff60b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.304+0000 7ff60b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.304+0000 7ff60b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.304+0000 7ff60b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.305+0000 7ff60bfff640 10 monclient(hunting): handle_auth_done global_id 5099 payload 274 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.305+0000 7ff60bfff640 1 -- 172.21.15.110:0/917363812 >> v2:172.21.15.110:3300/0 conn(0x7ff60c13fa10 msgr2=0x7ff60c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.305+0000 7ff60bfff640 1 --2- 172.21.15.110:0/917363812 >> v2:172.21.15.110:3300/0 conn(0x7ff60c13fa10 0x7ff60c1423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.305+0000 7ff60bfff640 1 -- 172.21.15.110:0/917363812 >> v1:172.21.15.110:6789/0 conn(0x7ff60c0a9ad0 legacy=0x7ff60c142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.305+0000 7ff60bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.305+0000 7ff60bfff640 1 monclient: found mon.noname-b 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.305+0000 7ff60bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.305+0000 7ff60bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.305+0000 7ff60bfff640 1 -- 172.21.15.110:0/917363812 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c0b3fb0 con 0x7ff60c0a96f0 2024-01-20T11:20:37.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.306+0000 7ff60bfff640 1 --2- 172.21.15.110:0/917363812 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a96f0 0x7ff60c13f4d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009870 tx=0x7ff5fc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc859328d8ed536b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:37.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.309+0000 7ff60affd640 1 -- 172.21.15.110:0/917363812 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc0317c0 con 0x7ff60c0a96f0 2024-01-20T11:20:37.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.309+0000 7ff60affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.309+0000 7ff60affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.309+0000 7ff60affd640 10 monclient: dump: 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.309+0000 7ff60affd640 10 monclient: _finish_auth 0 2024-01-20T11:20:37.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.309+0000 7ff60affd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:37.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.309+0000 7ff60affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:37.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.309+0000 7ff612390640 5 monclient: authenticate success, global_id 5099 2024-01-20T11:20:37.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.309+0000 7ff612390640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:37.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.310+0000 7ff60affd640 1 -- 172.21.15.110:0/917363812 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5fc031920 con 0x7ff60c0a96f0 2024-01-20T11:20:37.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.310+0000 7ff60affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:37.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.310+0000 7ff612390640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:37.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.310+0000 7ff60affd640 1 -- 172.21.15.110:0/917363812 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc031cb0 con 0x7ff60c0a96f0 2024-01-20T11:20:37.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.310+0000 7ff60affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:37.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.310+0000 7ff612390640 4 set_mon_vals no callback set 2024-01-20T11:20:37.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.310+0000 7ff60affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.310+0000 7ff60affd640 10 monclient: dump: 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 10 monclient: shutdown 2024-01-20T11:20:37.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:37.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 1 -- 172.21.15.110:0/917363812 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a96f0 msgr2=0x7ff60c13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 1 --2- 172.21.15.110:0/917363812 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a96f0 0x7ff60c13f4d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009870 tx=0x7ff5fc02b860 comp rx=0 tx=0).stop 2024-01-20T11:20:37.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 1 -- 172.21.15.110:0/917363812 shutdown_connections 2024-01-20T11:20:37.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 1 --2- 172.21.15.110:0/917363812 >> v2:172.21.15.110:3300/0 conn(0x7ff60c13fa10 0x7ff60c1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 1 --2- 172.21.15.110:0/917363812 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a96f0 0x7ff60c13f4d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 1 -- 172.21.15.110:0/917363812 >> 172.21.15.110:0/917363812 conn(0x7ff60c0b7bc0 msgr2=0x7ff60c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:37.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 1 -- 172.21.15.110:0/917363812 shutdown_connections 2024-01-20T11:20:37.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.311+0000 7ff612390640 1 -- 172.21.15.110:0/917363812 wait complete. 2024-01-20T11:20:37.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.312+0000 7ff612390640 10 monclient: build_initial_monmap 2024-01-20T11:20:37.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.312+0000 7ff612390640 10 monclient: monmap: 2024-01-20T11:20:37.316 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:37.316 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:37.316 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:37.316 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:37.316 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:37.316 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.316 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.312+0000 7ff612390640 1 librados: starting msgr at 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.312+0000 7ff612390640 1 librados: starting objecter 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.312+0000 7ff612390640 1 Processor -- start 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.312+0000 7ff612390640 1 -- start start 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.312+0000 7ff612390640 1 librados: setting wanted keys 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.312+0000 7ff612390640 1 librados: calling monclient init 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.312+0000 7ff612390640 10 monclient: init 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c0a96f0 0x7ff60c13ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient(hunting): picked mon.noname-a con 0x7ff60c0a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:37.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a9ad0 0x7ff60c133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient(hunting): picked mon.noname-c con 0x7ff60c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff60c137ff0 0x7ff60c1341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient(hunting): picked mon.noname-b con 0x7ff60c137ff0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c134720 con 0x7ff60c0a96f0 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1452e0 con 0x7ff60c137ff0 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c146000 con 0x7ff60c0a9ad0 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff612390640 10 monclient(hunting): authenticate will time out at 1680.663395s 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff60bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c0a96f0 0x7ff60c13ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.313+0000 7ff61092d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff60c137ff0 0x7ff60c1341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c0a96f0 0x7ff60c13ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40364/0 (socket says 172.21.15.110:40364) 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60bfff640 1 -- 172.21.15.110:0/3011296181 learned_addr learned my addr 172.21.15.110:0/3011296181 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60bfff640 10 monclient(hunting): get_auth_request con 0x7ff60c0a96f0 auth_method 0 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff61092d640 10 monclient(hunting): get_auth_request con 0x7ff60c137ff0 auth_method 0 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff61092d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff61092d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff61092d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60b7fe640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a9ad0 0x7ff60c133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff60bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff61092d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff61092d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.314+0000 7ff61092d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.315+0000 7ff60b7fe640 10 monclient(hunting): get_auth_request con 0x7ff60c0a9ad0 auth_method 0 2024-01-20T11:20:37.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.315+0000 7ff60b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.315+0000 7ff60b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.315+0000 7ff60b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.316+0000 7ff60b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.316+0000 7ff60b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.316+0000 7ff60b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 10 monclient(hunting): handle_auth_done global_id 5416 payload 824 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 1 -- 172.21.15.110:0/3011296181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a9ad0 msgr2=0x7ff60c133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a9ad0 0x7ff60c133ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 1 -- 172.21.15.110:0/3011296181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c0a96f0 msgr2=0x7ff60c13ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c0a96f0 0x7ff60c13ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 10 monclient: _finish_hunting 0 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 1 monclient: found mon.noname-b 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.317+0000 7ff61092d640 1 -- 172.21.15.110:0/3011296181 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0094d0 con 0x7ff60c137ff0 2024-01-20T11:20:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.320+0000 7ff61092d640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff60c137ff0 0x7ff60c1341e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff60400c780 tx=0x7ff60400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.320+0000 7ff608ff9640 10 client.?.objecter ms_handle_connect 0x7ff60c137ff0 2024-01-20T11:20:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.320+0000 7ff608ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:37.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 1 -- 172.21.15.110:0/3011296181 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60400aa50 con 0x7ff60c137ff0 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 10 monclient: dump: 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 10 monclient: _finish_auth 0 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 1 -- 172.21.15.110:0/3011296181 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60400abb0 con 0x7ff60c137ff0 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 1 -- 172.21.15.110:0/3011296181 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60400ce30 con 0x7ff60c137ff0 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff608ff9640 10 monclient: dump: 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.327 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff612390640 5 monclient: authenticate success, global_id 5416 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff612390640 10 monclient: _renew_subs 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff612390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c145a70 con 0x7ff60c137ff0 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff60a7fc640 4 set_mon_vals no callback set 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff612390640 10 client.5416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff612390640 10 monclient: _renew_subs 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff612390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c1d7070 con 0x7ff60c137ff0 2024-01-20T11:20:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.325+0000 7ff612390640 1 librados: init done 2024-01-20T11:20:37.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.326+0000 7ff612390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:37.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.326+0000 7ff612390640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:37.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.326+0000 7ff612390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:37.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.326+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c075490 con 0x7ff60c137ff0 2024-01-20T11:20:37.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.345+0000 7ff608ff9640 1 -- 172.21.15.110:0/3011296181 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff604024430 con 0x7ff60c137ff0 2024-01-20T11:20:37.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.345+0000 7ff608ff9640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff5f806c8e0 0x7ff5f806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.345+0000 7ff60bfff640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff5f806c8e0 0x7ff5f806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.345+0000 7ff60bfff640 10 monclient: get_auth_request con 0x7ff5f806c8e0 auth_method 0 2024-01-20T11:20:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.346+0000 7ff60bfff640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff5f806c8e0 0x7ff5f806eda0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0090b0 tx=0x7ff5fc03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.346+0000 7ff608ff9640 10 client.5416.objecter ms_handle_connect 0x7ff5f806c8e0 2024-01-20T11:20:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.348+0000 7ff608ff9640 1 -- 172.21.15.110:0/3011296181 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff604058620 con 0x7ff60c137ff0 2024-01-20T11:20:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.348+0000 7ff608ff9640 10 client.5416.objecter ms_dispatch 0x7ff60c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.348+0000 7ff608ff9640 3 client.5416.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.348+0000 7ff608ff9640 3 client.5416.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.348+0000 7ff608ff9640 20 client.5416.objecter dump_active .. 0 homeless 2024-01-20T11:20:37.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~99s 2024-01-20T11:20:37.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.576+0000 7ff608ff9640 1 -- 172.21.15.110:0/3011296181 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff604054960 con 0x7ff60c137ff0 2024-01-20T11:20:37.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.576+0000 7ff608ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:37.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.576+0000 7ff608ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:37.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.743+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff60c069850 con 0x7ff5f806c8e0 2024-01-20T11:20:37.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.746+0000 7ff608ff9640 1 -- 172.21.15.110:0/3011296181 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63866 (secure 0 0 0) 0x7ff60c069850 con 0x7ff5f806c8e0 2024-01-20T11:20:37.746 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:37.746 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:37.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 10 librados: watch_flush enter 2024-01-20T11:20:37.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 10 librados: watch_flush exit 2024-01-20T11:20:37.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 20 client.5416.objecter shutdown clearing up homeless session... 2024-01-20T11:20:37.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 10 client.5416.objecter successfully canceled tick 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff5f806c8e0 msgr2=0x7ff5f806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff5f806c8e0 0x7ff5f806eda0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0090b0 tx=0x7ff5fc03d040 comp rx=0 tx=0).stop 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 10 monclient: shutdown 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff60c137ff0 msgr2=0x7ff60c1341e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.747+0000 7ff612390640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff60c137ff0 0x7ff60c1341e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff60400c780 tx=0x7ff60400cc40 comp rx=0 tx=0).stop 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.748+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 shutdown_connections 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.748+0000 7ff612390640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff5f806c8e0 0x7ff5f806eda0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.748+0000 7ff612390640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff60c137ff0 0x7ff60c1341e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.748+0000 7ff612390640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0a9ad0 0x7ff60c133ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.748+0000 7ff612390640 1 --2- 172.21.15.110:0/3011296181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c0a96f0 0x7ff60c13ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.748+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 >> 172.21.15.110:0/3011296181 conn(0x7ff60c0b7bc0 msgr2=0x7ff60c0b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.748+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 shutdown_connections 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.748+0000 7ff612390640 1 -- 172.21.15.110:0/3011296181 wait complete. 2024-01-20T11:20:37.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.748+0000 7ff612390640 1 librados: shutdown 2024-01-20T11:20:37.758 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-01-20T11:20:36.772764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002400"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.0430000000000001},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.1100000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":6.8099999999999996},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":11.664}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":8.077},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.769}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":8.6370000000000005},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":12.483000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.154},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":14.558}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.2149999999999999},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":13.055}]}]},{"osd":7,"up_from":21,"seq":90194313285,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":10.974},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":10.342000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.284000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.4600000000000009},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.7919999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.643000000000001},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.147}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":13.757999999999999},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":7.8760000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":9.6890000000000001},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.409000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313285,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":7.7759999999999998,"5min":11.890000000000001,"15min":15.58},"min":{"1min":4.8479999999999999,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":18.129000000000001,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.2119999999999997},{"interface":"front","average":{"1min":7.5010000000000003,"5min":10.712999999999999,"15min":13.76},"min":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":12.83,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.4690000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":9.6029999999999998,"5min":12.752000000000001,"15min":15.332000000000001},"min":{"1min":7.8650000000000002,"5min":6.6879999999999997,"15min":6.6879999999999997},"max":{"1min":13.394,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.0500000000000007},{"interface":"front","average":{"1min":10.667,"5min":15.148999999999999,"15min":18.725000000000001},"min":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"max":{"1min":16.199000000000002,"5min":155.91200000000001,"15min":155.91200000000001},"last":13.528}]},{"osd":2,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":10.423,"5min":14.096,"15min":17.494},"min":{"1min":6.6150000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.574999999999999,"5min":145.553,"15min":145.553},"last":7.734},{"interface":"front","average":{"1min":10.608000000000001,"5min":14.492000000000001,"15min":17.513999999999999},"min":{"1min":7.9130000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":13.132,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.336}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.968999999999999,"5min":18.945,"15min":24.492000000000001},"min":{"1min":9.1210000000000004,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.34,"5min":284.00700000000001,"15min":284.00700000000001},"last":11.523},{"interface":"front","average":{"1min":12.920999999999999,"5min":19.042000000000002,"15min":25.161999999999999},"min":{"1min":7.8470000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":18.693000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.194000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":15.446,"15min":19.582999999999998},"min":{"1min":6.5579999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.975,"5min":146.744,"15min":146.744},"last":8.4770000000000003},{"interface":"front","average":{"1min":12.746,"5min":17.071999999999999,"15min":20.408999999999999},"min":{"1min":8.5969999999999995,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":15.551,"5min":150.542,"15min":150.542},"last":12.686999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":18.997,"15min":22.439},"min":{"1min":10.573,"5min":10.573,"15min":10.573},"max":{"1min":17.405999999999999,"5min":159.179,"15min":159.179},"last":10.516},{"interface":"front","average":{"1min":14.105,"5min":18.175000000000001,"15min":22.344999999999999},"min":{"1min":12.016,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.183,"5min":157.35599999999999,"15min":157.35599999999999},"last":17}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444099,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.9279999999999999},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":5.2430000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":10.452},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.544}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.7759999999999998},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":14.365}]},{"osd":4,"last update":"Sat Jan 20 11:19:34 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":15.468,"15min":17.170999999999999},"min":{"1min":8.7650000000000006,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":17.513999999999999,"5min":51.000999999999998,"15min":51.000999999999998},"last":11.946999999999999},{"interface":"front","average":{"1min":12.302,"5min":13.32,"15min":14.340999999999999},"min":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.219000000000001,"5min":48.299999999999997,"15min":48.299999999999997},"last":13.426}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":17.018000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.3579999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.8689999999999998},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":15.342000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444100,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":14.042},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":15.800000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":5.6920000000000002},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.4930000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":11.754},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.0920000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:19:32 2024","interfaces":[{"interface":"back","average":{"1min":12.242000000000001,"5min":13.125999999999999,"15min":13.904},"min":{"1min":6.5800000000000001,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":16.736999999999998,"5min":25.541,"15min":25.541},"last":6.9160000000000004},{"interface":"front","average":{"1min":13.788,"5min":14.808,"15min":15.161},"min":{"1min":9.6950000000000003,"5min":9.6950000000000003,"15min":9.6950000000000003},"max":{"1min":16.84,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.157}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.6790000000000003},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":12.081}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.372999999999999},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.907}]}]},{"osd":0,"up_from":17,"seq":73014444100,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":11.259},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":7.032}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.3029999999999999},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.333}]},{"osd":3,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":11.955,"5min":19.280000000000001,"15min":26.425999999999998},"min":{"1min":9.2829999999999995,"5min":8.1319999999999997,"15min":8.1319999999999997},"max":{"1min":15.148,"5min":358.738,"15min":358.738},"last":14.734999999999999},{"interface":"front","average":{"1min":10.994,"5min":11.833,"15min":12.781000000000001},"min":{"1min":7.9829999999999997,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":14.615,"5min":21.004999999999999,"15min":21.004999999999999},"last":7.6189999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":12.176,"5min":20.567,"15min":28.530999999999999},"min":{"1min":9.1530000000000005,"5min":8.6519999999999992,"15min":8.6519999999999992},"max":{"1min":17.266999999999999,"5min":372.02300000000002,"15min":372.02300000000002},"last":8.9610000000000003},{"interface":"front","average":{"1min":9.1760000000000002,"5min":17.986999999999998,"15min":25.765999999999998},"min":{"1min":5.1680000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":15.836,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.2669999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":15.926},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":12.307}]},{"osd":7,"last update":"Sat Jan 20 11:19:31 2024","interfaces":[{"interface":"back","average":{"1min":14.923,"5min":15.964,"15min":17.009},"min":{"1min":12.335000000000001,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.178999999999998,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.000999999999999},{"interface":"front","average":{"1min":11.413,"5min":12.173999999999999,"15min":12.367000000000001},"min":{"1min":7.6600000000000001,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.928000000000001,"5min":21.529,"15min":21.529},"last":11.927}]}]},{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":11.6},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.6369999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.3869999999999996},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.242000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.6150000000000002},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.0410000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.102},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.24}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.005000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.3559999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.949},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":7.4889999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:37.760 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:37.760 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:37.760 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:37.760 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:37.760 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:37.760 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:37.760 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:37.761 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:37.761 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.858+0000 7f98d489b640 10 monclient: get_monmap_and_config 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.858+0000 7f98d489b640 10 monclient: build_initial_monmap 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.858+0000 7f98d489b640 10 monclient: monmap: 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:37.858874+0000 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:37.858874+0000 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:37.858 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:37.859 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:37.859 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:37.859 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:37.859 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.858+0000 7f98d489b640 1 Processor -- start 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.858+0000 7f98d489b640 1 -- start start 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.858+0000 7f98d489b640 10 monclient: init 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f98d00b0ba0 0x7f98d00ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 10 monclient(hunting): picked mon.noname-c con 0x7f98d00b0ba0 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f98d00adc30 0x7f98d00af130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 10 monclient(hunting): picked mon.noname-a con 0x7f98d00adc30 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98d00ac4d0 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00ad530 con 0x7f98d00ac4d0 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d00b0f80 con 0x7f98d00adc30 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d00ad810 con 0x7f98d00b0ba0 2024-01-20T11:20:37.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.859+0000 7f98d489b640 10 monclient(hunting): authenticate will time out at 1681.209659s 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98c5d9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f98d00adc30 0x7f98d00af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98ce59c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f98d00b0ba0 0x7f98d00ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98ced9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f98d00ac4d0 0x7f98d013bf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:41488/0 (socket says 172.21.15.110:41488) 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98ced9d640 1 -- 172.21.15.110:0/873093545 learned_addr learned my addr 172.21.15.110:0/873093545 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98ce59c640 10 monclient(hunting): get_auth_request con 0x7f98d00b0ba0 auth_method 0 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98ce59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98c5d9b640 10 monclient(hunting): get_auth_request con 0x7f98d00adc30 auth_method 0 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98c5d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.860+0000 7f98c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.861+0000 7f98ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.861+0000 7f98ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.861+0000 7f98ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.861+0000 7f98c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.861+0000 7f98c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.861+0000 7f98c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.862+0000 7f98ce59c640 10 monclient(hunting): handle_auth_done global_id 5419 payload 274 2024-01-20T11:20:37.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.862+0000 7f98ce59c640 1 -- 172.21.15.110:0/873093545 >> v2:172.21.15.110:3300/0 conn(0x7f98d00adc30 msgr2=0x7f98d00af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:37.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.862+0000 7f98ce59c640 1 --2- 172.21.15.110:0/873093545 >> v2:172.21.15.110:3300/0 conn(0x7f98d00adc30 0x7f98d00af130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.862+0000 7f98ce59c640 1 -- 172.21.15.110:0/873093545 >> v1:172.21.15.110:6789/0 conn(0x7f98d00ac4d0 legacy=0x7f98d013bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.862+0000 7f98ce59c640 10 monclient: _finish_hunting 0 2024-01-20T11:20:37.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.862+0000 7f98ce59c640 1 monclient: found mon.noname-c 2024-01-20T11:20:37.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.862+0000 7f98ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:37.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.862+0000 7f98ce59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:37.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.862+0000 7f98ce59c640 1 -- 172.21.15.110:0/873093545 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00ab110 con 0x7f98d00b0ba0 2024-01-20T11:20:37.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.864+0000 7f98ce59c640 1 --2- 172.21.15.110:0/873093545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00b0ba0 0x7f98d00ab630 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f98d0067cc0 tx=0x7f98c002da50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76b9e622f2823125 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:37.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.868+0000 7f98cdd9b640 1 -- 172.21.15.110:0/873093545 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0031870 con 0x7f98d00b0ba0 2024-01-20T11:20:37.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.868+0000 7f98cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.868+0000 7f98cdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.868+0000 7f98cdd9b640 10 monclient: dump: 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.868+0000 7f98cdd9b640 10 monclient: _finish_auth 0 2024-01-20T11:20:37.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.868+0000 7f98cdd9b640 10 monclient: _check_auth_tickets 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.868+0000 7f98cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.868+0000 7f98cdd9b640 1 -- 172.21.15.110:0/873093545 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c00319d0 con 0x7f98d00b0ba0 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.868+0000 7f98d489b640 5 monclient: authenticate success, global_id 5419 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 4 set_mon_vals no callback set 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98cdd9b640 1 -- 172.21.15.110:0/873093545 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0031d60 con 0x7f98d00b0ba0 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98cdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98cdd9b640 10 monclient: dump: 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.870 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:37.871 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.871 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:37.871 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:37.871 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:37.871 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 10 monclient: shutdown 2024-01-20T11:20:37.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:37.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 1 -- 172.21.15.110:0/873093545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00b0ba0 msgr2=0x7f98d00ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 1 --2- 172.21.15.110:0/873093545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00b0ba0 0x7f98d00ab630 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f98d0067cc0 tx=0x7f98c002da50 comp rx=0 tx=0).stop 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 1 -- 172.21.15.110:0/873093545 shutdown_connections 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 1 --2- 172.21.15.110:0/873093545 >> v2:172.21.15.110:3300/0 conn(0x7f98d00adc30 0x7f98d00af130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 1 --2- 172.21.15.110:0/873093545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00b0ba0 0x7f98d00ab630 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 1 -- 172.21.15.110:0/873093545 >> 172.21.15.110:0/873093545 conn(0x7f98d00b3e60 msgr2=0x7f98d00b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 1 -- 172.21.15.110:0/873093545 shutdown_connections 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.869+0000 7f98d489b640 1 -- 172.21.15.110:0/873093545 wait complete. 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.870+0000 7f98d489b640 10 monclient: build_initial_monmap 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.870+0000 7f98d489b640 10 monclient: monmap: 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:37.875 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.870+0000 7f98d489b640 1 librados: starting msgr at 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.870+0000 7f98d489b640 1 librados: starting objecter 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 Processor -- start 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 -- start start 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 librados: setting wanted keys 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 librados: calling monclient init 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient: init 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00ac4d0 0x7f98d013cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient(hunting): picked mon.noname-b con 0x7f98d00ac4d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98d00b0ba0 0x7f98d013f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient(hunting): picked mon.noname-a con 0x7f98d00b0ba0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98d013f830 0x7f98d0142670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient(hunting): picked mon.noname-c con 0x7f98d013f830 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d013ee60 con 0x7f98d00b0ba0 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d013e5a0 con 0x7f98d00ac4d0 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d0142bb0 con 0x7f98d013f830 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:37.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.871+0000 7f98d489b640 10 monclient(hunting): authenticate will time out at 1681.221805s 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98c5d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98d00b0ba0 0x7f98d013f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ced9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98d013f830 0x7f98d0142670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ce59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00ac4d0 0x7f98d013cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98c5d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98d00b0ba0 0x7f98d013f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40392/0 (socket says 172.21.15.110:40392) 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98c5d9b640 1 -- 172.21.15.110:0/3249800168 learned_addr learned my addr 172.21.15.110:0/3249800168 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98c5d9b640 10 monclient(hunting): get_auth_request con 0x7f98d00b0ba0 auth_method 0 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98c5d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ce59c640 10 monclient(hunting): get_auth_request con 0x7f98d00ac4d0 auth_method 0 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ce59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ced9d640 10 monclient(hunting): get_auth_request con 0x7f98d013f830 auth_method 0 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ced9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.872+0000 7f98ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.873+0000 7f98ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.873+0000 7f98ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.873+0000 7f98ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.873+0000 7f98ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.873+0000 7f98ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.873+0000 7f98ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.873+0000 7f98c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.873+0000 7f98c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:37.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.873+0000 7f98c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:37.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 10 monclient(hunting): handle_auth_done global_id 5422 payload 824 2024-01-20T11:20:37.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 1 -- 172.21.15.110:0/3249800168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98d013f830 msgr2=0x7f98d0142670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98d013f830 0x7f98d0142670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 1 -- 172.21.15.110:0/3249800168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98d00b0ba0 msgr2=0x7f98d013f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98d00b0ba0 0x7f98d013f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 10 monclient: _finish_hunting 0 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 1 monclient: found mon.noname-b 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.875+0000 7f98ce59c640 1 -- 172.21.15.110:0/3249800168 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0009400 con 0x7f98d00ac4d0 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.877+0000 7f98ce59c640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00ac4d0 0x7f98d013cc50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f98c0035000 tx=0x7f98c0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.877+0000 7f98c77fe640 10 client.?.objecter ms_handle_connect 0x7f98d00ac4d0 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.877+0000 7f98c77fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 1 -- 172.21.15.110:0/3249800168 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c003e040 con 0x7f98d00ac4d0 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:37.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 10 monclient: dump: 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 1 -- 172.21.15.110:0/3249800168 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c0039d20 con 0x7f98d00ac4d0 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 1 -- 172.21.15.110:0/3249800168 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c002f640 con 0x7f98d00ac4d0 2024-01-20T11:20:37.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98d489b640 5 monclient: authenticate success, global_id 5422 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98d489b640 10 monclient: _renew_subs 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98d489b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d0142dd0 con 0x7f98d00ac4d0 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98cd59a640 4 set_mon_vals no callback set 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98c77fe640 10 monclient: dump: 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:37.884 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98d489b640 10 client.5422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98d489b640 10 monclient: _renew_subs 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98d489b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d01d2e80 con 0x7f98d00ac4d0 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.881+0000 7f98d489b640 1 librados: init done 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.882+0000 7f98d489b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.882+0000 7f98d489b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.882+0000 7f98d489b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:37.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:37.882+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d013ec30 con 0x7f98d00ac4d0 2024-01-20T11:20:38.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.124+0000 7f98c77fe640 1 -- 172.21.15.110:0/3249800168 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f98c002f7a0 con 0x7f98d00ac4d0 2024-01-20T11:20:38.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.124+0000 7f98c77fe640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f98b006c8b0 0x7f98b006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:38.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.124+0000 7f98c5d9b640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f98b006c8b0 0x7f98b006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:38.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.124+0000 7f98c5d9b640 10 monclient: get_auth_request con 0x7f98b006c8b0 auth_method 0 2024-01-20T11:20:38.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.125+0000 7f98c5d9b640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f98b006c8b0 0x7f98b006ed70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f98d01353f0 tx=0x7f98b8009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:38.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.125+0000 7f98c77fe640 10 client.5422.objecter ms_handle_connect 0x7f98b006c8b0 2024-01-20T11:20:38.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.127+0000 7f98c77fe640 1 -- 172.21.15.110:0/3249800168 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f98c007c8f0 con 0x7f98d00ac4d0 2024-01-20T11:20:38.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.127+0000 7f98c77fe640 10 client.5422.objecter ms_dispatch 0x7f98d0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:38.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.127+0000 7f98c77fe640 3 client.5422.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:38.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.127+0000 7f98c77fe640 3 client.5422.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:38.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.127+0000 7f98c77fe640 20 client.5422.objecter dump_active .. 0 homeless 2024-01-20T11:20:38.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.137+0000 7f98c77fe640 1 -- 172.21.15.110:0/3249800168 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f98c0078c30 con 0x7f98d00ac4d0 2024-01-20T11:20:38.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.137+0000 7f98c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:38.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.137+0000 7f98c77fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:38.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.382+0000 7f98d489b640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:38.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.382+0000 7f98d489b640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:38.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.382+0000 7f98d489b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:38.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.382+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f98d0075490 con 0x7f98d00ac4d0 2024-01-20T11:20:38.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.388+0000 7f98c77fe640 1 -- 172.21.15.110:0/3249800168 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f98c0049020 con 0x7f98d00ac4d0 2024-01-20T11:20:38.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.388+0000 7f98c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:38.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.388+0000 7f98c77fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:38.388 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:38.388 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":372,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 10 librados: watch_flush enter 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 10 librados: watch_flush exit 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 20 client.5422.objecter shutdown clearing up homeless session... 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 10 client.5422.objecter successfully canceled tick 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f98b006c8b0 msgr2=0x7f98b006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f98b006c8b0 0x7f98b006ed70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f98d01353f0 tx=0x7f98b8009190 comp rx=0 tx=0).stop 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 10 monclient: shutdown 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00ac4d0 msgr2=0x7f98d013cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00ac4d0 0x7f98d013cc50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f98c0035000 tx=0x7f98c0004730 comp rx=0 tx=0).stop 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 shutdown_connections 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f98b006c8b0 0x7f98b006ed70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98d013f830 0x7f98d0142670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98d00b0ba0 0x7f98d013f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 --2- 172.21.15.110:0/3249800168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d00ac4d0 0x7f98d013cc50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 >> 172.21.15.110:0/3249800168 conn(0x7f98d00b3e60 msgr2=0x7f98d0134970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:38.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 shutdown_connections 2024-01-20T11:20:38.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 -- 172.21.15.110:0/3249800168 wait complete. 2024-01-20T11:20:38.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.390+0000 7f98d489b640 1 librados: shutdown 2024-01-20T11:20:38.401 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:38.401 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.502+0000 7f4f0e8a2640 10 monclient: get_monmap_and_config 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.502+0000 7f4f0e8a2640 10 monclient: build_initial_monmap 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.502+0000 7f4f0e8a2640 10 monclient: monmap: 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:38.502936+0000 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:38.502936+0000 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:38.502 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:38.503 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:38.503 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 1 Processor -- start 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 1 -- start start 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient: init 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4f0806f920 0x7f4f0806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient(hunting): picked mon.noname-c con 0x7f4f0806f920 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4f0813d230 0x7f4f0813d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient(hunting): picked mon.noname-a con 0x7f4f0813d230 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f08143950 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f080715b0 con 0x7f4f08143950 2024-01-20T11:20:38.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f08071890 con 0x7f4f0813d230 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f08071b70 con 0x7f4f0806f920 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.503+0000 7f4f0e8a2640 10 monclient(hunting): authenticate will time out at 1681.853967s 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f077fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4f0813d230 0x7f4f0813d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f07fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4f0806f920 0x7f4f0806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f077fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4f0813d230 0x7f4f0813d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40402/0 (socket says 172.21.15.110:40402) 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f077fe640 1 -- 172.21.15.110:0/840900456 learned_addr learned my addr 172.21.15.110:0/840900456 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f077fe640 10 monclient(hunting): get_auth_request con 0x7f4f0813d230 auth_method 0 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f077fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f077fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:38.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f07fff640 10 monclient(hunting): get_auth_request con 0x7f4f0806f920 auth_method 0 2024-01-20T11:20:38.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:38.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f07fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:38.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.504+0000 7f4f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:38.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.505+0000 7f4f077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:38.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.505+0000 7f4f077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:38.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.505+0000 7f4f077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:38.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.505+0000 7f4f07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:38.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.505+0000 7f4f07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:38.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.505+0000 7f4f07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:38.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.506+0000 7f4f077fe640 10 monclient(hunting): handle_auth_done global_id 5355 payload 274 2024-01-20T11:20:38.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.506+0000 7f4f077fe640 1 -- 172.21.15.110:0/840900456 >> v2:172.21.15.134:3300/0 conn(0x7f4f0806f920 msgr2=0x7f4f0806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:38.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.506+0000 7f4f077fe640 1 --2- 172.21.15.110:0/840900456 >> v2:172.21.15.134:3300/0 conn(0x7f4f0806f920 0x7f4f0806fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.506+0000 7f4f077fe640 1 -- 172.21.15.110:0/840900456 >> v1:172.21.15.134:6789/0 conn(0x7f4f08143950 legacy=0x7f4f0813fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:38.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.506+0000 7f4f077fe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:38.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.506+0000 7f4f077fe640 1 monclient: found mon.noname-a 2024-01-20T11:20:38.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.506+0000 7f4f077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:38.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.506+0000 7f4f077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:38.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.506+0000 7f4f077fe640 1 -- 172.21.15.110:0/840900456 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08143430 con 0x7f4f0813d230 2024-01-20T11:20:38.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.508+0000 7f4f077fe640 1 --2- 172.21.15.110:0/840900456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0813d230 0x7f4f0813d680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4f08067350 tx=0x7f4ef802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4ce240a6042caa server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:38.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 1 -- 172.21.15.110:0/840900456 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8004230 con 0x7f4f0813d230 2024-01-20T11:20:38.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:38.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:38.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 10 monclient: dump: 2024-01-20T11:20:38.512 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:38.512 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:38.512 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 1 -- 172.21.15.110:0/840900456 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef8004390 con 0x7f4f0813d230 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f0e8a2640 5 monclient: authenticate success, global_id 5355 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f0e8a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:38.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 1 -- 172.21.15.110:0/840900456 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8003810 con 0x7f4f0813d230 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f0e8a2640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f0e8a2640 4 set_mon_vals no callback set 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.512+0000 7f4f06ffd640 10 monclient: dump: 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:38.514 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:38.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 10 monclient: shutdown 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/840900456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0813d230 msgr2=0x7f4f0813d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 1 --2- 172.21.15.110:0/840900456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0813d230 0x7f4f0813d680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4f08067350 tx=0x7f4ef802b8f0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/840900456 shutdown_connections 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 1 --2- 172.21.15.110:0/840900456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0813d230 0x7f4f0813d680 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 1 --2- 172.21.15.110:0/840900456 >> v2:172.21.15.134:3300/0 conn(0x7f4f0806f920 0x7f4f0806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/840900456 >> 172.21.15.110:0/840900456 conn(0x7f4f08066d60 msgr2=0x7f4f080b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/840900456 shutdown_connections 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.513+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/840900456 wait complete. 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.514+0000 7f4f0e8a2640 10 monclient: build_initial_monmap 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.514+0000 7f4f0e8a2640 10 monclient: monmap: 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:38.518 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.514+0000 7f4f0e8a2640 1 librados: starting msgr at 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.514+0000 7f4f0e8a2640 1 librados: starting objecter 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.514+0000 7f4f0e8a2640 1 Processor -- start 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.514+0000 7f4f0e8a2640 1 -- start start 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.514+0000 7f4f0e8a2640 1 librados: setting wanted keys 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.514+0000 7f4f0e8a2640 1 librados: calling monclient init 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.514+0000 7f4f0e8a2640 10 monclient: init 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:38.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f0806f920 0x7f4f08141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient(hunting): picked mon.noname-b con 0x7f4f0806f920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f0813d230 0x7f4f08133c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient(hunting): picked mon.noname-c con 0x7f4f0813d230 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f08143950 0x7f4f08134190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient(hunting): picked mon.noname-a con 0x7f4f08143950 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f08142e10 con 0x7f4f08143950 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f08142520 con 0x7f4f0806f920 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f08143010 con 0x7f4f0813d230 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0e8a2640 10 monclient(hunting): authenticate will time out at 1681.865426s 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f077fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f0813d230 0x7f4f08133c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:38.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f07fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f0806f920 0x7f4f08141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.515+0000 7f4f0ce3f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f08143950 0x7f4f08134190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f07fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f0806f920 0x7f4f08141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:34036/0 (socket says 172.21.15.110:34036) 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f07fff640 1 -- 172.21.15.110:0/2506030046 learned_addr learned my addr 172.21.15.110:0/2506030046 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f07fff640 10 monclient(hunting): get_auth_request con 0x7f4f0806f920 auth_method 0 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f07fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f077fe640 10 monclient(hunting): get_auth_request con 0x7f4f0813d230 auth_method 0 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f077fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f077fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f0ce3f640 10 monclient(hunting): get_auth_request con 0x7f4f08143950 auth_method 0 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f0ce3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f0ce3f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f0ce3f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:38.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:38.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:38.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:38.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:38.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:38.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:38.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f0ce3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:38.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f0ce3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:38.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.516+0000 7f4f0ce3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 10 monclient(hunting): handle_auth_done global_id 5428 payload 824 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 1 -- 172.21.15.110:0/2506030046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f0813d230 msgr2=0x7f4f08133c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f0813d230 0x7f4f08133c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 1 -- 172.21.15.110:0/2506030046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f08143950 msgr2=0x7f4f08134190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f08143950 0x7f4f08134190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 1 monclient: found mon.noname-b 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.519+0000 7f4f07fff640 1 -- 172.21.15.110:0/2506030046 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef8009460 con 0x7f4f0806f920 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.521+0000 7f4f07fff640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f0806f920 0x7f4f08141170 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00e9b0 tx=0x7f4efc00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:38.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.521+0000 7f4f04ff9640 10 client.?.objecter ms_handle_connect 0x7f4f0806f920 2024-01-20T11:20:38.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.521+0000 7f4f04ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 1 -- 172.21.15.110:0/2506030046 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc009670 con 0x7f4f0806f920 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 10 monclient: dump: 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:38.525 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 10 monclient: _finish_auth 0 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 1 -- 172.21.15.110:0/2506030046 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc00c9a0 con 0x7f4f0806f920 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f0e8a2640 5 monclient: authenticate success, global_id 5428 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f0e8a2640 10 monclient: _renew_subs 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f0e8a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f08142060 con 0x7f4f0806f920 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 1 -- 172.21.15.110:0/2506030046 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc00cd50 con 0x7f4f0806f920 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f04ff9640 10 monclient: dump: 2024-01-20T11:20:38.526 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.524+0000 7f4f067fc640 4 set_mon_vals no callback set 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.525+0000 7f4f0e8a2640 10 client.5428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.525+0000 7f4f0e8a2640 10 monclient: _renew_subs 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.525+0000 7f4f0e8a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.525+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f08141890 con 0x7f4f0806f920 2024-01-20T11:20:38.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.525+0000 7f4f0e8a2640 1 librados: init done 2024-01-20T11:20:38.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.525+0000 7f4f0e8a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:38.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.525+0000 7f4f0e8a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:38.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.525+0000 7f4f0e8a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:38.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.525+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f08075490 con 0x7f4f0806f920 2024-01-20T11:20:38.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.543+0000 7f4f04ff9640 1 -- 172.21.15.110:0/2506030046 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4efc013470 con 0x7f4f0806f920 2024-01-20T11:20:38.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.544+0000 7f4f04ff9640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c8b0 0x7f4ef406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:38.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.544+0000 7f4f077fe640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c8b0 0x7f4ef406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:38.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.544+0000 7f4f077fe640 10 monclient: get_auth_request con 0x7f4ef406c8b0 auth_method 0 2024-01-20T11:20:38.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.544+0000 7f4f04ff9640 10 client.5428.objecter ms_handle_connect 0x7f4ef406c8b0 2024-01-20T11:20:38.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.544+0000 7f4f077fe640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c8b0 0x7f4ef406ed70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8035000 tx=0x7f4ef8008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:38.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.546+0000 7f4f04ff9640 1 -- 172.21.15.110:0/2506030046 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4efc058a40 con 0x7f4f0806f920 2024-01-20T11:20:38.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.546+0000 7f4f04ff9640 10 client.5428.objecter ms_dispatch 0x7f4f08000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:38.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.546+0000 7f4f04ff9640 3 client.5428.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:38.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.546+0000 7f4f04ff9640 3 client.5428.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:38.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.546+0000 7f4f04ff9640 20 client.5428.objecter dump_active .. 0 homeless 2024-01-20T11:20:38.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.778+0000 7f4f04ff9640 1 -- 172.21.15.110:0/2506030046 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4efc054d80 con 0x7f4f0806f920 2024-01-20T11:20:38.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.778+0000 7f4f04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:38.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.778+0000 7f4f04ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:38.835 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:29.859 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:38.841 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:29.866 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:38.865 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:29.889 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:38.884 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:06:29.890 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:38.888 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:06:29.894 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:38.900 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:29.925 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:38.904 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:06:29.910 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:38.905 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:29.930 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:38.924 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:29.948 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:38.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.969+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f08141890 con 0x7f4ef406c8b0 2024-01-20T11:20:38.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.971+0000 7f4f04ff9640 1 -- 172.21.15.110:0/2506030046 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63943 (secure 0 0 0) 0x7f4f08141890 con 0x7f4ef406c8b0 2024-01-20T11:20:38.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:38.972 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 10 librados: watch_flush enter 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 10 librados: watch_flush exit 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 20 client.5428.objecter shutdown clearing up homeless session... 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 10 client.5428.objecter successfully canceled tick 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c8b0 msgr2=0x7f4ef406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c8b0 0x7f4ef406ed70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8035000 tx=0x7f4ef8008030 comp rx=0 tx=0).stop 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 10 monclient: shutdown 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f0806f920 msgr2=0x7f4f08141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f0806f920 0x7f4f08141170 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00e9b0 tx=0x7f4efc00ee70 comp rx=0 tx=0).stop 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 shutdown_connections 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c8b0 0x7f4ef406ed70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f08143950 0x7f4f08134190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f0813d230 0x7f4f08133c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 --2- 172.21.15.110:0/2506030046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f0806f920 0x7f4f08141170 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 >> 172.21.15.110:0/2506030046 conn(0x7f4f08066d60 msgr2=0x7f4f0813a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 shutdown_connections 2024-01-20T11:20:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 -- 172.21.15.110:0/2506030046 wait complete. 2024-01-20T11:20:38.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:38.974+0000 7f4f0e8a2640 1 librados: shutdown 2024-01-20T11:20:38.988 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-01-20T11:20:38.773141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.7320000000000002},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.944}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.358000000000001},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3740000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.4830000000000005},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.266}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.893000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":7.6769999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.961},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":11.629}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":14.548999999999999},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.140000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313285,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":10.974},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":10.342000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.284000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.4600000000000009},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.7919999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.643000000000001},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.147}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":13.757999999999999},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":7.8760000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":9.6890000000000001},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.409000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313286,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":4.867},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":12.581}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.4299999999999997},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.134}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":15.861000000000001},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":8.0459999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.282},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.428000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":14.904},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.041}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":10.816000000000001},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.132999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444100,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.085},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":5.3419999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.2490000000000006},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.898}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.862},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":7.6660000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.548999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8520000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.210000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.0579999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.4209999999999994},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.760999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444101,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.3810000000000002},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":12.819000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":15.959},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.2759999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":10.667999999999999},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":10.002000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":14.916},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.8769999999999998},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":16.780000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.978},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.684000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444101,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":9.4090000000000007},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":18.504999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":10.914999999999999},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.733000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.772},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.0960000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.798999999999999},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.7320000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":11.763999999999999},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":17.390000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.373000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":8.827}]}]},{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":11.6},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.6369999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.3869999999999996},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.242000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.6150000000000002},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.0410000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.102},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.24}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.005000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.3559999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.949},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":7.4889999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:41.991 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:42.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.091+0000 7f2ab2ce5640 10 monclient: get_monmap_and_config 2024-01-20T11:20:42.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.091+0000 7f2ab2ce5640 10 monclient: build_initial_monmap 2024-01-20T11:20:42.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.091+0000 7f2ab2ce5640 10 monclient: monmap: 2024-01-20T11:20:42.091 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:42.091 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:42.091827+0000 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:42.091827+0000 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:42.092 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 1 Processor -- start 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 1 -- start start 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient: init 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2aac0a96f0 0x7f2aac13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient(hunting): picked mon.noname-a con 0x7f2aac0a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2aac13fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a9ad0 0x7f2aac142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient(hunting): picked mon.noname-b con 0x7f2aac0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac0b25c0 con 0x7f2aac13fa10 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac0aaea0 con 0x7f2aac0a96f0 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0ab0a0 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.092+0000 7f2ab2ce5640 10 monclient(hunting): authenticate will time out at 1685.442861s 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab0a81640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2aac0a96f0 0x7f2aac13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab1282640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a9ad0 0x7f2aac142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2aabfff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2aac13fa10 0x7f2aac1423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:33766/0 (socket says 172.21.15.110:33766) 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2aabfff640 1 -- 172.21.15.110:0/2099854329 learned_addr learned my addr 172.21.15.110:0/2099854329 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab0a81640 10 monclient(hunting): get_auth_request con 0x7f2aac0a96f0 auth_method 0 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab0a81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab0a81640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab0a81640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab1282640 10 monclient(hunting): get_auth_request con 0x7f2aac0a9ad0 auth_method 0 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab1282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab1282640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.093+0000 7f2ab1282640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.094+0000 7f2ab0a81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.094+0000 7f2ab0a81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.094+0000 7f2ab0a81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.094+0000 7f2ab1282640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.094+0000 7f2ab1282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.094+0000 7f2ab1282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2aab7fe640 1 -- 172.21.15.110:0/2099854329 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462816642 0 0) 0x7f2aac0b25c0 con 0x7f2aac13fa10 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2aab7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2aab7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2aab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2aab7fe640 10 monclient(hunting): my global_id is 5431 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2aab7fe640 1 -- 172.21.15.110:0/2099854329 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a94003520 con 0x7f2aac13fa10 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2ab0a81640 10 monclient(hunting): handle_auth_done global_id 5361 payload 274 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2ab0a81640 1 -- 172.21.15.110:0/2099854329 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a9ad0 msgr2=0x7f2aac142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2ab0a81640 1 --2- 172.21.15.110:0/2099854329 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a9ad0 0x7f2aac142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2ab0a81640 1 -- 172.21.15.110:0/2099854329 >> v1:172.21.15.134:6789/0 conn(0x7f2aac13fa10 legacy=0x7f2aac1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2ab0a81640 10 monclient: _finish_hunting 0 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2ab0a81640 1 monclient: found mon.noname-a 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2ab0a81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2ab0a81640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:42.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.096+0000 7f2ab0a81640 1 -- 172.21.15.110:0/2099854329 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac0b3fb0 con 0x7f2aac0a96f0 2024-01-20T11:20:42.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.099+0000 7f2ab0a81640 1 --2- 172.21.15.110:0/2099854329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac0a96f0 0x7f2aac13f4d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0097a0 tx=0x7f2a9c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4851d73cc90ac7c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:42.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 1 -- 172.21.15.110:0/2099854329 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c02e070 con 0x7f2aac0a96f0 2024-01-20T11:20:42.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:42.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:42.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 10 monclient: dump: 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:42.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 1 -- 172.21.15.110:0/2099854329 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c0315a0 con 0x7f2aac0a96f0 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2ab2ce5640 5 monclient: authenticate success, global_id 5361 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2ab2ce5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2ab2ce5640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2ab2ce5640 4 set_mon_vals no callback set 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 1 -- 172.21.15.110:0/2099854329 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c031930 con 0x7f2aac0a96f0 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.102+0000 7f2aab7fe640 10 monclient: dump: 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.105 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 10 monclient: shutdown 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/2099854329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac0a96f0 msgr2=0x7f2aac13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 1 --2- 172.21.15.110:0/2099854329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac0a96f0 0x7f2aac13f4d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0097a0 tx=0x7f2a9c02b840 comp rx=0 tx=0).stop 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/2099854329 shutdown_connections 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 1 --2- 172.21.15.110:0/2099854329 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a9ad0 0x7f2aac142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 1 --2- 172.21.15.110:0/2099854329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac0a96f0 0x7f2aac13f4d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/2099854329 >> 172.21.15.110:0/2099854329 conn(0x7f2aac0b7bc0 msgr2=0x7f2aac0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:42.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/2099854329 shutdown_connections 2024-01-20T11:20:42.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.103+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/2099854329 wait complete. 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.104+0000 7f2ab2ce5640 10 monclient: build_initial_monmap 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.104+0000 7f2ab2ce5640 10 monclient: monmap: 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:42.110 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.104+0000 7f2ab2ce5640 1 librados: starting msgr at 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.104+0000 7f2ab2ce5640 1 librados: starting objecter 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.104+0000 7f2ab2ce5640 1 Processor -- start 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.104+0000 7f2ab2ce5640 1 -- start start 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.104+0000 7f2ab2ce5640 1 librados: setting wanted keys 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.104+0000 7f2ab2ce5640 1 librados: calling monclient init 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.104+0000 7f2ab2ce5640 10 monclient: init 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a96f0 0x7f2aac1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient(hunting): picked mon.noname-c con 0x7f2aac0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2aac0a9ad0 0x7f2aac132590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient(hunting): picked mon.noname-b con 0x7f2aac0a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac132ad0 0x7f2aac12dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient(hunting): picked mon.noname-a con 0x7f2aac132ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac133580 con 0x7f2aac132ad0 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac134310 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac133060 con 0x7f2aac0a96f0 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab2ce5640 10 monclient(hunting): authenticate will time out at 1685.455710s 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.105+0000 7f2ab1282640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac132ad0 0x7f2aac12dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab0a81640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a96f0 0x7f2aac1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2aabfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2aac0a9ad0 0x7f2aac132590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab1282640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac132ad0 0x7f2aac12dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58328/0 (socket says 172.21.15.110:58328) 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab1282640 1 -- 172.21.15.110:0/3349783173 learned_addr learned my addr 172.21.15.110:0/3349783173 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:42.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab1282640 10 monclient(hunting): get_auth_request con 0x7f2aac132ad0 auth_method 0 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab1282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab1282640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab1282640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab0a81640 10 monclient(hunting): get_auth_request con 0x7f2aac0a96f0 auth_method 0 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab0a81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab0a81640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2ab0a81640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2aabfff640 10 monclient(hunting): get_auth_request con 0x7f2aac0a9ad0 auth_method 0 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2aabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2aabfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.106+0000 7f2aabfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.107+0000 7f2ab1282640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.107+0000 7f2ab1282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.107+0000 7f2ab1282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.107+0000 7f2aabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.107+0000 7f2aabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.107+0000 7f2aabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.107+0000 7f2ab0a81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.107+0000 7f2ab0a81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.107+0000 7f2ab0a81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 10 monclient(hunting): handle_auth_done global_id 5434 payload 824 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 1 -- 172.21.15.110:0/3349783173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a96f0 msgr2=0x7f2aac1418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a96f0 0x7f2aac1418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 1 -- 172.21.15.110:0/3349783173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac132ad0 msgr2=0x7f2aac12dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac132ad0 0x7f2aac12dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 1 monclient: found mon.noname-b 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:42.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.109+0000 7f2aabfff640 1 -- 172.21.15.110:0/3349783173 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c009400 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.112+0000 7f2aabfff640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2aac0a9ad0 0x7f2aac132590 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0aaa10 tx=0x7f2aa0001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:42.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.112+0000 7f2aa97fa640 10 client.?.objecter ms_handle_connect 0x7f2aac0a9ad0 2024-01-20T11:20:42.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.112+0000 7f2aa97fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 1 -- 172.21.15.110:0/3349783173 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0005ce0 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 10 monclient: dump: 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:42.116 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 1 -- 172.21.15.110:0/3349783173 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa0005e40 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 1 -- 172.21.15.110:0/3349783173 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa00072f0 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aa97fa640 10 monclient: dump: 2024-01-20T11:20:42.117 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2ab2ce5640 5 monclient: authenticate success, global_id 5434 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2ab2ce5640 10 monclient: _renew_subs 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2ab2ce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac12e210 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.115+0000 7f2aaaffd640 4 set_mon_vals no callback set 2024-01-20T11:20:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.116+0000 7f2ab2ce5640 10 client.5434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.116+0000 7f2ab2ce5640 10 monclient: _renew_subs 2024-01-20T11:20:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.116+0000 7f2ab2ce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.116+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac12e6d0 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.116+0000 7f2ab2ce5640 1 librados: init done 2024-01-20T11:20:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.116+0000 7f2ab2ce5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.116+0000 7f2ab2ce5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.116+0000 7f2ab2ce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:42.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.116+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac0b3da0 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.353+0000 7f2aa97fa640 1 -- 172.21.15.110:0/3349783173 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2aa0007450 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.354+0000 7f2aa97fa640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a9806c8e0 0x7f2a9806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.354+0000 7f2ab0a81640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a9806c8e0 0x7f2a9806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.354+0000 7f2ab0a81640 10 monclient: get_auth_request con 0x7f2a9806c8e0 auth_method 0 2024-01-20T11:20:42.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.355+0000 7f2ab0a81640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a9806c8e0 0x7f2a9806eda0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c036000 tx=0x7f2a9c03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:42.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.355+0000 7f2aa97fa640 10 client.5434.objecter ms_handle_connect 0x7f2a9806c8e0 2024-01-20T11:20:42.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.357+0000 7f2aa97fa640 1 -- 172.21.15.110:0/3349783173 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2aa0007e20 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.357+0000 7f2aa97fa640 10 client.5434.objecter ms_dispatch 0x7f2aac000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:42.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.357+0000 7f2aa97fa640 3 client.5434.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:42.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.357+0000 7f2aa97fa640 3 client.5434.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:42.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.357+0000 7f2aa97fa640 20 client.5434.objecter dump_active .. 0 homeless 2024-01-20T11:20:42.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.369+0000 7f2aa97fa640 1 -- 172.21.15.110:0/3349783173 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2aa0053de0 con 0x7f2aac0a9ad0 2024-01-20T11:20:42.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.369+0000 7f2aa97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:42.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.369+0000 7f2aa97fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:42.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.548+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2aac075490 con 0x7f2a9806c8e0 2024-01-20T11:20:42.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.551+0000 7f2aa97fa640 1 -- 172.21.15.110:0/3349783173 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63943 (secure 0 0 0) 0x7f2aac075490 con 0x7f2a9806c8e0 2024-01-20T11:20:42.551 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:42.551 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 10 librados: watch_flush enter 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 10 librados: watch_flush exit 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 20 client.5434.objecter shutdown clearing up homeless session... 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 10 client.5434.objecter successfully canceled tick 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a9806c8e0 msgr2=0x7f2a9806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a9806c8e0 0x7f2a9806eda0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c036000 tx=0x7f2a9c03d040 comp rx=0 tx=0).stop 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 10 monclient: shutdown 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2aac0a9ad0 msgr2=0x7f2aac132590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2aac0a9ad0 0x7f2aac132590 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0aaa10 tx=0x7f2aa0001e70 comp rx=0 tx=0).stop 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 shutdown_connections 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2a9806c8e0 0x7f2a9806eda0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac132ad0 0x7f2aac12dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2aac0a9ad0 0x7f2aac132590 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 --2- 172.21.15.110:0/3349783173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac0a96f0 0x7f2aac1418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 >> 172.21.15.110:0/3349783173 conn(0x7f2aac0b7bc0 msgr2=0x7f2aac137310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:42.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 shutdown_connections 2024-01-20T11:20:42.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 -- 172.21.15.110:0/3349783173 wait complete. 2024-01-20T11:20:42.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.553+0000 7f2ab2ce5640 1 librados: shutdown 2024-01-20T11:20:42.563 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-01-20T11:20:40.773564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002427"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.7320000000000002},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.944}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.358000000000001},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3740000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.4830000000000005},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.266}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.893000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":7.6769999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.961},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":11.629}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":14.548999999999999},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.140000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313285,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":10.974},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":10.342000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.284000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":8.4600000000000009},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.7919999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":11.643000000000001},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.147}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":13.757999999999999},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":7.8760000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":9.6890000000000001},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":14.409000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313286,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":4.867},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":12.581}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.4299999999999997},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.134}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":15.861000000000001},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":8.0459999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":9.282},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":11.428000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":14.904},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.041}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":10.816000000000001},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.132999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444100,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.085},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":5.3419999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.2490000000000006},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.898}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.862},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":7.6660000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.548999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8520000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.210000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.0579999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.4209999999999994},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.760999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444101,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.3810000000000002},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":12.819000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":15.959},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.2759999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":10.667999999999999},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":10.002000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":14.916},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.8769999999999998},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":16.780000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.978},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.684000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444101,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":9.4090000000000007},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":18.504999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":10.914999999999999},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.733000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.772},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.0960000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.798999999999999},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.7320000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":11.763999999999999},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":17.390000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.373000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":8.827}]}]},{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":11.6},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.6369999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.3869999999999996},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.242000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.6150000000000002},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.0410000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.102},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":12.24}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.005000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.3559999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.949},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":7.4889999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:42.565 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:42.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.668+0000 7fa628683640 10 monclient: get_monmap_and_config 2024-01-20T11:20:42.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.668+0000 7fa628683640 10 monclient: build_initial_monmap 2024-01-20T11:20:42.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.668+0000 7fa628683640 10 monclient: monmap: 2024-01-20T11:20:42.668 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:42.668 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.668 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:42.669204+0000 2024-01-20T11:20:42.668 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:42.669204+0000 2024-01-20T11:20:42.669 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:42.669 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.669 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:42.669 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:42.669 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:42.669 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:42.669 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:42.669 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 1 Processor -- start 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 1 -- start start 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient: init 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa620145760 0x7fa620145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient(hunting): picked mon.noname-a con 0x7fa620145760 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa62013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient(hunting): picked mon.noname-b con 0x7fa6200a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa62013f6e0 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa620146250 con 0x7fa62013f6e0 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6200ab050 con 0x7fa620145760 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62013fb90 con 0x7fa6200a9d80 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.669+0000 7fa628683640 10 monclient(hunting): authenticate will time out at 1686.020158s 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.670+0000 7fa62641f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa620145760 0x7fa620145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.670+0000 7fa625c1e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa62013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.670+0000 7fa625c1e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa62013f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:51970/0 (socket says 172.21.15.110:51970) 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.670+0000 7fa625c1e640 1 -- 172.21.15.110:0/1574565612 learned_addr learned my addr 172.21.15.110:0/1574565612 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.670+0000 7fa625c1e640 10 monclient(hunting): get_auth_request con 0x7fa6200a9d80 auth_method 0 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.670+0000 7fa625c1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.670+0000 7fa625c1e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.670+0000 7fa625c1e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.671+0000 7fa62641f640 10 monclient(hunting): get_auth_request con 0x7fa620145760 auth_method 0 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.671+0000 7fa62641f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.671+0000 7fa62641f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.671+0000 7fa62641f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.671+0000 7fa625c1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.671+0000 7fa625c1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.671+0000 7fa625c1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.671+0000 7fa62641f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.671+0000 7fa62641f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.672+0000 7fa62641f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.673+0000 7fa625c1e640 10 monclient(hunting): handle_auth_done global_id 5117 payload 274 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.673+0000 7fa625c1e640 1 -- 172.21.15.110:0/1574565612 >> v2:172.21.15.110:3300/0 conn(0x7fa620145760 msgr2=0x7fa620145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.673+0000 7fa625c1e640 1 --2- 172.21.15.110:0/1574565612 >> v2:172.21.15.110:3300/0 conn(0x7fa620145760 0x7fa620145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.673+0000 7fa625c1e640 1 -- 172.21.15.110:0/1574565612 >> v1:172.21.15.110:6789/0 conn(0x7fa62013f6e0 legacy=0x7fa6200b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.673+0000 7fa625c1e640 10 monclient: _finish_hunting 0 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.673+0000 7fa625c1e640 1 monclient: found mon.noname-b 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.673+0000 7fa625c1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.673+0000 7fa625c1e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:42.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.673+0000 7fa625c1e640 1 -- 172.21.15.110:0/1574565612 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6200b20c0 con 0x7fa6200a9d80 2024-01-20T11:20:42.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.675+0000 7fa625c1e640 1 --2- 172.21.15.110:0/1574565612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa62013f1a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa610009870 tx=0x7fa61002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b72831a6832cfbe server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:42.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 1 -- 172.21.15.110:0/1574565612 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610030840 con 0x7fa6200a9d80 2024-01-20T11:20:42.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 10 monclient: dump: 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 10 monclient: _finish_auth 0 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 10 monclient: _check_auth_tickets 2024-01-20T11:20:42.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa628683640 5 monclient: authenticate success, global_id 5117 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 1 -- 172.21.15.110:0/1574565612 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6100309a0 con 0x7fa6200a9d80 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa628683640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa628683640 4 set_mon_vals no callback set 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 1 -- 172.21.15.110:0/1574565612 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610030d30 con 0x7fa6200a9d80 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.678+0000 7fa62541d640 10 monclient: dump: 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:42.681 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 10 monclient: shutdown 2024-01-20T11:20:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 1 -- 172.21.15.110:0/1574565612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 msgr2=0x7fa62013f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 1 --2- 172.21.15.110:0/1574565612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa62013f1a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa610009870 tx=0x7fa61002b980 comp rx=0 tx=0).stop 2024-01-20T11:20:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 1 -- 172.21.15.110:0/1574565612 shutdown_connections 2024-01-20T11:20:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 1 --2- 172.21.15.110:0/1574565612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa62013f1a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 1 --2- 172.21.15.110:0/1574565612 >> v2:172.21.15.110:3300/0 conn(0x7fa620145760 0x7fa620145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 1 -- 172.21.15.110:0/1574565612 >> 172.21.15.110:0/1574565612 conn(0x7fa6200663b0 msgr2=0x7fa6200667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 1 -- 172.21.15.110:0/1574565612 shutdown_connections 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.679+0000 7fa628683640 1 -- 172.21.15.110:0/1574565612 wait complete. 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.680+0000 7fa628683640 10 monclient: build_initial_monmap 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.680+0000 7fa628683640 10 monclient: monmap: 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.680+0000 7fa628683640 1 librados: starting msgr at 2024-01-20T11:20:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.680+0000 7fa628683640 1 librados: starting objecter 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.680+0000 7fa628683640 1 Processor -- start 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.680+0000 7fa628683640 1 -- start start 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.680+0000 7fa628683640 1 librados: setting wanted keys 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.680+0000 7fa628683640 1 librados: calling monclient init 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.680+0000 7fa628683640 10 monclient: init 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa620142650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient(hunting): picked mon.noname-c con 0x7fa6200a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62013f6e0 0x7fa620139610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient(hunting): picked mon.noname-b con 0x7fa62013f6e0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa620145760 0x7fa620139b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient(hunting): picked mon.noname-a con 0x7fa620145760 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa620138bc0 con 0x7fa620145760 2024-01-20T11:20:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa620138ea0 con 0x7fa62013f6e0 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200b7b00 con 0x7fa6200a9d80 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa628683640 10 monclient(hunting): authenticate will time out at 1686.031518s 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.681+0000 7fa62641f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa620142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa626c20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa620145760 0x7fa620139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa62641f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa620142650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:51984/0 (socket says 172.21.15.110:51984) 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa62641f640 1 -- 172.21.15.110:0/3805409278 learned_addr learned my addr 172.21.15.110:0/3805409278 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa62641f640 10 monclient(hunting): get_auth_request con 0x7fa6200a9d80 auth_method 0 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa62641f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa62641f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa62641f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa625c1e640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62013f6e0 0x7fa620139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa625c1e640 10 monclient(hunting): get_auth_request con 0x7fa62013f6e0 auth_method 0 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa625c1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa625c1e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa625c1e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa626c20640 10 monclient(hunting): get_auth_request con 0x7fa620145760 auth_method 0 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa626c20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa626c20640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa626c20640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa62641f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa62641f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa62641f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.682+0000 7fa625c1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.683+0000 7fa625c1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.683+0000 7fa625c1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.683+0000 7fa626c20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:42.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.683+0000 7fa626c20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:42.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.683+0000 7fa626c20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:42.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 10 monclient(hunting): handle_auth_done global_id 5120 payload 824 2024-01-20T11:20:42.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 1 -- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62013f6e0 msgr2=0x7fa620139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62013f6e0 0x7fa620139610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 1 -- 172.21.15.110:0/3805409278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa620145760 msgr2=0x7fa620139b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa620145760 0x7fa620139b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 10 monclient: _finish_hunting 0 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 1 monclient: found mon.noname-c 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.685+0000 7fa62641f640 1 -- 172.21.15.110:0/3805409278 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6100094d0 con 0x7fa6200a9d80 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.688+0000 7fa62641f640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa620142650 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00e870 tx=0x7fa61c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.688+0000 7fa616ffd640 10 client.?.objecter ms_handle_connect 0x7fa6200a9d80 2024-01-20T11:20:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.688+0000 7fa616ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 1 -- 172.21.15.110:0/3805409278 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0094f0 con 0x7fa6200a9d80 2024-01-20T11:20:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 10 monclient: dump: 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa628683640 5 monclient: authenticate success, global_id 5120 2024-01-20T11:20:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 1 -- 172.21.15.110:0/3805409278 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c00c940 con 0x7fa6200a9d80 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 1 -- 172.21.15.110:0/3805409278 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c00ccf0 con 0x7fa6200a9d80 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa616ffd640 10 monclient: dump: 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa624c1c640 4 set_mon_vals no callback set 2024-01-20T11:20:42.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa628683640 10 monclient: _renew_subs 2024-01-20T11:20:42.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa628683640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:42.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6200b7de0 con 0x7fa6200a9d80 2024-01-20T11:20:42.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa628683640 10 client.5120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:42.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa628683640 10 monclient: _renew_subs 2024-01-20T11:20:42.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa628683640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:42.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6200b7290 con 0x7fa6200a9d80 2024-01-20T11:20:42.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.691+0000 7fa628683640 1 librados: init done 2024-01-20T11:20:42.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.692+0000 7fa628683640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:42.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.692+0000 7fa628683640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:42.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.692+0000 7fa628683640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:42.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.692+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa620075490 con 0x7fa6200a9d80 2024-01-20T11:20:42.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.709+0000 7fa616ffd640 1 -- 172.21.15.110:0/3805409278 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa61c013470 con 0x7fa6200a9d80 2024-01-20T11:20:42.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.710+0000 7fa616ffd640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa60406c8b0 0x7fa60406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:42.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.710+0000 7fa625c1e640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa60406c8b0 0x7fa60406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:42.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.710+0000 7fa625c1e640 10 monclient: get_auth_request con 0x7fa60406c8b0 auth_method 0 2024-01-20T11:20:42.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.711+0000 7fa625c1e640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa60406c8b0 0x7fa60406ed70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa610036000 tx=0x7fa610008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:42.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.711+0000 7fa616ffd640 10 client.5120.objecter ms_handle_connect 0x7fa60406c8b0 2024-01-20T11:20:42.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.712+0000 7fa616ffd640 1 -- 172.21.15.110:0/3805409278 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa61c058a40 con 0x7fa6200a9d80 2024-01-20T11:20:42.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.712+0000 7fa616ffd640 10 client.5120.objecter ms_dispatch 0x7fa620000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:42.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.712+0000 7fa616ffd640 3 client.5120.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:42.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.712+0000 7fa616ffd640 3 client.5120.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:42.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.713+0000 7fa616ffd640 20 client.5120.objecter dump_active .. 0 homeless 2024-01-20T11:20:42.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.943+0000 7fa616ffd640 1 -- 172.21.15.110:0/3805409278 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa61c054d80 con 0x7fa6200a9d80 2024-01-20T11:20:42.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.943+0000 7fa616ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:42.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:42.943+0000 7fa616ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:43.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.125+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa6200a9950 con 0x7fa60406c8b0 2024-01-20T11:20:43.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.128+0000 7fa616ffd640 1 -- 172.21.15.110:0/3805409278 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63883 (secure 0 0 0) 0x7fa6200a9950 con 0x7fa60406c8b0 2024-01-20T11:20:43.128 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:43.129 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.129+0000 7fa628683640 10 librados: watch_flush enter 2024-01-20T11:20:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 10 librados: watch_flush exit 2024-01-20T11:20:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 20 client.5120.objecter shutdown clearing up homeless session... 2024-01-20T11:20:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 10 client.5120.objecter successfully canceled tick 2024-01-20T11:20:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa60406c8b0 msgr2=0x7fa60406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa60406c8b0 0x7fa60406ed70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa610036000 tx=0x7fa610008030 comp rx=0 tx=0).stop 2024-01-20T11:20:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 10 monclient: shutdown 2024-01-20T11:20:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 msgr2=0x7fa620142650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa620142650 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00e870 tx=0x7fa61c00ed30 comp rx=0 tx=0).stop 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 shutdown_connections 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa60406c8b0 0x7fa60406ed70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa620145760 0x7fa620139b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62013f6e0 0x7fa620139610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 --2- 172.21.15.110:0/3805409278 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6200a9d80 0x7fa620142650 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 >> 172.21.15.110:0/3805409278 conn(0x7fa6200663b0 msgr2=0x7fa620137750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 shutdown_connections 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 -- 172.21.15.110:0/3805409278 wait complete. 2024-01-20T11:20:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.130+0000 7fa628683640 1 librados: shutdown 2024-01-20T11:20:43.140 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-01-20T11:20:42.773965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002385"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.7320000000000002},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.944}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.358000000000001},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3740000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.4830000000000005},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.266}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.893000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":7.6769999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.961},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":11.629}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":14.548999999999999},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.140000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313286,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":11.484999999999999},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.1699999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":5.2679999999999998},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":7.8970000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.103},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.7619999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.2999999999999998},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.465}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.858000000000001},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":9.593}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.173999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.727}]}]},{"osd":3,"up_from":21,"seq":90194313287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.1459999999999999},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.181}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.318},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.7319999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":13.013},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.92}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.961},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.782}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":8.4280000000000008},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":11.592000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":16.382000000000001},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.292999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444100,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.085},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":5.3419999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.2490000000000006},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.898}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.862},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":7.6660000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.548999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8520000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.210000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.0579999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.4209999999999994},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.760999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444101,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.3810000000000002},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":12.819000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":15.959},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.2759999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":10.667999999999999},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":10.002000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":14.916},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.8769999999999998},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":16.780000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.978},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.684000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444101,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":9.4090000000000007},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":18.504999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":10.914999999999999},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.733000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.772},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.0960000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.798999999999999},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.7320000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":11.763999999999999},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":17.390000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.373000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":8.827}]}]},{"osd":6,"up_from":15,"seq":64424509510,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.923},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.1989999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.5419999999999998},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":15.055999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.375},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.583}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.257999999999999},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.1820000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.362},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.215}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.779},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.965}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:43.142 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:43.142 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:43.142 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:43.142 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:43.142 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:43.142 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:43.142 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:43.143 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:43.143 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.242+0000 7f1a49fe9640 10 monclient: get_monmap_and_config 2024-01-20T11:20:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.242+0000 7f1a49fe9640 10 monclient: build_initial_monmap 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.242+0000 7f1a49fe9640 10 monclient: monmap: 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:43.243506+0000 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:43.243506+0000 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:43.243 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.243+0000 7f1a49fe9640 1 Processor -- start 2024-01-20T11:20:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.243+0000 7f1a49fe9640 1 -- start start 2024-01-20T11:20:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.243+0000 7f1a49fe9640 10 monclient: init 2024-01-20T11:20:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440af3a0 0x7f1a4413b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 10 monclient(hunting): picked mon.noname-b con 0x7f1a440af3a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a440ab6d0 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a440adb30 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a44141930 con 0x7f1a440adb30 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a440ab2b0 con 0x7f1a440ab6d0 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a44142920 con 0x7f1a440af3a0 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a49fe9640 10 monclient(hunting): authenticate will time out at 1686.594633s 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.244+0000 7f1a437fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440af3a0 0x7f1a4413b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.245+0000 7f1a42ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f1a440ab6d0 0x7f1a440ad420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:33778/0 (socket says 172.21.15.110:33778) 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.245+0000 7f1a42ffd640 1 -- 172.21.15.110:0/328013085 learned_addr learned my addr 172.21.15.110:0/328013085 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.245+0000 7f1a43fff640 1 --1- 172.21.15.110:0/328013085 >> v1:172.21.15.110:6789/0 conn(0x7f1a440adb30 0x7f1a44148cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:52398/0 (socket says 172.21.15.110:52398) 2024-01-20T11:20:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.245+0000 7f1a437fe640 10 monclient(hunting): get_auth_request con 0x7f1a440af3a0 auth_method 0 2024-01-20T11:20:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.245+0000 7f1a437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.245+0000 7f1a437fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.245+0000 7f1a437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.245+0000 7f1a437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.245+0000 7f1a437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.246+0000 7f1a437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a427fc640 1 -- 172.21.15.110:0/328013085 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940820886 0 0) 0x7f1a440ab2b0 con 0x7f1a440ab6d0 2024-01-20T11:20:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a427fc640 10 monclient(hunting): _negotiate 2024-01-20T11:20:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a427fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a427fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a427fc640 10 monclient(hunting): my global_id is 5440 2024-01-20T11:20:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a427fc640 1 -- 172.21.15.110:0/328013085 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a30003520 con 0x7f1a440ab6d0 2024-01-20T11:20:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a437fe640 10 monclient(hunting): handle_auth_done global_id 5123 payload 274 2024-01-20T11:20:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a437fe640 1 -- 172.21.15.110:0/328013085 >> v1:172.21.15.134:6789/0 conn(0x7f1a440ab6d0 legacy=0x7f1a440ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a437fe640 1 -- 172.21.15.110:0/328013085 >> v1:172.21.15.110:6789/0 conn(0x7f1a440adb30 legacy=0x7f1a44148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a437fe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a437fe640 1 monclient: found mon.noname-b 2024-01-20T11:20:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a437fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.247+0000 7f1a437fe640 1 -- 172.21.15.110:0/328013085 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a440aca40 con 0x7f1a440af3a0 2024-01-20T11:20:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.249+0000 7f1a437fe640 1 --2- 172.21.15.110:0/328013085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440af3a0 0x7f1a4413b730 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400aed0 tx=0x7f1a3402d020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c2c532cd360708f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 1 -- 172.21.15.110:0/328013085 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3402dbc0 con 0x7f1a440af3a0 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 10 monclient: dump: 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 1 -- 172.21.15.110:0/328013085 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3402dd20 con 0x7f1a440af3a0 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 1 -- 172.21.15.110:0/328013085 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a34008660 con 0x7f1a440af3a0 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 5 monclient: authenticate success, global_id 5123 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 4 set_mon_vals no callback set 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a427fc640 10 monclient: dump: 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:43.255 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 10 monclient: shutdown 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 1 -- 172.21.15.110:0/328013085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440af3a0 msgr2=0x7f1a4413b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 1 --2- 172.21.15.110:0/328013085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440af3a0 0x7f1a4413b730 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400aed0 tx=0x7f1a3402d020 comp rx=0 tx=0).stop 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 1 -- 172.21.15.110:0/328013085 shutdown_connections 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 1 --2- 172.21.15.110:0/328013085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440af3a0 0x7f1a4413b730 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 1 -- 172.21.15.110:0/328013085 >> 172.21.15.110:0/328013085 conn(0x7f1a440b3e60 msgr2=0x7f1a440b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 1 -- 172.21.15.110:0/328013085 shutdown_connections 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.253+0000 7f1a49fe9640 1 -- 172.21.15.110:0/328013085 wait complete. 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.254+0000 7f1a49fe9640 10 monclient: build_initial_monmap 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.254+0000 7f1a49fe9640 10 monclient: monmap: 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.254+0000 7f1a49fe9640 1 librados: starting msgr at 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.254+0000 7f1a49fe9640 1 librados: starting objecter 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.254+0000 7f1a49fe9640 1 Processor -- start 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 1 -- start start 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 1 librados: setting wanted keys 2024-01-20T11:20:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 1 librados: calling monclient init 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient: init 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a440ab6d0 0x7f1a44071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient(hunting): picked mon.noname-b con 0x7f1a440ab6d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440adb30 0x7f1a4411c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient(hunting): picked mon.noname-c con 0x7f1a440adb30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a440af3a0 0x7f1a441d7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient(hunting): picked mon.noname-a con 0x7f1a440af3a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4411d6f0 con 0x7f1a440af3a0 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4411bf70 con 0x7f1a440ab6d0 2024-01-20T11:20:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4411b300 con 0x7f1a440adb30 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.255+0000 7f1a49fe9640 10 monclient(hunting): authenticate will time out at 1686.605837s 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a42ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440adb30 0x7f1a4411c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a43fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a440af3a0 0x7f1a441d7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a437fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a440ab6d0 0x7f1a44071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a42ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440adb30 0x7f1a4411c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52006/0 (socket says 172.21.15.110:52006) 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a42ffd640 1 -- 172.21.15.110:0/519647857 learned_addr learned my addr 172.21.15.110:0/519647857 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a42ffd640 10 monclient(hunting): get_auth_request con 0x7f1a440adb30 auth_method 0 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a42ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a437fe640 10 monclient(hunting): get_auth_request con 0x7f1a440ab6d0 auth_method 0 2024-01-20T11:20:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a437fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a43fff640 10 monclient(hunting): get_auth_request con 0x7f1a440af3a0 auth_method 0 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a43fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.256+0000 7f1a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.257+0000 7f1a437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.257+0000 7f1a437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.257+0000 7f1a437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.257+0000 7f1a42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.257+0000 7f1a42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.257+0000 7f1a42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.257+0000 7f1a43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.257+0000 7f1a43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.257+0000 7f1a43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 10 monclient(hunting): handle_auth_done global_id 5443 payload 824 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 1 -- 172.21.15.110:0/519647857 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440adb30 msgr2=0x7f1a4411c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440adb30 0x7f1a4411c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 1 -- 172.21.15.110:0/519647857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a440af3a0 msgr2=0x7f1a441d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a440af3a0 0x7f1a441d7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 1 monclient: found mon.noname-b 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.259+0000 7f1a437fe640 1 -- 172.21.15.110:0/519647857 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3400ab80 con 0x7f1a440ab6d0 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.262+0000 7f1a437fe640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a440ab6d0 0x7f1a44071880 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1a34033000 tx=0x7f1a34004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.262+0000 7f1a3bfff640 10 client.?.objecter ms_handle_connect 0x7f1a440ab6d0 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.262+0000 7f1a3bfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 1 -- 172.21.15.110:0/519647857 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3400a560 con 0x7f1a440ab6d0 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 10 monclient: dump: 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 10 monclient: _finish_auth 0 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a49fe9640 5 monclient: authenticate success, global_id 5443 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 1 -- 172.21.15.110:0/519647857 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3402de90 con 0x7f1a440ab6d0 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a49fe9640 10 monclient: _renew_subs 2024-01-20T11:20:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 1 -- 172.21.15.110:0/519647857 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a340406b0 con 0x7f1a440ab6d0 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a49fe9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a441d7b80 con 0x7f1a440ab6d0 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a41ffb640 4 set_mon_vals no callback set 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a3bfff640 10 monclient: dump: 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:43.269 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a49fe9640 10 client.5443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a49fe9640 10 monclient: _renew_subs 2024-01-20T11:20:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a49fe9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4411bcc0 con 0x7f1a440ab6d0 2024-01-20T11:20:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.265+0000 7f1a49fe9640 1 librados: init done 2024-01-20T11:20:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.266+0000 7f1a49fe9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.266+0000 7f1a49fe9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.266+0000 7f1a49fe9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.266+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a441d7d70 con 0x7f1a440ab6d0 2024-01-20T11:20:43.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.284+0000 7f1a3bfff640 1 -- 172.21.15.110:0/519647857 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1a34040810 con 0x7f1a440ab6d0 2024-01-20T11:20:43.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.285+0000 7f1a3bfff640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1a2406c8b0 0x7f1a2406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.285+0000 7f1a42ffd640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1a2406c8b0 0x7f1a2406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.285+0000 7f1a42ffd640 10 monclient: get_auth_request con 0x7f1a2406c8b0 auth_method 0 2024-01-20T11:20:43.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.285+0000 7f1a42ffd640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1a2406c8b0 0x7f1a2406ed70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1a4411daf0 tx=0x7f1a2c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:43.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.285+0000 7f1a3bfff640 10 client.5443.objecter ms_handle_connect 0x7f1a2406c8b0 2024-01-20T11:20:43.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.287+0000 7f1a3bfff640 1 -- 172.21.15.110:0/519647857 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1a3407bcd0 con 0x7f1a440ab6d0 2024-01-20T11:20:43.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.287+0000 7f1a3bfff640 10 client.5443.objecter ms_dispatch 0x7f1a44000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:43.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.287+0000 7f1a3bfff640 3 client.5443.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:43.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.287+0000 7f1a3bfff640 3 client.5443.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:43.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.287+0000 7f1a3bfff640 20 client.5443.objecter dump_active .. 0 homeless 2024-01-20T11:20:43.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.527+0000 7f1a3bfff640 1 -- 172.21.15.110:0/519647857 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1a34078010 con 0x7f1a440ab6d0 2024-01-20T11:20:43.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.527+0000 7f1a3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:43.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.527+0000 7f1a3bfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:43.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~106s 2024-01-20T11:20:43.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.768+0000 7f1a49fe9640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:43.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.768+0000 7f1a49fe9640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:43.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.768+0000 7f1a49fe9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:43.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.768+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1a44075490 con 0x7f1a440ab6d0 2024-01-20T11:20:43.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.774+0000 7f1a3bfff640 1 -- 172.21.15.110:0/519647857 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f1a34044520 con 0x7f1a440ab6d0 2024-01-20T11:20:43.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.774+0000 7f1a3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:43.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.774+0000 7f1a3bfff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:43.774 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:43.774 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":378,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:43.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 10 librados: watch_flush enter 2024-01-20T11:20:43.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 10 librados: watch_flush exit 2024-01-20T11:20:43.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 20 client.5443.objecter shutdown clearing up homeless session... 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 10 client.5443.objecter successfully canceled tick 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1a2406c8b0 msgr2=0x7f1a2406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1a2406c8b0 0x7f1a2406ed70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1a4411daf0 tx=0x7f1a2c009040 comp rx=0 tx=0).stop 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 10 monclient: shutdown 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a440ab6d0 msgr2=0x7f1a44071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a440ab6d0 0x7f1a44071880 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1a34033000 tx=0x7f1a34004a40 comp rx=0 tx=0).stop 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 shutdown_connections 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1a2406c8b0 0x7f1a2406ed70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a440af3a0 0x7f1a441d7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1a440adb30 0x7f1a4411c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 --2- 172.21.15.110:0/519647857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a440ab6d0 0x7f1a44071880 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 >> 172.21.15.110:0/519647857 conn(0x7f1a440b3e60 msgr2=0x7f1a440b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:43.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 shutdown_connections 2024-01-20T11:20:43.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 -- 172.21.15.110:0/519647857 wait complete. 2024-01-20T11:20:43.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.776+0000 7f1a49fe9640 1 librados: shutdown 2024-01-20T11:20:43.786 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:43.787 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.888+0000 7fc2f9b9e640 10 monclient: get_monmap_and_config 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.888+0000 7fc2f9b9e640 10 monclient: build_initial_monmap 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.888+0000 7fc2f9b9e640 10 monclient: monmap: 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:43.888626+0000 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:43.888626+0000 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:43.888 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:43.889 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:43.889 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:43.889 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.888+0000 7fc2f9b9e640 1 Processor -- start 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.888+0000 7fc2f9b9e640 1 -- start start 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.888+0000 7fc2f9b9e640 10 monclient: init 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc2f40b0c00 0x7fc2f40accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f40b0c00 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f413d0e0 0x7fc2f413d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f413d0e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2f4143200 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:43.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f406dcc0 con 0x7fc2f4143200 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f40ad450 con 0x7fc2f40b0c00 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f40ad8e0 con 0x7fc2f413d0e0 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.889+0000 7fc2f9b9e640 10 monclient(hunting): authenticate will time out at 1687.239684s 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f413d0e0 0x7fc2f413d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f37fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc2f40b0c00 0x7fc2f40accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f413d0e0 0x7fc2f413d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52008/0 (socket says 172.21.15.110:52008) 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 1 -- 172.21.15.110:0/3320495941 learned_addr learned my addr 172.21.15.110:0/3320495941 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 10 monclient(hunting): get_auth_request con 0x7fc2f413d0e0 auth_method 0 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f37fe640 10 monclient(hunting): get_auth_request con 0x7fc2f40b0c00 auth_method 0 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.890+0000 7fc2f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.891+0000 7fc2f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.891+0000 7fc2f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:43.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.891+0000 7fc2f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.892+0000 7fc2f2ffd640 10 monclient(hunting): handle_auth_done global_id 5129 payload 274 2024-01-20T11:20:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.892+0000 7fc2f2ffd640 1 -- 172.21.15.110:0/3320495941 >> v2:172.21.15.110:3300/0 conn(0x7fc2f40b0c00 msgr2=0x7fc2f40accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.892+0000 7fc2f2ffd640 1 --2- 172.21.15.110:0/3320495941 >> v2:172.21.15.110:3300/0 conn(0x7fc2f40b0c00 0x7fc2f40accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.892+0000 7fc2f2ffd640 1 -- 172.21.15.110:0/3320495941 >> v1:172.21.15.110:6789/0 conn(0x7fc2f4143200 legacy=0x7fc2f4143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.892+0000 7fc2f2ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:20:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.892+0000 7fc2f2ffd640 1 monclient: found mon.noname-b 2024-01-20T11:20:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.892+0000 7fc2f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.892+0000 7fc2f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:43.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.892+0000 7fc2f2ffd640 1 -- 172.21.15.110:0/3320495941 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f4143da0 con 0x7fc2f413d0e0 2024-01-20T11:20:43.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.894+0000 7fc2f2ffd640 1 --2- 172.21.15.110:0/3320495941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f413d0e0 0x7fc2f413d520 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009870 tx=0x7fc2e802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad22cab5d9cc7965 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:43.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 1 -- 172.21.15.110:0/3320495941 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8031640 con 0x7fc2f413d0e0 2024-01-20T11:20:43.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:43.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:43.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 10 monclient: dump: 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:43.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 1 -- 172.21.15.110:0/3320495941 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e80317a0 con 0x7fc2f413d0e0 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 1 -- 172.21.15.110:0/3320495941 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8031b30 con 0x7fc2f413d0e0 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f9b9e640 5 monclient: authenticate success, global_id 5129 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f9b9e640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f9b9e640 4 set_mon_vals no callback set 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.897+0000 7fc2f27fc640 10 monclient: dump: 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:43.899 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.900 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:43.900 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:43.900 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:43.900 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 10 monclient: shutdown 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/3320495941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f413d0e0 msgr2=0x7fc2f413d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 1 --2- 172.21.15.110:0/3320495941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f413d0e0 0x7fc2f413d520 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009870 tx=0x7fc2e802b8f0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/3320495941 shutdown_connections 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 1 --2- 172.21.15.110:0/3320495941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f413d0e0 0x7fc2f413d520 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 1 --2- 172.21.15.110:0/3320495941 >> v2:172.21.15.110:3300/0 conn(0x7fc2f40b0c00 0x7fc2f40accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/3320495941 >> 172.21.15.110:0/3320495941 conn(0x7fc2f40b3e60 msgr2=0x7fc2f40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/3320495941 shutdown_connections 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.898+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/3320495941 wait complete. 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.899+0000 7fc2f9b9e640 10 monclient: build_initial_monmap 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.899+0000 7fc2f9b9e640 10 monclient: monmap: 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:43.904 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.899+0000 7fc2f9b9e640 1 librados: starting msgr at 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.899+0000 7fc2f9b9e640 1 librados: starting objecter 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 Processor -- start 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 -- start start 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 librados: setting wanted keys 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 librados: calling monclient init 2024-01-20T11:20:43.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient: init 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f40b0c00 0x7fc2f4134990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f40b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2f413d0e0 0x7fc2f41486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f413d0e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f4143200 0x7fc2f4148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f4143200 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f40ac390 con 0x7fc2f413d0e0 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f40aa8c0 con 0x7fc2f4143200 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f40aa590 con 0x7fc2f40b0c00 2024-01-20T11:20:43.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.900+0000 7fc2f9b9e640 10 monclient(hunting): authenticate will time out at 1687.251000s 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f40b0c00 0x7fc2f4134990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f2ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2f413d0e0 0x7fc2f41486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f3fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f4143200 0x7fc2f4148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f40b0c00 0x7fc2f4134990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52014/0 (socket says 172.21.15.110:52014) 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f37fe640 1 -- 172.21.15.110:0/416230208 learned_addr learned my addr 172.21.15.110:0/416230208 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f37fe640 10 monclient(hunting): get_auth_request con 0x7fc2f40b0c00 auth_method 0 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f2ffd640 10 monclient(hunting): get_auth_request con 0x7fc2f413d0e0 auth_method 0 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f3fff640 10 monclient(hunting): get_auth_request con 0x7fc2f4143200 auth_method 0 2024-01-20T11:20:43.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:43.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:43.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.901+0000 7fc2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:43.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.902+0000 7fc2f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:43.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.902+0000 7fc2f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:43.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.902+0000 7fc2f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:43.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.902+0000 7fc2f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:43.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.902+0000 7fc2f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:43.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.902+0000 7fc2f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:43.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.902+0000 7fc2f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:43.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.902+0000 7fc2f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:43.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.902+0000 7fc2f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:43.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 10 monclient(hunting): handle_auth_done global_id 5132 payload 824 2024-01-20T11:20:43.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 1 -- 172.21.15.110:0/416230208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f4143200 msgr2=0x7fc2f4148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f4143200 0x7fc2f4148c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 1 -- 172.21.15.110:0/416230208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2f413d0e0 msgr2=0x7fc2f41486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:43.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2f413d0e0 0x7fc2f41486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:43.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:20:43.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 1 monclient: found mon.noname-c 2024-01-20T11:20:43.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.904+0000 7fc2f37fe640 1 -- 172.21.15.110:0/416230208 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80094d0 con 0x7fc2f40b0c00 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.906+0000 7fc2f37fe640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f40b0c00 0x7fc2f4134990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400e9e0 tx=0x7fc2e400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.906+0000 7fc2dbfff640 10 client.?.objecter ms_handle_connect 0x7fc2f40b0c00 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.906+0000 7fc2dbfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 1 -- 172.21.15.110:0/416230208 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4009670 con 0x7fc2f40b0c00 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 10 monclient: dump: 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.912 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 10 monclient: _finish_auth 0 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 1 -- 172.21.15.110:0/416230208 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e400c9d0 con 0x7fc2f40b0c00 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f9b9e640 5 monclient: authenticate success, global_id 5132 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f9b9e640 10 monclient: _renew_subs 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f9b9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f40ab6d0 con 0x7fc2f40b0c00 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:43.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 1 -- 172.21.15.110:0/416230208 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e400cd80 con 0x7fc2f40b0c00 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2dbfff640 10 monclient: dump: 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f9b9e640 10 client.5132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f9b9e640 10 monclient: _renew_subs 2024-01-20T11:20:43.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f9b9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:43.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f40aad50 con 0x7fc2f40b0c00 2024-01-20T11:20:43.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f1ffb640 4 set_mon_vals no callback set 2024-01-20T11:20:43.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.910+0000 7fc2f9b9e640 1 librados: init done 2024-01-20T11:20:43.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.911+0000 7fc2f9b9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:43.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.911+0000 7fc2f9b9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:43.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.911+0000 7fc2f9b9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:43.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.911+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f413de40 con 0x7fc2f40b0c00 2024-01-20T11:20:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.928+0000 7fc2dbfff640 1 -- 172.21.15.110:0/416230208 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc2e4013470 con 0x7fc2f40b0c00 2024-01-20T11:20:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.929+0000 7fc2dbfff640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2e006c860 0x7fc2e006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.929+0000 7fc2f2ffd640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2e006c860 0x7fc2e006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.929+0000 7fc2f2ffd640 10 monclient: get_auth_request con 0x7fc2e006c860 auth_method 0 2024-01-20T11:20:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.930+0000 7fc2dbfff640 10 client.5132.objecter ms_handle_connect 0x7fc2e006c860 2024-01-20T11:20:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.930+0000 7fc2f2ffd640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2e006c860 0x7fc2e006ed20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8036000 tx=0x7fc2e803c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:43.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.931+0000 7fc2dbfff640 1 -- 172.21.15.110:0/416230208 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc2e4058c20 con 0x7fc2f40b0c00 2024-01-20T11:20:43.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.931+0000 7fc2dbfff640 10 client.5132.objecter ms_dispatch 0x7fc2f4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:43.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.931+0000 7fc2dbfff640 3 client.5132.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:43.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.931+0000 7fc2dbfff640 3 client.5132.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:43.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:43.931+0000 7fc2dbfff640 20 client.5132.objecter dump_active .. 0 homeless 2024-01-20T11:20:44.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.156+0000 7fc2dbfff640 1 -- 172.21.15.110:0/416230208 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc2e4054f60 con 0x7fc2f40b0c00 2024-01-20T11:20:44.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.156+0000 7fc2dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:44.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.157+0000 7fc2dbfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:44.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.323+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2f414a190 con 0x7fc2e006c860 2024-01-20T11:20:44.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.326+0000 7fc2dbfff640 1 -- 172.21.15.110:0/416230208 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63883 (secure 0 0 0) 0x7fc2e8031440 con 0x7fc2e006c860 2024-01-20T11:20:44.326 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:44.327 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 10 librados: watch_flush enter 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 10 librados: watch_flush exit 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 20 client.5132.objecter shutdown clearing up homeless session... 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 10 client.5132.objecter successfully canceled tick 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2e006c860 msgr2=0x7fc2e006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2e006c860 0x7fc2e006ed20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8036000 tx=0x7fc2e803c040 comp rx=0 tx=0).stop 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 10 monclient: shutdown 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f40b0c00 msgr2=0x7fc2f4134990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f40b0c00 0x7fc2f4134990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400e9e0 tx=0x7fc2e400eea0 comp rx=0 tx=0).stop 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 shutdown_connections 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc2e006c860 0x7fc2e006ed20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f4143200 0x7fc2f4148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:44.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2f413d0e0 0x7fc2f41486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:44.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 --2- 172.21.15.110:0/416230208 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2f40b0c00 0x7fc2f4134990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:44.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 >> 172.21.15.110:0/416230208 conn(0x7fc2f40b3e60 msgr2=0x7fc2f413c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:44.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 shutdown_connections 2024-01-20T11:20:44.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.328+0000 7fc2f9b9e640 1 -- 172.21.15.110:0/416230208 wait complete. 2024-01-20T11:20:44.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:44.329+0000 7fc2f9b9e640 1 librados: shutdown 2024-01-20T11:20:44.339 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-01-20T11:20:42.773965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002385"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":4.7320000000000002},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.944}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.358000000000001},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3740000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.4830000000000005},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":10.266}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.893000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":7.6769999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":12.961},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":11.629}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":14.548999999999999},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.140000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313286,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":11.484999999999999},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.1699999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":5.2679999999999998},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":7.8970000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":12.103},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.7619999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.2999999999999998},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.465}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.858000000000001},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":9.593}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.173999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.727}]}]},{"osd":3,"up_from":21,"seq":90194313287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.1459999999999999},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.181}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.318},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.7319999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":13.013},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.92}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.961},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.782}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":8.4280000000000008},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":11.592000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":16.382000000000001},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.292999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444100,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":7.085},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":5.3419999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.2490000000000006},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":11.898}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.862},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":7.6660000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":14.548999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.8520000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.210000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":6.0579999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.4209999999999994},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.760999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444101,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.3810000000000002},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":12.819000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":15.959},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.2759999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":10.667999999999999},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":10.002000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":14.916},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.8769999999999998},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":16.780000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.978},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.684000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444101,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":9.4090000000000007},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":18.504999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":10.914999999999999},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.733000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.772},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":8.0960000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.798999999999999},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.7320000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":11.763999999999999},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":17.390000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":16.373000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":8.827}]}]},{"osd":6,"up_from":15,"seq":64424509510,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.923},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":4.1989999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.5419999999999998},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":15.055999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.375},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.583}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.257999999999999},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":9.1820000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.362},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.215}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":12.779},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.965}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:47.341 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:47.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.441+0000 7f7e36dd7640 10 monclient: get_monmap_and_config 2024-01-20T11:20:47.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.441+0000 7f7e36dd7640 10 monclient: build_initial_monmap 2024-01-20T11:20:47.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.441+0000 7f7e36dd7640 10 monclient: monmap: 2024-01-20T11:20:47.441 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:47.441 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:47.441 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:47.442023+0000 2024-01-20T11:20:47.441 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:47.442023+0000 2024-01-20T11:20:47.441 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:47.441 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:47.442 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:47.442 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:47.442 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:47.442 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:47.442 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:47.442 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:47.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 1 Processor -- start 2024-01-20T11:20:47.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 1 -- start start 2024-01-20T11:20:47.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient: init 2024-01-20T11:20:47.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:47.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e30145760 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7e300a9d80 0x7f7e30145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient(hunting): picked mon.noname-a con 0x7f7e300a9d80 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7e3013f8b0 0x7f7e300b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient(hunting): picked mon.noname-c con 0x7f7e3013f8b0 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e30146250 con 0x7f7e30145760 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e300ab050 con 0x7f7e300a9d80 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e300b20c0 con 0x7f7e3013f8b0 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.442+0000 7f7e36dd7640 10 monclient(hunting): authenticate will time out at 1690.792966s 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.443+0000 7f7e2ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7e300a9d80 0x7f7e30145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.443+0000 7f7e35374640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7e3013f8b0 0x7f7e300b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:47.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.443+0000 7f7e2ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7e300a9d80 0x7f7e30145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58384/0 (socket says 172.21.15.110:58384) 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.443+0000 7f7e2ffff640 1 -- 172.21.15.110:0/1557276516 learned_addr learned my addr 172.21.15.110:0/1557276516 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.443+0000 7f7e2ffff640 10 monclient(hunting): get_auth_request con 0x7f7e300a9d80 auth_method 0 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.443+0000 7f7e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.443+0000 7f7e2ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.443+0000 7f7e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e35374640 10 monclient(hunting): get_auth_request con 0x7f7e3013f8b0 auth_method 0 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e35374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e35374640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e35374640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e35374640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:47.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e35374640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:47.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.444+0000 7f7e35374640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:47.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.445+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_done global_id 5388 payload 274 2024-01-20T11:20:47.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.445+0000 7f7e2ffff640 1 -- 172.21.15.110:0/1557276516 >> v2:172.21.15.134:3300/0 conn(0x7f7e3013f8b0 msgr2=0x7f7e300b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:47.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.445+0000 7f7e2ffff640 1 --2- 172.21.15.110:0/1557276516 >> v2:172.21.15.134:3300/0 conn(0x7f7e3013f8b0 0x7f7e300b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.445+0000 7f7e2ffff640 1 -- 172.21.15.110:0/1557276516 >> v1:172.21.15.134:6789/0 conn(0x7f7e30145760 legacy=0x7f7e3013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:47.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.445+0000 7f7e2ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:47.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.445+0000 7f7e2ffff640 1 monclient: found mon.noname-a 2024-01-20T11:20:47.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.445+0000 7f7e2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:47.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.445+0000 7f7e2ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:47.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.445+0000 7f7e2ffff640 1 -- 172.21.15.110:0/1557276516 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e300b2bb0 con 0x7f7e300a9d80 2024-01-20T11:20:47.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.447+0000 7f7e2ffff640 1 --2- 172.21.15.110:0/1557276516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e300a9d80 0x7f7e30145b40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009870 tx=0x7f7e1c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457acc7fec25eade server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 1 -- 172.21.15.110:0/1557276516 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c030860 con 0x7f7e300a9d80 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 10 monclient: dump: 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:47.451 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 1 -- 172.21.15.110:0/1557276516 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c0309c0 con 0x7f7e300a9d80 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 1 -- 172.21.15.110:0/1557276516 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c030d50 con 0x7f7e300a9d80 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 5 monclient: authenticate success, global_id 5388 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 4 set_mon_vals no callback set 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e2f7fe640 10 monclient: dump: 2024-01-20T11:20:47.452 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:47.453 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:47.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 10 monclient: shutdown 2024-01-20T11:20:47.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:47.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 1 -- 172.21.15.110:0/1557276516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e300a9d80 msgr2=0x7f7e30145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 1 --2- 172.21.15.110:0/1557276516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e300a9d80 0x7f7e30145b40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009870 tx=0x7f7e1c02b9a0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 1 -- 172.21.15.110:0/1557276516 shutdown_connections 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 1 --2- 172.21.15.110:0/1557276516 >> v2:172.21.15.134:3300/0 conn(0x7f7e3013f8b0 0x7f7e300b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 1 --2- 172.21.15.110:0/1557276516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e300a9d80 0x7f7e30145b40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 1 -- 172.21.15.110:0/1557276516 >> 172.21.15.110:0/1557276516 conn(0x7f7e300663b0 msgr2=0x7f7e300667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 1 -- 172.21.15.110:0/1557276516 shutdown_connections 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.451+0000 7f7e36dd7640 1 -- 172.21.15.110:0/1557276516 wait complete. 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.452+0000 7f7e36dd7640 10 monclient: build_initial_monmap 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.452+0000 7f7e36dd7640 10 monclient: monmap: 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 1 librados: starting msgr at 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 1 librados: starting objecter 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 1 Processor -- start 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 1 -- start start 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 1 librados: setting wanted keys 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 1 librados: calling monclient init 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 10 monclient: init 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e300a9d80 0x7f7e30138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.453+0000 7f7e36dd7640 10 monclient(hunting): picked mon.noname-c con 0x7f7e300a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e3013f8b0 0x7f7e301dab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3013f8b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e30145760 0x7f7e301db090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 10 monclient(hunting): picked mon.noname-b con 0x7f7e30145760 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e301387c0 con 0x7f7e3013f8b0 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e301db5d0 con 0x7f7e30145760 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e30139c40 con 0x7f7e300a9d80 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e36dd7640 10 monclient(hunting): authenticate will time out at 1690.804230s 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e2ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e3013f8b0 0x7f7e301dab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e35374640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e30145760 0x7f7e301db090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e34b73640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e300a9d80 0x7f7e30138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e35374640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e30145760 0x7f7e301db090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:36004/0 (socket says 172.21.15.110:36004) 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e35374640 1 -- 172.21.15.110:0/231344497 learned_addr learned my addr 172.21.15.110:0/231344497 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e35374640 10 monclient(hunting): get_auth_request con 0x7f7e30145760 auth_method 0 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e35374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e35374640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.454+0000 7f7e35374640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e2ffff640 10 monclient(hunting): get_auth_request con 0x7f7e3013f8b0 auth_method 0 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e2ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:47.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e34b73640 10 monclient(hunting): get_auth_request con 0x7f7e300a9d80 auth_method 0 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e34b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e34b73640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e34b73640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e35374640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e35374640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e35374640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e34b73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e34b73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.455+0000 7f7e34b73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 10 monclient(hunting): handle_auth_done global_id 5455 payload 824 2024-01-20T11:20:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 1 -- 172.21.15.110:0/231344497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e300a9d80 msgr2=0x7f7e30138280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e300a9d80 0x7f7e30138280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 1 -- 172.21.15.110:0/231344497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e3013f8b0 msgr2=0x7f7e301dab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e3013f8b0 0x7f7e301dab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 10 monclient: _finish_hunting 0 2024-01-20T11:20:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 1 monclient: found mon.noname-b 2024-01-20T11:20:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.458+0000 7f7e35374640 1 -- 172.21.15.110:0/231344497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0094d0 con 0x7f7e30145760 2024-01-20T11:20:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.460+0000 7f7e35374640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e30145760 0x7f7e301db090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800efc0 tx=0x7f7e2800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.460+0000 7f7e2d7fa640 10 client.?.objecter ms_handle_connect 0x7f7e30145760 2024-01-20T11:20:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.460+0000 7f7e2d7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 1 -- 172.21.15.110:0/231344497 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28009ca0 con 0x7f7e30145760 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 10 monclient: dump: 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 1 -- 172.21.15.110:0/231344497 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2800f040 con 0x7f7e30145760 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 1 -- 172.21.15.110:0/231344497 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e280146c0 con 0x7f7e30145760 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e36dd7640 5 monclient: authenticate success, global_id 5455 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e36dd7640 10 monclient: _renew_subs 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e36dd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e30139f20 con 0x7f7e30145760 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2e7fc640 4 set_mon_vals no callback set 2024-01-20T11:20:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.463+0000 7f7e2d7fa640 10 monclient: dump: 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.464+0000 7f7e36dd7640 10 client.5455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.464+0000 7f7e36dd7640 10 monclient: _renew_subs 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.464+0000 7f7e36dd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.464+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e30142b30 con 0x7f7e30145760 2024-01-20T11:20:47.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.464+0000 7f7e36dd7640 1 librados: init done 2024-01-20T11:20:47.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.465+0000 7f7e36dd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:47.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.465+0000 7f7e36dd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:47.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.465+0000 7f7e36dd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:47.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.465+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e30138900 con 0x7f7e30145760 2024-01-20T11:20:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.482+0000 7f7e2d7fa640 1 -- 172.21.15.110:0/231344497 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7e28014820 con 0x7f7e30145760 2024-01-20T11:20:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.483+0000 7f7e2d7fa640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7e1406c8b0 0x7f7e1406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:47.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.483+0000 7f7e34b73640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7e1406c8b0 0x7f7e1406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:47.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.483+0000 7f7e34b73640 10 monclient: get_auth_request con 0x7f7e1406c8b0 auth_method 0 2024-01-20T11:20:47.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.483+0000 7f7e2d7fa640 10 client.5455.objecter ms_handle_connect 0x7f7e1406c8b0 2024-01-20T11:20:47.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.483+0000 7f7e34b73640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7e1406c8b0 0x7f7e1406ed70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7e24000f60 tx=0x7f7e24009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:47.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.486+0000 7f7e2d7fa640 1 -- 172.21.15.110:0/231344497 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7e28059480 con 0x7f7e30145760 2024-01-20T11:20:47.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.486+0000 7f7e2d7fa640 10 client.5455.objecter ms_dispatch 0x7f7e30000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:47.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.486+0000 7f7e2d7fa640 3 client.5455.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:47.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.486+0000 7f7e2d7fa640 3 client.5455.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:47.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.486+0000 7f7e2d7fa640 20 client.5455.objecter dump_active .. 0 homeless 2024-01-20T11:20:47.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.721+0000 7f7e2d7fa640 1 -- 172.21.15.110:0/231344497 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7e280557c0 con 0x7f7e30145760 2024-01-20T11:20:47.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.721+0000 7f7e2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:47.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.721+0000 7f7e2d7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:47.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.890+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e300b2200 con 0x7f7e1406c8b0 2024-01-20T11:20:47.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.892+0000 7f7e2d7fa640 1 -- 172.21.15.110:0/231344497 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63897 (secure 0 0 0) 0x7f7e24001d10 con 0x7f7e1406c8b0 2024-01-20T11:20:47.893 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:47.893 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 10 librados: watch_flush enter 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 10 librados: watch_flush exit 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 20 client.5455.objecter shutdown clearing up homeless session... 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 10 client.5455.objecter successfully canceled tick 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7e1406c8b0 msgr2=0x7f7e1406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7e1406c8b0 0x7f7e1406ed70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7e24000f60 tx=0x7f7e24009040 comp rx=0 tx=0).stop 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 10 monclient: shutdown 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e30145760 msgr2=0x7f7e301db090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e30145760 0x7f7e301db090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800efc0 tx=0x7f7e2800c490 comp rx=0 tx=0).stop 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 shutdown_connections 2024-01-20T11:20:47.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7e1406c8b0 0x7f7e1406ed70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e30145760 0x7f7e301db090 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e3013f8b0 0x7f7e301dab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 --2- 172.21.15.110:0/231344497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e300a9d80 0x7f7e30138280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:47.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 >> 172.21.15.110:0/231344497 conn(0x7f7e300663b0 msgr2=0x7f7e300b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:47.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 shutdown_connections 2024-01-20T11:20:47.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.894+0000 7f7e36dd7640 1 -- 172.21.15.110:0/231344497 wait complete. 2024-01-20T11:20:47.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:47.895+0000 7f7e36dd7640 1 librados: shutdown 2024-01-20T11:20:47.905 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-01-20T11:20:46.774937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002534"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.8239999999999998},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.9420000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.1059999999999999},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.176}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.096},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.266999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.901},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":14.773999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":9.6020000000000003},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":12.858000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":7.6980000000000004},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.481999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313287,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":8.4239999999999995},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7759999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.8220000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.0310000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.218999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.5609999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.6999999999999993},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.816000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.378},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":14.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.868},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":10.917999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.1459999999999999},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.181}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.318},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.7319999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":13.013},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.92}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.961},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.782}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":8.4280000000000008},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":11.592000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":16.382000000000001},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.292999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444101,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.6319999999999997},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":13.903}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":17.024000000000001},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.2260000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.919},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":7.6429999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.8879999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.932}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":18.132000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":11.981999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.131},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":15.871}]}]},{"osd":1,"up_from":17,"seq":73014444102,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.242},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":9.1359999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.7439999999999998},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":6.2370000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":13.121},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.3680000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":10.321},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":8.3680000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.2140000000000004},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":12.414999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.566000000000001},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.654}]}]},{"osd":0,"up_from":17,"seq":73014444102,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":6.0810000000000004},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":8.9960000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":11.074999999999999},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":4.819}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":7.3179999999999996},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.18}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.332000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.919}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":8.3079999999999998},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":11.734999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.504},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":14.991}]}]},{"osd":6,"up_from":15,"seq":64424509511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":8.3000000000000007},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.1870000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.9039999999999999},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":14.162000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.4489999999999998},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3160000000000007}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.970000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.244}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.891},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":9.9309999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.962999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.85}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:47.907 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:48.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.005+0000 7fd8434d4640 10 monclient: get_monmap_and_config 2024-01-20T11:20:48.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.005+0000 7fd8434d4640 10 monclient: build_initial_monmap 2024-01-20T11:20:48.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.005+0000 7fd8434d4640 10 monclient: monmap: 2024-01-20T11:20:48.005 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:48.005881+0000 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:48.005881+0000 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.005+0000 7fd8434d4640 1 Processor -- start 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 1 -- start start 2024-01-20T11:20:48.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient: init 2024-01-20T11:20:48.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0af3a0 0x7fd83c13b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient(hunting): picked mon.noname-b con 0x7fd83c0af3a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd83c0ab6d0 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd83c0adb30 0x7fd83c13bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient(hunting): picked mon.noname-a con 0x7fd83c0adb30 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c141930 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83c142920 con 0x7fd83c0adb30 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c0af780 con 0x7fd83c0af3a0 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:48.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.006+0000 7fd8434d4640 10 monclient(hunting): authenticate will time out at 1691.356820s 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841270640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0af3a0 0x7fd83c13b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd840a6f640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd83c0ab6d0 0x7fd83c0ad420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:33804/0 (socket says 172.21.15.110:33804) 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841a71640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd83c0adb30 0x7fd83c13bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd840a6f640 1 -- 172.21.15.110:0/55501965 learned_addr learned my addr 172.21.15.110:0/55501965 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841a71640 10 monclient(hunting): get_auth_request con 0x7fd83c0adb30 auth_method 0 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841a71640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841a71640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841270640 10 monclient(hunting): get_auth_request con 0x7fd83c0af3a0 auth_method 0 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841270640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841270640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.007+0000 7fd841270640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.008+0000 7fd841a71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.008+0000 7fd841a71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.008+0000 7fd841a71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.008+0000 7fd841270640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.008+0000 7fd841270640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.008+0000 7fd841270640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.009+0000 7fd833fff640 1 -- 172.21.15.110:0/55501965 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472077674 0 0) 0x7fd83c141930 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.009+0000 7fd833fff640 10 monclient(hunting): _negotiate 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.009+0000 7fd833fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.009+0000 7fd833fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.009+0000 7fd833fff640 10 monclient(hunting): my global_id is 5458 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.009+0000 7fd833fff640 1 -- 172.21.15.110:0/55501965 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd828003520 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.010+0000 7fd841a71640 10 monclient(hunting): handle_auth_done global_id 5394 payload 274 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.010+0000 7fd841a71640 1 -- 172.21.15.110:0/55501965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0af3a0 msgr2=0x7fd83c13b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:48.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.010+0000 7fd841a71640 1 --2- 172.21.15.110:0/55501965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0af3a0 0x7fd83c13b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.010+0000 7fd841a71640 1 -- 172.21.15.110:0/55501965 >> v1:172.21.15.134:6789/0 conn(0x7fd83c0ab6d0 legacy=0x7fd83c0ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.010+0000 7fd841a71640 10 monclient: _finish_hunting 0 2024-01-20T11:20:48.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.010+0000 7fd841a71640 1 monclient: found mon.noname-a 2024-01-20T11:20:48.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.010+0000 7fd841a71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:48.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.010+0000 7fd841a71640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:48.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.010+0000 7fd841a71640 1 -- 172.21.15.110:0/55501965 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c0ab2b0 con 0x7fd83c0adb30 2024-01-20T11:20:48.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.013+0000 7fd841a71640 1 --2- 172.21.15.110:0/55501965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0adb30 0x7fd83c13bc70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd8380097a0 tx=0x7fd83802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8c1553056b0445 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:48.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 1 -- 172.21.15.110:0/55501965 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838031830 con 0x7fd83c0adb30 2024-01-20T11:20:48.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:48.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 10 monclient: dump: 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 10 monclient: _finish_auth 0 2024-01-20T11:20:48.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 1 -- 172.21.15.110:0/55501965 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd838031990 con 0x7fd83c0adb30 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 1 -- 172.21.15.110:0/55501965 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838031d20 con 0x7fd83c0adb30 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd833fff640 10 monclient: dump: 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:48.018 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:48.019 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:48.019 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd8434d4640 5 monclient: authenticate success, global_id 5394 2024-01-20T11:20:48.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd8434d4640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:48.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.016+0000 7fd8434d4640 4 set_mon_vals no callback set 2024-01-20T11:20:48.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 10 monclient: shutdown 2024-01-20T11:20:48.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:48.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 1 -- 172.21.15.110:0/55501965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0adb30 msgr2=0x7fd83c13bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 1 --2- 172.21.15.110:0/55501965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0adb30 0x7fd83c13bc70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd8380097a0 tx=0x7fd83802b8d0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 1 -- 172.21.15.110:0/55501965 shutdown_connections 2024-01-20T11:20:48.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 1 --2- 172.21.15.110:0/55501965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0adb30 0x7fd83c13bc70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 1 --2- 172.21.15.110:0/55501965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0af3a0 0x7fd83c13b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 1 -- 172.21.15.110:0/55501965 >> 172.21.15.110:0/55501965 conn(0x7fd83c0b3e60 msgr2=0x7fd83c0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:48.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 1 -- 172.21.15.110:0/55501965 shutdown_connections 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.017+0000 7fd8434d4640 1 -- 172.21.15.110:0/55501965 wait complete. 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.018+0000 7fd8434d4640 10 monclient: build_initial_monmap 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.018+0000 7fd8434d4640 10 monclient: monmap: 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.018+0000 7fd8434d4640 1 librados: starting msgr at 2024-01-20T11:20:48.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.018+0000 7fd8434d4640 1 librados: starting objecter 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.018+0000 7fd8434d4640 1 Processor -- start 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.018+0000 7fd8434d4640 1 -- start start 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.018+0000 7fd8434d4640 1 librados: setting wanted keys 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.018+0000 7fd8434d4640 1 librados: calling monclient init 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.018+0000 7fd8434d4640 10 monclient: init 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd83c0ab6d0 0x7fd83c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient(hunting): picked mon.noname-b con 0x7fd83c0ab6d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0adb30 0x7fd83c13fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient(hunting): picked mon.noname-c con 0x7fd83c0adb30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0af3a0 0x7fd83c1401e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient(hunting): picked mon.noname-a con 0x7fd83c0af3a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:48.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c13db50 con 0x7fd83c0af3a0 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c13dfe0 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c140720 con 0x7fd83c0adb30 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd8434d4640 10 monclient(hunting): authenticate will time out at 1691.369524s 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd841a71640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0af3a0 0x7fd83c1401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd840a6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0adb30 0x7fd83c13fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.019+0000 7fd841270640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd83c0ab6d0 0x7fd83c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841a71640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0af3a0 0x7fd83c1401e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58418/0 (socket says 172.21.15.110:58418) 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841270640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd83c0ab6d0 0x7fd83c13cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:36008/0 (socket says 172.21.15.110:36008) 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841a71640 1 -- 172.21.15.110:0/2654986720 learned_addr learned my addr 172.21.15.110:0/2654986720 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841a71640 10 monclient(hunting): get_auth_request con 0x7fd83c0af3a0 auth_method 0 2024-01-20T11:20:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841a71640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841a71640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841270640 10 monclient(hunting): get_auth_request con 0x7fd83c0ab6d0 auth_method 0 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841270640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841270640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841270640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd840a6f640 10 monclient(hunting): get_auth_request con 0x7fd83c0adb30 auth_method 0 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd840a6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd840a6f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd840a6f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841270640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841270640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841270640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841a71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841a71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd841a71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd840a6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.020+0000 7fd840a6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.021+0000 7fd840a6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 10 monclient(hunting): handle_auth_done global_id 5461 payload 824 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 1 -- 172.21.15.110:0/2654986720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0adb30 msgr2=0x7fd83c13fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0adb30 0x7fd83c13fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 1 -- 172.21.15.110:0/2654986720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0af3a0 msgr2=0x7fd83c1401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0af3a0 0x7fd83c1401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 10 monclient: _finish_hunting 0 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 1 monclient: found mon.noname-b 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.023+0000 7fd841270640 1 -- 172.21.15.110:0/2654986720 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838009400 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.025+0000 7fd841270640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd83c0ab6d0 0x7fd83c13cd70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd83400ebc0 tx=0x7fd83400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.025+0000 7fd831ffb640 10 client.?.objecter ms_handle_connect 0x7fd83c0ab6d0 2024-01-20T11:20:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.025+0000 7fd831ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 1 -- 172.21.15.110:0/2654986720 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83400cda0 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 10 monclient: dump: 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 10 monclient: _finish_auth 0 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 1 -- 172.21.15.110:0/2654986720 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83400f040 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 1 -- 172.21.15.110:0/2654986720 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8340146c0 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8434d4640 5 monclient: authenticate success, global_id 5461 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8434d4640 10 monclient: _renew_subs 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8434d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83c13f160 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8337fe640 4 set_mon_vals no callback set 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd831ffb640 10 monclient: dump: 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8434d4640 10 client.5461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8434d4640 10 monclient: _renew_subs 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8434d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:48.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83c13d2b0 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.029+0000 7fd8434d4640 1 librados: init done 2024-01-20T11:20:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.030+0000 7fd8434d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.030+0000 7fd8434d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.030+0000 7fd8434d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.030+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83c075490 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.268+0000 7fd831ffb640 1 -- 172.21.15.110:0/2654986720 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd834014820 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.269+0000 7fd831ffb640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd82006c860 0x7fd82006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.269+0000 7fd840a6f640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd82006c860 0x7fd82006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.269+0000 7fd840a6f640 10 monclient: get_auth_request con 0x7fd82006c860 auth_method 0 2024-01-20T11:20:48.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.270+0000 7fd840a6f640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd82006c860 0x7fd82006ed20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd83c13f440 tx=0x7fd82c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:48.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.270+0000 7fd831ffb640 10 client.5461.objecter ms_handle_connect 0x7fd82006c860 2024-01-20T11:20:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.272+0000 7fd831ffb640 1 -- 172.21.15.110:0/2654986720 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd8340594c0 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.272+0000 7fd831ffb640 10 client.5461.objecter ms_dispatch 0x7fd83c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.272+0000 7fd831ffb640 3 client.5461.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.272+0000 7fd831ffb640 3 client.5461.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.272+0000 7fd831ffb640 20 client.5461.objecter dump_active .. 0 homeless 2024-01-20T11:20:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.282+0000 7fd831ffb640 1 -- 172.21.15.110:0/2654986720 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd834055800 con 0x7fd83c0ab6d0 2024-01-20T11:20:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.282+0000 7fd831ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.282+0000 7fd831ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.457+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd83c0abf00 con 0x7fd82006c860 2024-01-20T11:20:48.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.460+0000 7fd831ffb640 1 -- 172.21.15.110:0/2654986720 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63897 (secure 0 0 0) 0x7fd82c001580 con 0x7fd82006c860 2024-01-20T11:20:48.460 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 10 librados: watch_flush enter 2024-01-20T11:20:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 10 librados: watch_flush exit 2024-01-20T11:20:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 20 client.5461.objecter shutdown clearing up homeless session... 2024-01-20T11:20:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 10 client.5461.objecter successfully canceled tick 2024-01-20T11:20:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd82006c860 msgr2=0x7fd82006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd82006c860 0x7fd82006ed20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd83c13f440 tx=0x7fd82c009040 comp rx=0 tx=0).stop 2024-01-20T11:20:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 10 monclient: shutdown 2024-01-20T11:20:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd83c0ab6d0 msgr2=0x7fd83c13cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd83c0ab6d0 0x7fd83c13cd70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd83400ebc0 tx=0x7fd83400c490 comp rx=0 tx=0).stop 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 shutdown_connections 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd82006c860 0x7fd82006ed20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd83c0af3a0 0x7fd83c1401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd83c0adb30 0x7fd83c13fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 --2- 172.21.15.110:0/2654986720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd83c0ab6d0 0x7fd83c13cd70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 >> 172.21.15.110:0/2654986720 conn(0x7fd83c0b3e60 msgr2=0x7fd83c139650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 shutdown_connections 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 -- 172.21.15.110:0/2654986720 wait complete. 2024-01-20T11:20:48.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.462+0000 7fd8434d4640 1 librados: shutdown 2024-01-20T11:20:48.472 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-01-20T11:20:46.774937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002534"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":8.8239999999999998},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.9420000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.1059999999999999},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":10.176}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.096},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.266999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.901},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":14.773999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":9.6020000000000003},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":12.858000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":7.6980000000000004},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.481999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313287,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":8.4239999999999995},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7759999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.8220000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.0310000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.218999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.5609999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.6999999999999993},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.816000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.378},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":14.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.868},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":10.917999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313287,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.1459999999999999},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.181}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.318},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.7319999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":13.013},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.92}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.961},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.782}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":8.4280000000000008},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":11.592000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":16.382000000000001},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.292999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444101,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.6319999999999997},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":13.903}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":17.024000000000001},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.2260000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.919},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":7.6429999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.8879999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.932}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":18.132000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":11.981999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.131},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":15.871}]}]},{"osd":1,"up_from":17,"seq":73014444102,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.242},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":9.1359999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":9.7439999999999998},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":6.2370000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":13.121},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.3680000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":10.321},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":8.3680000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.2140000000000004},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":12.414999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":10.566000000000001},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":13.654}]}]},{"osd":0,"up_from":17,"seq":73014444102,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":6.0810000000000004},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":8.9960000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":11.074999999999999},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":4.819}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":7.3179999999999996},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.18}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":12.332000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":12.919}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":8.3079999999999998},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":11.734999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":13.504},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":14.991}]}]},{"osd":6,"up_from":15,"seq":64424509511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":8.3000000000000007},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.1870000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.9039999999999999},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":14.162000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.4489999999999998},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3160000000000007}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.970000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.244}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.891},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":9.9309999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.962999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.85}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:48.474 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:48.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:48.474 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:48.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:48.474 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:48.475 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:48.475 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:48.475 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:48.475 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:48.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.573+0000 7f421f22b640 10 monclient: get_monmap_and_config 2024-01-20T11:20:48.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.573+0000 7f421f22b640 10 monclient: build_initial_monmap 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.573+0000 7f421f22b640 10 monclient: monmap: 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:48.574341+0000 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:48.574341+0000 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:48.574 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.574+0000 7f421f22b640 1 Processor -- start 2024-01-20T11:20:48.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.574+0000 7f421f22b640 1 -- start start 2024-01-20T11:20:48.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.574+0000 7f421f22b640 10 monclient: init 2024-01-20T11:20:48.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4218066620 0x7f421813bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 10 monclient(hunting): picked mon.noname-b con 0x7f4218066620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f421806df70 0x7f4218145440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 10 monclient(hunting): picked mon.noname-a con 0x7f421806df70 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4218145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42180adf80 con 0x7f4218145980 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421806e690 con 0x7f421806df70 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4218066a00 con 0x7f4218066620 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421f22b640 10 monclient(hunting): authenticate will time out at 1691.925276s 2024-01-20T11:20:48.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f421cfc7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4218066620 0x7f421813bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.575+0000 7f4217fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f421806df70 0x7f4218145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f421cfc7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4218066620 0x7f421813bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52048/0 (socket says 172.21.15.110:52048) 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f421cfc7640 1 -- 172.21.15.110:0/3900257136 learned_addr learned my addr 172.21.15.110:0/3900257136 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f421cfc7640 10 monclient(hunting): get_auth_request con 0x7f4218066620 auth_method 0 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f421cfc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f421cfc7640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f421cfc7640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f4217fff640 10 monclient(hunting): get_auth_request con 0x7f421806df70 auth_method 0 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f4217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f4217fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f4217fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f421cfc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f421cfc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.576+0000 7f421cfc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.577+0000 7f4217fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.577+0000 7f4217fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.577+0000 7f4217fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.578+0000 7f421cfc7640 10 monclient(hunting): handle_auth_done global_id 5144 payload 274 2024-01-20T11:20:48.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.578+0000 7f421cfc7640 1 -- 172.21.15.110:0/3900257136 >> v2:172.21.15.110:3300/0 conn(0x7f421806df70 msgr2=0x7f4218145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:48.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.578+0000 7f421cfc7640 1 --2- 172.21.15.110:0/3900257136 >> v2:172.21.15.110:3300/0 conn(0x7f421806df70 0x7f4218145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.578+0000 7f421cfc7640 1 -- 172.21.15.110:0/3900257136 >> v1:172.21.15.110:6789/0 conn(0x7f4218145980 legacy=0x7f42180af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.578+0000 7f421cfc7640 10 monclient: _finish_hunting 0 2024-01-20T11:20:48.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.578+0000 7f421cfc7640 1 monclient: found mon.noname-b 2024-01-20T11:20:48.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.578+0000 7f421cfc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:48.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.578+0000 7f421cfc7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:48.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.578+0000 7f421cfc7640 1 -- 172.21.15.110:0/3900257136 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421806f650 con 0x7f4218066620 2024-01-20T11:20:48.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.580+0000 7f421cfc7640 1 --2- 172.21.15.110:0/3900257136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4218066620 0x7f421813bcc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f420c009870 tx=0x7f420c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fba7f5b0314a2acb server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:48.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 1 -- 172.21.15.110:0/3900257136 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c031640 con 0x7f4218066620 2024-01-20T11:20:48.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 10 monclient: dump: 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:48.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 1 -- 172.21.15.110:0/3900257136 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420c0317a0 con 0x7f4218066620 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 5 monclient: authenticate success, global_id 5144 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 4 set_mon_vals no callback set 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 1 -- 172.21.15.110:0/3900257136 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c031b30 con 0x7f4218066620 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f42177fe640 10 monclient: dump: 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.586 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 10 monclient: shutdown 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 1 -- 172.21.15.110:0/3900257136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4218066620 msgr2=0x7f421813bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 1 --2- 172.21.15.110:0/3900257136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4218066620 0x7f421813bcc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f420c009870 tx=0x7f420c02b8f0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 1 -- 172.21.15.110:0/3900257136 shutdown_connections 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.584+0000 7f421f22b640 1 --2- 172.21.15.110:0/3900257136 >> v2:172.21.15.110:3300/0 conn(0x7f421806df70 0x7f4218145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.585+0000 7f421f22b640 1 --2- 172.21.15.110:0/3900257136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4218066620 0x7f421813bcc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.585+0000 7f421f22b640 1 -- 172.21.15.110:0/3900257136 >> 172.21.15.110:0/3900257136 conn(0x7f42180b09e0 msgr2=0x7f42180b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:48.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.585+0000 7f421f22b640 1 -- 172.21.15.110:0/3900257136 shutdown_connections 2024-01-20T11:20:48.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.585+0000 7f421f22b640 1 -- 172.21.15.110:0/3900257136 wait complete. 2024-01-20T11:20:48.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.585+0000 7f421f22b640 10 monclient: build_initial_monmap 2024-01-20T11:20:48.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.585+0000 7f421f22b640 10 monclient: monmap: 2024-01-20T11:20:48.590 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:48.590 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:48.590 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:48.590 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:48.590 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.585+0000 7f421f22b640 1 librados: starting msgr at 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.585+0000 7f421f22b640 1 librados: starting objecter 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 Processor -- start 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 -- start start 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 librados: setting wanted keys 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 librados: calling monclient init 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient: init 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:48.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4218066620 0x7f42181daa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient(hunting): picked mon.noname-b con 0x7f4218066620 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f421806df70 0x7f42181daf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient(hunting): picked mon.noname-c con 0x7f421806df70 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4218145980 0x7f421813d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient(hunting): picked mon.noname-a con 0x7f4218145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421813f530 con 0x7f4218145980 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42181db480 con 0x7f4218066620 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4218135f60 con 0x7f421806df70 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:48.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.586+0000 7f421f22b640 10 monclient(hunting): authenticate will time out at 1691.937148s 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f4217fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f421806df70 0x7f42181daf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421d7c8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4218145980 0x7f421813d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421cfc7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4218066620 0x7f42181daa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f4217fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f421806df70 0x7f42181daf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52058/0 (socket says 172.21.15.110:52058) 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f4217fff640 1 -- 172.21.15.110:0/766921388 learned_addr learned my addr 172.21.15.110:0/766921388 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f4217fff640 10 monclient(hunting): get_auth_request con 0x7f421806df70 auth_method 0 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f4217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f4217fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f4217fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421cfc7640 10 monclient(hunting): get_auth_request con 0x7f4218066620 auth_method 0 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421cfc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421cfc7640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421cfc7640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421d7c8640 10 monclient(hunting): get_auth_request con 0x7f4218145980 auth_method 0 2024-01-20T11:20:48.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421d7c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421d7c8640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.587+0000 7f421d7c8640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.588+0000 7f4217fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.588+0000 7f4217fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.588+0000 7f4217fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.588+0000 7f421cfc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.588+0000 7f421cfc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.588+0000 7f421cfc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.588+0000 7f421d7c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.588+0000 7f421d7c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:48.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.588+0000 7f421d7c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:48.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 10 monclient(hunting): handle_auth_done global_id 5147 payload 824 2024-01-20T11:20:48.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 1 -- 172.21.15.110:0/766921388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4218066620 msgr2=0x7f42181daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4218066620 0x7f42181daa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 1 -- 172.21.15.110:0/766921388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4218145980 msgr2=0x7f421813d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:48.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4218145980 0x7f421813d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:48.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:48.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 1 monclient: found mon.noname-c 2024-01-20T11:20:48.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:48.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:20:48.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.591+0000 7f4217fff640 1 -- 172.21.15.110:0/766921388 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c0094d0 con 0x7f421806df70 2024-01-20T11:20:48.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.593+0000 7f4217fff640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f421806df70 0x7f42181daf40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f420400e9b0 tx=0x7f420400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:48.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.593+0000 7f42157fa640 10 client.?.objecter ms_handle_connect 0x7f421806df70 2024-01-20T11:20:48.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.593+0000 7f42157fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:48.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 1 -- 172.21.15.110:0/766921388 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204009670 con 0x7f421806df70 2024-01-20T11:20:48.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:48.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 10 monclient: dump: 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:48.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 1 -- 172.21.15.110:0/766921388 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420400c9a0 con 0x7f421806df70 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 1 -- 172.21.15.110:0/766921388 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420400cd50 con 0x7f421806df70 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f42157fa640 10 monclient: dump: 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:48.599 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f421f22b640 5 monclient: authenticate success, global_id 5147 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f4216ffd640 4 set_mon_vals no callback set 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f421f22b640 10 monclient: _renew_subs 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f421f22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421813dd90 con 0x7f421806df70 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f421f22b640 10 client.5147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f421f22b640 10 monclient: _renew_subs 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f421f22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421813e250 con 0x7f421806df70 2024-01-20T11:20:48.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.597+0000 7f421f22b640 1 librados: init done 2024-01-20T11:20:48.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.598+0000 7f421f22b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:48.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.598+0000 7f421f22b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:48.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.598+0000 7f421f22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:48.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.598+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421813e530 con 0x7f421806df70 2024-01-20T11:20:48.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.616+0000 7f42157fa640 1 -- 172.21.15.110:0/766921388 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4204013470 con 0x7f421806df70 2024-01-20T11:20:48.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.616+0000 7f42157fa640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41fc06c8e0 0x7f41fc06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:48.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.616+0000 7f421cfc7640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41fc06c8e0 0x7f41fc06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:48.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.616+0000 7f421cfc7640 10 monclient: get_auth_request con 0x7f41fc06c8e0 auth_method 0 2024-01-20T11:20:48.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.617+0000 7f42157fa640 10 client.5147.objecter ms_handle_connect 0x7f41fc06c8e0 2024-01-20T11:20:48.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.617+0000 7f421cfc7640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41fc06c8e0 0x7f41fc06eda0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f420c009870 tx=0x7f420c03c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:48.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.620+0000 7f42157fa640 1 -- 172.21.15.110:0/766921388 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4204058c40 con 0x7f421806df70 2024-01-20T11:20:48.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.620+0000 7f42157fa640 10 client.5147.objecter ms_dispatch 0x7f4218000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:48.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.620+0000 7f42157fa640 3 client.5147.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:48.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.620+0000 7f42157fa640 3 client.5147.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:48.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.620+0000 7f42157fa640 20 client.5147.objecter dump_active .. 0 homeless 2024-01-20T11:20:48.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.854+0000 7f42157fa640 1 -- 172.21.15.110:0/766921388 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4204054f80 con 0x7f421806df70 2024-01-20T11:20:48.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.854+0000 7f42157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:48.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:48.854+0000 7f42157fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:49.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.125+0000 7f421f22b640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:49.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.125+0000 7f421f22b640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:49.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.125+0000 7f421f22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:49.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.125+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f4218072c10 con 0x7f421806df70 2024-01-20T11:20:49.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.131+0000 7f42157fa640 1 -- 172.21.15.110:0/766921388 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f4204021520 con 0x7f421806df70 2024-01-20T11:20:49.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.131+0000 7f42157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:49.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.131+0000 7f42157fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:49.131 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:49.131 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":383,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 10 librados: watch_flush enter 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 10 librados: watch_flush exit 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 20 client.5147.objecter shutdown clearing up homeless session... 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 10 client.5147.objecter successfully canceled tick 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41fc06c8e0 msgr2=0x7f41fc06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41fc06c8e0 0x7f41fc06eda0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f420c009870 tx=0x7f420c03c040 comp rx=0 tx=0).stop 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 10 monclient: shutdown 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f421806df70 msgr2=0x7f42181daf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f421806df70 0x7f42181daf40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f420400e9b0 tx=0x7f420400ee70 comp rx=0 tx=0).stop 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 shutdown_connections 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41fc06c8e0 0x7f41fc06eda0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4218145980 0x7f421813d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f421806df70 0x7f42181daf40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 --2- 172.21.15.110:0/766921388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4218066620 0x7f42181daa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 >> 172.21.15.110:0/766921388 conn(0x7f42180b09e0 msgr2=0x7f42180afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:49.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 shutdown_connections 2024-01-20T11:20:49.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 -- 172.21.15.110:0/766921388 wait complete. 2024-01-20T11:20:49.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.133+0000 7f421f22b640 1 librados: shutdown 2024-01-20T11:20:49.144 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:49.144 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.247+0000 7f797247c640 10 monclient: get_monmap_and_config 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.247+0000 7f797247c640 10 monclient: build_initial_monmap 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.247+0000 7f797247c640 10 monclient: monmap: 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:49.247728+0000 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:49.247728+0000 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:49.247 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:49.248 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:49.248 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:49.248 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:49.248 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:49.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 1 Processor -- start 2024-01-20T11:20:49.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 1 -- start start 2024-01-20T11:20:49.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 10 monclient: init 2024-01-20T11:20:49.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:49.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-01-20T11:20:49.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f796c0a96f0 0x7f796c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:49.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 10 monclient(hunting): picked mon.noname-a con 0x7f796c0a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:49.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f796c13fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:49.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f796c0a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:49.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796c0b25c0 con 0x7f796c13fa10 2024-01-20T11:20:49.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796c0ab0a0 con 0x7f796c0a9ad0 2024-01-20T11:20:49.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:49.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c0aaea0 con 0x7f796c0a96f0 2024-01-20T11:20:49.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:49.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.248+0000 7f797247c640 10 monclient(hunting): authenticate will time out at 1692.598868s 2024-01-20T11:20:49.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.249+0000 7f796bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f796c0a96f0 0x7f796c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:49.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.249+0000 7f7970a19640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f796c0a9ad0 0x7f796c14c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:33818/0 (socket says 172.21.15.110:33818) 2024-01-20T11:20:49.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.249+0000 7f7970a19640 1 -- 172.21.15.110:0/176351581 learned_addr learned my addr 172.21.15.110:0/176351581 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:49.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.249+0000 7f796bfff640 10 monclient(hunting): get_auth_request con 0x7f796c0a96f0 auth_method 0 2024-01-20T11:20:49.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.249+0000 7f796bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:49.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.249+0000 7f796bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:49.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.249+0000 7f796bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:49.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.250+0000 7f796bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:49.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.250+0000 7f796bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:49.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.250+0000 7f796bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.251+0000 7f796b7fe640 1 -- 172.21.15.110:0/176351581 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032220143 0 0) 0x7f796c0ab0a0 con 0x7f796c0a9ad0 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.251+0000 7f796b7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.251+0000 7f796b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.251+0000 7f796b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.251+0000 7f796b7fe640 10 monclient(hunting): my global_id is 5467 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.251+0000 7f796b7fe640 1 -- 172.21.15.110:0/176351581 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7958003520 con 0x7f796c0a9ad0 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.252+0000 7f796bfff640 10 monclient(hunting): handle_auth_done global_id 5406 payload 274 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.252+0000 7f796bfff640 1 -- 172.21.15.110:0/176351581 >> v1:172.21.15.134:6789/0 conn(0x7f796c0a9ad0 legacy=0x7f796c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.252+0000 7f796bfff640 1 -- 172.21.15.110:0/176351581 >> v1:172.21.15.110:6789/0 conn(0x7f796c13fa10 legacy=0x7f796c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:49.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.252+0000 7f796bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:49.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.252+0000 7f796bfff640 1 monclient: found mon.noname-a 2024-01-20T11:20:49.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.252+0000 7f796bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:49.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.252+0000 7f796bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:49.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.252+0000 7f796bfff640 1 -- 172.21.15.110:0/176351581 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c0b3fb0 con 0x7f796c0a96f0 2024-01-20T11:20:49.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.254+0000 7f796bfff640 1 --2- 172.21.15.110:0/176351581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c0a96f0 0x7f796c13f4d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f795c0097a0 tx=0x7f795c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c2cddaafa9bb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:49.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 1 -- 172.21.15.110:0/176351581 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c031830 con 0x7f796c0a96f0 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 10 monclient: dump: 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 1 -- 172.21.15.110:0/176351581 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c031990 con 0x7f796c0a96f0 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 5 monclient: authenticate success, global_id 5406 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 1 -- 172.21.15.110:0/176351581 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c031d20 con 0x7f796c0a96f0 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 4 set_mon_vals no callback set 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f796b7fe640 10 monclient: dump: 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:49.262 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:49.263 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:49.263 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:49.263 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:49.263 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:49.263 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:49.263 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:49.263 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:49.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 10 monclient: shutdown 2024-01-20T11:20:49.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:49.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 1 -- 172.21.15.110:0/176351581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c0a96f0 msgr2=0x7f796c13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 1 --2- 172.21.15.110:0/176351581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c0a96f0 0x7f796c13f4d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f795c0097a0 tx=0x7f795c02b8d0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 1 -- 172.21.15.110:0/176351581 shutdown_connections 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 1 --2- 172.21.15.110:0/176351581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c0a96f0 0x7f796c13f4d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 1 -- 172.21.15.110:0/176351581 >> 172.21.15.110:0/176351581 conn(0x7f796c0b7bc0 msgr2=0x7f796c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 1 -- 172.21.15.110:0/176351581 shutdown_connections 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.261+0000 7f797247c640 1 -- 172.21.15.110:0/176351581 wait complete. 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.262+0000 7f797247c640 10 monclient: build_initial_monmap 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.262+0000 7f797247c640 10 monclient: monmap: 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:49.271 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.262+0000 7f797247c640 1 librados: starting msgr at 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.262+0000 7f797247c640 1 librados: starting objecter 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 Processor -- start 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 -- start start 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 librados: setting wanted keys 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 librados: calling monclient init 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient: init 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:49.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f796c0a96f0 0x7f796c1cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient(hunting): picked mon.noname-b con 0x7f796c0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f796c0a9ad0 0x7f796c13eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient(hunting): picked mon.noname-c con 0x7f796c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c1402e0 0x7f796c1da8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient(hunting): picked mon.noname-a con 0x7f796c1402e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c13fe00 con 0x7f796c1402e0 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c13e320 con 0x7f796c0a96f0 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c141b20 con 0x7f796c0a9ad0 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.263+0000 7f797247c640 10 monclient(hunting): authenticate will time out at 1692.613955s 2024-01-20T11:20:49.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7970a19640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c1402e0 0x7f796c1da8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7963fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f796c0a9ad0 0x7f796c13eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7970a19640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c1402e0 0x7f796c1da8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58452/0 (socket says 172.21.15.110:58452) 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7970a19640 1 -- 172.21.15.110:0/3537928989 learned_addr learned my addr 172.21.15.110:0/3537928989 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7970a19640 10 monclient(hunting): get_auth_request con 0x7f796c1402e0 auth_method 0 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7970a19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7970a19640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7970a19640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f796bfff640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f796c0a96f0 0x7f796c1cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7963fff640 10 monclient(hunting): get_auth_request con 0x7f796c0a9ad0 auth_method 0 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7963fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7963fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f7963fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f796bfff640 10 monclient(hunting): get_auth_request con 0x7f796c0a96f0 auth_method 0 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f796bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:49.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f796bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.264+0000 7f796bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.265+0000 7f7970a19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.265+0000 7f7970a19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.265+0000 7f7970a19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.265+0000 7f7963fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.265+0000 7f7963fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.265+0000 7f7963fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.265+0000 7f796bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.265+0000 7f796bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.265+0000 7f796bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 10 monclient(hunting): handle_auth_done global_id 5412 payload 824 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 1 -- 172.21.15.110:0/3537928989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f796c0a9ad0 msgr2=0x7f796c13eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f796c0a9ad0 0x7f796c13eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 1 -- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f796c0a96f0 msgr2=0x7f796c1cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f796c0a96f0 0x7f796c1cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 10 monclient: _finish_hunting 0 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 1 monclient: found mon.noname-a 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f7970a19640 1 -- 172.21.15.110:0/3537928989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c009400 con 0x7f796c1402e0 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.268+0000 7f796bfff640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f796c0a96f0 0x7f796c1cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.270+0000 7f7970a19640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c1402e0 0x7f796c1da8a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f796400cbb0 tx=0x7f796400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.270+0000 7f79697fa640 10 client.?.objecter ms_handle_connect 0x7f796c1402e0 2024-01-20T11:20:49.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.270+0000 7f79697fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 1 -- 172.21.15.110:0/3537928989 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400ad40 con 0x7f796c1402e0 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 10 monclient: dump: 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:49.278 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 10 monclient: _finish_auth 0 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 10 monclient: _check_auth_tickets 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 1 -- 172.21.15.110:0/3537928989 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7964012400 con 0x7f796c1402e0 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f797247c640 5 monclient: authenticate success, global_id 5412 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f797247c640 10 monclient: _renew_subs 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f797247c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c141e00 con 0x7f796c1402e0 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 1 -- 172.21.15.110:0/3537928989 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400aea0 con 0x7f796c1402e0 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:49.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f79697fa640 10 monclient: dump: 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f796affd640 4 set_mon_vals no callback set 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f797247c640 10 client.5412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f797247c640 10 monclient: _renew_subs 2024-01-20T11:20:49.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f797247c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:49.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c1dafc0 con 0x7f796c1402e0 2024-01-20T11:20:49.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.273+0000 7f797247c640 1 librados: init done 2024-01-20T11:20:49.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.274+0000 7f797247c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:49.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.274+0000 7f797247c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:49.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.274+0000 7f797247c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:49.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.274+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c13ffc0 con 0x7f796c1402e0 2024-01-20T11:20:49.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.292+0000 7f79697fa640 1 -- 172.21.15.110:0/3537928989 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7964007070 con 0x7f796c1402e0 2024-01-20T11:20:49.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.292+0000 7f79697fa640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f794c06c860 0x7f794c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:49.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.293+0000 7f796bfff640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f794c06c860 0x7f794c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:49.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.293+0000 7f796bfff640 10 monclient: get_auth_request con 0x7f794c06c860 auth_method 0 2024-01-20T11:20:49.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.293+0000 7f79697fa640 10 client.5412.objecter ms_handle_connect 0x7f794c06c860 2024-01-20T11:20:49.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.293+0000 7f796bfff640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f794c06c860 0x7f794c06ed20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f795c036000 tx=0x7f795c03c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.295+0000 7f79697fa640 1 -- 172.21.15.110:0/3537928989 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f79640585d0 con 0x7f796c1402e0 2024-01-20T11:20:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.295+0000 7f79697fa640 10 client.5412.objecter ms_dispatch 0x7f796c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.295+0000 7f79697fa640 3 client.5412.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.295+0000 7f79697fa640 3 client.5412.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.295+0000 7f79697fa640 20 client.5412.objecter dump_active .. 0 homeless 2024-01-20T11:20:49.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.524+0000 7f79697fa640 1 -- 172.21.15.110:0/3537928989 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7964054910 con 0x7f796c1402e0 2024-01-20T11:20:49.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.524+0000 7f79697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:49.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.524+0000 7f79697fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:49.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.697+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f796c075490 con 0x7f794c06c860 2024-01-20T11:20:49.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.700+0000 7f79697fa640 1 -- 172.21.15.110:0/3537928989 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63945 (secure 0 0 0) 0x7f796c075490 con 0x7f794c06c860 2024-01-20T11:20:49.700 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:49.701 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.701+0000 7f797247c640 10 librados: watch_flush enter 2024-01-20T11:20:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.701+0000 7f797247c640 10 librados: watch_flush exit 2024-01-20T11:20:49.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.701+0000 7f797247c640 20 client.5412.objecter shutdown clearing up homeless session... 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.701+0000 7f797247c640 10 client.5412.objecter successfully canceled tick 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f794c06c860 msgr2=0x7f794c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f794c06c860 0x7f794c06ed20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f795c036000 tx=0x7f795c03c040 comp rx=0 tx=0).stop 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 10 monclient: shutdown 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c1402e0 msgr2=0x7f796c1da8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c1402e0 0x7f796c1da8a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f796400cbb0 tx=0x7f796400a430 comp rx=0 tx=0).stop 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 shutdown_connections 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f794c06c860 0x7f794c06ed20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f796c1402e0 0x7f796c1da8a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f796c0a9ad0 0x7f796c13eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 --2- 172.21.15.110:0/3537928989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f796c0a96f0 0x7f796c1cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 >> 172.21.15.110:0/3537928989 conn(0x7f796c0b7bc0 msgr2=0x7f796c0b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:49.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 shutdown_connections 2024-01-20T11:20:49.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 -- 172.21.15.110:0/3537928989 wait complete. 2024-01-20T11:20:49.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:49.702+0000 7f797247c640 1 librados: shutdown 2024-01-20T11:20:49.712 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-01-20T11:20:48.775279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002514"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.5879999999999992},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.7140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.1},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.6890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.843999999999999},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.117000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.9950000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":12.819000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.449999999999999},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":10.548999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.9269999999999996},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.569000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313287,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":8.4239999999999995},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.7759999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":7.8220000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.0310000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.218999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":6.5609999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.6999999999999993},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.816000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.378},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":14.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":12.868},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":10.917999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313288,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.218},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":6.7140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":11.708},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.7249999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":8.2449999999999992},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.965999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.789},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":10.487}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":13.634},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":15.618}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":16.486999999999998},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.077999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444102,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.3140000000000001},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":6.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.0899999999999999},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.661}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.865},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.390000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.638999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.4350000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.945},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.291}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.382},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":16.507999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444103,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.9160000000000004},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":11.869}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.513999999999999},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.3119999999999994}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":13.69},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.5869999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":15.439},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.117000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.4120000000000008},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":14.478999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.9909999999999997},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.199999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444103,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":13.045999999999999},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":6.4189999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.3440000000000003},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.794}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":14.526999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.8260000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":7.0780000000000003},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.0830000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":15.285},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":10.398}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":10.98},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":8.8010000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":8.3000000000000007},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.1870000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.9039999999999999},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":14.162000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.4489999999999998},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.3160000000000007}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.970000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.244}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":12.891},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":9.9309999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.962999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":15.85}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:49.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~112s 2024-01-20T11:20:52.714 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:52.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.821+0000 7f1cdc5f0640 10 monclient: get_monmap_and_config 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.821+0000 7f1cdc5f0640 10 monclient: build_initial_monmap 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.821+0000 7f1cdc5f0640 10 monclient: monmap: 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:52.822200+0000 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:52.822200+0000 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:52.822 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:52.823 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:52.823 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:52.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 1 Processor -- start 2024-01-20T11:20:52.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 1 -- start start 2024-01-20T11:20:52.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient: init 2024-01-20T11:20:52.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:52.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-01-20T11:20:52.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1cd40b0ba0 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:52.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd40adc30 0x7f1cd40ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:52.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient(hunting): picked mon.noname-b con 0x7f1cd40adc30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:52.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1cd40ac4d0 0x7f1cd413bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:52.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient(hunting): picked mon.noname-a con 0x7f1cd40ac4d0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:52.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd40ad530 con 0x7f1cd40b0ba0 2024-01-20T11:20:52.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:52.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd40af840 con 0x7f1cd40ac4d0 2024-01-20T11:20:52.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:52.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40b0f80 con 0x7f1cd40adc30 2024-01-20T11:20:52.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:52.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.822+0000 7f1cdc5f0640 10 monclient(hunting): authenticate will time out at 1696.173144s 2024-01-20T11:20:52.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.823+0000 7f1cd9b8b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd40adc30 0x7f1cd40ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:52.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.823+0000 7f1cda38c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f1cd40b0ba0 0x7f1cd40af130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:36924/0 (socket says 172.21.15.110:36924) 2024-01-20T11:20:52.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.823+0000 7f1cda38c640 1 -- 172.21.15.110:0/3107610158 learned_addr learned my addr 172.21.15.110:0/3107610158 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:52.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.823+0000 7f1cd9b8b640 10 monclient(hunting): get_auth_request con 0x7f1cd40adc30 auth_method 0 2024-01-20T11:20:52.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.823+0000 7f1cd9b8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:52.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.823+0000 7f1cd9b8b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:52.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.823+0000 7f1cd9b8b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:52.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cdab8d640 1 --2- 172.21.15.110:0/3107610158 >> v2:172.21.15.110:3300/0 conn(0x7f1cd40ac4d0 0x7f1cd413bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:52.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cdab8d640 10 monclient(hunting): get_auth_request con 0x7f1cd40ac4d0 auth_method 0 2024-01-20T11:20:52.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cdab8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:52.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cdab8d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:52.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cdab8d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:52.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cd9b8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:52.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cd9b8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:52.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cd9b8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:52.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cdab8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:52.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cdab8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:52.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.824+0000 7f1cdab8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:52.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.826+0000 7f1cd9b8b640 10 monclient(hunting): handle_auth_done global_id 5153 payload 274 2024-01-20T11:20:52.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.826+0000 7f1cd9b8b640 1 -- 172.21.15.110:0/3107610158 >> v2:172.21.15.110:3300/0 conn(0x7f1cd40ac4d0 msgr2=0x7f1cd413bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:52.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.826+0000 7f1cd9b8b640 1 --2- 172.21.15.110:0/3107610158 >> v2:172.21.15.110:3300/0 conn(0x7f1cd40ac4d0 0x7f1cd413bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:52.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.826+0000 7f1cd9b8b640 1 -- 172.21.15.110:0/3107610158 >> v1:172.21.15.134:6789/0 conn(0x7f1cd40b0ba0 legacy=0x7f1cd40af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:52.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.826+0000 7f1cd9b8b640 10 monclient: _finish_hunting 0 2024-01-20T11:20:52.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.826+0000 7f1cd9b8b640 1 monclient: found mon.noname-b 2024-01-20T11:20:52.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.826+0000 7f1cd9b8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:52.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.826+0000 7f1cd9b8b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:52.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.826+0000 7f1cd9b8b640 1 -- 172.21.15.110:0/3107610158 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40ad810 con 0x7f1cd40adc30 2024-01-20T11:20:52.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.828+0000 7f1cd9b8b640 1 --2- 172.21.15.110:0/3107610158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd40adc30 0x7f1cd40ab630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40097a0 tx=0x7f1cc402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4c61b63214a5a3c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:52.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 1 -- 172.21.15.110:0/3107610158 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc402e070 con 0x7f1cd40adc30 2024-01-20T11:20:52.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 10 monclient: dump: 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 10 monclient: _finish_auth 0 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 10 monclient: _check_auth_tickets 2024-01-20T11:20:52.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 1 -- 172.21.15.110:0/3107610158 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc40315a0 con 0x7f1cd40adc30 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cdc5f0640 5 monclient: authenticate success, global_id 5153 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cdc5f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cdc5f0640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cdc5f0640 4 set_mon_vals no callback set 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 1 -- 172.21.15.110:0/3107610158 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4031930 con 0x7f1cd40adc30 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.831+0000 7f1cd938a640 10 monclient: dump: 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:52.840 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 10 monclient: shutdown 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/3107610158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd40adc30 msgr2=0x7f1cd40ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 1 --2- 172.21.15.110:0/3107610158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd40adc30 0x7f1cd40ab630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40097a0 tx=0x7f1cc402b840 comp rx=0 tx=0).stop 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/3107610158 shutdown_connections 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 1 --2- 172.21.15.110:0/3107610158 >> v2:172.21.15.110:3300/0 conn(0x7f1cd40ac4d0 0x7f1cd413bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:52.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 1 --2- 172.21.15.110:0/3107610158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd40adc30 0x7f1cd40ab630 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/3107610158 >> 172.21.15.110:0/3107610158 conn(0x7f1cd40b3e60 msgr2=0x7f1cd40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/3107610158 shutdown_connections 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.832+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/3107610158 wait complete. 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.833+0000 7f1cdc5f0640 10 monclient: build_initial_monmap 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.833+0000 7f1cdc5f0640 10 monclient: monmap: 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:52.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.833+0000 7f1cdc5f0640 1 librados: starting msgr at 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.833+0000 7f1cdc5f0640 1 librados: starting objecter 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.833+0000 7f1cdc5f0640 1 Processor -- start 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.833+0000 7f1cdc5f0640 1 -- start start 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.833+0000 7f1cdc5f0640 1 librados: setting wanted keys 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.833+0000 7f1cdc5f0640 1 librados: calling monclient init 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.833+0000 7f1cdc5f0640 10 monclient: init 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cd40ac4d0 0x7f1cd40b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient(hunting): picked mon.noname-a con 0x7f1cd40ac4d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cd40b0ba0 0x7f1cd406f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient(hunting): picked mon.noname-b con 0x7f1cd40b0ba0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd406ee60 0x7f1cd406f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient(hunting): picked mon.noname-c con 0x7f1cd406ee60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:52.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd406fe30 con 0x7f1cd40ac4d0 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4070f00 con 0x7f1cd40b0ba0 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4072210 con 0x7f1cd406ee60 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdc5f0640 10 monclient(hunting): authenticate will time out at 1696.184298s 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdab8d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd406ee60 0x7f1cd406f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cda38c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cd40ac4d0 0x7f1cd40b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cd9b8b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cd40b0ba0 0x7f1cd406f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdab8d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd406ee60 0x7f1cd406f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:39578/0 (socket says 172.21.15.110:39578) 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdab8d640 1 -- 172.21.15.110:0/913393838 learned_addr learned my addr 172.21.15.110:0/913393838 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:52.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdab8d640 10 monclient(hunting): get_auth_request con 0x7f1cd406ee60 auth_method 0 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdab8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdab8d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cdab8d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cda38c640 10 monclient(hunting): get_auth_request con 0x7f1cd40ac4d0 auth_method 0 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cda38c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cda38c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.834+0000 7f1cda38c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cd9b8b640 10 monclient(hunting): get_auth_request con 0x7f1cd40b0ba0 auth_method 0 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cd9b8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cd9b8b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cd9b8b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cdab8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cdab8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cdab8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:52.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cda38c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cda38c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cda38c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cd9b8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cd9b8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.835+0000 7f1cd9b8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 10 monclient(hunting): handle_auth_done global_id 5476 payload 824 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 1 -- 172.21.15.110:0/913393838 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd406ee60 msgr2=0x7f1cd406f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd406ee60 0x7f1cd406f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 1 -- 172.21.15.110:0/913393838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cd40ac4d0 msgr2=0x7f1cd40b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cd40ac4d0 0x7f1cd40b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 10 monclient: _finish_hunting 0 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 1 monclient: found mon.noname-b 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:52.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.838+0000 7f1cd9b8b640 1 -- 172.21.15.110:0/913393838 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc4009400 con 0x7f1cd40b0ba0 2024-01-20T11:20:52.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.841+0000 7f1cd9b8b640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cd40b0ba0 0x7f1cd406f3b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4033000 tx=0x7f1cc40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:52.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.841+0000 7f1ccaffd640 10 client.?.objecter ms_handle_connect 0x7f1cd40b0ba0 2024-01-20T11:20:52.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.841+0000 7f1ccaffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:52.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 1 -- 172.21.15.110:0/913393838 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4032840 con 0x7f1cd40b0ba0 2024-01-20T11:20:52.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:52.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 10 monclient: dump: 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:52.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 1 -- 172.21.15.110:0/913393838 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc40329c0 con 0x7f1cd40b0ba0 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 1 -- 172.21.15.110:0/913393838 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4032d70 con 0x7f1cd40b0ba0 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1ccaffd640 10 monclient: dump: 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:52.850 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:52.851 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:52.851 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:52.851 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:52.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cd8b89640 4 set_mon_vals no callback set 2024-01-20T11:20:52.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cdc5f0640 5 monclient: authenticate success, global_id 5476 2024-01-20T11:20:52.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cdc5f0640 10 monclient: _renew_subs 2024-01-20T11:20:52.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cdc5f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:52.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd40711b0 con 0x7f1cd40b0ba0 2024-01-20T11:20:52.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cdc5f0640 10 client.5476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:52.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cdc5f0640 10 monclient: _renew_subs 2024-01-20T11:20:52.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cdc5f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:52.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd4071aa0 con 0x7f1cd40b0ba0 2024-01-20T11:20:52.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.845+0000 7f1cdc5f0640 1 librados: init done 2024-01-20T11:20:52.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.846+0000 7f1cdc5f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:52.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.846+0000 7f1cdc5f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:52.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.846+0000 7f1cdc5f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:52.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:52.846+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd4075490 con 0x7f1cd40b0ba0 2024-01-20T11:20:53.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.087+0000 7f1ccaffd640 1 -- 172.21.15.110:0/913393838 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1cc404e430 con 0x7f1cd40b0ba0 2024-01-20T11:20:53.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.087+0000 7f1ccaffd640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1cb806c8e0 0x7f1cb806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:53.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.087+0000 7f1cda38c640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1cb806c8e0 0x7f1cb806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:53.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.087+0000 7f1cda38c640 10 monclient: get_auth_request con 0x7f1cb806c8e0 auth_method 0 2024-01-20T11:20:53.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.088+0000 7f1cda38c640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1cb806c8e0 0x7f1cb806eda0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0007fd0 tx=0x7f1cd0007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:53.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.088+0000 7f1ccaffd640 10 client.5476.objecter ms_handle_connect 0x7f1cb806c8e0 2024-01-20T11:20:53.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.090+0000 7f1ccaffd640 1 -- 172.21.15.110:0/913393838 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1cc407a420 con 0x7f1cd40b0ba0 2024-01-20T11:20:53.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.090+0000 7f1ccaffd640 10 client.5476.objecter ms_dispatch 0x7f1cd4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:53.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.090+0000 7f1ccaffd640 3 client.5476.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:53.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.090+0000 7f1ccaffd640 3 client.5476.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:53.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.090+0000 7f1ccaffd640 20 client.5476.objecter dump_active .. 0 homeless 2024-01-20T11:20:53.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.099+0000 7f1ccaffd640 1 -- 172.21.15.110:0/913393838 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1cc4076760 con 0x7f1cd40b0ba0 2024-01-20T11:20:53.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.099+0000 7f1ccaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:53.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.099+0000 7f1ccaffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:53.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.273+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1cd4069970 con 0x7f1cb806c8e0 2024-01-20T11:20:53.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.276+0000 7f1ccaffd640 1 -- 172.21.15.110:0/913393838 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63893 (secure 0 0 0) 0x7f1cd4069970 con 0x7f1cb806c8e0 2024-01-20T11:20:53.276 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:53.277 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:53.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 10 librados: watch_flush enter 2024-01-20T11:20:53.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 10 librados: watch_flush exit 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 20 client.5476.objecter shutdown clearing up homeless session... 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 10 client.5476.objecter successfully canceled tick 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1cb806c8e0 msgr2=0x7f1cb806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1cb806c8e0 0x7f1cb806eda0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0007fd0 tx=0x7f1cd0007e80 comp rx=0 tx=0).stop 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 10 monclient: shutdown 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cd40b0ba0 msgr2=0x7f1cd406f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cd40b0ba0 0x7f1cd406f3b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4033000 tx=0x7f1cc40046c0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 shutdown_connections 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1cb806c8e0 0x7f1cb806eda0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cd406ee60 0x7f1cd406f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cd40b0ba0 0x7f1cd406f3b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 --2- 172.21.15.110:0/913393838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cd40ac4d0 0x7f1cd40b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 >> 172.21.15.110:0/913393838 conn(0x7f1cd40b3e60 msgr2=0x7f1cd413a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:53.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 shutdown_connections 2024-01-20T11:20:53.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 -- 172.21.15.110:0/913393838 wait complete. 2024-01-20T11:20:53.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.279+0000 7f1cdc5f0640 1 librados: shutdown 2024-01-20T11:20:53.292 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-01-20T11:20:52.776159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002594"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.5879999999999992},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.7140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.1},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.6890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.843999999999999},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.117000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.9950000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":12.819000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.449999999999999},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":10.548999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.9269999999999996},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.569000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313288,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.2539999999999996},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.521000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":12.863},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":6.4409999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.73},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":5.6260000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.992},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.641999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.224},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":8.0459999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.462999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.8320000000000007}]}]},{"osd":3,"up_from":21,"seq":90194313289,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.4649999999999999},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.024}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.7929999999999993},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.407}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":7.7969999999999997},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.096}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.683},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.563000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":11.904},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.407}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":15.227},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.351000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444102,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.3140000000000001},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":6.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.0899999999999999},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.661}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.865},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.390000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.638999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.4350000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.945},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.291}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.382},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":16.507999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444103,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.9160000000000004},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":11.869}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.513999999999999},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.3119999999999994}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":13.69},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.5869999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":15.439},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.117000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.4120000000000008},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":14.478999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.9909999999999997},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.199999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444103,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":13.045999999999999},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":6.4189999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.3440000000000003},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.794}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":14.526999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.8260000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":7.0780000000000003},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.0830000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":15.285},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":10.398}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":10.98},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":8.8010000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509512,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.407},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":12.438000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.0069999999999997},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":8.7449999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":17.491},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.4100000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.002000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":15.882999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.722000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.274}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.782},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.131}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:53.294 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:53.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.395+0000 7f635e761640 10 monclient: get_monmap_and_config 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.395+0000 7f635e761640 10 monclient: build_initial_monmap 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.395+0000 7f635e761640 10 monclient: monmap: 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:53.395648+0000 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:53.395648+0000 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:53.396 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:53.397 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:53.397 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:53.397 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:53.397 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:53.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.395+0000 7f635e761640 1 Processor -- start 2024-01-20T11:20:53.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.395+0000 7f635e761640 1 -- start start 2024-01-20T11:20:53.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.395+0000 7f635e761640 10 monclient: init 2024-01-20T11:20:53.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:53.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-01-20T11:20:53.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b4d10 0x7f63580b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:53.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 10 monclient(hunting): picked mon.noname-b con 0x7f63580b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f63580b17c0 0x7f63580b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 10 monclient(hunting): picked mon.noname-a con 0x7f63580b17c0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6358145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63580b2120 con 0x7f6358145980 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63580b50f0 con 0x7f63580b17c0 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63580aade0 con 0x7f63580b4d10 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f635e761640 10 monclient(hunting): authenticate will time out at 1696.746576s 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.396+0000 7f6357fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b4d10 0x7f63580b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:53.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.397+0000 7f635ccfe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6358145980 0x7f6358145e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:36938/0 (socket says 172.21.15.110:36938) 2024-01-20T11:20:53.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.397+0000 7f635ccfe640 1 -- 172.21.15.110:0/3194313095 learned_addr learned my addr 172.21.15.110:0/3194313095 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:53.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.397+0000 7f6357fff640 10 monclient(hunting): get_auth_request con 0x7f63580b4d10 auth_method 0 2024-01-20T11:20:53.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.397+0000 7f6357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:53.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.397+0000 7f6357fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:53.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.397+0000 7f6357fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:53.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.398+0000 7f6357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:53.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.398+0000 7f6357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:53.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.398+0000 7f6357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.399+0000 7f6356ffd640 1 -- 172.21.15.110:0/3194313095 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715849766 0 0) 0x7f63580b2120 con 0x7f6358145980 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.399+0000 7f6356ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.399+0000 7f6356ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.399+0000 7f6356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.399+0000 7f6356ffd640 10 monclient(hunting): my global_id is 5479 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.399+0000 7f6356ffd640 1 -- 172.21.15.110:0/3194313095 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6340003520 con 0x7f6358145980 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.400+0000 7f6357fff640 10 monclient(hunting): handle_auth_done global_id 5159 payload 274 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.400+0000 7f6357fff640 1 -- 172.21.15.110:0/3194313095 >> v2:172.21.15.110:3300/0 conn(0x7f63580b17c0 msgr2=0x7f63580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.400+0000 7f6357fff640 1 --2- 172.21.15.110:0/3194313095 >> v2:172.21.15.110:3300/0 conn(0x7f63580b17c0 0x7f63580b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.400+0000 7f6357fff640 1 -- 172.21.15.110:0/3194313095 >> v1:172.21.15.134:6789/0 conn(0x7f6358145980 legacy=0x7f6358145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.400+0000 7f6357fff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:53.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.400+0000 7f6357fff640 1 monclient: found mon.noname-b 2024-01-20T11:20:53.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.400+0000 7f6357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:53.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.400+0000 7f6357fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:53.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.400+0000 7f6357fff640 1 -- 172.21.15.110:0/3194313095 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6358145460 con 0x7f63580b4d10 2024-01-20T11:20:53.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.402+0000 7f6357fff640 1 --2- 172.21.15.110:0/3194313095 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b4d10 0x7f63580b3da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f63480097a0 tx=0x7f634802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=789b2e7745a45900 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:53.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 1 -- 172.21.15.110:0/3194313095 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634802e070 con 0x7f63580b4d10 2024-01-20T11:20:53.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:53.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 10 monclient: dump: 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:53.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 1 -- 172.21.15.110:0/3194313095 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63480315a0 con 0x7f63580b4d10 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f635e761640 5 monclient: authenticate success, global_id 5159 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f635e761640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 1 -- 172.21.15.110:0/3194313095 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348031930 con 0x7f63580b4d10 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f635e761640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f635e761640 4 set_mon_vals no callback set 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.405+0000 7f6356ffd640 10 monclient: dump: 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:53.408 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:53.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 10 monclient: shutdown 2024-01-20T11:20:53.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:53.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 1 -- 172.21.15.110:0/3194313095 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b4d10 msgr2=0x7f63580b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 1 --2- 172.21.15.110:0/3194313095 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b4d10 0x7f63580b3da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f63480097a0 tx=0x7f634802b840 comp rx=0 tx=0).stop 2024-01-20T11:20:53.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 1 -- 172.21.15.110:0/3194313095 shutdown_connections 2024-01-20T11:20:53.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 1 --2- 172.21.15.110:0/3194313095 >> v2:172.21.15.110:3300/0 conn(0x7f63580b17c0 0x7f63580b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 1 --2- 172.21.15.110:0/3194313095 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b4d10 0x7f63580b3da0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 1 -- 172.21.15.110:0/3194313095 >> 172.21.15.110:0/3194313095 conn(0x7f6358066590 msgr2=0x7f63580669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 1 -- 172.21.15.110:0/3194313095 shutdown_connections 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.406+0000 7f635e761640 1 -- 172.21.15.110:0/3194313095 wait complete. 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.407+0000 7f635e761640 10 monclient: build_initial_monmap 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.407+0000 7f635e761640 10 monclient: monmap: 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.407+0000 7f635e761640 1 librados: starting msgr at 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.407+0000 7f635e761640 1 librados: starting objecter 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.407+0000 7f635e761640 1 Processor -- start 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 1 -- start start 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 1 librados: setting wanted keys 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 1 librados: calling monclient init 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient: init 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b17c0 0x7f6358146970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient(hunting): picked mon.noname-c con 0x7f63580b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63580b4d10 0x7f6358140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient(hunting): picked mon.noname-b con 0x7f63580b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:53.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6358145980 0x7f63581408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient(hunting): picked mon.noname-a con 0x7f6358145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63581434c0 con 0x7f6358145980 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6358141ac0 con 0x7f63580b4d10 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63581479b0 con 0x7f63580b17c0 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f635e761640 10 monclient(hunting): authenticate will time out at 1696.758760s 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.408+0000 7f6357fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b17c0 0x7f6358146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f635ccfe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6358145980 0x7f63581408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f6357fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b17c0 0x7f6358146970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:39590/0 (socket says 172.21.15.110:39590) 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f6357fff640 1 -- 172.21.15.110:0/155634187 learned_addr learned my addr 172.21.15.110:0/155634187 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f6357fff640 10 monclient(hunting): get_auth_request con 0x7f63580b17c0 auth_method 0 2024-01-20T11:20:53.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f6357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f6357fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f6357fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f63577fe640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63580b4d10 0x7f6358140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f635ccfe640 10 monclient(hunting): get_auth_request con 0x7f6358145980 auth_method 0 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f635ccfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f635ccfe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f635ccfe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f63577fe640 10 monclient(hunting): get_auth_request con 0x7f63580b4d10 auth_method 0 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f63577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f63577fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f63577fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f6357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f6357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.409+0000 7f6357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.410+0000 7f635ccfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:53.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.410+0000 7f635ccfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:53.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.410+0000 7f635ccfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:53.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.410+0000 7f63577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:53.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.410+0000 7f63577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:53.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.410+0000 7f63577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 10 monclient(hunting): handle_auth_done global_id 5162 payload 824 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 1 -- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63580b4d10 msgr2=0x7f6358140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63580b4d10 0x7f6358140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 1 -- 172.21.15.110:0/155634187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6358145980 msgr2=0x7f63581408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6358145980 0x7f63581408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 10 monclient: _finish_hunting 0 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 1 monclient: found mon.noname-c 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:20:53.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.412+0000 7f6357fff640 1 -- 172.21.15.110:0/155634187 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6348009400 con 0x7f63580b17c0 2024-01-20T11:20:53.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.414+0000 7f6357fff640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b17c0 0x7f6358146970 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f63480098c0 tx=0x7f63480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:53.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.414+0000 7f6354ff9640 10 client.?.objecter ms_handle_connect 0x7f63580b17c0 2024-01-20T11:20:53.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.414+0000 7f6354ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:53.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 1 -- 172.21.15.110:0/155634187 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348032840 con 0x7f63580b17c0 2024-01-20T11:20:53.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:53.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:53.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 10 monclient: dump: 2024-01-20T11:20:53.419 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 10 monclient: _finish_auth 0 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 1 -- 172.21.15.110:0/155634187 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63480329c0 con 0x7f63580b17c0 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f635e761640 5 monclient: authenticate success, global_id 5162 2024-01-20T11:20:53.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f635e761640 10 monclient: _renew_subs 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f635e761640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6358147c90 con 0x7f63580b17c0 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 1 -- 172.21.15.110:0/155634187 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348032d70 con 0x7f63580b17c0 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f6354ff9640 10 monclient: dump: 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:53.421 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f63567fc640 4 set_mon_vals no callback set 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f635e761640 10 client.5162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f635e761640 10 monclient: _renew_subs 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f635e761640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6358148020 con 0x7f63580b17c0 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.418+0000 7f635e761640 1 librados: init done 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.419+0000 7f635e761640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.419+0000 7f635e761640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.419+0000 7f635e761640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:20:53.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.419+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6358143a80 con 0x7f63580b17c0 2024-01-20T11:20:53.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.436+0000 7f6354ff9640 1 -- 172.21.15.110:0/155634187 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f634804e470 con 0x7f63580b17c0 2024-01-20T11:20:53.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.437+0000 7f6354ff9640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f634406c860 0x7f634406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:53.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.437+0000 7f63577fe640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f634406c860 0x7f634406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:53.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.437+0000 7f63577fe640 10 monclient: get_auth_request con 0x7f634406c860 auth_method 0 2024-01-20T11:20:53.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.437+0000 7f63577fe640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f634406c860 0x7f634406ed20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6358146600 tx=0x7f634c009110 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:53.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.437+0000 7f6354ff9640 10 client.5162.objecter ms_handle_connect 0x7f634406c860 2024-01-20T11:20:53.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.440+0000 7f6354ff9640 1 -- 172.21.15.110:0/155634187 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f634807a6b0 con 0x7f63580b17c0 2024-01-20T11:20:53.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.440+0000 7f6354ff9640 10 client.5162.objecter ms_dispatch 0x7f6358000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:53.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.440+0000 7f6354ff9640 3 client.5162.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:53.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.440+0000 7f6354ff9640 3 client.5162.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:53.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.440+0000 7f6354ff9640 20 client.5162.objecter dump_active .. 0 homeless 2024-01-20T11:20:53.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.678+0000 7f6354ff9640 1 -- 172.21.15.110:0/155634187 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f63480762f0 con 0x7f63580b17c0 2024-01-20T11:20:53.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.678+0000 7f6354ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:53.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.678+0000 7f6354ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:53.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.875+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6358075490 con 0x7f634406c860 2024-01-20T11:20:53.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.878+0000 7f6354ff9640 1 -- 172.21.15.110:0/155634187 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63893 (secure 0 0 0) 0x7f6358075490 con 0x7f634406c860 2024-01-20T11:20:53.878 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:53.878 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 10 librados: watch_flush enter 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 10 librados: watch_flush exit 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 20 client.5162.objecter shutdown clearing up homeless session... 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 10 client.5162.objecter successfully canceled tick 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f634406c860 msgr2=0x7f634406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f634406c860 0x7f634406ed20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6358146600 tx=0x7f634c009110 comp rx=0 tx=0).stop 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 10 monclient: shutdown 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b17c0 msgr2=0x7f6358146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b17c0 0x7f6358146970 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f63480098c0 tx=0x7f63480046c0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 shutdown_connections 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f634406c860 0x7f634406ed20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6358145980 0x7f63581408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63580b4d10 0x7f6358140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 --2- 172.21.15.110:0/155634187 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63580b17c0 0x7f6358146970 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:53.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 >> 172.21.15.110:0/155634187 conn(0x7f6358066590 msgr2=0x7f635813b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:53.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 shutdown_connections 2024-01-20T11:20:53.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 -- 172.21.15.110:0/155634187 wait complete. 2024-01-20T11:20:53.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.880+0000 7f635e761640 1 librados: shutdown 2024-01-20T11:20:53.890 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-01-20T11:20:52.776159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002594"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":9.5879999999999992},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.7140000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":11.1},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.6890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":13.843999999999999},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.117000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.9950000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":12.819000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.449999999999999},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":10.548999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.9269999999999996},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.569000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313288,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.2539999999999996},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.521000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":12.863},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":6.4409999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.73},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":5.6260000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.992},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.641999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.224},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":8.0459999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.462999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.8320000000000007}]}]},{"osd":3,"up_from":21,"seq":90194313289,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.4649999999999999},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.024}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.7929999999999993},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.407}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":7.7969999999999997},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.096}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.683},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.563000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":11.904},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.407}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":15.227},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.351000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444102,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.3140000000000001},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":6.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":9.0899999999999999},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":12.661}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.865},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.390000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.638999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.4350000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.945},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":14.291}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.382},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":16.507999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444103,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.9160000000000004},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":11.869}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.513999999999999},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.3119999999999994}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":13.69},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.5869999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":15.439},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.117000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.4120000000000008},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":14.478999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":9.9909999999999997},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":16.199999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444103,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":13.045999999999999},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":6.4189999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":7.3440000000000003},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":13.794}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":14.526999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":9.8260000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":7.0780000000000003},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.0830000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":15.285},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":10.398}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":10.98},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":8.8010000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509512,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.407},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":12.438000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.0069999999999997},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":8.7449999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":17.491},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.4100000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.002000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":15.882999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.722000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.274}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.782},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.131}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:53.892 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:53.892 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:53.892 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:53.892 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:53.892 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:53.892 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:53.893 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:53.893 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:53.893 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:53.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.992+0000 7f9aeb725640 10 monclient: get_monmap_and_config 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.992+0000 7f9aeb725640 10 monclient: build_initial_monmap 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.992+0000 7f9aeb725640 10 monclient: monmap: 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:53.993312+0000 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:53.993312+0000 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:53.993 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:53.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.993+0000 7f9aeb725640 1 Processor -- start 2024-01-20T11:20:53.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.993+0000 7f9aeb725640 1 -- start start 2024-01-20T11:20:53.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.993+0000 7f9aeb725640 10 monclient: init 2024-01-20T11:20:53.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.993+0000 7f9aeb725640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:53.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-01-20T11:20:53.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ae4066620 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:53.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ae406df70 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:53.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9ae4145b50 0x7f9ae414d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:53.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 10 monclient(hunting): picked mon.noname-a con 0x7f9ae4145b50 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:53.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae40adf80 con 0x7f9ae406df70 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae406f650 con 0x7f9ae4066620 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae4072880 con 0x7f9ae4145b50 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9aeb725640 10 monclient(hunting): authenticate will time out at 1697.344251s 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9ae9cc2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9ae4145b50 0x7f9ae414d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9ae94c1640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9ae4066620 0x7f9ae413bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:36952/0 (socket says 172.21.15.110:36952) 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.994+0000 7f9ae94c1640 1 -- 172.21.15.110:0/4215345973 learned_addr learned my addr 172.21.15.110:0/4215345973 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.995+0000 7f9ae9cc2640 10 monclient(hunting): get_auth_request con 0x7f9ae4145b50 auth_method 0 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.995+0000 7f9ae9cc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.995+0000 7f9ae9cc2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.995+0000 7f9ae9cc2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:53.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.995+0000 7f9ae9cc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:53.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.995+0000 7f9ae9cc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:53.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.995+0000 7f9ae9cc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:53.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9adbfff640 1 -- 172.21.15.110:0/4215345973 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702773802 0 0) 0x7f9ae406f650 con 0x7f9ae4066620 2024-01-20T11:20:53.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9adbfff640 10 monclient(hunting): _negotiate 2024-01-20T11:20:53.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9adbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:53.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:53.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9adbfff640 10 monclient(hunting): my global_id is 5485 2024-01-20T11:20:53.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9adbfff640 1 -- 172.21.15.110:0/4215345973 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad0003520 con 0x7f9ae4066620 2024-01-20T11:20:53.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9ae9cc2640 10 monclient(hunting): handle_auth_done global_id 5424 payload 274 2024-01-20T11:20:53.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9ae9cc2640 1 -- 172.21.15.110:0/4215345973 >> v1:172.21.15.134:6789/0 conn(0x7f9ae4066620 legacy=0x7f9ae413bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9ae9cc2640 1 -- 172.21.15.110:0/4215345973 >> v1:172.21.15.110:6789/0 conn(0x7f9ae406df70 legacy=0x7f9ae4145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:53.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9ae9cc2640 10 monclient: _finish_hunting 0 2024-01-20T11:20:53.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9ae9cc2640 1 monclient: found mon.noname-a 2024-01-20T11:20:53.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9ae9cc2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:53.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9ae9cc2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:53.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:53.997+0000 7f9ae9cc2640 1 -- 172.21.15.110:0/4215345973 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae4072be0 con 0x7f9ae4145b50 2024-01-20T11:20:54.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.001+0000 7f9ae9cc2640 1 --2- 172.21.15.110:0/4215345973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4145b50 0x7f9ae414d7e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d90 tx=0x7f9adc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06a26d970939b95 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:54.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 1 -- 172.21.15.110:0/4215345973 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc02cc60 con 0x7f9ae4145b50 2024-01-20T11:20:54.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:54.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:54.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 10 monclient: dump: 2024-01-20T11:20:54.006 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 10 monclient: _finish_auth 0 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 10 monclient: _check_auth_tickets 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 1 -- 172.21.15.110:0/4215345973 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc02cdc0 con 0x7f9ae4145b50 2024-01-20T11:20:54.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 1 -- 172.21.15.110:0/4215345973 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc034660 con 0x7f9ae4145b50 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9adbfff640 10 monclient: dump: 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:54.008 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9aeb725640 5 monclient: authenticate success, global_id 5424 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9aeb725640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.005+0000 7f9aeb725640 4 set_mon_vals no callback set 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.006+0000 7f9aeb725640 10 monclient: shutdown 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.006+0000 7f9aeb725640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.006+0000 7f9aeb725640 1 -- 172.21.15.110:0/4215345973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4145b50 msgr2=0x7f9ae414d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.006+0000 7f9aeb725640 1 --2- 172.21.15.110:0/4215345973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4145b50 0x7f9ae414d7e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009d90 tx=0x7f9adc02beb0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.006+0000 7f9aeb725640 1 -- 172.21.15.110:0/4215345973 shutdown_connections 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.006+0000 7f9aeb725640 1 --2- 172.21.15.110:0/4215345973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4145b50 0x7f9ae414d7e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.006+0000 7f9aeb725640 1 -- 172.21.15.110:0/4215345973 >> 172.21.15.110:0/4215345973 conn(0x7f9ae40b09e0 msgr2=0x7f9ae40b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.006+0000 7f9aeb725640 1 -- 172.21.15.110:0/4215345973 shutdown_connections 2024-01-20T11:20:54.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.006+0000 7f9aeb725640 1 -- 172.21.15.110:0/4215345973 wait complete. 2024-01-20T11:20:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.007+0000 7f9aeb725640 10 monclient: build_initial_monmap 2024-01-20T11:20:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.007+0000 7f9aeb725640 10 monclient: monmap: 2024-01-20T11:20:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.013 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.007+0000 7f9aeb725640 1 librados: starting msgr at 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.007+0000 7f9aeb725640 1 librados: starting objecter 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.007+0000 7f9aeb725640 1 Processor -- start 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.007+0000 7f9aeb725640 1 -- start start 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.007+0000 7f9aeb725640 1 librados: setting wanted keys 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.007+0000 7f9aeb725640 1 librados: calling monclient init 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.007+0000 7f9aeb725640 10 monclient: init 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4066620 0x7f9ae413ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient(hunting): picked mon.noname-a con 0x7f9ae4066620 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ae406df70 0x7f9ae4142870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient(hunting): picked mon.noname-b con 0x7f9ae406df70 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ae4145b50 0x7f9ae41408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient(hunting): picked mon.noname-c con 0x7f9ae4145b50 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4141f20 con 0x7f9ae4066620 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4142de0 con 0x7f9ae406df70 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4142180 con 0x7f9ae4145b50 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9aeb725640 10 monclient(hunting): authenticate will time out at 1697.358426s 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9ae94c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4066620 0x7f9ae413ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9ae8cc0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ae406df70 0x7f9ae4142870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9ae9cc2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ae4145b50 0x7f9ae41408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.008+0000 7f9ae94c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4066620 0x7f9ae413ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:37526/0 (socket says 172.21.15.110:37526) 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae94c1640 1 -- 172.21.15.110:0/1724351167 learned_addr learned my addr 172.21.15.110:0/1724351167 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae94c1640 10 monclient(hunting): get_auth_request con 0x7f9ae4066620 auth_method 0 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae94c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae94c1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae94c1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae8cc0640 10 monclient(hunting): get_auth_request con 0x7f9ae406df70 auth_method 0 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae8cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae8cc0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae8cc0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae9cc2640 10 monclient(hunting): get_auth_request con 0x7f9ae4145b50 auth_method 0 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae9cc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae9cc2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae9cc2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:54.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae94c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae94c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae94c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae8cc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae8cc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae8cc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae9cc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae9cc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.009+0000 7f9ae9cc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 10 monclient(hunting): handle_auth_done global_id 5488 payload 824 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 1 -- 172.21.15.110:0/1724351167 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ae4145b50 msgr2=0x7f9ae41408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ae4145b50 0x7f9ae41408a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 1 -- 172.21.15.110:0/1724351167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4066620 msgr2=0x7f9ae413ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4066620 0x7f9ae413ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 10 monclient: _finish_hunting 0 2024-01-20T11:20:54.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 1 monclient: found mon.noname-b 2024-01-20T11:20:54.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:54.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:54.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae8cc0640 1 -- 172.21.15.110:0/1724351167 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc009a70 con 0x7f9ae406df70 2024-01-20T11:20:54.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.012+0000 7f9ae94c1640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4066620 0x7f9ae413ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:20:54.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.014+0000 7f9ae8cc0640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ae406df70 0x7f9ae4142870 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c340 tx=0x7f9ad400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:54.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.014+0000 7f9ad9ffb640 10 client.?.objecter ms_handle_connect 0x7f9ae406df70 2024-01-20T11:20:54.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.014+0000 7f9ad9ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:54.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.017+0000 7f9ad9ffb640 1 -- 172.21.15.110:0/1724351167 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40052d0 con 0x7f9ae406df70 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 10 monclient: dump: 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 10 monclient: _finish_auth 0 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 1 -- 172.21.15.110:0/1724351167 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad4005450 con 0x7f9ae406df70 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 1 -- 172.21.15.110:0/1724351167 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4007260 con 0x7f9ae406df70 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9aeb725640 5 monclient: authenticate success, global_id 5488 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9aeb725640 10 monclient: _renew_subs 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9aeb725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae4142440 con 0x7f9ae406df70 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9adb7fe640 4 set_mon_vals no callback set 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9ad9ffb640 10 monclient: dump: 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:54.021 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9aeb725640 10 client.5488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9aeb725640 10 monclient: _renew_subs 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9aeb725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae4140de0 con 0x7f9ae406df70 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.018+0000 7f9aeb725640 1 librados: init done 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.019+0000 7f9aeb725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:54.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.019+0000 7f9aeb725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:54.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.019+0000 7f9aeb725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:54.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.019+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae4143060 con 0x7f9ae406df70 2024-01-20T11:20:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.256+0000 7f9ad9ffb640 1 -- 172.21.15.110:0/1724351167 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9ad40073c0 con 0x7f9ae406df70 2024-01-20T11:20:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.256+0000 7f9ad9ffb640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9ac806c8e0 0x7f9ac806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.256+0000 7f9ae94c1640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9ac806c8e0 0x7f9ac806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.256+0000 7f9ae94c1640 10 monclient: get_auth_request con 0x7f9ac806c8e0 auth_method 0 2024-01-20T11:20:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.257+0000 7f9ad9ffb640 10 client.5488.objecter ms_handle_connect 0x7f9ac806c8e0 2024-01-20T11:20:54.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.257+0000 7f9ae94c1640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9ac806c8e0 0x7f9ac806eda0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0005880 tx=0x7f9ae0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:54.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.258+0000 7f9ad9ffb640 1 -- 172.21.15.110:0/1724351167 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9ad4057a00 con 0x7f9ae406df70 2024-01-20T11:20:54.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.258+0000 7f9ad9ffb640 10 client.5488.objecter ms_dispatch 0x7f9ae4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:54.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.258+0000 7f9ad9ffb640 3 client.5488.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:54.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.258+0000 7f9ad9ffb640 3 client.5488.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:54.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.258+0000 7f9ad9ffb640 20 client.5488.objecter dump_active .. 0 homeless 2024-01-20T11:20:54.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.270+0000 7f9ad9ffb640 1 -- 172.21.15.110:0/1724351167 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9ad4053d40 con 0x7f9ae406df70 2024-01-20T11:20:54.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.270+0000 7f9ad9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:54.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.270+0000 7f9ad9ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.550+0000 7f9aeb725640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.550+0000 7f9aeb725640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.550+0000 7f9aeb725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.550+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f9ae41405c0 con 0x7f9ae406df70 2024-01-20T11:20:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.556+0000 7f9ad9ffb640 1 -- 172.21.15.110:0/1724351167 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f9ad4020080 con 0x7f9ae406df70 2024-01-20T11:20:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.556+0000 7f9ad9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.556+0000 7f9ad9ffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:54.556 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:54.556 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":389,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 10 librados: watch_flush enter 2024-01-20T11:20:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 10 librados: watch_flush exit 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 20 client.5488.objecter shutdown clearing up homeless session... 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 10 client.5488.objecter successfully canceled tick 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9ac806c8e0 msgr2=0x7f9ac806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9ac806c8e0 0x7f9ac806eda0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0005880 tx=0x7f9ae0009040 comp rx=0 tx=0).stop 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 10 monclient: shutdown 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ae406df70 msgr2=0x7f9ae4142870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ae406df70 0x7f9ae4142870 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c340 tx=0x7f9ad400c800 comp rx=0 tx=0).stop 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 shutdown_connections 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9ac806c8e0 0x7f9ac806eda0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ae4145b50 0x7f9ae41408a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ae406df70 0x7f9ae4142870 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 --2- 172.21.15.110:0/1724351167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ae4066620 0x7f9ae413ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 >> 172.21.15.110:0/1724351167 conn(0x7f9ae40b09e0 msgr2=0x7f9ae40b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:54.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 shutdown_connections 2024-01-20T11:20:54.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 -- 172.21.15.110:0/1724351167 wait complete. 2024-01-20T11:20:54.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.558+0000 7f9aeb725640 1 librados: shutdown 2024-01-20T11:20:54.568 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:54.569 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.667+0000 7f6fabc68640 10 monclient: get_monmap_and_config 2024-01-20T11:20:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.668+0000 7f6fabc68640 10 monclient: build_initial_monmap 2024-01-20T11:20:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.668+0000 7f6fabc68640 10 monclient: monmap: 2024-01-20T11:20:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:54.668611+0000 2024-01-20T11:20:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:54.668611+0000 2024-01-20T11:20:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:54.669 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.668+0000 7f6fabc68640 1 Processor -- start 2024-01-20T11:20:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 1 -- start start 2024-01-20T11:20:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 10 monclient: init 2024-01-20T11:20:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-20T11:20:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fa40b4d10 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fa40b1ed0 addr v1:172.21.15.110:6789/0 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa40b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 10 monclient(hunting): picked mon.noname-b con 0x7f6fa4146090 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa40aade0 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa4145460 con 0x7f6fa40b4d10 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40b50f0 con 0x7f6fa4146090 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.669+0000 7f6fabc68640 10 monclient(hunting): authenticate will time out at 1698.019853s 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.670+0000 7f6fa9a04640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6fa40b4d10 0x7f6fa40b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:36956/0 (socket says 172.21.15.110:36956) 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.670+0000 7f6fa9a04640 1 -- 172.21.15.110:0/2126864879 learned_addr learned my addr 172.21.15.110:0/2126864879 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.670+0000 7f6faa205640 1 --2- 172.21.15.110:0/2126864879 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa40b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:54.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.671+0000 7f6faa205640 10 monclient(hunting): get_auth_request con 0x7f6fa4146090 auth_method 0 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.671+0000 7f6faa205640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.671+0000 7f6faa205640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.671+0000 7f6faa205640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.671+0000 7f6faa205640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.671+0000 7f6faa205640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.671+0000 7f6faa205640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.672+0000 7f6fa8a02640 1 -- 172.21.15.110:0/2126864879 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33459703 0 0) 0x7f6fa4145460 con 0x7f6fa40b4d10 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.672+0000 7f6fa8a02640 10 monclient(hunting): _negotiate 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.672+0000 7f6fa8a02640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.672+0000 7f6fa8a02640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.672+0000 7f6fa8a02640 10 monclient(hunting): my global_id is 5491 2024-01-20T11:20:54.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.672+0000 7f6fa8a02640 1 -- 172.21.15.110:0/2126864879 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f90003520 con 0x7f6fa40b4d10 2024-01-20T11:20:54.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.673+0000 7f6faa205640 10 monclient(hunting): handle_auth_done global_id 5168 payload 274 2024-01-20T11:20:54.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.673+0000 7f6faa205640 1 -- 172.21.15.110:0/2126864879 >> v1:172.21.15.134:6789/0 conn(0x7f6fa40b4d10 legacy=0x7f6fa40b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.673+0000 7f6faa205640 1 -- 172.21.15.110:0/2126864879 >> v1:172.21.15.110:6789/0 conn(0x7f6fa40b1ed0 legacy=0x7f6fa4145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.673+0000 7f6faa205640 10 monclient: _finish_hunting 0 2024-01-20T11:20:54.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.673+0000 7f6faa205640 1 monclient: found mon.noname-b 2024-01-20T11:20:54.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.673+0000 7f6faa205640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:54.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.673+0000 7f6faa205640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:20:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.673+0000 7f6faa205640 1 -- 172.21.15.110:0/2126864879 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa40b2660 con 0x7f6fa4146090 2024-01-20T11:20:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.675+0000 7f6faa205640 1 --2- 172.21.15.110:0/2126864879 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa40b3da0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009df0 tx=0x7f6f9c02be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=176c107611a97dc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:54.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 1 -- 172.21.15.110:0/2126864879 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c02cc20 con 0x7f6fa4146090 2024-01-20T11:20:54.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:54.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:54.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 10 monclient: dump: 2024-01-20T11:20:54.696 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:54.696 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:54.696 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 10 monclient: _finish_auth 0 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 10 monclient: _check_auth_tickets 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 1 -- 172.21.15.110:0/2126864879 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c02cd80 con 0x7f6fa4146090 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fabc68640 5 monclient: authenticate success, global_id 5168 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fabc68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:54.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 1 -- 172.21.15.110:0/2126864879 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c034660 con 0x7f6fa4146090 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fabc68640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fabc68640 4 set_mon_vals no callback set 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.696+0000 7f6fa8a02640 10 monclient: dump: 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:54.698 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.697+0000 7f6fabc68640 10 monclient: shutdown 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.697+0000 7f6fabc68640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.697+0000 7f6fabc68640 1 -- 172.21.15.110:0/2126864879 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 msgr2=0x7f6fa40b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.697+0000 7f6fabc68640 1 --2- 172.21.15.110:0/2126864879 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa40b3da0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009df0 tx=0x7f6f9c02be70 comp rx=0 tx=0).stop 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.697+0000 7f6fabc68640 1 -- 172.21.15.110:0/2126864879 shutdown_connections 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.697+0000 7f6fabc68640 1 --2- 172.21.15.110:0/2126864879 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa40b3da0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.697+0000 7f6fabc68640 1 -- 172.21.15.110:0/2126864879 >> 172.21.15.110:0/2126864879 conn(0x7f6fa4066590 msgr2=0x7f6fa40669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.697+0000 7f6fabc68640 1 -- 172.21.15.110:0/2126864879 shutdown_connections 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.697+0000 7f6fabc68640 1 -- 172.21.15.110:0/2126864879 wait complete. 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.698+0000 7f6fabc68640 10 monclient: build_initial_monmap 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.698+0000 7f6fabc68640 10 monclient: monmap: 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:54.702 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.698+0000 7f6fabc68640 1 librados: starting msgr at 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.698+0000 7f6fabc68640 1 librados: starting objecter 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.698+0000 7f6fabc68640 1 Processor -- start 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.698+0000 7f6fabc68640 1 -- start start 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.698+0000 7f6fabc68640 1 librados: setting wanted keys 2024-01-20T11:20:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.698+0000 7f6fabc68640 1 librados: calling monclient init 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.698+0000 7f6fabc68640 10 monclient: init 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fa40b1ed0 0x7f6fa413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient(hunting): picked mon.noname-b con 0x7f6fa40b1ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fa40b4d10 0x7f6fa40ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient(hunting): picked mon.noname-a con 0x7f6fa40b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa413e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient(hunting): picked mon.noname-c con 0x7f6fa4146090 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40abee0 con 0x7f6fa40b4d10 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa413dff0 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa413dcf0 con 0x7f6fa4146090 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6fabc68640 10 monclient(hunting): authenticate will time out at 1698.049446s 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.699+0000 7f6faa205640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa413e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6fa9a04640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fa40b1ed0 0x7f6fa413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6faa205640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa413e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:39612/0 (socket says 172.21.15.110:39612) 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6faa205640 1 -- 172.21.15.110:0/1864580793 learned_addr learned my addr 172.21.15.110:0/1864580793 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6faa205640 10 monclient(hunting): get_auth_request con 0x7f6fa4146090 auth_method 0 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6faa205640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6faa205640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6faa205640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6fa9a04640 10 monclient(hunting): get_auth_request con 0x7f6fa40b1ed0 auth_method 0 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6fa9a04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6fa9a04640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6fa9a04640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6faa205640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6faa205640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6faa205640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6fa9a04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6fa9a04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.700+0000 7f6fa9a04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 10 monclient(hunting): handle_auth_done global_id 5494 payload 824 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 1 -- 172.21.15.110:0/1864580793 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 msgr2=0x7f6fa413e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa413e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 1 -- 172.21.15.110:0/1864580793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fa40b4d10 msgr2=0x7f6fa40ac300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fa40b4d10 0x7f6fa40ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 10 monclient: _finish_hunting 0 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 1 monclient: found mon.noname-b 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.703+0000 7f6fa9a04640 1 -- 172.21.15.110:0/1864580793 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c009a70 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.705+0000 7f6fa9a04640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fa40b1ed0 0x7f6fa413d7b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c430 tx=0x7f6fa000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:54.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.705+0000 7f6f967fc640 10 client.?.objecter ms_handle_connect 0x7f6fa40b1ed0 2024-01-20T11:20:54.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.705+0000 7f6f967fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:54.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 1 -- 172.21.15.110:0/1864580793 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0005ce0 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 10 monclient: dump: 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:54.710 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 1 -- 172.21.15.110:0/1864580793 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0005e40 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6fabc68640 5 monclient: authenticate success, global_id 5494 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6fabc68640 10 monclient: _renew_subs 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6fabc68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa413c410 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 1 -- 172.21.15.110:0/1864580793 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00072d0 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f967fc640 10 monclient: dump: 2024-01-20T11:20:54.711 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6f97fff640 4 set_mon_vals no callback set 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6fabc68640 10 client.5494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6fabc68640 10 monclient: _renew_subs 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6fabc68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa413ea70 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.709+0000 7f6fabc68640 1 librados: init done 2024-01-20T11:20:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.710+0000 7f6fabc68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.710+0000 7f6fabc68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.710+0000 7f6fabc68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.710+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa4075490 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.729+0000 7f6f967fc640 1 -- 172.21.15.110:0/1864580793 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6fa0007430 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.729+0000 7f6f967fc640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f8806c8b0 0x7f6f8806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:54.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.730+0000 7f6fa9203640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f8806c8b0 0x7f6f8806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:54.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.730+0000 7f6fa9203640 10 monclient: get_auth_request con 0x7f6f8806c8b0 auth_method 0 2024-01-20T11:20:54.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.730+0000 7f6fa9203640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f8806c8b0 0x7f6f8806ed70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6f98000f60 tx=0x7f6f980079d0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:54.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.730+0000 7f6f967fc640 10 client.5494.objecter ms_handle_connect 0x7f6f8806c8b0 2024-01-20T11:20:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.732+0000 7f6f967fc640 1 -- 172.21.15.110:0/1864580793 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6fa0057d10 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.732+0000 7f6f967fc640 10 client.5494.objecter ms_dispatch 0x7f6fa4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.732+0000 7f6f967fc640 3 client.5494.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.732+0000 7f6f967fc640 3 client.5494.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.732+0000 7f6f967fc640 20 client.5494.objecter dump_active .. 0 homeless 2024-01-20T11:20:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.966+0000 7f6f967fc640 1 -- 172.21.15.110:0/1864580793 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6fa0054050 con 0x7f6fa40b1ed0 2024-01-20T11:20:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.966+0000 7f6f967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:54.966+0000 7f6f967fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:55.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.135+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fa40a9cf0 con 0x7f6f8806c8b0 2024-01-20T11:20:55.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.138+0000 7f6f967fc640 1 -- 172.21.15.110:0/1864580793 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63770 (secure 0 0 0) 0x7f6f98005080 con 0x7f6f8806c8b0 2024-01-20T11:20:55.138 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:55.138 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:55.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.139+0000 7f6fabc68640 10 librados: watch_flush enter 2024-01-20T11:20:55.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.139+0000 7f6fabc68640 10 librados: watch_flush exit 2024-01-20T11:20:55.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.139+0000 7f6fabc68640 20 client.5494.objecter shutdown clearing up homeless session... 2024-01-20T11:20:55.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.139+0000 7f6fabc68640 10 client.5494.objecter successfully canceled tick 2024-01-20T11:20:55.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.139+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f8806c8b0 msgr2=0x7f6f8806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:55.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f8806c8b0 0x7f6f8806ed70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6f98000f60 tx=0x7f6f980079d0 comp rx=0 tx=0).stop 2024-01-20T11:20:55.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 10 monclient: shutdown 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fa40b1ed0 msgr2=0x7f6fa413d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fa40b1ed0 0x7f6fa413d7b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c430 tx=0x7f6fa000c8f0 comp rx=0 tx=0).stop 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 shutdown_connections 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f8806c8b0 0x7f6f8806ed70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fa4146090 0x7f6fa413e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fa40b4d10 0x7f6fa40ac300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 --2- 172.21.15.110:0/1864580793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fa40b1ed0 0x7f6fa413d7b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 >> 172.21.15.110:0/1864580793 conn(0x7f6fa4066590 msgr2=0x7f6fa413f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 shutdown_connections 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 -- 172.21.15.110:0/1864580793 wait complete. 2024-01-20T11:20:55.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:55.140+0000 7f6fabc68640 1 librados: shutdown 2024-01-20T11:20:55.150 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-01-20T11:20:54.776548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002583"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":10.631},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":9.9260000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.9660000000000002},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.489000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":7.125},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.904}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.1619999999999999},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":13.772}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.012},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":13.082000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.25},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.808999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313288,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":9.2539999999999996},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":14.521000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":12.863},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":6.4409999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.73},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":5.6260000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":6.992},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.641999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":11.224},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":8.0459999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":10.462999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.8320000000000007}]}]},{"osd":3,"up_from":21,"seq":90194313289,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.4649999999999999},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.024}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.7929999999999993},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.407}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":7.7969999999999997},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.096}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.683},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.563000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":11.904},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.407}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":15.227},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.351000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444103,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.881},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":10.048999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.6299999999999999},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":15.073}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.305},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.3300000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":8.4019999999999992},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.976000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.956},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.6120000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.2460000000000004},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":15.861000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444104,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.8190000000000008},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":5.4809999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.8440000000000003},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.92}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":15.74},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.6669999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":8.2080000000000002},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.683}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":10.445},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":11.206}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":14.412000000000001},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.73}]}]},{"osd":0,"up_from":17,"seq":73014444104,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":9.4079999999999995},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":5.8499999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.468},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.464}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":12.398999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.689}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.721},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.4399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":12.997},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":14.804}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.411},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":9.8620000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509512,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.407},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":12.438000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":6.0069999999999997},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":8.7449999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":17.491},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.4100000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.002000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":15.882999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.722000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":7.274}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.782},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.131}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:56.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~118s 2024-01-20T11:20:58.152 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.253+0000 7f02d8bdc640 10 monclient: get_monmap_and_config 2024-01-20T11:20:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.253+0000 7f02d8bdc640 10 monclient: build_initial_monmap 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.253+0000 7f02d8bdc640 10 monclient: monmap: 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:58.254398+0000 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:58.254398+0000 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:58.254 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.254+0000 7f02d8bdc640 1 Processor -- start 2024-01-20T11:20:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.254+0000 7f02d8bdc640 1 -- start start 2024-01-20T11:20:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.254+0000 7f02d8bdc640 10 monclient: init 2024-01-20T11:20:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-01-20T11:20:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02d406a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f02d4072330 0x7f02d4145c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 10 monclient(hunting): picked mon.noname-a con 0x7f02d4072330 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f02d413ed20 0x7f02d413f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 10 monclient(hunting): picked mon.noname-c con 0x7f02d413ed20 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d40b1010 con 0x7f02d406a860 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d40afe00 con 0x7f02d4072330 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d4146170 con 0x7f02d413ed20 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d8bdc640 10 monclient(hunting): authenticate will time out at 1701.605444s 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.255+0000 7f02d1d9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f02d4072330 0x7f02d4145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d2d9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f02d413ed20 0x7f02d413f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d1d9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f02d4072330 0x7f02d4145c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:37550/0 (socket says 172.21.15.110:37550) 2024-01-20T11:20:58.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d1d9b640 1 -- 172.21.15.110:0/233086128 learned_addr learned my addr 172.21.15.110:0/233086128 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d1d9b640 10 monclient(hunting): get_auth_request con 0x7f02d4072330 auth_method 0 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d2d9d640 10 monclient(hunting): get_auth_request con 0x7f02d413ed20 auth_method 0 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:58.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.256+0000 7f02d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.257+0000 7f02d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.257+0000 7f02d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.257+0000 7f02d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.258+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_done global_id 5436 payload 274 2024-01-20T11:20:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.258+0000 7f02d1d9b640 1 -- 172.21.15.110:0/233086128 >> v2:172.21.15.134:3300/0 conn(0x7f02d413ed20 msgr2=0x7f02d413f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.258+0000 7f02d1d9b640 1 --2- 172.21.15.110:0/233086128 >> v2:172.21.15.134:3300/0 conn(0x7f02d413ed20 0x7f02d413f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.258+0000 7f02d1d9b640 1 -- 172.21.15.110:0/233086128 >> v1:172.21.15.134:6789/0 conn(0x7f02d406a860 legacy=0x7f02d4145520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.258+0000 7f02d1d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:20:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.258+0000 7f02d1d9b640 1 monclient: found mon.noname-a 2024-01-20T11:20:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.258+0000 7f02d1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.258+0000 7f02d1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.258+0000 7f02d1d9b640 1 -- 172.21.15.110:0/233086128 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d406e480 con 0x7f02d4072330 2024-01-20T11:20:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.260+0000 7f02d1d9b640 1 --2- 172.21.15.110:0/233086128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d4072330 0x7f02d4145c30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f02d406b140 tx=0x7f02c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c65a136b71c1f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:58.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 1 -- 172.21.15.110:0/233086128 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8030860 con 0x7f02d4072330 2024-01-20T11:20:58.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:58.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:58.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 10 monclient: dump: 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 10 monclient: _finish_auth 0 2024-01-20T11:20:58.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 10 monclient: _check_auth_tickets 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 1 -- 172.21.15.110:0/233086128 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c80309c0 con 0x7f02d4072330 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 1 -- 172.21.15.110:0/233086128 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8030d50 con 0x7f02d4072330 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d159a640 10 monclient: dump: 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:58.265 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d8bdc640 5 monclient: authenticate success, global_id 5436 2024-01-20T11:20:58.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d8bdc640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:58.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.263+0000 7f02d8bdc640 4 set_mon_vals no callback set 2024-01-20T11:20:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 10 monclient: shutdown 2024-01-20T11:20:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 1 -- 172.21.15.110:0/233086128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d4072330 msgr2=0x7f02d4145c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 1 --2- 172.21.15.110:0/233086128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d4072330 0x7f02d4145c30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f02d406b140 tx=0x7f02c802b9a0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 1 -- 172.21.15.110:0/233086128 shutdown_connections 2024-01-20T11:20:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 1 --2- 172.21.15.110:0/233086128 >> v2:172.21.15.134:3300/0 conn(0x7f02d413ed20 0x7f02d413f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 1 --2- 172.21.15.110:0/233086128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d4072330 0x7f02d4145c30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 1 -- 172.21.15.110:0/233086128 >> 172.21.15.110:0/233086128 conn(0x7f02d40b47d0 msgr2=0x7f02d40b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 1 -- 172.21.15.110:0/233086128 shutdown_connections 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.264+0000 7f02d8bdc640 1 -- 172.21.15.110:0/233086128 wait complete. 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.265+0000 7f02d8bdc640 10 monclient: build_initial_monmap 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.265+0000 7f02d8bdc640 10 monclient: monmap: 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.265+0000 7f02d8bdc640 1 librados: starting msgr at 2024-01-20T11:20:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.265+0000 7f02d8bdc640 1 librados: starting objecter 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.265+0000 7f02d8bdc640 1 Processor -- start 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.265+0000 7f02d8bdc640 1 -- start start 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.265+0000 7f02d8bdc640 1 librados: setting wanted keys 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.265+0000 7f02d8bdc640 1 librados: calling monclient init 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.265+0000 7f02d8bdc640 10 monclient: init 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02d406a860 0x7f02d4138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient(hunting): picked mon.noname-c con 0x7f02d406a860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d4072330 0x7f02d406fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient(hunting): picked mon.noname-b con 0x7f02d4072330 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d413ed20 0x7f02d40704e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient(hunting): picked mon.noname-a con 0x7f02d413ed20 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d406f8d0 con 0x7f02d413ed20 2024-01-20T11:20:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d406f5c0 con 0x7f02d4072330 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d4071d00 con 0x7f02d406a860 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d8bdc640 10 monclient(hunting): authenticate will time out at 1701.616666s 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.266+0000 7f02d2d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d413ed20 0x7f02d40704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02d406a860 0x7f02d4138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d1d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d4072330 0x7f02d406fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d2d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d413ed20 0x7f02d40704e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:37552/0 (socket says 172.21.15.110:37552) 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d2d9d640 1 -- 172.21.15.110:0/28516528 learned_addr learned my addr 172.21.15.110:0/28516528 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d2d9d640 10 monclient(hunting): get_auth_request con 0x7f02d413ed20 auth_method 0 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d259c640 10 monclient(hunting): get_auth_request con 0x7f02d406a860 auth_method 0 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d1d9b640 10 monclient(hunting): get_auth_request con 0x7f02d4072330 auth_method 0 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.267+0000 7f02d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.268+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.268+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.268+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 10 monclient(hunting): handle_auth_done global_id 5500 payload 824 2024-01-20T11:20:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 1 -- 172.21.15.110:0/28516528 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02d406a860 msgr2=0x7f02d4138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02d406a860 0x7f02d4138910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 1 -- 172.21.15.110:0/28516528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d413ed20 msgr2=0x7f02d40704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d413ed20 0x7f02d40704e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:20:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 1 monclient: found mon.noname-b 2024-01-20T11:20:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.271+0000 7f02d1d9b640 1 -- 172.21.15.110:0/28516528 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c80094d0 con 0x7f02d4072330 2024-01-20T11:20:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.273+0000 7f02d1d9b640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d4072330 0x7f02d406fcf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f02d40ae990 tx=0x7f02c8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.273+0000 7f02c2ffd640 10 client.?.objecter ms_handle_connect 0x7f02d4072330 2024-01-20T11:20:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.273+0000 7f02c2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.277+0000 7f02c2ffd640 1 -- 172.21.15.110:0/28516528 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8039c90 con 0x7f02d4072330 2024-01-20T11:20:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 10 monclient: dump: 2024-01-20T11:20:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 1 -- 172.21.15.110:0/28516528 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c8039e10 con 0x7f02d4072330 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d8bdc640 5 monclient: authenticate success, global_id 5500 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d8bdc640 10 monclient: _renew_subs 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d8bdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d4070a20 con 0x7f02d4072330 2024-01-20T11:20:58.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 1 -- 172.21.15.110:0/28516528 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c80446c0 con 0x7f02d4072330 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02c2ffd640 10 monclient: dump: 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:58.280 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d8bdc640 10 client.5500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:58.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d0d99640 4 set_mon_vals no callback set 2024-01-20T11:20:58.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d8bdc640 10 monclient: _renew_subs 2024-01-20T11:20:58.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d8bdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:58.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d41d7810 con 0x7f02d4072330 2024-01-20T11:20:58.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.278+0000 7f02d8bdc640 1 librados: init done 2024-01-20T11:20:58.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.279+0000 7f02d8bdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:58.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.279+0000 7f02d8bdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.279+0000 7f02d8bdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.279+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d406fa10 con 0x7f02d4072330 2024-01-20T11:20:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.299+0000 7f02c2ffd640 1 -- 172.21.15.110:0/28516528 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f02c804d3b0 con 0x7f02d4072330 2024-01-20T11:20:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.299+0000 7f02c2ffd640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f02b406c8b0 0x7f02b406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.299+0000 7f02d259c640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f02b406c8b0 0x7f02b406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.299+0000 7f02d259c640 10 monclient: get_auth_request con 0x7f02b406c8b0 auth_method 0 2024-01-20T11:20:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.300+0000 7f02c2ffd640 10 client.5500.objecter ms_handle_connect 0x7f02b406c8b0 2024-01-20T11:20:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.300+0000 7f02d259c640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f02b406c8b0 0x7f02b406ed70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f02c4001780 tx=0x7f02c4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:58.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.522+0000 7f02c2ffd640 1 -- 172.21.15.110:0/28516528 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f02c803e070 con 0x7f02d4072330 2024-01-20T11:20:58.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.522+0000 7f02c2ffd640 10 client.5500.objecter ms_dispatch 0x7f02d4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:58.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.522+0000 7f02c2ffd640 3 client.5500.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:58.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.522+0000 7f02c2ffd640 3 client.5500.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:58.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.523+0000 7f02c2ffd640 20 client.5500.objecter dump_active .. 0 homeless 2024-01-20T11:20:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.534+0000 7f02c2ffd640 1 -- 172.21.15.110:0/28516528 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f02c8077530 con 0x7f02d4072330 2024-01-20T11:20:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.534+0000 7f02c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.534+0000 7f02c2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:58.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.714+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f02d40affe0 con 0x7f02b406c8b0 2024-01-20T11:20:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.716+0000 7f02c2ffd640 1 -- 172.21.15.110:0/28516528 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63820 (secure 0 0 0) 0x7f02c4001420 con 0x7f02b406c8b0 2024-01-20T11:20:58.717 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:58.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 10 librados: watch_flush enter 2024-01-20T11:20:58.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 10 librados: watch_flush exit 2024-01-20T11:20:58.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 20 client.5500.objecter shutdown clearing up homeless session... 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 10 client.5500.objecter successfully canceled tick 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f02b406c8b0 msgr2=0x7f02b406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f02b406c8b0 0x7f02b406ed70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f02c4001780 tx=0x7f02c4009040 comp rx=0 tx=0).stop 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 10 monclient: shutdown 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d4072330 msgr2=0x7f02d406fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d4072330 0x7f02d406fcf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f02d40ae990 tx=0x7f02c8004730 comp rx=0 tx=0).stop 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 shutdown_connections 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f02b406c8b0 0x7f02b406ed70 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02d413ed20 0x7f02d40704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d4072330 0x7f02d406fcf0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 --2- 172.21.15.110:0/28516528 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02d406a860 0x7f02d4138910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 >> 172.21.15.110:0/28516528 conn(0x7f02d40b47d0 msgr2=0x7f02d413b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:58.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 shutdown_connections 2024-01-20T11:20:58.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 -- 172.21.15.110:0/28516528 wait complete. 2024-01-20T11:20:58.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.718+0000 7f02d8bdc640 1 librados: shutdown 2024-01-20T11:20:58.730 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-01-20T11:20:56.776963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002504"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":10.631},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":9.9260000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":7.9660000000000002},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.489000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":7.125},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.904}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.1619999999999999},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":13.772}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.012},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":13.082000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.25},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.808999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313289,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.1760000000000002},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.8460000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.54},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.4589999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.268000000000001},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.959}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.6379999999999999},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.0180000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.632},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":14.321999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.106999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.252000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313289,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.4649999999999999},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.024}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.7929999999999993},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.407}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":7.7969999999999997},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.096}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.683},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":12.563000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":11.904},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.407}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":15.227},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.351000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444103,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":12.881},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":10.048999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.6299999999999999},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":15.073}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":14.305},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.3300000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":8.4019999999999992},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.976000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":11.956},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":7.6120000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":9.2460000000000004},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":15.861000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444104,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.8190000000000008},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":5.4809999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.8440000000000003},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":11.92}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":15.74},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.6669999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":8.2080000000000002},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.683}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":10.445},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":11.206}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":14.412000000000001},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.73}]}]},{"osd":0,"up_from":17,"seq":73014444104,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":9.4079999999999995},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":5.8499999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.468},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":11.464}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":12.398999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":13.689}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.721},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":8.4399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":12.997},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":14.804}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.411},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":9.8620000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.4589999999999996},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.1849999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.3900000000000006},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":13.420999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.353999999999999},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.8219999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.169},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":6.9240000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.957000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":10.682}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":8.1419999999999995},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.906000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:58.731 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:20:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.833+0000 7efdf41f5640 10 monclient: get_monmap_and_config 2024-01-20T11:20:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.833+0000 7efdf41f5640 10 monclient: build_initial_monmap 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.833+0000 7efdf41f5640 10 monclient: monmap: 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:58.834027+0000 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:58.834027+0000 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.833+0000 7efdf41f5640 1 Processor -- start 2024-01-20T11:20:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 1 -- start start 2024-01-20T11:20:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient: init 2024-01-20T11:20:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-01-20T11:20:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efdec0a96f0 0x7efdec13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient(hunting): picked mon.noname-c con 0x7efdec0a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:20:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec13fa10 0x7efdec1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient(hunting): picked mon.noname-b con 0x7efdec13fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efdec0a9ad0 0x7efdec142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient(hunting): picked mon.noname-a con 0x7efdec0a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdec0aaea0 con 0x7efdec0a9ad0 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdec0b25c0 con 0x7efdec0a96f0 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdec0ab0a0 con 0x7efdec13fa10 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.834+0000 7efdf41f5640 10 monclient(hunting): authenticate will time out at 1702.184976s 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.835+0000 7efdf1790640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec13fa10 0x7efdec1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.835+0000 7efdf1f91640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efdec0a96f0 0x7efdec13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.835+0000 7efdf1f91640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efdec0a96f0 0x7efdec13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:47848/0 (socket says 172.21.15.110:47848) 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.835+0000 7efdf1f91640 1 -- 172.21.15.110:0/156187450 learned_addr learned my addr 172.21.15.110:0/156187450 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.835+0000 7efdf1f91640 10 monclient(hunting): get_auth_request con 0x7efdec0a96f0 auth_method 0 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.835+0000 7efdf1f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.835+0000 7efdf1f91640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.835+0000 7efdf1f91640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1790640 10 monclient(hunting): get_auth_request con 0x7efdec13fa10 auth_method 0 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1790640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1790640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1790640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1790640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1790640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.836+0000 7efdf1790640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:58.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 10 monclient(hunting): handle_auth_done global_id 5503 payload 274 2024-01-20T11:20:58.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 1 -- 172.21.15.110:0/156187450 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec13fa10 msgr2=0x7efdec1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 1 --2- 172.21.15.110:0/156187450 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec13fa10 0x7efdec1423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 1 -- 172.21.15.110:0/156187450 >> v2:172.21.15.110:3300/0 conn(0x7efdec0a9ad0 msgr2=0x7efdec142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 1 --2- 172.21.15.110:0/156187450 >> v2:172.21.15.110:3300/0 conn(0x7efdec0a9ad0 0x7efdec142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 10 monclient: _finish_hunting 0 2024-01-20T11:20:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 1 monclient: found mon.noname-c 2024-01-20T11:20:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:20:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.838+0000 7efdf1f91640 1 -- 172.21.15.110:0/156187450 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdec0b3fb0 con 0x7efdec0a96f0 2024-01-20T11:20:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.840+0000 7efdf1f91640 1 --2- 172.21.15.110:0/156187450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec0a96f0 0x7efdec13f4d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efdec0b5720 tx=0x7efde402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a35fdaa746ca0b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:58.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 1 -- 172.21.15.110:0/156187450 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde402c9e0 con 0x7efdec0a96f0 2024-01-20T11:20:58.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 10 monclient: dump: 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 10 monclient: _finish_auth 0 2024-01-20T11:20:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 10 monclient: _check_auth_tickets 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 1 -- 172.21.15.110:0/156187450 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde402cb40 con 0x7efdec0a96f0 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 1 -- 172.21.15.110:0/156187450 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde40343f0 con 0x7efdec0a96f0 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf41f5640 5 monclient: authenticate success, global_id 5503 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf41f5640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf41f5640 4 set_mon_vals no callback set 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.843+0000 7efdf0f8f640 10 monclient: dump: 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:58.846 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 10 monclient: shutdown 2024-01-20T11:20:58.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:58.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 1 -- 172.21.15.110:0/156187450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec0a96f0 msgr2=0x7efdec13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:58.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 1 --2- 172.21.15.110:0/156187450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec0a96f0 0x7efdec13f4d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efdec0b5720 tx=0x7efde402be40 comp rx=0 tx=0).stop 2024-01-20T11:20:58.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 1 -- 172.21.15.110:0/156187450 shutdown_connections 2024-01-20T11:20:58.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 1 --2- 172.21.15.110:0/156187450 >> v2:172.21.15.110:3300/0 conn(0x7efdec0a9ad0 0x7efdec142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 1 --2- 172.21.15.110:0/156187450 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec13fa10 0x7efdec1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 1 --2- 172.21.15.110:0/156187450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec0a96f0 0x7efdec13f4d0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 1 -- 172.21.15.110:0/156187450 >> 172.21.15.110:0/156187450 conn(0x7efdec0b7bc0 msgr2=0x7efdec0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 1 -- 172.21.15.110:0/156187450 shutdown_connections 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.844+0000 7efdf41f5640 1 -- 172.21.15.110:0/156187450 wait complete. 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.845+0000 7efdf41f5640 10 monclient: build_initial_monmap 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.845+0000 7efdf41f5640 10 monclient: monmap: 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.845+0000 7efdf41f5640 1 librados: starting msgr at 2024-01-20T11:20:58.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.845+0000 7efdf41f5640 1 librados: starting objecter 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.845+0000 7efdf41f5640 1 Processor -- start 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.845+0000 7efdf41f5640 1 -- start start 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.845+0000 7efdf41f5640 1 librados: setting wanted keys 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.845+0000 7efdf41f5640 1 librados: calling monclient init 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.845+0000 7efdf41f5640 10 monclient: init 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec0a96f0 0x7efdec0b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient(hunting): picked mon.noname-c con 0x7efdec0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdec0a9ad0 0x7efdec144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient(hunting): picked mon.noname-a con 0x7efdec0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec141930 0x7efdec0af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient(hunting): picked mon.noname-b con 0x7efdec141930 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec1d7430 con 0x7efdec0a9ad0 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec141e50 con 0x7efdec141930 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdec0ad1a0 con 0x7efdec0a96f0 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf41f5640 10 monclient(hunting): authenticate will time out at 1702.196536s 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.846+0000 7efdf1f91640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec0a96f0 0x7efdec0b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf2792640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec141930 0x7efdec0af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf1f91640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec0a96f0 0x7efdec0b06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:39656/0 (socket says 172.21.15.110:39656) 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf1f91640 1 -- 172.21.15.110:0/1936162951 learned_addr learned my addr 172.21.15.110:0/1936162951 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf1f91640 10 monclient(hunting): get_auth_request con 0x7efdec0a96f0 auth_method 0 2024-01-20T11:20:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf1f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf1f91640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf1f91640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf2792640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec141930 0x7efdec0af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:47856/0 (socket says 172.21.15.110:47856) 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf2792640 10 monclient(hunting): get_auth_request con 0x7efdec141930 auth_method 0 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf2792640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf2792640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf2792640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf1f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf1f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf1f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf2792640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf2792640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.847+0000 7efdf2792640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 10 monclient(hunting): handle_auth_done global_id 5506 payload 824 2024-01-20T11:20:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 1 -- 172.21.15.110:0/1936162951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec0a96f0 msgr2=0x7efdec0b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec0a96f0 0x7efdec0b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 1 -- 172.21.15.110:0/1936162951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdec0a9ad0 msgr2=0x7efdec144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdec0a9ad0 0x7efdec144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 10 monclient: _finish_hunting 0 2024-01-20T11:20:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 1 monclient: found mon.noname-b 2024-01-20T11:20:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:20:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.850+0000 7efdf2792640 1 -- 172.21.15.110:0/1936162951 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde4009b30 con 0x7efdec141930 2024-01-20T11:20:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.852+0000 7efdf2792640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec141930 0x7efdec0af910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efde800b3e0 tx=0x7efde800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.852+0000 7efde27fc640 10 client.?.objecter ms_handle_connect 0x7efdec141930 2024-01-20T11:20:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.852+0000 7efde27fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 1 -- 172.21.15.110:0/1936162951 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8011830 con 0x7efdec141930 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 10 monclient: dump: 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 10 monclient: _finish_auth 0 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 1 -- 172.21.15.110:0/1936162951 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde8011990 con 0x7efdec141930 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 1 -- 172.21.15.110:0/1936162951 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8011d80 con 0x7efdec141930 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde27fc640 10 monclient: dump: 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efdf41f5640 5 monclient: authenticate success, global_id 5506 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efdf41f5640 10 monclient: _renew_subs 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efdf41f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efde3fff640 4 set_mon_vals no callback set 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdec0afe50 con 0x7efdec141930 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efdf41f5640 10 client.5506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:58.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efdf41f5640 10 monclient: _renew_subs 2024-01-20T11:20:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.855+0000 7efdf41f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.856+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdec0b0310 con 0x7efdec141930 2024-01-20T11:20:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.856+0000 7efdf41f5640 1 librados: init done 2024-01-20T11:20:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.857+0000 7efdf41f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.857+0000 7efdf41f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.857+0000 7efdf41f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:20:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.857+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdec0b3da0 con 0x7efdec141930 2024-01-20T11:20:58.867 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:49.892 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:58.872 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:49.896 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.875+0000 7efde27fc640 1 -- 172.21.15.110:0/1936162951 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7efde8023430 con 0x7efdec141930 2024-01-20T11:20:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.875+0000 7efde27fc640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd006c8b0 0x7efdd006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:58.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.876+0000 7efdf1f91640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd006c8b0 0x7efdd006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:58.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.876+0000 7efdf1f91640 10 monclient: get_auth_request con 0x7efdd006c8b0 auth_method 0 2024-01-20T11:20:58.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.876+0000 7efdf1f91640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd006c8b0 0x7efdd006ed70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efde402be80 tx=0x7efde4030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:58.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.876+0000 7efde27fc640 10 client.5506.objecter ms_handle_connect 0x7efdd006c8b0 2024-01-20T11:20:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.878+0000 7efde27fc640 1 -- 172.21.15.110:0/1936162951 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efde8058a80 con 0x7efdec141930 2024-01-20T11:20:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.878+0000 7efde27fc640 10 client.5506.objecter ms_dispatch 0x7efdec000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.878+0000 7efde27fc640 3 client.5506.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.878+0000 7efde27fc640 3 client.5506.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:58.878+0000 7efde27fc640 20 client.5506.objecter dump_active .. 0 homeless 2024-01-20T11:20:58.887 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:49.912 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:58.935 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:49.959 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:58.938 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:49.963 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:58.954 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:49.979 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.114+0000 7efde27fc640 1 -- 172.21.15.110:0/1936162951 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7efde80546c0 con 0x7efdec141930 2024-01-20T11:20:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.114+0000 7efde27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:59.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.114+0000 7efde27fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:59.159 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:06:50.165 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:59.161 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:06:50.168 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:59.224 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:06:50.230 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:20:59.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.287+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efdec0b4400 con 0x7efdd006c8b0 2024-01-20T11:20:59.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.290+0000 7efde27fc640 1 -- 172.21.15.110:0/1936162951 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63922 (secure 0 0 0) 0x7efde4009510 con 0x7efdd006c8b0 2024-01-20T11:20:59.290 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:59.290 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:20:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.291+0000 7efdf41f5640 10 librados: watch_flush enter 2024-01-20T11:20:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.291+0000 7efdf41f5640 10 librados: watch_flush exit 2024-01-20T11:20:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.291+0000 7efdf41f5640 20 client.5506.objecter shutdown clearing up homeless session... 2024-01-20T11:20:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.291+0000 7efdf41f5640 10 client.5506.objecter successfully canceled tick 2024-01-20T11:20:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd006c8b0 msgr2=0x7efdd006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd006c8b0 0x7efdd006ed70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efde402be80 tx=0x7efde4030000 comp rx=0 tx=0).stop 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 10 monclient: shutdown 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec141930 msgr2=0x7efdec0af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec141930 0x7efdec0af910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efde800b3e0 tx=0x7efde800b8a0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 shutdown_connections 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd006c8b0 0x7efdd006ed70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdec141930 0x7efdec0af910 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdec0a9ad0 0x7efdec144860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 --2- 172.21.15.110:0/1936162951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdec0a96f0 0x7efdec0b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 >> 172.21.15.110:0/1936162951 conn(0x7efdec0b7bc0 msgr2=0x7efdec1423e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 shutdown_connections 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 -- 172.21.15.110:0/1936162951 wait complete. 2024-01-20T11:20:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.292+0000 7efdf41f5640 1 librados: shutdown 2024-01-20T11:20:59.302 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-01-20T11:20:58.777335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002397"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":3.9180000000000001},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.3639999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6449999999999996},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.9000000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.961},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.917999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.292},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":12.617000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.695},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":9.9540000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.917},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.612}]}]},{"osd":7,"up_from":21,"seq":90194313289,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.1760000000000002},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.8460000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.54},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.4589999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.268000000000001},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.959}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.6379999999999999},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.0180000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.632},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":14.321999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.106999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.252000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313290,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.3490000000000002},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.2000000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":13.385},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.708}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":8.6530000000000005},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.548999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.471},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":7.9770000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":10.042999999999999},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":16.724}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":17.856000000000002},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.053000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444104,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.0410000000000004},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":8.1319999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.2750000000000004},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.8399999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.887},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.683999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.394},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":17.379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":14.813000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.343},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":16.425999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444105,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":9.0370000000000008},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":15.316000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.762},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.6159999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":11.019},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.8049999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":13.606},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.643000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.1609999999999996},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":14.507}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.3710000000000004},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.9}]}]},{"osd":0,"up_from":17,"seq":73014444105,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":4.6429999999999998},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":8.468}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9580000000000002},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":10.718}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.334},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.318}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.092000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.7569999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":13.949999999999999},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":12.554}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.898999999999999},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":10.048999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.4589999999999996},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.1849999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.3900000000000006},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":13.420999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.353999999999999},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.8219999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.169},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":6.9240000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.957000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":10.682}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":8.1419999999999995},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.906000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:20:59.303 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:20:59.303 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:16:52.990307+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:16:52.990307+0000', 'last_peered': '2024-01-20T11:16:52.990307+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:16:52.990307+0000', 'last_undegraded': '2024-01-20T11:16:52.990307+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:20:59.303 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:20:59.303 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:20:59.303 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:20:59.304 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:20:59.304 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:20:59.304 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:20:59.304 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:20:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.404+0000 7fe7ac615640 10 monclient: get_monmap_and_config 2024-01-20T11:20:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.404+0000 7fe7ac615640 10 monclient: build_initial_monmap 2024-01-20T11:20:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.404+0000 7fe7ac615640 10 monclient: monmap: 2024-01-20T11:20:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:20:59.404658+0000 2024-01-20T11:20:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:20:59.404658+0000 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.404+0000 7fe7ac615640 1 Processor -- start 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.404+0000 7fe7ac615640 1 -- start start 2024-01-20T11:20:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.404+0000 7fe7ac615640 10 monclient: init 2024-01-20T11:20:59.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7a40a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe7a40a9ad0 0x7fe7a41423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 10 monclient(hunting): picked mon.noname-a con 0x7fe7a40a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a4142920 0x7fe7a414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 10 monclient(hunting): picked mon.noname-b con 0x7fe7a4142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a40b25c0 con 0x7fe7a40a96f0 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a40aaea0 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a413fbe0 con 0x7fe7a4142920 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7ac615640 10 monclient(hunting): authenticate will time out at 1702.755596s 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.405+0000 7fe7aabb2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a4142920 0x7fe7a414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7a9bb0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe7a40a9ad0 0x7fe7a41423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7aabb2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a4142920 0x7fe7a414c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:39670/0 (socket says 172.21.15.110:39670) 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7aabb2640 1 -- 172.21.15.110:0/2768811926 learned_addr learned my addr 172.21.15.110:0/2768811926 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7aabb2640 10 monclient(hunting): get_auth_request con 0x7fe7a4142920 auth_method 0 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7aabb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7aabb2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7aabb2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7a9bb0640 10 monclient(hunting): get_auth_request con 0x7fe7a40a9ad0 auth_method 0 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7a9bb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7a9bb0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7a9bb0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7aabb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7aabb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:59.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.406+0000 7fe7aabb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:59.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.407+0000 7fe7a9bb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:59.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.407+0000 7fe7a9bb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:59.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.407+0000 7fe7a9bb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7a9bb0640 10 monclient(hunting): handle_auth_done global_id 5442 payload 274 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7a9bb0640 1 -- 172.21.15.110:0/2768811926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a4142920 msgr2=0x7fe7a414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7a9bb0640 1 --2- 172.21.15.110:0/2768811926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a4142920 0x7fe7a414c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7a9bb0640 1 -- 172.21.15.110:0/2768811926 >> v1:172.21.15.134:6789/0 conn(0x7fe7a40a96f0 legacy=0x7fe7a413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7a9bb0640 10 monclient: _finish_hunting 0 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7a9bb0640 1 monclient: found mon.noname-a 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7a9bb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7a9bb0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7a9bb0640 1 -- 172.21.15.110:0/2768811926 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a40ab0a0 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.408+0000 7fe7aabb2640 1 --2- 172.21.15.110:0/2768811926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a4142920 0x7fe7a414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:20:59.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.410+0000 7fe7a9bb0640 1 --2- 172.21.15.110:0/2768811926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 0x7fe7a41423e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe7940097a0 tx=0x7fe79402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae2f44108988d14 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:59.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 1 -- 172.21.15.110:0/2768811926 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79402f050 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:59.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:59.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 10 monclient: dump: 2024-01-20T11:20:59.414 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 10 monclient: _finish_auth 0 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 10 monclient: _check_auth_tickets 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 1 -- 172.21.15.110:0/2768811926 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe794031700 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 1 -- 172.21.15.110:0/2768811926 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe794031a90 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7a93af640 10 monclient: dump: 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7ac615640 5 monclient: authenticate success, global_id 5442 2024-01-20T11:20:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7ac615640 10 monclient: get_monmap_and_config success 2024-01-20T11:20:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.414+0000 7fe7ac615640 4 set_mon_vals no callback set 2024-01-20T11:20:59.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 10 monclient: shutdown 2024-01-20T11:20:59.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:59.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 1 -- 172.21.15.110:0/2768811926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 msgr2=0x7fe7a41423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 1 --2- 172.21.15.110:0/2768811926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 0x7fe7a41423e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe7940097a0 tx=0x7fe79402b7a0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 1 -- 172.21.15.110:0/2768811926 shutdown_connections 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 1 --2- 172.21.15.110:0/2768811926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a4142920 0x7fe7a414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 1 --2- 172.21.15.110:0/2768811926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 0x7fe7a41423e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 1 -- 172.21.15.110:0/2768811926 >> 172.21.15.110:0/2768811926 conn(0x7fe7a40b7bc0 msgr2=0x7fe7a40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 1 -- 172.21.15.110:0/2768811926 shutdown_connections 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.415+0000 7fe7ac615640 1 -- 172.21.15.110:0/2768811926 wait complete. 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.416+0000 7fe7ac615640 10 monclient: build_initial_monmap 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.416+0000 7fe7ac615640 10 monclient: monmap: 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:20:59.420 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.416+0000 7fe7ac615640 1 librados: starting msgr at 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.416+0000 7fe7ac615640 1 librados: starting objecter 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.416+0000 7fe7ac615640 1 Processor -- start 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.416+0000 7fe7ac615640 1 -- start start 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.416+0000 7fe7ac615640 1 librados: setting wanted keys 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.416+0000 7fe7ac615640 1 librados: calling monclient init 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.416+0000 7fe7ac615640 10 monclient: init 2024-01-20T11:20:59.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient: _reopen_session rank -1 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a40a96f0 0x7fe7a4139440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient(hunting): picked mon.noname-c con 0x7fe7a40a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 0x7fe7a41dad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient(hunting): picked mon.noname-a con 0x7fe7a40a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7a4142920 0x7fe7a41db2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient(hunting): picked mon.noname-b con 0x7fe7a4142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a40af990 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a40afb00 con 0x7fe7a4142920 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a40afcc0 con 0x7fe7a40a96f0 2024-01-20T11:20:59.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient(hunting): _renew_subs 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7ac615640 10 monclient(hunting): authenticate will time out at 1702.767487s 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7a9bb0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 0x7fe7a41dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.417+0000 7fe7aa3b1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a40a96f0 0x7fe7a4139440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7a9bb0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 0x7fe7a41dad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:37582/0 (socket says 172.21.15.110:37582) 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7a9bb0640 1 -- 172.21.15.110:0/281367511 learned_addr learned my addr 172.21.15.110:0/281367511 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7a9bb0640 10 monclient(hunting): get_auth_request con 0x7fe7a40a9ad0 auth_method 0 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7a9bb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7a9bb0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7a9bb0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7aa3b1640 10 monclient(hunting): get_auth_request con 0x7fe7a40a96f0 auth_method 0 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7aa3b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7aa3b1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:20:59.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7aa3b1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:20:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7a9bb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7a9bb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7a9bb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7aa3b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:20:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7aa3b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:20:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.418+0000 7fe7aa3b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 10 monclient(hunting): handle_auth_done global_id 5445 payload 824 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 1 -- 172.21.15.110:0/281367511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a40a96f0 msgr2=0x7fe7a4139440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a40a96f0 0x7fe7a4139440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 1 -- 172.21.15.110:0/281367511 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7a4142920 msgr2=0x7fe7a41db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7a4142920 0x7fe7a41db2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 10 monclient: _finish_hunting 0 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 1 monclient: found mon.noname-a 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.421+0000 7fe7a9bb0640 1 -- 172.21.15.110:0/281367511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a00094d0 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.423+0000 7fe7a9bb0640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 0x7fe7a41dad60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe7940098c0 tx=0x7fe7940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.423+0000 7fe79affd640 10 client.?.objecter ms_handle_connect 0x7fe7a40a9ad0 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.423+0000 7fe79affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe79affd640 1 -- 172.21.15.110:0/281367511 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe794033ab0 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe79affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe79affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe79affd640 10 monclient: dump: 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:59.427 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe79affd640 10 monclient: _finish_auth 0 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe79affd640 10 monclient: _check_auth_tickets 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe79affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe7ac615640 5 monclient: authenticate success, global_id 5445 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe7ac615640 10 monclient: _renew_subs 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe7ac615640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe794009400 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe7ac615640 10 client.5445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe7ac615640 10 monclient: _renew_subs 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe7ac615640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a40b0200 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.425+0000 7fe7ac615640 1 librados: init done 2024-01-20T11:20:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.426+0000 7fe7ac615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.426+0000 7fe7ac615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.426+0000 7fe7ac615640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.426+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a411fe10 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.427+0000 7fe79affd640 1 -- 172.21.15.110:0/281367511 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe794033c10 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.427+0000 7fe79affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.427+0000 7fe79affd640 1 -- 172.21.15.110:0/281367511 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79402f020 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.427+0000 7fe79affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.427+0000 7fe79affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.427+0000 7fe79affd640 10 monclient: dump: 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:20:59.430 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:20:59.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.427+0000 7fe7a8bae640 4 set_mon_vals no callback set 2024-01-20T11:20:59.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.667+0000 7fe79affd640 1 -- 172.21.15.110:0/281367511 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe79403a050 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.667+0000 7fe79affd640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe78806c860 0x7fe78806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:20:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.668+0000 7fe7aa3b1640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe78806c860 0x7fe78806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:20:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.668+0000 7fe7aa3b1640 10 monclient: get_auth_request con 0x7fe78806c860 auth_method 0 2024-01-20T11:20:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.668+0000 7fe7aa3b1640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe78806c860 0x7fe78806ed20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe79c001ef0 tx=0x7fe79c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:20:59.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.668+0000 7fe79affd640 10 client.5445.objecter ms_handle_connect 0x7fe78806c860 2024-01-20T11:20:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.695+0000 7fe79affd640 1 -- 172.21.15.110:0/281367511 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe79407a610 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.695+0000 7fe79affd640 10 client.5445.objecter ms_dispatch 0x7fe7a4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:20:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.695+0000 7fe79affd640 3 client.5445.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:20:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.695+0000 7fe79affd640 3 client.5445.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:20:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.695+0000 7fe79affd640 20 client.5445.objecter dump_active .. 0 homeless 2024-01-20T11:20:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.695+0000 7fe79affd640 1 -- 172.21.15.110:0/281367511 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe794076250 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.695+0000 7fe79affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:20:59.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.695+0000 7fe79affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:20:59.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.939+0000 7fe7ac615640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:20:59.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.939+0000 7fe7ac615640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:59.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.939+0000 7fe7ac615640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:20:59.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.939+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fe7a40af6b0 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.944+0000 7fe79affd640 1 -- 172.21.15.110:0/281367511 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fe794038520 con 0x7fe7a40a9ad0 2024-01-20T11:20:59.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.944+0000 7fe79affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:20:59.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.944+0000 7fe79affd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:20:59.944 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:20:59.945 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":394,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:20:59.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 10 librados: watch_flush enter 2024-01-20T11:20:59.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 10 librados: watch_flush exit 2024-01-20T11:20:59.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 20 client.5445.objecter shutdown clearing up homeless session... 2024-01-20T11:20:59.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 10 client.5445.objecter successfully canceled tick 2024-01-20T11:20:59.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe78806c860 msgr2=0x7fe78806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:59.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe78806c860 0x7fe78806ed20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe79c001ef0 tx=0x7fe79c009040 comp rx=0 tx=0).stop 2024-01-20T11:20:59.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 10 monclient: shutdown 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 msgr2=0x7fe7a41dad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 0x7fe7a41dad60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe7940098c0 tx=0x7fe7940046c0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 shutdown_connections 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe78806c860 0x7fe78806ed20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7a4142920 0x7fe7a41db2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7a40a9ad0 0x7fe7a41dad60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 --2- 172.21.15.110:0/281367511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7a40a96f0 0x7fe7a4139440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.946+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 >> 172.21.15.110:0/281367511 conn(0x7fe7a40b7bc0 msgr2=0x7fe7a413b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.947+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 shutdown_connections 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.947+0000 7fe7ac615640 1 -- 172.21.15.110:0/281367511 wait complete. 2024-01-20T11:20:59.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:20:59.947+0000 7fe7ac615640 1 librados: shutdown 2024-01-20T11:20:59.957 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:20:59.957 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.055+0000 7fa9ba4c2640 10 monclient: get_monmap_and_config 2024-01-20T11:21:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.055+0000 7fa9ba4c2640 10 monclient: build_initial_monmap 2024-01-20T11:21:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.055+0000 7fa9ba4c2640 10 monclient: monmap: 2024-01-20T11:21:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:00.056334+0000 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:00.056334+0000 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:00.056 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:00.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.056+0000 7fa9ba4c2640 1 Processor -- start 2024-01-20T11:21:00.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.056+0000 7fa9ba4c2640 1 -- start start 2024-01-20T11:21:00.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.056+0000 7fa9ba4c2640 10 monclient: init 2024-01-20T11:21:00.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:00.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-20T11:21:00.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9b406f920 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:00.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9b406fd00 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b4144060 0x7fa9b414b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 10 monclient(hunting): picked mon.noname-b con 0x7fa9b4144060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b40715b0 con 0x7fa9b406fd00 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b4071890 con 0x7fa9b406f920 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4071b70 con 0x7fa9b4144060 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9ba4c2640 10 monclient(hunting): authenticate will time out at 1703.407458s 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9b8a5f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b4144060 0x7fa9b414b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9b37fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa9b406fd00 0x7fa9b4143950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44726/0 (socket says 172.21.15.110:44726) 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.057+0000 7fa9b37fe640 1 -- 172.21.15.110:0/3231714216 learned_addr learned my addr 172.21.15.110:0/3231714216 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.058+0000 7fa9b8a5f640 10 monclient(hunting): get_auth_request con 0x7fa9b4144060 auth_method 0 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.058+0000 7fa9b8a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.058+0000 7fa9b8a5f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.058+0000 7fa9b8a5f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.058+0000 7fa9b8a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.058+0000 7fa9b8a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.058+0000 7fa9b8a5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b2ffd640 1 -- 172.21.15.110:0/3231714216 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378432292 0 0) 0x7fa9b4071890 con 0x7fa9b406f920 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b2ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b2ffd640 10 monclient(hunting): my global_id is 5509 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b2ffd640 1 -- 172.21.15.110:0/3231714216 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa99c003520 con 0x7fa9b406f920 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b8a5f640 10 monclient(hunting): handle_auth_done global_id 5189 payload 274 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b8a5f640 1 -- 172.21.15.110:0/3231714216 >> v1:172.21.15.134:6789/0 conn(0x7fa9b406f920 legacy=0x7fa9b413d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b8a5f640 1 -- 172.21.15.110:0/3231714216 >> v1:172.21.15.110:6789/0 conn(0x7fa9b406fd00 legacy=0x7fa9b4143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b8a5f640 10 monclient: _finish_hunting 0 2024-01-20T11:21:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b8a5f640 1 monclient: found mon.noname-b 2024-01-20T11:21:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b8a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b8a5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.060+0000 7fa9b8a5f640 1 -- 172.21.15.110:0/3231714216 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b4143430 con 0x7fa9b4144060 2024-01-20T11:21:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.063+0000 7fa9b8a5f640 1 --2- 172.21.15.110:0/3231714216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b4144060 0x7fa9b414b730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009dc0 tx=0x7fa9ac02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc221c2e76860c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:00.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.064+0000 7fa9b2ffd640 1 -- 172.21.15.110:0/3231714216 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac02cc40 con 0x7fa9b4144060 2024-01-20T11:21:00.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.064+0000 7fa9b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:00.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.064+0000 7fa9b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.064+0000 7fa9b2ffd640 10 monclient: dump: 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:00.065 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.064+0000 7fa9b2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:21:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.064+0000 7fa9b2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.064+0000 7fa9b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.065+0000 7fa9ba4c2640 5 monclient: authenticate success, global_id 5189 2024-01-20T11:21:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.065+0000 7fa9ba4c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.066+0000 7fa9b2ffd640 1 -- 172.21.15.110:0/3231714216 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ac02cda0 con 0x7fa9b4144060 2024-01-20T11:21:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.066+0000 7fa9b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.066+0000 7fa9b2ffd640 1 -- 172.21.15.110:0/3231714216 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac034640 con 0x7fa9b4144060 2024-01-20T11:21:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.066+0000 7fa9b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.066+0000 7fa9b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.066+0000 7fa9b2ffd640 10 monclient: dump: 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.066+0000 7fa9ba4c2640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:00.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.066+0000 7fa9ba4c2640 4 set_mon_vals no callback set 2024-01-20T11:21:00.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.067+0000 7fa9ba4c2640 10 monclient: shutdown 2024-01-20T11:21:00.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.067+0000 7fa9ba4c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:00.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.067+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/3231714216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b4144060 msgr2=0x7fa9b414b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:00.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.067+0000 7fa9ba4c2640 1 --2- 172.21.15.110:0/3231714216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b4144060 0x7fa9b414b730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009dc0 tx=0x7fa9ac02be90 comp rx=0 tx=0).stop 2024-01-20T11:21:00.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.067+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/3231714216 shutdown_connections 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.067+0000 7fa9ba4c2640 1 --2- 172.21.15.110:0/3231714216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b4144060 0x7fa9b414b730 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.067+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/3231714216 >> 172.21.15.110:0/3231714216 conn(0x7fa9b4066d60 msgr2=0x7fa9b40b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.067+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/3231714216 shutdown_connections 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.067+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/3231714216 wait complete. 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.068+0000 7fa9ba4c2640 10 monclient: build_initial_monmap 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.068+0000 7fa9ba4c2640 10 monclient: monmap: 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:00.073 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.068+0000 7fa9ba4c2640 1 librados: starting msgr at 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.068+0000 7fa9ba4c2640 1 librados: starting objecter 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.068+0000 7fa9ba4c2640 1 Processor -- start 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 1 -- start start 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 1 librados: setting wanted keys 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 1 librados: calling monclient init 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient: init 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b406f920 0x7fa9b413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient(hunting): picked mon.noname-c con 0x7fa9b406f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:00.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9b406fd00 0x7fa9b4131aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient(hunting): picked mon.noname-a con 0x7fa9b406fd00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9b4144060 0x7fa9b4131fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient(hunting): picked mon.noname-b con 0x7fa9b4144060 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4133dd0 con 0x7fa9b406fd00 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b413fe90 con 0x7fa9b4144060 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4132520 con 0x7fa9b406f920 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.069+0000 7fa9ba4c2640 10 monclient(hunting): authenticate will time out at 1703.419966s 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b406f920 0x7fa9b413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9b406fd00 0x7fa9b4131aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:00.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b8a5f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9b4144060 0x7fa9b4131fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b406f920 0x7fa9b413ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50764/0 (socket says 172.21.15.110:50764) 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b3fff640 1 -- 172.21.15.110:0/287144231 learned_addr learned my addr 172.21.15.110:0/287144231 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b3fff640 10 monclient(hunting): get_auth_request con 0x7fa9b406f920 auth_method 0 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b8a5f640 10 monclient(hunting): get_auth_request con 0x7fa9b4144060 auth_method 0 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b8a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b8a5f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b8a5f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b37fe640 10 monclient(hunting): get_auth_request con 0x7fa9b406fd00 auth_method 0 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.070+0000 7fa9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:00.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.071+0000 7fa9b8a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:00.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.071+0000 7fa9b8a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:00.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.071+0000 7fa9b8a5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:00.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.071+0000 7fa9b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:00.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.071+0000 7fa9b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:00.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.071+0000 7fa9b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:00.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.071+0000 7fa9b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:00.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.071+0000 7fa9b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:00.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.071+0000 7fa9b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.073+0000 7fa9b8a5f640 10 monclient(hunting): handle_auth_done global_id 5512 payload 824 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.073+0000 7fa9b8a5f640 1 -- 172.21.15.110:0/287144231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b406f920 msgr2=0x7fa9b413ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.073+0000 7fa9b8a5f640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b406f920 0x7fa9b413ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.073+0000 7fa9b8a5f640 1 -- 172.21.15.110:0/287144231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9b406fd00 msgr2=0x7fa9b4131aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.073+0000 7fa9b8a5f640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9b406fd00 0x7fa9b4131aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.074+0000 7fa9b8a5f640 10 monclient: _finish_hunting 0 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.074+0000 7fa9b8a5f640 1 monclient: found mon.noname-b 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.074+0000 7fa9b8a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.074+0000 7fa9b8a5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:00.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.074+0000 7fa9b8a5f640 1 -- 172.21.15.110:0/287144231 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac009a70 con 0x7fa9b4144060 2024-01-20T11:21:00.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.076+0000 7fa9b8a5f640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9b4144060 0x7fa9b4131fe0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac033000 tx=0x7fa9ac004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:00.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.076+0000 7fa9b0ff9640 10 client.?.objecter ms_handle_connect 0x7fa9b4144060 2024-01-20T11:21:00.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.076+0000 7fa9b0ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:00.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 1 -- 172.21.15.110:0/287144231 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac004800 con 0x7fa9b4144060 2024-01-20T11:21:00.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:00.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 10 monclient: dump: 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 10 monclient: _finish_auth 0 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:21:00.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 1 -- 172.21.15.110:0/287144231 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ac003e10 con 0x7fa9b4144060 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9ba4c2640 5 monclient: authenticate success, global_id 5512 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9ba4c2640 10 monclient: _renew_subs 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9ba4c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b41400e0 con 0x7fa9b4144060 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 1 -- 172.21.15.110:0/287144231 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac034680 con 0x7fa9b4144060 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b0ff9640 10 monclient: dump: 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:00.082 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9b27fc640 4 set_mon_vals no callback set 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9ba4c2640 10 client.5512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9ba4c2640 10 monclient: _renew_subs 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9ba4c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b4136760 con 0x7fa9b4144060 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.080+0000 7fa9ba4c2640 1 librados: init done 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.081+0000 7fa9ba4c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.081+0000 7fa9ba4c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.081+0000 7fa9ba4c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:00.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.081+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b4075490 con 0x7fa9b4144060 2024-01-20T11:21:00.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.100+0000 7fa9b0ff9640 1 -- 172.21.15.110:0/287144231 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa9ac0347e0 con 0x7fa9b4144060 2024-01-20T11:21:00.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.100+0000 7fa9b0ff9640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa9a006c8b0 0x7fa9a006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:00.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.100+0000 7fa9b3fff640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa9a006c8b0 0x7fa9a006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:00.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.100+0000 7fa9b3fff640 10 monclient: get_auth_request con 0x7fa9a006c8b0 auth_method 0 2024-01-20T11:21:00.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.101+0000 7fa9b0ff9640 10 client.5512.objecter ms_handle_connect 0x7fa9a006c8b0 2024-01-20T11:21:00.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.101+0000 7fa9b3fff640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa9a006c8b0 0x7fa9a006ed70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4007fd0 tx=0x7fa9a4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:00.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.323+0000 7fa9b0ff9640 1 -- 172.21.15.110:0/287144231 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa9ac07aff0 con 0x7fa9b4144060 2024-01-20T11:21:00.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.323+0000 7fa9b0ff9640 10 client.5512.objecter ms_dispatch 0x7fa9b4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:00.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.323+0000 7fa9b0ff9640 3 client.5512.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:00.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.323+0000 7fa9b0ff9640 3 client.5512.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:00.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.323+0000 7fa9b0ff9640 20 client.5512.objecter dump_active .. 0 homeless 2024-01-20T11:21:00.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.334+0000 7fa9b0ff9640 1 -- 172.21.15.110:0/287144231 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa9ac077330 con 0x7fa9b4144060 2024-01-20T11:21:00.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.334+0000 7fa9b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:00.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.334+0000 7fa9b0ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:00.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.513+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9b40b00a0 con 0x7fa9a006c8b0 2024-01-20T11:21:00.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.516+0000 7fa9b0ff9640 1 -- 172.21.15.110:0/287144231 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63922 (secure 0 0 0) 0x7fa9b40b00a0 con 0x7fa9a006c8b0 2024-01-20T11:21:00.516 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:00.517 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:00.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 10 librados: watch_flush enter 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 10 librados: watch_flush exit 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 20 client.5512.objecter shutdown clearing up homeless session... 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 10 client.5512.objecter successfully canceled tick 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa9a006c8b0 msgr2=0x7fa9a006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa9a006c8b0 0x7fa9a006ed70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4007fd0 tx=0x7fa9a4009040 comp rx=0 tx=0).stop 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 10 monclient: shutdown 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9b4144060 msgr2=0x7fa9b4131fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9b4144060 0x7fa9b4131fe0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac033000 tx=0x7fa9ac004a40 comp rx=0 tx=0).stop 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 shutdown_connections 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa9a006c8b0 0x7fa9a006ed70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9b4144060 0x7fa9b4131fe0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:00.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9b406fd00 0x7fa9b4131aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:00.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 --2- 172.21.15.110:0/287144231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9b406f920 0x7fa9b413ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:00.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 >> 172.21.15.110:0/287144231 conn(0x7fa9b4066d60 msgr2=0x7fa9b40b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:00.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 shutdown_connections 2024-01-20T11:21:00.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 -- 172.21.15.110:0/287144231 wait complete. 2024-01-20T11:21:00.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:00.518+0000 7fa9ba4c2640 1 librados: shutdown 2024-01-20T11:21:00.528 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-01-20T11:20:58.777335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002397"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:16:52.990307+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:16:52.990307+0000","last_peered":"2024-01-20T11:16:52.990307+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:16:52.990307+0000","last_undegraded":"2024-01-20T11:16:52.990307+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":3.9180000000000001},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.3639999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6449999999999996},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.9000000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.961},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.917999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.292},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":12.617000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.695},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":9.9540000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.917},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.612}]}]},{"osd":7,"up_from":21,"seq":90194313289,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.1760000000000002},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.8460000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.54},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":8.4589999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.268000000000001},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":10.959}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":9.6379999999999999},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":9.0180000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.632},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":14.321999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.106999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.252000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313290,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":7.3490000000000002},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":5.2000000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":13.385},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":10.708}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":8.6530000000000005},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":15.548999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.471},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":7.9770000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":10.042999999999999},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":16.724}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":17.856000000000002},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.053000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444104,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.0410000000000004},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":8.1319999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.2750000000000004},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.8399999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.887},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.683999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.394},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":17.379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":14.813000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.343},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":16.425999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444105,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":9.0370000000000008},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":15.316000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.762},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.6159999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":11.019},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.8049999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":13.606},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.643000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.1609999999999996},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":14.507}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.3710000000000004},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.9}]}]},{"osd":0,"up_from":17,"seq":73014444105,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":4.6429999999999998},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":8.468}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9580000000000002},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":10.718}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.334},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.318}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.092000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.7569999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":13.949999999999999},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":12.554}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.898999999999999},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":10.048999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.4589999999999996},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.1849999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.3900000000000006},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":13.420999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.353999999999999},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":8.8219999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":14.169},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":6.9240000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.957000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":10.682}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":8.1419999999999995},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":14.906000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:02.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~124s 2024-01-20T11:21:03.531 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.630+0000 7f620d7c9640 10 monclient: get_monmap_and_config 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.630+0000 7f620d7c9640 10 monclient: build_initial_monmap 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.630+0000 7f620d7c9640 10 monclient: monmap: 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:03.631540+0000 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:03.631540+0000 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:03.631 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.631+0000 7f620d7c9640 1 Processor -- start 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.631+0000 7f620d7c9640 1 -- start start 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.631+0000 7f620d7c9640 10 monclient: init 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f620806bb00 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f62080af790 0x7f62080ae410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 10 monclient(hunting): picked mon.noname-c con 0x7f62080af790 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62080ae950 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620806bee0 con 0x7f62080ae950 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62080ada30 con 0x7f620806bb00 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62080add10 con 0x7f62080af790 2024-01-20T11:21:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.632+0000 7f620d7c9640 10 monclient(hunting): authenticate will time out at 1706.982665s 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.633+0000 7f61fffff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f62080af790 0x7f62080ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.633+0000 7f62077fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f62080ae950 0x7f62080aedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44736/0 (socket says 172.21.15.110:44736) 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.633+0000 7f62077fe640 1 -- 172.21.15.110:0/2204878489 learned_addr learned my addr 172.21.15.110:0/2204878489 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.633+0000 7f61fffff640 10 monclient(hunting): get_auth_request con 0x7f62080af790 auth_method 0 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.633+0000 7f61fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.633+0000 7f61fffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.633+0000 7f61fffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.634+0000 7f61fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.634+0000 7f61fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.634+0000 7f61fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:03.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.635+0000 7f62067fc640 1 -- 172.21.15.110:0/2204878489 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352485067 0 0) 0x7f620806bee0 con 0x7f62080ae950 2024-01-20T11:21:03.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.635+0000 7f62067fc640 10 monclient(hunting): _negotiate 2024-01-20T11:21:03.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.635+0000 7f62067fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:03.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.635+0000 7f62067fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:03.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.635+0000 7f62067fc640 10 monclient(hunting): my global_id is 5454 2024-01-20T11:21:03.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.635+0000 7f62067fc640 1 -- 172.21.15.110:0/2204878489 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f4003520 con 0x7f62080ae950 2024-01-20T11:21:03.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.636+0000 7f61fffff640 10 monclient(hunting): handle_auth_done global_id 5515 payload 274 2024-01-20T11:21:03.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.636+0000 7f61fffff640 1 -- 172.21.15.110:0/2204878489 >> v1:172.21.15.134:6789/0 conn(0x7f620806bb00 legacy=0x7f62080b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:03.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.636+0000 7f61fffff640 1 -- 172.21.15.110:0/2204878489 >> v1:172.21.15.110:6789/0 conn(0x7f62080ae950 legacy=0x7f62080aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:03.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.636+0000 7f61fffff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:03.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.636+0000 7f61fffff640 1 monclient: found mon.noname-c 2024-01-20T11:21:03.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.636+0000 7f61fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:03.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.636+0000 7f61fffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:21:03.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.636+0000 7f61fffff640 1 -- 172.21.15.110:0/2204878489 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62080adff0 con 0x7f62080af790 2024-01-20T11:21:03.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.641+0000 7f61fffff640 1 --2- 172.21.15.110:0/2204878489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62080af790 0x7f62080ae410 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009790 tx=0x7f61f002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f59c6748081380e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:03.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 1 -- 172.21.15.110:0/2204878489 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0031830 con 0x7f62080af790 2024-01-20T11:21:03.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:03.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 10 monclient: dump: 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:03.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 1 -- 172.21.15.110:0/2204878489 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f0031990 con 0x7f62080af790 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f620d7c9640 5 monclient: authenticate success, global_id 5515 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f620d7c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f620d7c9640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 1 -- 172.21.15.110:0/2204878489 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0031d20 con 0x7f62080af790 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f620d7c9640 4 set_mon_vals no callback set 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.645+0000 7f62067fc640 10 monclient: dump: 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:03.647 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:03.648 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:03.648 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:03.648 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:03.648 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:03.648 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:03.648 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:03.648 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:03.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.646+0000 7f620d7c9640 10 monclient: shutdown 2024-01-20T11:21:03.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.646+0000 7f620d7c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:03.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.646+0000 7f620d7c9640 1 -- 172.21.15.110:0/2204878489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62080af790 msgr2=0x7f62080ae410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:03.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.646+0000 7f620d7c9640 1 --2- 172.21.15.110:0/2204878489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62080af790 0x7f62080ae410 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009790 tx=0x7f61f002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:21:03.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.646+0000 7f620d7c9640 1 -- 172.21.15.110:0/2204878489 shutdown_connections 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.646+0000 7f620d7c9640 1 --2- 172.21.15.110:0/2204878489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62080af790 0x7f62080ae410 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.646+0000 7f620d7c9640 1 -- 172.21.15.110:0/2204878489 >> 172.21.15.110:0/2204878489 conn(0x7f6208066d60 msgr2=0x7f62080b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.646+0000 7f620d7c9640 1 -- 172.21.15.110:0/2204878489 shutdown_connections 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.646+0000 7f620d7c9640 1 -- 172.21.15.110:0/2204878489 wait complete. 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.647+0000 7f620d7c9640 10 monclient: build_initial_monmap 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.647+0000 7f620d7c9640 10 monclient: monmap: 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:03.652 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.647+0000 7f620d7c9640 1 librados: starting msgr at 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.647+0000 7f620d7c9640 1 librados: starting objecter 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.647+0000 7f620d7c9640 1 Processor -- start 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.647+0000 7f620d7c9640 1 -- start start 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.647+0000 7f620d7c9640 1 librados: setting wanted keys 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.647+0000 7f620d7c9640 1 librados: calling monclient init 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.647+0000 7f620d7c9640 10 monclient: init 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806bb00 0x7f620811fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient(hunting): picked mon.noname-c con 0x7f620806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62080ae950 0x7f6208120920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient(hunting): picked mon.noname-a con 0x7f62080ae950 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:03.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62080af790 0x7f62081d74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient(hunting): picked mon.noname-b con 0x7f62080af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62081d6fc0 con 0x7f62080ae950 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62081221f0 con 0x7f62080af790 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620811d5a0 con 0x7f620806bb00 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.648+0000 7f620d7c9640 10 monclient(hunting): authenticate will time out at 1706.998689s 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62080ae950 0x7f6208120920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f6206ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806bb00 0x7f620811fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f62077fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62080af790 0x7f62081d74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62080ae950 0x7f6208120920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39604/0 (socket says 172.21.15.110:39604) 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 1 -- 172.21.15.110:0/1188864846 learned_addr learned my addr 172.21.15.110:0/1188864846 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:03.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 10 monclient(hunting): get_auth_request con 0x7f62080ae950 auth_method 0 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f6206ffd640 10 monclient(hunting): get_auth_request con 0x7f620806bb00 auth_method 0 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f6206ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f6206ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f6206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f62077fe640 10 monclient(hunting): get_auth_request con 0x7f62080af790 auth_method 0 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f62077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f62077fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f62077fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.649+0000 7f61fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.650+0000 7f62077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.650+0000 7f62077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.650+0000 7f62077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.650+0000 7f6206ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.650+0000 7f6206ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.650+0000 7f6206ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 10 monclient(hunting): handle_auth_done global_id 5457 payload 824 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 1 -- 172.21.15.110:0/1188864846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806bb00 msgr2=0x7f620811fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806bb00 0x7f620811fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 1 -- 172.21.15.110:0/1188864846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62080af790 msgr2=0x7f62081d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62080af790 0x7f62081d74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 1 monclient: found mon.noname-a 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.652+0000 7f61fffff640 1 -- 172.21.15.110:0/1188864846 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f00093a0 con 0x7f62080ae950 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.655+0000 7f61fffff640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62080ae950 0x7f6208120920 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f61f00098b0 tx=0x7f61f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.655+0000 7f61ff7fe640 10 client.?.objecter ms_handle_connect 0x7f62080ae950 2024-01-20T11:21:03.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.655+0000 7f61ff7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:03.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 1 -- 172.21.15.110:0/1188864846 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f003a6f0 con 0x7f62080ae950 2024-01-20T11:21:03.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:03.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:03.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 10 monclient: dump: 2024-01-20T11:21:03.659 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:03.659 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:03.659 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 1 -- 172.21.15.110:0/1188864846 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f003a870 con 0x7f62080ae950 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 1 -- 172.21.15.110:0/1188864846 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f003ac20 con 0x7f62080ae950 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f620d7c9640 5 monclient: authenticate success, global_id 5457 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f620d7c9640 10 monclient: _renew_subs 2024-01-20T11:21:03.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f620d7c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6208120e60 con 0x7f62080ae950 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f6205ffb640 4 set_mon_vals no callback set 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f61ff7fe640 10 monclient: dump: 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:03.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f620d7c9640 10 client.5457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:03.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f620d7c9640 10 monclient: _renew_subs 2024-01-20T11:21:03.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f620d7c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:03.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62081d7bc0 con 0x7f62080ae950 2024-01-20T11:21:03.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.659+0000 7f620d7c9640 1 librados: init done 2024-01-20T11:21:03.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.660+0000 7f620d7c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:03.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.660+0000 7f620d7c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:03.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.660+0000 7f620d7c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:03.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.660+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6208075490 con 0x7f62080ae950 2024-01-20T11:21:03.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.904+0000 7f61ff7fe640 1 -- 172.21.15.110:0/1188864846 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f61f0005990 con 0x7f62080ae950 2024-01-20T11:21:03.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.904+0000 7f61ff7fe640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f61e806c860 0x7f61e806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:03.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.905+0000 7f6206ffd640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f61e806c860 0x7f61e806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:03.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.905+0000 7f6206ffd640 10 monclient: get_auth_request con 0x7f61e806c860 auth_method 0 2024-01-20T11:21:03.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.905+0000 7f6206ffd640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f61e806c860 0x7f61e806ed20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f61f8000f60 tx=0x7f61f8009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:03.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.906+0000 7f61ff7fe640 10 client.5457.objecter ms_handle_connect 0x7f61e806c860 2024-01-20T11:21:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.916+0000 7f61ff7fe640 1 -- 172.21.15.110:0/1188864846 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f61f0079e90 con 0x7f62080ae950 2024-01-20T11:21:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.916+0000 7f61ff7fe640 10 client.5457.objecter ms_dispatch 0x7f6208000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.916+0000 7f61ff7fe640 3 client.5457.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.916+0000 7f61ff7fe640 3 client.5457.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.916+0000 7f61ff7fe640 20 client.5457.objecter dump_active .. 0 homeless 2024-01-20T11:21:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.916+0000 7f61ff7fe640 1 -- 172.21.15.110:0/1188864846 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f61f00761d0 con 0x7f62080ae950 2024-01-20T11:21:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.916+0000 7f61ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:03.916+0000 7f61ff7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:04.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.094+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62080b14e0 con 0x7f61e806c860 2024-01-20T11:21:04.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.097+0000 7f61ff7fe640 1 -- 172.21.15.110:0/1188864846 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63912 (secure 0 0 0) 0x7f62080b14e0 con 0x7f61e806c860 2024-01-20T11:21:04.097 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:04.098 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:04.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 10 librados: watch_flush enter 2024-01-20T11:21:04.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 10 librados: watch_flush exit 2024-01-20T11:21:04.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 20 client.5457.objecter shutdown clearing up homeless session... 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 10 client.5457.objecter successfully canceled tick 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f61e806c860 msgr2=0x7f61e806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f61e806c860 0x7f61e806ed20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f61f8000f60 tx=0x7f61f8009040 comp rx=0 tx=0).stop 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 10 monclient: shutdown 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62080ae950 msgr2=0x7f6208120920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62080ae950 0x7f6208120920 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f61f00098b0 tx=0x7f61f00046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 shutdown_connections 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f61e806c860 0x7f61e806ed20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62080af790 0x7f62081d74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62080ae950 0x7f6208120920 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 --2- 172.21.15.110:0/1188864846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806bb00 0x7f620811fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 >> 172.21.15.110:0/1188864846 conn(0x7f6208066d60 msgr2=0x7f62080b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:04.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 shutdown_connections 2024-01-20T11:21:04.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 -- 172.21.15.110:0/1188864846 wait complete. 2024-01-20T11:21:04.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.099+0000 7f620d7c9640 1 librados: shutdown 2024-01-20T11:21:04.110 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-01-20T11:21:02.778270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002567"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":3.9180000000000001},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.3639999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6449999999999996},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.9000000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.961},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.917999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.292},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":12.617000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.695},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":9.9540000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.917},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.612}]}]},{"osd":7,"up_from":21,"seq":90194313290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.0910000000000002},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.7810000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.7770000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.388999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.8559999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.262}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4269999999999996},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.000999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.390000000000001},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":15.092000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.855},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.603999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313291,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4029999999999996},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.0300000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.8209999999999997},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.237}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.939},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":19.343}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.997},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.486000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":8.2289999999999992},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":17.888999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":14.087},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.789999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444104,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.0410000000000004},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":8.1319999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.2750000000000004},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.8399999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.887},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.683999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.394},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":17.379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":14.813000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.343},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":16.425999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444105,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":9.0370000000000008},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":15.316000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.762},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.6159999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":11.019},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.8049999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":13.606},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.643000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.1609999999999996},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":14.507}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.3710000000000004},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.9}]}]},{"osd":0,"up_from":17,"seq":73014444105,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":4.6429999999999998},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":8.468}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9580000000000002},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":10.718}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.334},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.318}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.092000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.7569999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":13.949999999999999},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":12.554}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.898999999999999},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":10.048999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509514,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":6.4379999999999997},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.0150000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":10.223000000000001},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":8.3620000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.7379999999999995},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.542}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":17.699000000000002},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.528}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.255000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.521999999999998},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":18.591000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:04.111 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:04.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.210+0000 7ff4c8ea9640 10 monclient: get_monmap_and_config 2024-01-20T11:21:04.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.210+0000 7ff4c8ea9640 10 monclient: build_initial_monmap 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.210+0000 7ff4c8ea9640 10 monclient: monmap: 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:04.211411+0000 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:04.211411+0000 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:04.211 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.211+0000 7ff4c8ea9640 1 Processor -- start 2024-01-20T11:21:04.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.211+0000 7ff4c8ea9640 1 -- start start 2024-01-20T11:21:04.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.211+0000 7ff4c8ea9640 10 monclient: init 2024-01-20T11:21:04.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4c406a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff4c413d800 0x7ff4c4143390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c413d800 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c40b0620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c40b10f0 con 0x7ff4c406a860 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c406ac40 con 0x7ff4c413d800 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4072150 con 0x7ff4c40b0620 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c8ea9640 10 monclient(hunting): authenticate will time out at 1707.562505s 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c2d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.212+0000 7ff4c1d9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff4c413d800 0x7ff4c4143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c259c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff4c406a860 0x7ff4c413d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:47376/0 (socket says 172.21.15.110:47376) 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c259c640 1 -- 172.21.15.110:0/714370803 learned_addr learned my addr 172.21.15.110:0/714370803 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c2d9d640 10 monclient(hunting): get_auth_request con 0x7ff4c40b0620 auth_method 0 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c1d9b640 10 monclient(hunting): get_auth_request con 0x7ff4c413d800 auth_method 0 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:04.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:04.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.213+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:04.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_done global_id 5198 payload 274 2024-01-20T11:21:04.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c159a640 1 -- 172.21.15.110:0/714370803 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498021926 0 0) 0x7ff4c40b10f0 con 0x7ff4c406a860 2024-01-20T11:21:04.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c2d9d640 1 -- 172.21.15.110:0/714370803 >> v2:172.21.15.110:3300/0 conn(0x7ff4c413d800 msgr2=0x7ff4c4143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c2d9d640 1 --2- 172.21.15.110:0/714370803 >> v2:172.21.15.110:3300/0 conn(0x7ff4c413d800 0x7ff4c4143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c2d9d640 1 -- 172.21.15.110:0/714370803 >> v1:172.21.15.134:6789/0 conn(0x7ff4c406a860 legacy=0x7ff4c413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c2d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c2d9d640 1 monclient: found mon.noname-b 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c2d9d640 1 -- 172.21.15.110:0/714370803 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4071e70 con 0x7ff4c40b0620 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c1d9b640 1 --2- 172.21.15.110:0/714370803 >> v2:172.21.15.110:3300/0 conn(0x7ff4c413d800 0x7ff4c4143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.215+0000 7ff4c159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:21:04.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.217+0000 7ff4c2d9d640 1 --2- 172.21.15.110:0/714370803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41438d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009d90 tx=0x7ff4bc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5134bad974dca944 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:04.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 1 -- 172.21.15.110:0/714370803 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc02cc50 con 0x7ff4c40b0620 2024-01-20T11:21:04.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:04.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 10 monclient: dump: 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 10 monclient: _finish_auth 0 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 10 monclient: _check_auth_tickets 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 1 -- 172.21.15.110:0/714370803 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc02cdb0 con 0x7ff4c40b0620 2024-01-20T11:21:04.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c8ea9640 5 monclient: authenticate success, global_id 5198 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c8ea9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c8ea9640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c8ea9640 4 set_mon_vals no callback set 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 1 -- 172.21.15.110:0/714370803 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc034660 con 0x7ff4c40b0620 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.220+0000 7ff4c159a640 10 monclient: dump: 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:04.222 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:04.223 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 10 monclient: shutdown 2024-01-20T11:21:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/714370803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 msgr2=0x7ff4c41438d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 1 --2- 172.21.15.110:0/714370803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41438d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009d90 tx=0x7ff4bc02bea0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/714370803 shutdown_connections 2024-01-20T11:21:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 1 --2- 172.21.15.110:0/714370803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41438d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 1 --2- 172.21.15.110:0/714370803 >> v2:172.21.15.110:3300/0 conn(0x7ff4c413d800 0x7ff4c4143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/714370803 >> 172.21.15.110:0/714370803 conn(0x7ff4c40b47d0 msgr2=0x7ff4c40b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:04.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/714370803 shutdown_connections 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.221+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/714370803 wait complete. 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.222+0000 7ff4c8ea9640 10 monclient: build_initial_monmap 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.222+0000 7ff4c8ea9640 10 monclient: monmap: 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 librados: starting msgr at 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 librados: starting objecter 2024-01-20T11:21:04.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 Processor -- start 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 -- start start 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 librados: setting wanted keys 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 librados: calling monclient init 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient: init 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c406a860 0x7ff4c406f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c406a860 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41d6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c40b0620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4c413d800 0x7ff4c41d7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c413d800 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4137e80 con 0x7ff4c406a860 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c406ffd0 con 0x7ff4c413d800 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c41d7980 con 0x7ff4c40b0620 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.223+0000 7ff4c8ea9640 10 monclient(hunting): authenticate will time out at 1707.573910s 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c1d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41d6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c406a860 0x7ff4c406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c2d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4c413d800 0x7ff4c41d7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c1d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41d6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50802/0 (socket says 172.21.15.110:50802) 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c1d9b640 1 -- 172.21.15.110:0/3078331431 learned_addr learned my addr 172.21.15.110:0/3078331431 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c1d9b640 10 monclient(hunting): get_auth_request con 0x7ff4c40b0620 auth_method 0 2024-01-20T11:21:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c259c640 10 monclient(hunting): get_auth_request con 0x7ff4c406a860 auth_method 0 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c2d9d640 10 monclient(hunting): get_auth_request con 0x7ff4c413d800 auth_method 0 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.224+0000 7ff4c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.225+0000 7ff4c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.225+0000 7ff4c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:04.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.225+0000 7ff4c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:04.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.225+0000 7ff4c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:04.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.225+0000 7ff4c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:04.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.225+0000 7ff4c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:04.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.225+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:04.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.225+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:04.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.225+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:04.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 10 monclient(hunting): handle_auth_done global_id 5201 payload 824 2024-01-20T11:21:04.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 1 -- 172.21.15.110:0/3078331431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4c413d800 msgr2=0x7ff4c41d7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4c413d800 0x7ff4c41d7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 1 -- 172.21.15.110:0/3078331431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c406a860 msgr2=0x7ff4c406f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c406a860 0x7ff4c406f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 10 monclient: _finish_hunting 0 2024-01-20T11:21:04.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 1 monclient: found mon.noname-c 2024-01-20T11:21:04.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:04.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:04.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c1d9b640 1 -- 172.21.15.110:0/3078331431 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b8009400 con 0x7ff4c40b0620 2024-01-20T11:21:04.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.227+0000 7ff4c259c640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c406a860 0x7ff4c406f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:04.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.229+0000 7ff4c1d9b640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41d6f00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800e960 tx=0x7ff4b800ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:04.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.229+0000 7ff4b2ffd640 10 client.?.objecter ms_handle_connect 0x7ff4c40b0620 2024-01-20T11:21:04.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.229+0000 7ff4b2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:04.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 1 -- 172.21.15.110:0/3078331431 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b800ca00 con 0x7ff4c40b0620 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 10 monclient: dump: 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:04.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 1 -- 172.21.15.110:0/3078331431 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b800cb60 con 0x7ff4c40b0620 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c8ea9640 5 monclient: authenticate success, global_id 5201 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c8ea9640 10 monclient: _renew_subs 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c8ea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc009a70 con 0x7ff4c40b0620 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 1 -- 172.21.15.110:0/3078331431 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8013470 con 0x7ff4c40b0620 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4b2ffd640 10 monclient: dump: 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.236 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c0d99640 4 set_mon_vals no callback set 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c8ea9640 10 client.5201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c8ea9640 10 monclient: _renew_subs 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c8ea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c406fca0 con 0x7ff4c40b0620 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.233+0000 7ff4c8ea9640 1 librados: init done 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.234+0000 7ff4c8ea9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.234+0000 7ff4c8ea9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.234+0000 7ff4c8ea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:04.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.234+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c4070190 con 0x7ff4c40b0620 2024-01-20T11:21:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.480+0000 7ff4b2ffd640 1 -- 172.21.15.110:0/3078331431 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff4b80135d0 con 0x7ff4c40b0620 2024-01-20T11:21:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.480+0000 7ff4b2ffd640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4a406c8b0 0x7ff4a406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.480+0000 7ff4c259c640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4a406c8b0 0x7ff4a406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.480+0000 7ff4c259c640 10 monclient: get_auth_request con 0x7ff4a406c8b0 auth_method 0 2024-01-20T11:21:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.481+0000 7ff4b2ffd640 10 client.5201.objecter ms_handle_connect 0x7ff4a406c8b0 2024-01-20T11:21:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.481+0000 7ff4c259c640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4a406c8b0 0x7ff4a406ed70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4007f70 tx=0x7ff4b4007f00 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:04.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.496+0000 7ff4b2ffd640 1 -- 172.21.15.110:0/3078331431 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff4b8058bf0 con 0x7ff4c40b0620 2024-01-20T11:21:04.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.496+0000 7ff4b2ffd640 10 client.5201.objecter ms_dispatch 0x7ff4c4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:04.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.496+0000 7ff4b2ffd640 3 client.5201.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:04.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.496+0000 7ff4b2ffd640 3 client.5201.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:04.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.496+0000 7ff4b2ffd640 20 client.5201.objecter dump_active .. 0 homeless 2024-01-20T11:21:04.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.496+0000 7ff4b2ffd640 1 -- 172.21.15.110:0/3078331431 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff4b8054f30 con 0x7ff4c40b0620 2024-01-20T11:21:04.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.496+0000 7ff4b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:04.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.496+0000 7ff4b2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:04.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.668+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4c4143e10 con 0x7ff4a406c8b0 2024-01-20T11:21:04.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.671+0000 7ff4b2ffd640 1 -- 172.21.15.110:0/3078331431 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63912 (secure 0 0 0) 0x7ff4b4001cf0 con 0x7ff4a406c8b0 2024-01-20T11:21:04.671 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:04.672 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:04.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 10 librados: watch_flush enter 2024-01-20T11:21:04.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 10 librados: watch_flush exit 2024-01-20T11:21:04.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 20 client.5201.objecter shutdown clearing up homeless session... 2024-01-20T11:21:04.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 10 client.5201.objecter successfully canceled tick 2024-01-20T11:21:04.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4a406c8b0 msgr2=0x7ff4a406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4a406c8b0 0x7ff4a406ed70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4007f70 tx=0x7ff4b4007f00 comp rx=0 tx=0).stop 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 10 monclient: shutdown 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 msgr2=0x7ff4c41d6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41d6f00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800e960 tx=0x7ff4b800ee20 comp rx=0 tx=0).stop 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 shutdown_connections 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4a406c8b0 0x7ff4a406ed70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4c413d800 0x7ff4c41d7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40b0620 0x7ff4c41d6f00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 --2- 172.21.15.110:0/3078331431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c406a860 0x7ff4c406f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 >> 172.21.15.110:0/3078331431 conn(0x7ff4c40b47d0 msgr2=0x7ff4c40b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 shutdown_connections 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 -- 172.21.15.110:0/3078331431 wait complete. 2024-01-20T11:21:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.673+0000 7ff4c8ea9640 1 librados: shutdown 2024-01-20T11:21:04.683 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-01-20T11:21:02.778270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002567"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":3.9180000000000001},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.3639999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6449999999999996},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":6.9000000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.961},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.917999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":11.292},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":12.617000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.695},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":9.9540000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.917},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.612}]}]},{"osd":7,"up_from":21,"seq":90194313290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.0910000000000002},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.7810000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.7770000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.388999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.8559999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.262}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4269999999999996},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.000999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.390000000000001},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":15.092000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.855},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.603999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313291,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4029999999999996},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.0300000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.8209999999999997},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.237}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.939},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":19.343}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.997},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.486000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":8.2289999999999992},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":17.888999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":14.087},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.789999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444104,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.0410000000000004},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":8.1319999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":5.2750000000000004},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":9.8399999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":13.887},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":10.683999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.394},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":17.379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":14.813000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.343},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":16.425999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444105,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":9.0370000000000008},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":15.316000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.762},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.6159999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":11.019},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":7.8049999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":13.606},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.643000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":6.1609999999999996},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":14.507}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":8.3710000000000004},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":15.9}]}]},{"osd":0,"up_from":17,"seq":73014444105,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":4.6429999999999998},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":8.468}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.9580000000000002},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":10.718}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.334},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.318}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":15.092000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":7.7569999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":13.949999999999999},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":12.554}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.898999999999999},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":10.048999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509514,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":6.4379999999999997},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.0150000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":10.223000000000001},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":8.3620000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.7379999999999995},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.542}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":17.699000000000002},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.528}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.255000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.521999999999998},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":18.591000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:04.685 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:04.685 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:04.685 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:04.685 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:04.685 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:04.685 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:04.685 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:04.685 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:04.686 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:04.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.790+0000 7fd43aea2640 10 monclient: get_monmap_and_config 2024-01-20T11:21:04.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.790+0000 7fd43aea2640 10 monclient: build_initial_monmap 2024-01-20T11:21:04.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.790+0000 7fd43aea2640 10 monclient: monmap: 2024-01-20T11:21:04.790 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:04.790 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.790 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:04.791153+0000 2024-01-20T11:21:04.790 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:04.791153+0000 2024-01-20T11:21:04.790 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:04.791 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.791 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:04.791 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:04.791 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:04.791 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:04.791 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:04.791 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 1 Processor -- start 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 1 -- start start 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 10 monclient: init 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4340a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4340a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd434142af0 0x7fd43414d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 10 monclient(hunting): picked mon.noname-c con 0x7fd434142af0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:04.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.791+0000 7fd43aea2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4340b25c0 con 0x7fd4340a9ad0 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.792+0000 7fd43aea2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd434142f50 con 0x7fd4340a96f0 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.792+0000 7fd43aea2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.792+0000 7fd43aea2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4340aaea0 con 0x7fd434142af0 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.792+0000 7fd43aea2640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.792+0000 7fd43aea2640 10 monclient(hunting): authenticate will time out at 1708.142206s 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.792+0000 7fd43943f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd434142af0 0x7fd43414d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.792+0000 7fd438c3e640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd4340a96f0 0x7fd43413f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:47388/0 (socket says 172.21.15.110:47388) 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.792+0000 7fd438c3e640 1 -- 172.21.15.110:0/113122244 learned_addr learned my addr 172.21.15.110:0/113122244 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.793+0000 7fd43943f640 10 monclient(hunting): get_auth_request con 0x7fd434142af0 auth_method 0 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.793+0000 7fd43943f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.793+0000 7fd43943f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:04.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.793+0000 7fd43943f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:04.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.794+0000 7fd43943f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:04.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.794+0000 7fd43943f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:04.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.794+0000 7fd43943f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.796+0000 7fd43943f640 10 monclient(hunting): handle_auth_done global_id 5530 payload 274 2024-01-20T11:21:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.796+0000 7fd43943f640 1 -- 172.21.15.110:0/113122244 >> v1:172.21.15.134:6789/0 conn(0x7fd4340a96f0 legacy=0x7fd43413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.796+0000 7fd43943f640 1 -- 172.21.15.110:0/113122244 >> v1:172.21.15.110:6789/0 conn(0x7fd4340a9ad0 legacy=0x7fd4341423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.796+0000 7fd43943f640 10 monclient: _finish_hunting 0 2024-01-20T11:21:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.796+0000 7fd43943f640 1 monclient: found mon.noname-c 2024-01-20T11:21:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.796+0000 7fd43943f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.796+0000 7fd43943f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:21:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.796+0000 7fd43943f640 1 -- 172.21.15.110:0/113122244 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43413fbe0 con 0x7fd434142af0 2024-01-20T11:21:04.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.800+0000 7fd43943f640 1 --2- 172.21.15.110:0/113122244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 0x7fd43414d860 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009d90 tx=0x7fd42c02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c56fb7033d1abee server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.803+0000 7fd4337fe640 1 -- 172.21.15.110:0/113122244 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c02cbd0 con 0x7fd434142af0 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.803+0000 7fd4337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.803+0000 7fd4337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.803+0000 7fd4337fe640 10 monclient: dump: 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:04.804 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:04.804 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.803+0000 7fd4337fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:04.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.803+0000 7fd4337fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:04.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.803+0000 7fd4337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:04.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.803+0000 7fd43aea2640 5 monclient: authenticate success, global_id 5530 2024-01-20T11:21:04.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.803+0000 7fd43aea2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.804+0000 7fd4337fe640 1 -- 172.21.15.110:0/113122244 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42c02cd30 con 0x7fd434142af0 2024-01-20T11:21:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.804+0000 7fd4337fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.804+0000 7fd4337fe640 1 -- 172.21.15.110:0/113122244 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c02cea0 con 0x7fd434142af0 2024-01-20T11:21:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.804+0000 7fd4337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.804+0000 7fd4337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.804+0000 7fd4337fe640 10 monclient: dump: 2024-01-20T11:21:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.804+0000 7fd43aea2640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.804+0000 7fd43aea2640 4 set_mon_vals no callback set 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.805+0000 7fd43aea2640 10 monclient: shutdown 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.805+0000 7fd43aea2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.805+0000 7fd43aea2640 1 -- 172.21.15.110:0/113122244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 msgr2=0x7fd43414d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.805+0000 7fd43aea2640 1 --2- 172.21.15.110:0/113122244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 0x7fd43414d860 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009d90 tx=0x7fd42c02be20 comp rx=0 tx=0).stop 2024-01-20T11:21:04.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.805+0000 7fd43aea2640 1 -- 172.21.15.110:0/113122244 shutdown_connections 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.805+0000 7fd43aea2640 1 --2- 172.21.15.110:0/113122244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 0x7fd43414d860 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.805+0000 7fd43aea2640 1 -- 172.21.15.110:0/113122244 >> 172.21.15.110:0/113122244 conn(0x7fd4340b7bc0 msgr2=0x7fd4340b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.805+0000 7fd43aea2640 1 -- 172.21.15.110:0/113122244 shutdown_connections 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.805+0000 7fd43aea2640 1 -- 172.21.15.110:0/113122244 wait complete. 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.806+0000 7fd43aea2640 10 monclient: build_initial_monmap 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.806+0000 7fd43aea2640 10 monclient: monmap: 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.808 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.806+0000 7fd43aea2640 1 librados: starting msgr at 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.806+0000 7fd43aea2640 1 librados: starting objecter 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.806+0000 7fd43aea2640 1 Processor -- start 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.806+0000 7fd43aea2640 1 -- start start 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.806+0000 7fd43aea2640 1 librados: setting wanted keys 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.806+0000 7fd43aea2640 1 librados: calling monclient init 2024-01-20T11:21:04.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.806+0000 7fd43aea2640 10 monclient: init 2024-01-20T11:21:04.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4340a96f0 0x7fd434131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient(hunting): picked mon.noname-c con 0x7fd4340a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340a9ad0 0x7fd434132430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient(hunting): picked mon.noname-a con 0x7fd4340a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 0x7fd43412b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient(hunting): picked mon.noname-b con 0x7fd434142af0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43412fe90 con 0x7fd4340a9ad0 2024-01-20T11:21:04.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd434131480 con 0x7fd434142af0 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd434131760 con 0x7fd4340a96f0 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43aea2640 10 monclient(hunting): authenticate will time out at 1708.157352s 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd433fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340a9ad0 0x7fd434132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd43943f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 0x7fd43412b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.807+0000 7fd438c3e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4340a96f0 0x7fd434131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd433fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340a9ad0 0x7fd434132430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39616/0 (socket says 172.21.15.110:39616) 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd438c3e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4340a96f0 0x7fd434131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50814/0 (socket says 172.21.15.110:50814) 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd433fff640 1 -- 172.21.15.110:0/1722659804 learned_addr learned my addr 172.21.15.110:0/1722659804 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd433fff640 10 monclient(hunting): get_auth_request con 0x7fd4340a9ad0 auth_method 0 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd433fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd433fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd433fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd438c3e640 10 monclient(hunting): get_auth_request con 0x7fd4340a96f0 auth_method 0 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd438c3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd438c3e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd438c3e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd43943f640 10 monclient(hunting): get_auth_request con 0x7fd434142af0 auth_method 0 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd43943f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd43943f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd43943f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd433fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd433fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd433fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd438c3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd438c3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:04.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd438c3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd43943f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd43943f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.808+0000 7fd43943f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 10 monclient(hunting): handle_auth_done global_id 5466 payload 824 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 1 -- 172.21.15.110:0/1722659804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4340a96f0 msgr2=0x7fd434131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4340a96f0 0x7fd434131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 1 -- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 msgr2=0x7fd43412b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 0x7fd43412b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 1 monclient: found mon.noname-a 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd433fff640 1 -- 172.21.15.110:0/1722659804 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c009a70 con 0x7fd4340a9ad0 2024-01-20T11:21:04.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.811+0000 7fd43943f640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 0x7fd43412b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:04.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.813+0000 7fd433fff640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340a9ad0 0x7fd434132430 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd42000c400 tx=0x7fd42000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.813+0000 7fd4317fa640 10 client.?.objecter ms_handle_connect 0x7fd4340a9ad0 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.813+0000 7fd4317fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 1 -- 172.21.15.110:0/1722659804 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4200035f0 con 0x7fd4340a9ad0 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 10 monclient: dump: 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.817 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 10 monclient: _finish_auth 0 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 10 monclient: _check_auth_tickets 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 1 -- 172.21.15.110:0/1722659804 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd420003750 con 0x7fd4340a9ad0 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 1 -- 172.21.15.110:0/1722659804 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd420003b40 con 0x7fd4340a9ad0 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd43aea2640 5 monclient: authenticate success, global_id 5466 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd43aea2640 10 monclient: _renew_subs 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd43aea2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd434130320 con 0x7fd4340a9ad0 2024-01-20T11:21:04.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd4317fa640 10 monclient: dump: 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.816+0000 7fd432ffd640 4 set_mon_vals no callback set 2024-01-20T11:21:04.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.817+0000 7fd43aea2640 10 client.5466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:04.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.817+0000 7fd43aea2640 10 monclient: _renew_subs 2024-01-20T11:21:04.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.817+0000 7fd43aea2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:04.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.817+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd434132970 con 0x7fd4340a9ad0 2024-01-20T11:21:04.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.817+0000 7fd43aea2640 1 librados: init done 2024-01-20T11:21:04.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.817+0000 7fd43aea2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:04.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.817+0000 7fd43aea2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:04.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.817+0000 7fd43aea2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:04.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:04.817+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd434132b60 con 0x7fd4340a9ad0 2024-01-20T11:21:05.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.055+0000 7fd4317fa640 1 -- 172.21.15.110:0/1722659804 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd420003ca0 con 0x7fd4340a9ad0 2024-01-20T11:21:05.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.056+0000 7fd4317fa640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd41806c8b0 0x7fd41806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:05.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.056+0000 7fd438c3e640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd41806c8b0 0x7fd41806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:05.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.056+0000 7fd438c3e640 10 monclient: get_auth_request con 0x7fd41806c8b0 auth_method 0 2024-01-20T11:21:05.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.057+0000 7fd438c3e640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd41806c8b0 0x7fd41806ed70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd428005880 tx=0x7fd428009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:05.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.057+0000 7fd4317fa640 10 client.5466.objecter ms_handle_connect 0x7fd41806c8b0 2024-01-20T11:21:05.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.070+0000 7fd4317fa640 1 -- 172.21.15.110:0/1722659804 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd4200010e0 con 0x7fd4340a9ad0 2024-01-20T11:21:05.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.070+0000 7fd4317fa640 10 client.5466.objecter ms_dispatch 0x7fd434000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:05.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.070+0000 7fd4317fa640 3 client.5466.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:05.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.070+0000 7fd4317fa640 3 client.5466.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:05.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.070+0000 7fd4317fa640 20 client.5466.objecter dump_active .. 0 homeless 2024-01-20T11:21:05.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.070+0000 7fd4317fa640 1 -- 172.21.15.110:0/1722659804 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd420054370 con 0x7fd4340a9ad0 2024-01-20T11:21:05.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.070+0000 7fd4317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:05.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.070+0000 7fd4317fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:05.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.306+0000 7fd43aea2640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:05.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.306+0000 7fd43aea2640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:05.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.306+0000 7fd43aea2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:05.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.306+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd434075490 con 0x7fd4340a9ad0 2024-01-20T11:21:05.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.312+0000 7fd4317fa640 1 -- 172.21.15.110:0/1722659804 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fd420057990 con 0x7fd4340a9ad0 2024-01-20T11:21:05.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.312+0000 7fd4317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:05.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.312+0000 7fd4317fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:05.312 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:05.312 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":399,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 10 librados: watch_flush enter 2024-01-20T11:21:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 10 librados: watch_flush exit 2024-01-20T11:21:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 20 client.5466.objecter shutdown clearing up homeless session... 2024-01-20T11:21:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 10 client.5466.objecter successfully canceled tick 2024-01-20T11:21:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd41806c8b0 msgr2=0x7fd41806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd41806c8b0 0x7fd41806ed70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd428005880 tx=0x7fd428009040 comp rx=0 tx=0).stop 2024-01-20T11:21:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 10 monclient: shutdown 2024-01-20T11:21:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340a9ad0 msgr2=0x7fd434132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340a9ad0 0x7fd434132430 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd42000c400 tx=0x7fd42000c8c0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 shutdown_connections 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd41806c8b0 0x7fd41806ed70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd434142af0 0x7fd43412b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340a9ad0 0x7fd434132430 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 --2- 172.21.15.110:0/1722659804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4340a96f0 0x7fd434131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 >> 172.21.15.110:0/1722659804 conn(0x7fd4340b7bc0 msgr2=0x7fd43413f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 shutdown_connections 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 -- 172.21.15.110:0/1722659804 wait complete. 2024-01-20T11:21:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.314+0000 7fd43aea2640 1 librados: shutdown 2024-01-20T11:21:05.325 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:05.325 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.424+0000 7f6fc95d7640 10 monclient: get_monmap_and_config 2024-01-20T11:21:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.424+0000 7f6fc95d7640 10 monclient: build_initial_monmap 2024-01-20T11:21:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.424+0000 7f6fc95d7640 10 monclient: monmap: 2024-01-20T11:21:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:05.424886+0000 2024-01-20T11:21:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:05.424886+0000 2024-01-20T11:21:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:05.425 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:05.425 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:05.425 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:05.425 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:05.425 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:05.425 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:05.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.424+0000 7f6fc95d7640 1 Processor -- start 2024-01-20T11:21:05.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.424+0000 7f6fc95d7640 1 -- start start 2024-01-20T11:21:05.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.424+0000 7f6fc95d7640 10 monclient: init 2024-01-20T11:21:05.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:05.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-01-20T11:21:05.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6fc406a860 0x7f6fc413f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:05.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 10 monclient(hunting): picked mon.noname-a con 0x7f6fc406a860 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:05.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6fc40af730 0x7f6fc41455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 10 monclient(hunting): picked mon.noname-c con 0x7f6fc40af730 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fc413f990 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc40afc50 con 0x7f6fc413f990 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc406e450 con 0x7f6fc406a860 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc40b0170 con 0x7f6fc40af730 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.425+0000 7f6fc95d7640 10 monclient(hunting): authenticate will time out at 1708.775814s 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.426+0000 7f6fc2ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6fc406a860 0x7f6fc413f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.426+0000 7f6fc2ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6fc406a860 0x7f6fc413f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39630/0 (socket says 172.21.15.110:39630) 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.426+0000 7f6fc2ffd640 1 -- 172.21.15.110:0/1561773756 learned_addr learned my addr 172.21.15.110:0/1561773756 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.426+0000 7f6fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f6fc406a860 auth_method 0 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.426+0000 7f6fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.426+0000 7f6fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.426+0000 7f6fc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.426+0000 7f6fc27fc640 1 --2- 172.21.15.110:0/1561773756 >> v2:172.21.15.134:3300/0 conn(0x7f6fc40af730 0x7f6fc41455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.427+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.427+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.427+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.427+0000 7f6fc27fc640 10 monclient(hunting): get_auth_request con 0x7f6fc40af730 auth_method 0 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.427+0000 7f6fc27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.427+0000 7f6fc27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:05.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.427+0000 7f6fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:05.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.428+0000 7f6fc27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:05.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.428+0000 7f6fc27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:05.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.428+0000 7f6fc27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:05.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.429+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_done global_id 5469 payload 274 2024-01-20T11:21:05.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.429+0000 7f6fc2ffd640 1 -- 172.21.15.110:0/1561773756 >> v2:172.21.15.134:3300/0 conn(0x7f6fc40af730 msgr2=0x7f6fc41455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:05.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.429+0000 7f6fc2ffd640 1 --2- 172.21.15.110:0/1561773756 >> v2:172.21.15.134:3300/0 conn(0x7f6fc40af730 0x7f6fc41455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.429+0000 7f6fc2ffd640 1 -- 172.21.15.110:0/1561773756 >> v1:172.21.15.134:6789/0 conn(0x7f6fc413f990 legacy=0x7f6fc4145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:05.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.429+0000 7f6fc2ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:05.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.429+0000 7f6fc2ffd640 1 monclient: found mon.noname-a 2024-01-20T11:21:05.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.429+0000 7f6fc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:05.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.429+0000 7f6fc2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:05.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.429+0000 7f6fc2ffd640 1 -- 172.21.15.110:0/1561773756 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc406ac40 con 0x7f6fc406a860 2024-01-20T11:21:05.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.431+0000 7f6fc2ffd640 1 --2- 172.21.15.110:0/1561773756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc406a860 0x7f6fc413f450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6fc40b57a0 tx=0x7f6fb402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cae056bf7158cf1 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 1 -- 172.21.15.110:0/1561773756 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb402c9f0 con 0x7f6fc406a860 2024-01-20T11:21:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 10 monclient: dump: 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 10 monclient: _finish_auth 0 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:21:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 1 -- 172.21.15.110:0/1561773756 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb402cb50 con 0x7f6fc406a860 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 1 -- 172.21.15.110:0/1561773756 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb40343f0 con 0x7f6fc406a860 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc1ffb640 10 monclient: dump: 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:05.437 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:05.437 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:05.437 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:05.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc95d7640 5 monclient: authenticate success, global_id 5469 2024-01-20T11:21:05.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc95d7640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:05.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.434+0000 7f6fc95d7640 4 set_mon_vals no callback set 2024-01-20T11:21:05.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 10 monclient: shutdown 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 1 -- 172.21.15.110:0/1561773756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc406a860 msgr2=0x7f6fc413f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 1 --2- 172.21.15.110:0/1561773756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc406a860 0x7f6fc413f450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6fc40b57a0 tx=0x7f6fb402be50 comp rx=0 tx=0).stop 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 1 -- 172.21.15.110:0/1561773756 shutdown_connections 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 1 --2- 172.21.15.110:0/1561773756 >> v2:172.21.15.134:3300/0 conn(0x7f6fc40af730 0x7f6fc41455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 1 --2- 172.21.15.110:0/1561773756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc406a860 0x7f6fc413f450 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 1 -- 172.21.15.110:0/1561773756 >> 172.21.15.110:0/1561773756 conn(0x7f6fc40b47d0 msgr2=0x7f6fc40b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 1 -- 172.21.15.110:0/1561773756 shutdown_connections 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.435+0000 7f6fc95d7640 1 -- 172.21.15.110:0/1561773756 wait complete. 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.436+0000 7f6fc95d7640 10 monclient: build_initial_monmap 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.436+0000 7f6fc95d7640 10 monclient: monmap: 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:05.441 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.436+0000 7f6fc95d7640 1 librados: starting msgr at 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.436+0000 7f6fc95d7640 1 librados: starting objecter 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.436+0000 7f6fc95d7640 1 Processor -- start 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.436+0000 7f6fc95d7640 1 -- start start 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.436+0000 7f6fc95d7640 1 librados: setting wanted keys 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.436+0000 7f6fc95d7640 1 librados: calling monclient init 2024-01-20T11:21:05.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.436+0000 7f6fc95d7640 10 monclient: init 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fc406a860 0x7f6fc41402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient(hunting): picked mon.noname-c con 0x7f6fc406a860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fc40af730 0x7f6fc4135030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient(hunting): picked mon.noname-b con 0x7f6fc40af730 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc413f990 0x7f6fc4142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient(hunting): picked mon.noname-a con 0x7f6fc413f990 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4141f50 con 0x7f6fc413f990 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc41421f0 con 0x7f6fc40af730 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4140830 con 0x7f6fc406a860 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc95d7640 10 monclient(hunting): authenticate will time out at 1708.787670s 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.437+0000 7f6fc37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc413f990 0x7f6fc4142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fc406a860 0x7f6fc41402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fc40af730 0x7f6fc4135030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc413f990 0x7f6fc4142710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39642/0 (socket says 172.21.15.110:39642) 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc37fe640 1 -- 172.21.15.110:0/3932710951 learned_addr learned my addr 172.21.15.110:0/3932710951 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc37fe640 10 monclient(hunting): get_auth_request con 0x7f6fc413f990 auth_method 0 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:05.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f6fc406a860 auth_method 0 2024-01-20T11:21:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc27fc640 10 monclient(hunting): get_auth_request con 0x7f6fc40af730 auth_method 0 2024-01-20T11:21:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.438+0000 7f6fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.439+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:05.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.439+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:05.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.439+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:05.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.439+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:05.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.439+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:05.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.439+0000 7f6fc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:05.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.439+0000 7f6fc27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:05.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.439+0000 7f6fc27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:05.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.439+0000 7f6fc27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:05.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 10 monclient(hunting): handle_auth_done global_id 5472 payload 824 2024-01-20T11:21:05.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 1 -- 172.21.15.110:0/3932710951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fc406a860 msgr2=0x7f6fc41402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:05.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fc406a860 0x7f6fc41402f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 1 -- 172.21.15.110:0/3932710951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fc40af730 msgr2=0x7f6fc4135030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fc40af730 0x7f6fc4135030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 1 monclient: found mon.noname-a 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.442+0000 7f6fc37fe640 1 -- 172.21.15.110:0/3932710951 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb4009b30 con 0x7f6fc413f990 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.445+0000 7f6fc37fe640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc413f990 0x7f6fc4142710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c3a0 tx=0x7f6fbc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.445+0000 7f6fab7fe640 10 client.?.objecter ms_handle_connect 0x7f6fc413f990 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.445+0000 7f6fab7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 1 -- 172.21.15.110:0/3932710951 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0052d0 con 0x7f6fc413f990 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 10 monclient: dump: 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:05.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 1 -- 172.21.15.110:0/3932710951 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc005430 con 0x7f6fc413f990 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 1 -- 172.21.15.110:0/3932710951 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc003830 con 0x7f6fc413f990 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc95d7640 5 monclient: authenticate success, global_id 5472 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc95d7640 10 monclient: _renew_subs 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc95d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc41da560 con 0x7f6fc413f990 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fab7fe640 10 monclient: dump: 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc17fa640 4 set_mon_vals no callback set 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc95d7640 10 client.5472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc95d7640 10 monclient: _renew_subs 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc95d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc4135750 con 0x7f6fc413f990 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.449+0000 7f6fc95d7640 1 librados: init done 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.450+0000 7f6fc95d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.450+0000 7f6fc95d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.450+0000 7f6fc95d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:05.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.450+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc4075490 con 0x7f6fc413f990 2024-01-20T11:21:05.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.471+0000 7f6fab7fe640 1 -- 172.21.15.110:0/3932710951 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6fbc003990 con 0x7f6fc413f990 2024-01-20T11:21:05.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.472+0000 7f6fab7fe640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fb006c8b0 0x7f6fb006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:05.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.472+0000 7f6fc2ffd640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fb006c8b0 0x7f6fb006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:05.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.472+0000 7f6fc2ffd640 10 monclient: get_auth_request con 0x7f6fb006c8b0 auth_method 0 2024-01-20T11:21:05.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.473+0000 7f6fc2ffd640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fb006c8b0 0x7f6fb006ed70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4035000 tx=0x7f6fb4030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:05.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.473+0000 7f6fab7fe640 10 client.5472.objecter ms_handle_connect 0x7f6fb006c8b0 2024-01-20T11:21:05.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.475+0000 7f6fab7fe640 1 -- 172.21.15.110:0/3932710951 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6fbc057cd0 con 0x7f6fc413f990 2024-01-20T11:21:05.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.475+0000 7f6fab7fe640 10 client.5472.objecter ms_dispatch 0x7f6fc4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:05.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.475+0000 7f6fab7fe640 3 client.5472.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:05.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.475+0000 7f6fab7fe640 3 client.5472.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:05.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.475+0000 7f6fab7fe640 20 client.5472.objecter dump_active .. 0 homeless 2024-01-20T11:21:05.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.707+0000 7f6fab7fe640 1 -- 172.21.15.110:0/3932710951 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6fbc054010 con 0x7f6fc413f990 2024-01-20T11:21:05.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.707+0000 7f6fab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:05.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.707+0000 7f6fab7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:05.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.898+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fc4140a50 con 0x7f6fb006c8b0 2024-01-20T11:21:05.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.901+0000 7f6fab7fe640 1 -- 172.21.15.110:0/3932710951 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63899 (secure 0 0 0) 0x7f6fb403f440 con 0x7f6fb006c8b0 2024-01-20T11:21:05.902 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:05.902 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.903+0000 7f6fc95d7640 10 librados: watch_flush enter 2024-01-20T11:21:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.903+0000 7f6fc95d7640 10 librados: watch_flush exit 2024-01-20T11:21:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.903+0000 7f6fc95d7640 20 client.5472.objecter shutdown clearing up homeless session... 2024-01-20T11:21:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 10 client.5472.objecter successfully canceled tick 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fb006c8b0 msgr2=0x7f6fb006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fb006c8b0 0x7f6fb006ed70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4035000 tx=0x7f6fb4030000 comp rx=0 tx=0).stop 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 10 monclient: shutdown 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc413f990 msgr2=0x7f6fc4142710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc413f990 0x7f6fc4142710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c3a0 tx=0x7f6fbc00c860 comp rx=0 tx=0).stop 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 shutdown_connections 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fb006c8b0 0x7f6fb006ed70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fc413f990 0x7f6fc4142710 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fc40af730 0x7f6fc4135030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 --2- 172.21.15.110:0/3932710951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fc406a860 0x7f6fc41402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 >> 172.21.15.110:0/3932710951 conn(0x7f6fc40b47d0 msgr2=0x7f6fc40adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 shutdown_connections 2024-01-20T11:21:05.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 -- 172.21.15.110:0/3932710951 wait complete. 2024-01-20T11:21:05.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:05.904+0000 7f6fc95d7640 1 librados: shutdown 2024-01-20T11:21:05.915 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-01-20T11:21:04.778879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002720"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":17.597999999999999},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":15.567}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.8559999999999999},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":7.9539999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":9.1820000000000004},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":13.526}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":10.584},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":16.452000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":11.5},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":14.206}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":7.1429999999999998},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":12.102}]}]},{"osd":7,"up_from":21,"seq":90194313290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.0910000000000002},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.7810000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.7770000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.388999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.8559999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.262}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4269999999999996},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.000999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.390000000000001},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":15.092000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.855},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.603999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313291,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.4029999999999996},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.0300000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":9.8209999999999997},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.237}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.939},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":19.343}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":12.997},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.486000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":8.2289999999999992},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":17.888999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":14.087},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":16.789999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444105,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":6.0839999999999996},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":7.8970000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":6.7060000000000004},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":14.175000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":15.045},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":8.875}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":15.601000000000001},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":10.221}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":10.801},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.254}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.914999999999999},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":16.683}]}]},{"osd":1,"up_from":17,"seq":73014444106,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0510000000000002},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":13.531000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":14.145},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":9.2059999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":12.721},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":5.2519999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":15.526999999999999},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":10.116}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":8.1549999999999994},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":16.343}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":11.141999999999999},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":17.097999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444106,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":5.7610000000000001},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":7.798}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.2110000000000003},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.287000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":6.4889999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.875999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.555999999999999},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":9.9280000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":7.2270000000000003},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":15.131}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.409000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":11.631}]}]},{"osd":6,"up_from":15,"seq":64424509514,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":6.4379999999999997},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.0150000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":10.223000000000001},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":8.3620000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":9.7379999999999995},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.542}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":17.699000000000002},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":13.528}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":14.255000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":15.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.521999999999998},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":18.591000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:08.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~130s 2024-01-20T11:21:08.917 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.019+0000 7f78ca421640 10 monclient: get_monmap_and_config 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.019+0000 7f78ca421640 10 monclient: build_initial_monmap 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.019+0000 7f78ca421640 10 monclient: monmap: 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:09.020906+0000 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:09.020906+0000 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.020 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:09.021 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.020+0000 7f78ca421640 1 Processor -- start 2024-01-20T11:21:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.020+0000 7f78ca421640 1 -- start start 2024-01-20T11:21:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.020+0000 7f78ca421640 10 monclient: init 2024-01-20T11:21:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.020+0000 7f78ca421640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.020+0000 7f78ca421640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-01-20T11:21:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.020+0000 7f78ca421640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f78c40a96f0 0x7f78c413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.020+0000 7f78ca421640 10 monclient(hunting): picked mon.noname-c con 0x7f78c40a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78c413fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f78c40a9ad0 0x7f78c4142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 10 monclient(hunting): picked mon.noname-a con 0x7f78c40a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40b25c0 con 0x7f78c413fa10 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40aaea0 con 0x7f78c40a9ad0 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40ab0a0 con 0x7f78c40a96f0 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78ca421640 10 monclient(hunting): authenticate will time out at 1712.372250s 2024-01-20T11:21:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.021+0000 7f78c3fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f78c40a96f0 0x7f78c413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c3fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f78c40a96f0 0x7f78c413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:38904/0 (socket says 172.21.15.110:38904) 2024-01-20T11:21:09.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c3fff640 1 -- 172.21.15.110:0/3558704779 learned_addr learned my addr 172.21.15.110:0/3558704779 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:09.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c3fff640 10 monclient(hunting): get_auth_request con 0x7f78c40a96f0 auth_method 0 2024-01-20T11:21:09.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.022+0000 7f78c89be640 1 --2- 172.21.15.110:0/3558704779 >> v2:172.21.15.110:3300/0 conn(0x7f78c40a9ad0 0x7f78c4142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.023+0000 7f78c89be640 10 monclient(hunting): get_auth_request con 0x7f78c40a9ad0 auth_method 0 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c89be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c89be640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:09.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c89be640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:09.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c3fff640 10 monclient(hunting): handle_auth_done global_id 5545 payload 274 2024-01-20T11:21:09.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c3fff640 1 -- 172.21.15.110:0/3558704779 >> v2:172.21.15.110:3300/0 conn(0x7f78c40a9ad0 msgr2=0x7f78c4142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:09.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c3fff640 1 --2- 172.21.15.110:0/3558704779 >> v2:172.21.15.110:3300/0 conn(0x7f78c40a9ad0 0x7f78c4142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c3fff640 1 -- 172.21.15.110:0/3558704779 >> v1:172.21.15.110:6789/0 conn(0x7f78c413fa10 legacy=0x7f78c41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:09.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:09.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c3fff640 1 monclient: found mon.noname-c 2024-01-20T11:21:09.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:09.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:21:09.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.024+0000 7f78c3fff640 1 -- 172.21.15.110:0/3558704779 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c40b3fb0 con 0x7f78c40a96f0 2024-01-20T11:21:09.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.026+0000 7f78c3fff640 1 --2- 172.21.15.110:0/3558704779 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 0x7f78c413f4d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f78c40b5720 tx=0x7f78b402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84cb934d0f3b4f04 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:09.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 1 -- 172.21.15.110:0/3558704779 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40317e0 con 0x7f78c40a96f0 2024-01-20T11:21:09.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:09.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:09.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 10 monclient: dump: 2024-01-20T11:21:09.031 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 1 -- 172.21.15.110:0/3558704779 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b4031940 con 0x7f78c40a96f0 2024-01-20T11:21:09.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 1 -- 172.21.15.110:0/3558704779 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4031cd0 con 0x7f78c40a96f0 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78ca421640 5 monclient: authenticate success, global_id 5545 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78ca421640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78ca421640 4 set_mon_vals no callback set 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.030+0000 7f78c37fe640 10 monclient: dump: 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:09.033 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:09.034 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:09.034 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 10 monclient: shutdown 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 1 -- 172.21.15.110:0/3558704779 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 msgr2=0x7f78c413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 1 --2- 172.21.15.110:0/3558704779 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 0x7f78c413f4d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f78c40b5720 tx=0x7f78b402b880 comp rx=0 tx=0).stop 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 1 -- 172.21.15.110:0/3558704779 shutdown_connections 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 1 --2- 172.21.15.110:0/3558704779 >> v2:172.21.15.110:3300/0 conn(0x7f78c40a9ad0 0x7f78c4142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 1 --2- 172.21.15.110:0/3558704779 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 0x7f78c413f4d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 1 -- 172.21.15.110:0/3558704779 >> 172.21.15.110:0/3558704779 conn(0x7f78c40b7bc0 msgr2=0x7f78c40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 1 -- 172.21.15.110:0/3558704779 shutdown_connections 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.031+0000 7f78ca421640 1 -- 172.21.15.110:0/3558704779 wait complete. 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.032+0000 7f78ca421640 10 monclient: build_initial_monmap 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.032+0000 7f78ca421640 10 monclient: monmap: 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:09.037 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.032+0000 7f78ca421640 1 librados: starting msgr at 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.032+0000 7f78ca421640 1 librados: starting objecter 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.032+0000 7f78ca421640 1 Processor -- start 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.032+0000 7f78ca421640 1 -- start start 2024-01-20T11:21:09.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.032+0000 7f78ca421640 1 librados: setting wanted keys 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.032+0000 7f78ca421640 1 librados: calling monclient init 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.032+0000 7f78ca421640 10 monclient: init 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 0x7f78c413eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient(hunting): picked mon.noname-b con 0x7f78c40a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78c40a9ad0 0x7f78c4134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient(hunting): picked mon.noname-a con 0x7f78c40a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78c4134c20 0x7f78c41da470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient(hunting): picked mon.noname-c con 0x7f78c4134c20 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c4134770 con 0x7f78c40a9ad0 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c41351d0 con 0x7f78c40a96f0 2024-01-20T11:21:09.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c41da9b0 con 0x7f78c4134c20 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78ca421640 10 monclient(hunting): authenticate will time out at 1712.384417s 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78c89be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78c4134c20 0x7f78c41da470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.033+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 0x7f78c413eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c89be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78c4134c20 0x7f78c41da470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50840/0 (socket says 172.21.15.110:50840) 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 0x7f78c413eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:38916/0 (socket says 172.21.15.110:38916) 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c89be640 1 -- 172.21.15.110:0/189959842 learned_addr learned my addr 172.21.15.110:0/189959842 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c89be640 10 monclient(hunting): get_auth_request con 0x7f78c4134c20 auth_method 0 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c89be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c89be640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c89be640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c3fff640 10 monclient(hunting): get_auth_request con 0x7f78c40a96f0 auth_method 0 2024-01-20T11:21:09.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:09.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:09.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:09.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:09.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:09.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:09.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c89be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:09.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c89be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:09.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.034+0000 7f78c89be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 10 monclient(hunting): handle_auth_done global_id 5548 payload 824 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 1 -- 172.21.15.110:0/189959842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78c4134c20 msgr2=0x7f78c41da470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78c4134c20 0x7f78c41da470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 1 -- 172.21.15.110:0/189959842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78c40a9ad0 msgr2=0x7f78c4134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78c40a9ad0 0x7f78c4134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 1 monclient: found mon.noname-b 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.037+0000 7f78c3fff640 1 -- 172.21.15.110:0/189959842 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b40094d0 con 0x7f78c40a96f0 2024-01-20T11:21:09.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.039+0000 7f78c3fff640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 0x7f78c413eb20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f78c41430c0 tx=0x7f78b40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:09.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.039+0000 7f78c17fa640 10 client.?.objecter ms_handle_connect 0x7f78c40a96f0 2024-01-20T11:21:09.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.039+0000 7f78c17fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:09.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 1 -- 172.21.15.110:0/189959842 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4032880 con 0x7f78c40a96f0 2024-01-20T11:21:09.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:09.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 10 monclient: dump: 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 10 monclient: _finish_auth 0 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 10 monclient: _check_auth_tickets 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:09.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 1 -- 172.21.15.110:0/189959842 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b4032a20 con 0x7f78c40a96f0 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78ca421640 5 monclient: authenticate success, global_id 5548 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78ca421640 10 monclient: _renew_subs 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78ca421640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c41dac30 con 0x7f78c40a96f0 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 1 -- 172.21.15.110:0/189959842 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4032dd0 con 0x7f78c40a96f0 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c17fa640 10 monclient: dump: 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.046 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78c2ffd640 4 set_mon_vals no callback set 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78ca421640 10 client.5548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78ca421640 10 monclient: _renew_subs 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78ca421640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c41db0b0 con 0x7f78c40a96f0 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.043+0000 7f78ca421640 1 librados: init done 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.044+0000 7f78ca421640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.044+0000 7f78ca421640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.044+0000 7f78ca421640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:09.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.044+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c4136920 con 0x7f78c40a96f0 2024-01-20T11:21:09.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.280+0000 7f78c17fa640 1 -- 172.21.15.110:0/189959842 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f78b40453f0 con 0x7f78c40a96f0 2024-01-20T11:21:09.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.281+0000 7f78c17fa640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f78a406c8b0 0x7f78a406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.281+0000 7f78bbfff640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f78a406c8b0 0x7f78a406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.281+0000 7f78bbfff640 10 monclient: get_auth_request con 0x7f78a406c8b0 auth_method 0 2024-01-20T11:21:09.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.282+0000 7f78bbfff640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f78a406c8b0 0x7f78a406ed70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f78ac006fd0 tx=0x7f78ac006bc0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:09.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.282+0000 7f78c17fa640 10 client.5548.objecter ms_handle_connect 0x7f78a406c8b0 2024-01-20T11:21:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.284+0000 7f78c17fa640 1 -- 172.21.15.110:0/189959842 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f78b407ac80 con 0x7f78c40a96f0 2024-01-20T11:21:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.284+0000 7f78c17fa640 10 client.5548.objecter ms_dispatch 0x7f78c4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.284+0000 7f78c17fa640 3 client.5548.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.284+0000 7f78c17fa640 3 client.5548.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.284+0000 7f78c17fa640 20 client.5548.objecter dump_active .. 0 homeless 2024-01-20T11:21:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.299+0000 7f78c17fa640 1 -- 172.21.15.110:0/189959842 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f78b40768c0 con 0x7f78c40a96f0 2024-01-20T11:21:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.299+0000 7f78c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.299+0000 7f78c17fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.468+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f78c4136920 con 0x7f78a406c8b0 2024-01-20T11:21:09.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.471+0000 7f78c17fa640 1 -- 172.21.15.110:0/189959842 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63967 (secure 0 0 0) 0x7f78c4136920 con 0x7f78a406c8b0 2024-01-20T11:21:09.472 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:09.473 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.472+0000 7f78ca421640 10 librados: watch_flush enter 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.472+0000 7f78ca421640 10 librados: watch_flush exit 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.472+0000 7f78ca421640 20 client.5548.objecter shutdown clearing up homeless session... 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.472+0000 7f78ca421640 10 client.5548.objecter successfully canceled tick 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f78a406c8b0 msgr2=0x7f78a406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f78a406c8b0 0x7f78a406ed70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f78ac006fd0 tx=0x7f78ac006bc0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 10 monclient: shutdown 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 msgr2=0x7f78c413eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 0x7f78c413eb20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f78c41430c0 tx=0x7f78b40046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 shutdown_connections 2024-01-20T11:21:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f78a406c8b0 0x7f78a406ed70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78c4134c20 0x7f78c41da470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78c40a9ad0 0x7f78c4134230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 --2- 172.21.15.110:0/189959842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78c40a96f0 0x7f78c413eb20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 >> 172.21.15.110:0/189959842 conn(0x7f78c40b7bc0 msgr2=0x7f78c40b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 shutdown_connections 2024-01-20T11:21:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 -- 172.21.15.110:0/189959842 wait complete. 2024-01-20T11:21:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.473+0000 7f78ca421640 1 librados: shutdown 2024-01-20T11:21:09.484 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-01-20T11:21:08.779677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002714"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.0019999999999998},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.5269999999999992},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.8130000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":16.091000000000001},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.398}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.470000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":15.244999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.391999999999999},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":10.792999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":12.131},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":16.920000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313291,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.0910000000000002},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.7810000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.7770000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.388999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.8559999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.262}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4269999999999996},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.000999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.390000000000001},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":15.092000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.855},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.603999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.718},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.0459999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.653},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":15.43}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.068},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.052}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.968},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.3610000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":11.374000000000001},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":16.402999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":17.318000000000001},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.282}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444106,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.4420000000000002},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":6.3239999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.1099999999999994},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.118}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":7.1859999999999999},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.561}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.186999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.097000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":8.7899999999999991}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.478999999999999},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":14.33}]}]},{"osd":1,"up_from":17,"seq":73014444107,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":9.9640000000000004},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":5.8380000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.4390000000000001},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.702999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":12.119999999999999},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.6150000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":15.676},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.894}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.3260000000000005},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":7.2859999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.872},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.59}]}]},{"osd":0,"up_from":17,"seq":73014444107,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":5.883},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":17.379000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.798},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.8399999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":12.744999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.156000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.983000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.132999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":14.954000000000001},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":15.6}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":18.472000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":14.268000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":6.7699999999999996},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":13.348000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.9809999999999999},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.763999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.952999999999999},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.5359999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.109999999999999},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.891}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.672000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.8230000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.4359999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.539}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:09.486 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:09.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.585+0000 7f65fd9c0640 10 monclient: get_monmap_and_config 2024-01-20T11:21:09.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.585+0000 7f65fd9c0640 10 monclient: build_initial_monmap 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.585+0000 7f65fd9c0640 10 monclient: monmap: 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:09.587500+0000 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:09.587500+0000 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:09.587 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.586+0000 7f65fd9c0640 1 Processor -- start 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.586+0000 7f65fd9c0640 1 -- start start 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.586+0000 7f65fd9c0640 10 monclient: init 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65f8145760 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f65f80a9d80 0x7f65f8145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 10 monclient(hunting): picked mon.noname-c con 0x7f65f80a9d80 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f80b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 10 monclient(hunting): picked mon.noname-b con 0x7f65f813f8b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f8146250 con 0x7f65f8145760 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f80ab050 con 0x7f65f80a9d80 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80b20c0 con 0x7f65f813f8b0 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65fd9c0640 10 monclient(hunting): authenticate will time out at 1712.938426s 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65f77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f80b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.587+0000 7f65effff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f65f80a9d80 0x7f65f8145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65f77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f80b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33750/0 (socket says 172.21.15.110:33750) 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65f77fe640 1 -- 172.21.15.110:0/3369733883 learned_addr learned my addr 172.21.15.110:0/3369733883 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65f77fe640 10 monclient(hunting): get_auth_request con 0x7f65f813f8b0 auth_method 0 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65f77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65effff640 10 monclient(hunting): get_auth_request con 0x7f65f80a9d80 auth_method 0 2024-01-20T11:21:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65effff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65effff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.588+0000 7f65f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.589+0000 7f65effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.589+0000 7f65effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.589+0000 7f65effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.590+0000 7f65f77fe640 10 monclient(hunting): handle_auth_done global_id 5213 payload 274 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.590+0000 7f65f77fe640 1 -- 172.21.15.110:0/3369733883 >> v2:172.21.15.134:3300/0 conn(0x7f65f80a9d80 msgr2=0x7f65f8145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.590+0000 7f65f77fe640 1 --2- 172.21.15.110:0/3369733883 >> v2:172.21.15.134:3300/0 conn(0x7f65f80a9d80 0x7f65f8145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.590+0000 7f65f77fe640 1 -- 172.21.15.110:0/3369733883 >> v1:172.21.15.134:6789/0 conn(0x7f65f8145760 legacy=0x7f65f813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.590+0000 7f65f77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.590+0000 7f65f77fe640 1 monclient: found mon.noname-b 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.590+0000 7f65f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.590+0000 7f65f77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.590+0000 7f65f77fe640 1 -- 172.21.15.110:0/3369733883 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f80b2bb0 con 0x7f65f813f8b0 2024-01-20T11:21:09.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.592+0000 7f65f77fe640 1 --2- 172.21.15.110:0/3369733883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f80b2670 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009e50 tx=0x7f65f002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da7b4f834158b71c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:09.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 1 -- 172.21.15.110:0/3369733883 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f002cb00 con 0x7f65f813f8b0 2024-01-20T11:21:09.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:09.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:09.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 10 monclient: dump: 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 1 -- 172.21.15.110:0/3369733883 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f002cc60 con 0x7f65f813f8b0 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65fd9c0640 5 monclient: authenticate success, global_id 5213 2024-01-20T11:21:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65fd9c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65fd9c0640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 1 -- 172.21.15.110:0/3369733883 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f00343f0 con 0x7f65f813f8b0 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65fd9c0640 4 set_mon_vals no callback set 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.595+0000 7f65f67fc640 10 monclient: dump: 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:09.599 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 10 monclient: shutdown 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 1 -- 172.21.15.110:0/3369733883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 msgr2=0x7f65f80b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 1 --2- 172.21.15.110:0/3369733883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f80b2670 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009e50 tx=0x7f65f002bf60 comp rx=0 tx=0).stop 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 1 -- 172.21.15.110:0/3369733883 shutdown_connections 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 1 --2- 172.21.15.110:0/3369733883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f80b2670 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 1 --2- 172.21.15.110:0/3369733883 >> v2:172.21.15.134:3300/0 conn(0x7f65f80a9d80 0x7f65f8145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 1 -- 172.21.15.110:0/3369733883 >> 172.21.15.110:0/3369733883 conn(0x7f65f80663b0 msgr2=0x7f65f80667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 1 -- 172.21.15.110:0/3369733883 shutdown_connections 2024-01-20T11:21:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.596+0000 7f65fd9c0640 1 -- 172.21.15.110:0/3369733883 wait complete. 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.597+0000 7f65fd9c0640 10 monclient: build_initial_monmap 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.597+0000 7f65fd9c0640 10 monclient: monmap: 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.597+0000 7f65fd9c0640 1 librados: starting msgr at 2024-01-20T11:21:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.597+0000 7f65fd9c0640 1 librados: starting objecter 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.597+0000 7f65fd9c0640 1 Processor -- start 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.597+0000 7f65fd9c0640 1 -- start start 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.597+0000 7f65fd9c0640 1 librados: setting wanted keys 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.597+0000 7f65fd9c0640 1 librados: calling monclient init 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.597+0000 7f65fd9c0640 10 monclient: init 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65f80a9d80 0x7f65f80b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient(hunting): picked mon.noname-b con 0x7f65f80a9d80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f8139180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient(hunting): picked mon.noname-c con 0x7f65f813f8b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65f8145760 0x7f65f81dabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient(hunting): picked mon.noname-a con 0x7f65f8145760 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f813a7d0 con 0x7f65f8145760 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80b6500 con 0x7f65f80a9d80 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80b7760 con 0x7f65f813f8b0 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65fd9c0640 10 monclient(hunting): authenticate will time out at 1712.949350s 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65effff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f8139180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65f77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65f8145760 0x7f65f81dabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.598+0000 7f65f6ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65f80a9d80 0x7f65f80b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65effff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f8139180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33764/0 (socket says 172.21.15.110:33764) 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65effff640 1 -- 172.21.15.110:0/614288238 learned_addr learned my addr 172.21.15.110:0/614288238 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65effff640 10 monclient(hunting): get_auth_request con 0x7f65f813f8b0 auth_method 0 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65effff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65effff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f6ffd640 10 monclient(hunting): get_auth_request con 0x7f65f80a9d80 auth_method 0 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f77fe640 10 monclient(hunting): get_auth_request con 0x7f65f8145760 auth_method 0 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.599+0000 7f65f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 10 monclient(hunting): handle_auth_done global_id 5554 payload 824 2024-01-20T11:21:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 1 -- 172.21.15.110:0/614288238 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 msgr2=0x7f65f8139180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f8139180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 1 -- 172.21.15.110:0/614288238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65f8145760 msgr2=0x7f65f81dabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65f8145760 0x7f65f81dabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 1 monclient: found mon.noname-b 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.602+0000 7f65f6ffd640 1 -- 172.21.15.110:0/614288238 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f0009b30 con 0x7f65f80a9d80 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.604+0000 7f65f6ffd640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65f80a9d80 0x7f65f80b5c10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f65e800c390 tx=0x7f65e800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.604+0000 7f65f4ff9640 10 client.?.objecter ms_handle_connect 0x7f65f80a9d80 2024-01-20T11:21:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.604+0000 7f65f4ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:09.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.607+0000 7f65f4ff9640 1 -- 172.21.15.110:0/614288238 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e80052d0 con 0x7f65f80a9d80 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 10 monclient: dump: 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 10 monclient: _finish_auth 0 2024-01-20T11:21:09.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 1 -- 172.21.15.110:0/614288238 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e8005430 con 0x7f65f80a9d80 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65fd9c0640 5 monclient: authenticate success, global_id 5554 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65fd9c0640 10 monclient: _renew_subs 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65fd9c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f80b7a40 con 0x7f65f80a9d80 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 1 -- 172.21.15.110:0/614288238 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e8003830 con 0x7f65f80a9d80 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f4ff9640 10 monclient: dump: 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.611 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65f5ffb640 4 set_mon_vals no callback set 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65fd9c0640 10 client.5554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65fd9c0640 10 monclient: _renew_subs 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65fd9c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f80b6750 con 0x7f65f80a9d80 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.608+0000 7f65fd9c0640 1 librados: init done 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.609+0000 7f65fd9c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.609+0000 7f65fd9c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.609+0000 7f65fd9c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:09.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.609+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f80b7e60 con 0x7f65f80a9d80 2024-01-20T11:21:09.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.626+0000 7f65f4ff9640 1 -- 172.21.15.110:0/614288238 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f65e8003990 con 0x7f65f80a9d80 2024-01-20T11:21:09.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.626+0000 7f65f4ff9640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f65d806c860 0x7f65d806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:09.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.627+0000 7f65effff640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f65d806c860 0x7f65d806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.627+0000 7f65effff640 10 monclient: get_auth_request con 0x7f65d806c860 auth_method 0 2024-01-20T11:21:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.627+0000 7f65f4ff9640 10 client.5554.objecter ms_handle_connect 0x7f65d806c860 2024-01-20T11:21:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.627+0000 7f65effff640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f65d806c860 0x7f65d806ed20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f65f8142ba0 tx=0x7f65e0009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.629+0000 7f65f4ff9640 1 -- 172.21.15.110:0/614288238 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f65e8057990 con 0x7f65f80a9d80 2024-01-20T11:21:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.629+0000 7f65f4ff9640 10 client.5554.objecter ms_dispatch 0x7f65f8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.629+0000 7f65f4ff9640 3 client.5554.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.629+0000 7f65f4ff9640 3 client.5554.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.629+0000 7f65f4ff9640 20 client.5554.objecter dump_active .. 0 homeless 2024-01-20T11:21:09.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.860+0000 7f65f4ff9640 1 -- 172.21.15.110:0/614288238 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f65e8053cd0 con 0x7f65f80a9d80 2024-01-20T11:21:09.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.860+0000 7f65f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:09.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:09.860+0000 7f65f4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:10.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.029+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65f80b7e60 con 0x7f65d806c860 2024-01-20T11:21:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.032+0000 7f65f4ff9640 1 -- 172.21.15.110:0/614288238 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63967 (secure 0 0 0) 0x7f65f80b7e60 con 0x7f65d806c860 2024-01-20T11:21:10.033 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:10.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 10 librados: watch_flush enter 2024-01-20T11:21:10.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 10 librados: watch_flush exit 2024-01-20T11:21:10.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 20 client.5554.objecter shutdown clearing up homeless session... 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 10 client.5554.objecter successfully canceled tick 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f65d806c860 msgr2=0x7f65d806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f65d806c860 0x7f65d806ed20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f65f8142ba0 tx=0x7f65e0009190 comp rx=0 tx=0).stop 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 10 monclient: shutdown 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65f80a9d80 msgr2=0x7f65f80b5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65f80a9d80 0x7f65f80b5c10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f65e800c390 tx=0x7f65e800c850 comp rx=0 tx=0).stop 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 shutdown_connections 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f65d806c860 0x7f65d806ed20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65f8145760 0x7f65f81dabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65f813f8b0 0x7f65f8139180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 --2- 172.21.15.110:0/614288238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65f80a9d80 0x7f65f80b5c10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 >> 172.21.15.110:0/614288238 conn(0x7f65f80663b0 msgr2=0x7f65f813cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:10.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 shutdown_connections 2024-01-20T11:21:10.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 -- 172.21.15.110:0/614288238 wait complete. 2024-01-20T11:21:10.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.034+0000 7f65fd9c0640 1 librados: shutdown 2024-01-20T11:21:10.046 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-01-20T11:21:08.779677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002714"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.0019999999999998},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.5269999999999992},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.8130000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":16.091000000000001},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.398}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.470000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":15.244999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.391999999999999},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":10.792999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":12.131},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":16.920000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313291,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.0910000000000002},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.7810000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.7770000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.388999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.8559999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.262}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4269999999999996},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.000999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.390000000000001},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":15.092000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.855},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.603999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.718},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.0459999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.653},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":15.43}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.068},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.052}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.968},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.3610000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":11.374000000000001},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":16.402999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":17.318000000000001},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.282}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444106,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.4420000000000002},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":6.3239999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.1099999999999994},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.118}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":7.1859999999999999},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.561}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.186999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.097000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":8.7899999999999991}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.478999999999999},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":14.33}]}]},{"osd":1,"up_from":17,"seq":73014444107,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":9.9640000000000004},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":5.8380000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.4390000000000001},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.702999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":12.119999999999999},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.6150000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":15.676},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.894}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.3260000000000005},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":7.2859999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.872},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.59}]}]},{"osd":0,"up_from":17,"seq":73014444107,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":5.883},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":17.379000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.798},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.8399999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":12.744999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.156000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.983000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.132999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":14.954000000000001},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":15.6}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":18.472000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":14.268000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":6.7699999999999996},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":13.348000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.9809999999999999},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.763999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.952999999999999},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.5359999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.109999999999999},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.891}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.672000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.8230000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.4359999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.539}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:10.048 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:10.048 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:10.048 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:10.048 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:10.048 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:10.048 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:10.048 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:10.048 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:10.049 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.152+0000 7f9d4997c640 10 monclient: get_monmap_and_config 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.152+0000 7f9d4997c640 10 monclient: build_initial_monmap 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.152+0000 7f9d4997c640 10 monclient: monmap: 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:10.153694+0000 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:10.153694+0000 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:10.154 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.152+0000 7f9d4997c640 1 Processor -- start 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.152+0000 7f9d4997c640 1 -- start start 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.152+0000 7f9d4997c640 10 monclient: init 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d440a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d440a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d44142af0 0x7f9d4414d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 10 monclient(hunting): picked mon.noname-b con 0x7f9d44142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d440b25c0 con 0x7f9d440a96f0 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d44142f50 con 0x7f9d440a9ad0 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:10.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440aaea0 con 0x7f9d44142af0 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.153+0000 7f9d4997c640 10 monclient(hunting): authenticate will time out at 1713.504599s 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.154+0000 7f9d42ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9d440a96f0 0x7f9d4413f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36840/0 (socket says 172.21.15.110:36840) 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.154+0000 7f9d42ffd640 1 -- 172.21.15.110:0/2244033130 learned_addr learned my addr 172.21.15.110:0/2244033130 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.154+0000 7f9d437fe640 1 --2- 172.21.15.110:0/2244033130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d44142af0 0x7f9d4414d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.154+0000 7f9d437fe640 10 monclient(hunting): get_auth_request con 0x7f9d44142af0 auth_method 0 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.154+0000 7f9d437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.154+0000 7f9d437fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.154+0000 7f9d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.155+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.155+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:10.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.155+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:10.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d41ffb640 1 -- 172.21.15.110:0/2244033130 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269317985 0 0) 0x7f9d38001110 con 0x7f9d440a9ad0 2024-01-20T11:21:10.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d41ffb640 10 monclient(hunting): _negotiate 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d41ffb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d41ffb640 10 monclient(hunting): my global_id is 5557 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d41ffb640 1 -- 172.21.15.110:0/2244033130 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2c003520 con 0x7f9d440a9ad0 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d437fe640 10 monclient(hunting): handle_auth_done global_id 5219 payload 274 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d437fe640 1 -- 172.21.15.110:0/2244033130 >> v1:172.21.15.134:6789/0 conn(0x7f9d440a9ad0 legacy=0x7f9d441423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d437fe640 1 -- 172.21.15.110:0/2244033130 >> v1:172.21.15.110:6789/0 conn(0x7f9d440a96f0 legacy=0x7f9d4413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d437fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d437fe640 1 monclient: found mon.noname-b 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d437fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:10.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.156+0000 7f9d437fe640 1 -- 172.21.15.110:0/2244033130 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4413fbe0 con 0x7f9d44142af0 2024-01-20T11:21:10.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.158+0000 7f9d437fe640 1 --2- 172.21.15.110:0/2244033130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d44142af0 0x7f9d4414d860 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0097d0 tx=0x7f9d3c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fdaf8ea22ad0a26 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:10.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 1 -- 172.21.15.110:0/2244033130 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c031800 con 0x7f9d44142af0 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 10 monclient: dump: 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:10.163 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 10 monclient: _finish_auth 0 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 1 -- 172.21.15.110:0/2244033130 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c031960 con 0x7f9d44142af0 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 1 -- 172.21.15.110:0/2244033130 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c031cf0 con 0x7f9d44142af0 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 5 monclient: authenticate success, global_id 5219 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 4 set_mon_vals no callback set 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:10.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d41ffb640 10 monclient: dump: 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:10.165 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 10 monclient: shutdown 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 1 -- 172.21.15.110:0/2244033130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d44142af0 msgr2=0x7f9d4414d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 1 --2- 172.21.15.110:0/2244033130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d44142af0 0x7f9d4414d860 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0097d0 tx=0x7f9d3c02b8a0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 1 -- 172.21.15.110:0/2244033130 shutdown_connections 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 1 --2- 172.21.15.110:0/2244033130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d44142af0 0x7f9d4414d860 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 1 -- 172.21.15.110:0/2244033130 >> 172.21.15.110:0/2244033130 conn(0x7f9d440b7bc0 msgr2=0x7f9d440b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 1 -- 172.21.15.110:0/2244033130 shutdown_connections 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.162+0000 7f9d4997c640 1 -- 172.21.15.110:0/2244033130 wait complete. 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.163+0000 7f9d4997c640 10 monclient: build_initial_monmap 2024-01-20T11:21:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.163+0000 7f9d4997c640 10 monclient: monmap: 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.163+0000 7f9d4997c640 1 librados: starting msgr at 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.163+0000 7f9d4997c640 1 librados: starting objecter 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.163+0000 7f9d4997c640 1 Processor -- start 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 1 -- start start 2024-01-20T11:21:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 1 librados: setting wanted keys 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 1 librados: calling monclient init 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient: init 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d440a96f0 0x7f9d441d78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient(hunting): picked mon.noname-c con 0x7f9d440a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d440a9ad0 0x7f9d44135dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient(hunting): picked mon.noname-a con 0x7f9d440a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d44142af0 0x7f9d44136300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient(hunting): picked mon.noname-b con 0x7f9d44142af0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44136840 con 0x7f9d440a9ad0 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d441d73a0 con 0x7f9d44142af0 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440ab9d0 con 0x7f9d440a96f0 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.164+0000 7f9d4997c640 10 monclient(hunting): authenticate will time out at 1713.515851s 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d42ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d440a96f0 0x7f9d441d78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d427fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d440a9ad0 0x7f9d44135dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d437fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d44142af0 0x7f9d44136300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d42ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d440a96f0 0x7f9d441d78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33778/0 (socket says 172.21.15.110:33778) 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d42ffd640 1 -- 172.21.15.110:0/1356145324 learned_addr learned my addr 172.21.15.110:0/1356145324 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d42ffd640 10 monclient(hunting): get_auth_request con 0x7f9d440a96f0 auth_method 0 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d42ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d427fc640 10 monclient(hunting): get_auth_request con 0x7f9d440a9ad0 auth_method 0 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:10.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d427fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d427fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d437fe640 10 monclient(hunting): get_auth_request con 0x7f9d44142af0 auth_method 0 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d437fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.165+0000 7f9d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.166+0000 7f9d42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.166+0000 7f9d42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.166+0000 7f9d42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.166+0000 7f9d427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.166+0000 7f9d427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.166+0000 7f9d427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.166+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.166+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:10.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.166+0000 7f9d437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 10 monclient(hunting): handle_auth_done global_id 5560 payload 824 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 1 -- 172.21.15.110:0/1356145324 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d440a96f0 msgr2=0x7f9d441d78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d440a96f0 0x7f9d441d78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 1 -- 172.21.15.110:0/1356145324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d440a9ad0 msgr2=0x7f9d44135dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d440a9ad0 0x7f9d44135dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 1 monclient: found mon.noname-b 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:10.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.168+0000 7f9d437fe640 1 -- 172.21.15.110:0/1356145324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c009400 con 0x7f9d44142af0 2024-01-20T11:21:10.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.170+0000 7f9d437fe640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d44142af0 0x7f9d44136300 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c033000 tx=0x7f9d3c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.170+0000 7f9d2bfff640 10 client.?.objecter ms_handle_connect 0x7f9d44142af0 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.170+0000 7f9d2bfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 1 -- 172.21.15.110:0/1356145324 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c03aca0 con 0x7f9d44142af0 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 10 monclient: dump: 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.177 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 10 monclient: _finish_auth 0 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 1 -- 172.21.15.110:0/1356145324 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c03ae00 con 0x7f9d44142af0 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 1 -- 172.21.15.110:0/1356145324 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c044640 con 0x7f9d44142af0 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d4997c640 5 monclient: authenticate success, global_id 5560 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d2bfff640 10 monclient: dump: 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:10.178 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d4997c640 10 monclient: _renew_subs 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d4997c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d440abb90 con 0x7f9d44142af0 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d417fa640 4 set_mon_vals no callback set 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d4997c640 10 client.5560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d4997c640 10 monclient: _renew_subs 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d4997c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:10.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d44138880 con 0x7f9d44142af0 2024-01-20T11:21:10.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.174+0000 7f9d4997c640 1 librados: init done 2024-01-20T11:21:10.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.175+0000 7f9d4997c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:10.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.175+0000 7f9d4997c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:10.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.175+0000 7f9d4997c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:10.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.175+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d44075490 con 0x7f9d44142af0 2024-01-20T11:21:10.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.192+0000 7f9d2bfff640 1 -- 172.21.15.110:0/1356145324 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9d3c0433a0 con 0x7f9d44142af0 2024-01-20T11:21:10.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.192+0000 7f9d2bfff640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9d3006c890 0x7f9d3006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.193+0000 7f9d42ffd640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9d3006c890 0x7f9d3006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:10.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.193+0000 7f9d42ffd640 10 monclient: get_auth_request con 0x7f9d3006c890 auth_method 0 2024-01-20T11:21:10.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.193+0000 7f9d42ffd640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9d3006c890 0x7f9d3006ed50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001f30 tx=0x7f9d34001ac0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:10.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.193+0000 7f9d2bfff640 10 client.5560.objecter ms_handle_connect 0x7f9d3006c890 2024-01-20T11:21:10.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.195+0000 7f9d2bfff640 1 -- 172.21.15.110:0/1356145324 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9d3c02e070 con 0x7f9d44142af0 2024-01-20T11:21:10.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.195+0000 7f9d2bfff640 10 client.5560.objecter ms_dispatch 0x7f9d44000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:10.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.195+0000 7f9d2bfff640 3 client.5560.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:10.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.195+0000 7f9d2bfff640 3 client.5560.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:10.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.196+0000 7f9d2bfff640 20 client.5560.objecter dump_active .. 0 homeless 2024-01-20T11:21:10.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.433+0000 7f9d2bfff640 1 -- 172.21.15.110:0/1356145324 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9d3c076c60 con 0x7f9d44142af0 2024-01-20T11:21:10.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.433+0000 7f9d2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:10.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.433+0000 7f9d2bfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:10.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.672+0000 7f9d4997c640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:10.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.672+0000 7f9d4997c640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:10.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.672+0000 7f9d4997c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:10.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.672+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f9d440b65a0 con 0x7f9d44142af0 2024-01-20T11:21:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.678+0000 7f9d2bfff640 1 -- 172.21.15.110:0/1356145324 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f9d3c07a280 con 0x7f9d44142af0 2024-01-20T11:21:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.678+0000 7f9d2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.678+0000 7f9d2bfff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:10.679 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:10.679 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":405,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:10.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.679+0000 7f9d4997c640 10 librados: watch_flush enter 2024-01-20T11:21:10.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 10 librados: watch_flush exit 2024-01-20T11:21:10.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 20 client.5560.objecter shutdown clearing up homeless session... 2024-01-20T11:21:10.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 10 client.5560.objecter successfully canceled tick 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9d3006c890 msgr2=0x7f9d3006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9d3006c890 0x7f9d3006ed50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001f30 tx=0x7f9d34001ac0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 10 monclient: shutdown 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d44142af0 msgr2=0x7f9d44136300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d44142af0 0x7f9d44136300 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c033000 tx=0x7f9d3c0046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 shutdown_connections 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9d3006c890 0x7f9d3006ed50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d44142af0 0x7f9d44136300 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d440a9ad0 0x7f9d44135dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 --2- 172.21.15.110:0/1356145324 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d440a96f0 0x7f9d441d78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 >> 172.21.15.110:0/1356145324 conn(0x7f9d440b7bc0 msgr2=0x7f9d440b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 shutdown_connections 2024-01-20T11:21:10.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 -- 172.21.15.110:0/1356145324 wait complete. 2024-01-20T11:21:10.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.680+0000 7f9d4997c640 1 librados: shutdown 2024-01-20T11:21:10.693 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:10.693 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.803+0000 7f45e377d640 10 monclient: get_monmap_and_config 2024-01-20T11:21:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.803+0000 7f45e377d640 10 monclient: build_initial_monmap 2024-01-20T11:21:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.803+0000 7f45e377d640 10 monclient: monmap: 2024-01-20T11:21:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:10.804929+0000 2024-01-20T11:21:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:10.804929+0000 2024-01-20T11:21:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:10.805 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.803+0000 7f45e377d640 1 Processor -- start 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 1 -- start start 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient: init 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient(hunting): picked mon.noname-b con 0x7f45dc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45dc13fa10 0x7f45dc1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient(hunting): picked mon.noname-a con 0x7f45dc13fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45dc0a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc0b25c0 con 0x7f45dc0a9ad0 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45dc0aaea0 con 0x7f45dc13fa10 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0ab0a0 con 0x7f45dc0a96f0 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.804+0000 7f45e377d640 10 monclient(hunting): authenticate will time out at 1714.155887s 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.805+0000 7f45e1519640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.805+0000 7f45e1519640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33792/0 (socket says 172.21.15.110:33792) 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.805+0000 7f45e1519640 1 -- 172.21.15.110:0/4039425461 learned_addr learned my addr 172.21.15.110:0/4039425461 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.805+0000 7f45e1519640 10 monclient(hunting): get_auth_request con 0x7f45dc0a96f0 auth_method 0 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.805+0000 7f45e1519640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.805+0000 7f45e1519640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.805+0000 7f45e1519640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.806+0000 7f45e1519640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.806+0000 7f45e1519640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.806+0000 7f45e1519640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.807+0000 7f45e1519640 10 monclient(hunting): handle_auth_done global_id 5225 payload 274 2024-01-20T11:21:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.807+0000 7f45e1519640 1 -- 172.21.15.110:0/4039425461 >> v2:172.21.15.110:3300/0 conn(0x7f45dc13fa10 msgr2=0x7f45dc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.807+0000 7f45e1519640 1 --2- 172.21.15.110:0/4039425461 >> v2:172.21.15.110:3300/0 conn(0x7f45dc13fa10 0x7f45dc1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.807+0000 7f45e1519640 1 -- 172.21.15.110:0/4039425461 >> v1:172.21.15.110:6789/0 conn(0x7f45dc0a9ad0 legacy=0x7f45dc142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.807+0000 7f45e1519640 10 monclient: _finish_hunting 0 2024-01-20T11:21:10.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.807+0000 7f45e1519640 1 monclient: found mon.noname-b 2024-01-20T11:21:10.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.807+0000 7f45e1519640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:10.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.807+0000 7f45e1519640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:10.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.807+0000 7f45e1519640 1 -- 172.21.15.110:0/4039425461 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc0b3fb0 con 0x7f45dc0a96f0 2024-01-20T11:21:10.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.809+0000 7f45e1519640 1 --2- 172.21.15.110:0/4039425461 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc13f4d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009870 tx=0x7f45d802b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5c92ed0df1ff79b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:10.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.811+0000 7f45d3fff640 1 -- 172.21.15.110:0/4039425461 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d80317e0 con 0x7f45dc0a96f0 2024-01-20T11:21:10.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.811+0000 7f45d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:10.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.811+0000 7f45d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:10.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.811+0000 7f45d3fff640 10 monclient: dump: 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.811+0000 7f45d3fff640 10 monclient: _finish_auth 0 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.811+0000 7f45d3fff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.811+0000 7f45d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.811+0000 7f45e377d640 5 monclient: authenticate success, global_id 5225 2024-01-20T11:21:10.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.811+0000 7f45e377d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:10.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.813+0000 7f45d3fff640 1 -- 172.21.15.110:0/4039425461 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d8031940 con 0x7f45dc0a96f0 2024-01-20T11:21:10.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.813+0000 7f45d3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:10.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.813+0000 7f45d3fff640 1 -- 172.21.15.110:0/4039425461 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d8031cd0 con 0x7f45dc0a96f0 2024-01-20T11:21:10.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.813+0000 7f45d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:10.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.813+0000 7f45d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.813+0000 7f45d3fff640 10 monclient: dump: 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.813+0000 7f45e377d640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.813+0000 7f45e377d640 4 set_mon_vals no callback set 2024-01-20T11:21:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 10 monclient: shutdown 2024-01-20T11:21:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 1 -- 172.21.15.110:0/4039425461 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 msgr2=0x7f45dc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 1 --2- 172.21.15.110:0/4039425461 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc13f4d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009870 tx=0x7f45d802b880 comp rx=0 tx=0).stop 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 1 -- 172.21.15.110:0/4039425461 shutdown_connections 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 1 --2- 172.21.15.110:0/4039425461 >> v2:172.21.15.110:3300/0 conn(0x7f45dc13fa10 0x7f45dc1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 1 --2- 172.21.15.110:0/4039425461 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc13f4d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 1 -- 172.21.15.110:0/4039425461 >> 172.21.15.110:0/4039425461 conn(0x7f45dc0b7bc0 msgr2=0x7f45dc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 1 -- 172.21.15.110:0/4039425461 shutdown_connections 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.814+0000 7f45e377d640 1 -- 172.21.15.110:0/4039425461 wait complete. 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.815+0000 7f45e377d640 10 monclient: build_initial_monmap 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.815+0000 7f45e377d640 10 monclient: monmap: 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.815+0000 7f45e377d640 1 librados: starting msgr at 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.815+0000 7f45e377d640 1 librados: starting objecter 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 Processor -- start 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 -- start start 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 librados: setting wanted keys 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 librados: calling monclient init 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient: init 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc1381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient(hunting): picked mon.noname-c con 0x7f45dc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45dc0a9ad0 0x7f45dc1daeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient(hunting): picked mon.noname-b con 0x7f45dc0a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45dc1460c0 0x7f45dc133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient(hunting): picked mon.noname-a con 0x7f45dc1460c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc139830 con 0x7f45dc1460c0 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1da780 con 0x7f45dc0a9ad0 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1daa20 con 0x7f45dc0a96f0 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.816+0000 7f45e377d640 10 monclient(hunting): authenticate will time out at 1714.167773s 2024-01-20T11:21:10.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc1381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e0d18640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45dc0a9ad0 0x7f45dc1daeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc1381f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33802/0 (socket says 172.21.15.110:33802) 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 1 -- 172.21.15.110:0/2568693781 learned_addr learned my addr 172.21.15.110:0/2568693781 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 10 monclient(hunting): get_auth_request con 0x7f45dc0a96f0 auth_method 0 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e0d18640 10 monclient(hunting): get_auth_request con 0x7f45dc0a9ad0 auth_method 0 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e0d18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e0d18640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e0d18640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.817+0000 7f45e1519640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.818+0000 7f45e0d18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.818+0000 7f45e0d18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.818+0000 7f45e0d18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 10 monclient(hunting): handle_auth_done global_id 5228 payload 824 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 1 -- 172.21.15.110:0/2568693781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45dc0a9ad0 msgr2=0x7f45dc1daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45dc0a9ad0 0x7f45dc1daeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 1 -- 172.21.15.110:0/2568693781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45dc1460c0 msgr2=0x7f45dc133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45dc1460c0 0x7f45dc133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 10 monclient: _finish_hunting 0 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 1 monclient: found mon.noname-c 2024-01-20T11:21:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.820+0000 7f45e1519640 1 -- 172.21.15.110:0/2568693781 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d80094d0 con 0x7f45dc0a96f0 2024-01-20T11:21:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.822+0000 7f45e1519640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc1381f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f45d802b8c0 tx=0x7f45d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.822+0000 7f45d1ffb640 10 client.?.objecter ms_handle_connect 0x7f45dc0a96f0 2024-01-20T11:21:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.822+0000 7f45d1ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:10.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 1 -- 172.21.15.110:0/2568693781 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d8032880 con 0x7f45dc0a96f0 2024-01-20T11:21:10.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:10.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:10.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 10 monclient: dump: 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 10 monclient: _finish_auth 0 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 1 -- 172.21.15.110:0/2568693781 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d8032a00 con 0x7f45dc0a96f0 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45e377d640 5 monclient: authenticate success, global_id 5228 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 1 -- 172.21.15.110:0/2568693781 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d8032db0 con 0x7f45dc0a96f0 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45e377d640 10 monclient: _renew_subs 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45e377d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc1db3f0 con 0x7f45dc0a96f0 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d37fe640 4 set_mon_vals no callback set 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45d1ffb640 10 monclient: dump: 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45e377d640 10 client.5228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45e377d640 10 monclient: _renew_subs 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45e377d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc143300 con 0x7f45dc0a96f0 2024-01-20T11:21:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.826+0000 7f45e377d640 1 librados: init done 2024-01-20T11:21:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.827+0000 7f45e377d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.827+0000 7f45e377d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.827+0000 7f45e377d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.827+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45dc075490 con 0x7f45dc0a96f0 2024-01-20T11:21:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.846+0000 7f45d1ffb640 1 -- 172.21.15.110:0/2568693781 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f45d804e450 con 0x7f45dc0a96f0 2024-01-20T11:21:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.846+0000 7f45d1ffb640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f45c006c8b0 0x7f45c006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.846+0000 7f45e0d18640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f45c006c8b0 0x7f45c006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.846+0000 7f45e0d18640 10 monclient: get_auth_request con 0x7f45c006c8b0 auth_method 0 2024-01-20T11:21:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.847+0000 7f45d1ffb640 10 client.5228.objecter ms_handle_connect 0x7f45c006c8b0 2024-01-20T11:21:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.847+0000 7f45e0d18640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f45c006c8b0 0x7f45c006ed70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f45dc1390a0 tx=0x7f45cc0090e0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.850+0000 7f45d1ffb640 1 -- 172.21.15.110:0/2568693781 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f45d807b1d0 con 0x7f45dc0a96f0 2024-01-20T11:21:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.850+0000 7f45d1ffb640 10 client.5228.objecter ms_dispatch 0x7f45dc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.850+0000 7f45d1ffb640 3 client.5228.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.850+0000 7f45d1ffb640 3 client.5228.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:10.850+0000 7f45d1ffb640 20 client.5228.objecter dump_active .. 0 homeless 2024-01-20T11:21:11.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.087+0000 7f45d1ffb640 1 -- 172.21.15.110:0/2568693781 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f45d8076e10 con 0x7f45dc0a96f0 2024-01-20T11:21:11.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.087+0000 7f45d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:11.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.087+0000 7f45d1ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.318+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f45dc1dab60 con 0x7f45c006c8b0 2024-01-20T11:21:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.321+0000 7f45d1ffb640 1 -- 172.21.15.110:0/2568693781 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63967 (secure 0 0 0) 0x7f45cc009530 con 0x7f45c006c8b0 2024-01-20T11:21:11.322 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:11.323 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 10 librados: watch_flush enter 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 10 librados: watch_flush exit 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 20 client.5228.objecter shutdown clearing up homeless session... 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 10 client.5228.objecter successfully canceled tick 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f45c006c8b0 msgr2=0x7f45c006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f45c006c8b0 0x7f45c006ed70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f45dc1390a0 tx=0x7f45cc0090e0 comp rx=0 tx=0).stop 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 10 monclient: shutdown 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 msgr2=0x7f45dc1381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc1381f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f45d802b8c0 tx=0x7f45d80046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 shutdown_connections 2024-01-20T11:21:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f45c006c8b0 0x7f45c006ed70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45dc1460c0 0x7f45dc133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45dc0a9ad0 0x7f45dc1daeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 --2- 172.21.15.110:0/2568693781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45dc0a96f0 0x7f45dc1381f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 >> 172.21.15.110:0/2568693781 conn(0x7f45dc0b7bc0 msgr2=0x7f45dc13c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 shutdown_connections 2024-01-20T11:21:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 -- 172.21.15.110:0/2568693781 wait complete. 2024-01-20T11:21:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:11.323+0000 7f45e377d640 1 librados: shutdown 2024-01-20T11:21:11.334 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-01-20T11:21:10.780200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002864"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.0019999999999998},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":7.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.5269999999999992},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.8130000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":16.091000000000001},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.398}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":13.470000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":15.244999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.391999999999999},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":10.792999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":12.131},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":16.920000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313291,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.0910000000000002},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.7810000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.7770000000000001},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":12.388999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":7.8559999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.262}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":8.4269999999999996},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":13.000999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":10.390000000000001},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":15.092000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.855},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":11.603999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.718},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.0459999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.653},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":15.43}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.068},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":12.052}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":13.968},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.3610000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":11.374000000000001},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":16.402999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":17.318000000000001},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":13.282}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444106,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":9.4420000000000002},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":6.3239999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":8.1099999999999994},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":10.118}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":7.1859999999999999},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":11.561}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":13.186999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":15.279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":16.097000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":8.7899999999999991}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":12.478999999999999},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":14.33}]}]},{"osd":1,"up_from":17,"seq":73014444107,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":9.9640000000000004},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":5.8380000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.4390000000000001},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":10.702999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":12.119999999999999},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.6150000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":15.676},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.894}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.3260000000000005},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":7.2859999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":12.872},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":14.59}]}]},{"osd":0,"up_from":17,"seq":73014444107,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":5.883},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":17.379000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":6.798},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.8399999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":12.744999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":10.156000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.983000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.132999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":14.954000000000001},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":15.6}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":18.472000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":14.268000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":6.7699999999999996},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":13.348000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":5.9809999999999999},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":10.763999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":13.952999999999999},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":7.5359999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":10.109999999999999},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.891}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":15.672000000000001},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":8.8230000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":9.4359999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":11.539}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:14.336 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:14.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.437+0000 7f3aedbd7640 10 monclient: get_monmap_and_config 2024-01-20T11:21:14.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.437+0000 7f3aedbd7640 10 monclient: build_initial_monmap 2024-01-20T11:21:14.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.437+0000 7f3aedbd7640 10 monclient: monmap: 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:14.439015+0000 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:14.439015+0000 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:14.439 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:14.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 1 Processor -- start 2024-01-20T11:21:14.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 1 -- start start 2024-01-20T11:21:14.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient: init 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ae8066620 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae806df70 0x7f3ae8145440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient(hunting): picked mon.noname-b con 0x7f3ae806df70 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3ae8145980 0x7f3ae814c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient(hunting): picked mon.noname-a con 0x7f3ae8145980 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae80adf80 con 0x7f3ae8066620 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae8072880 con 0x7f3ae8145980 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae806e690 con 0x7f3ae806df70 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:14.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.438+0000 7f3aedbd7640 10 monclient(hunting): authenticate will time out at 1717.789943s 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae806df70 0x7f3ae8145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3ae8145980 0x7f3ae814c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae77fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f3ae8066620 0x7f3ae813bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:41684/0 (socket says 172.21.15.110:41684) 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae77fe640 1 -- 172.21.15.110:0/1230713720 learned_addr learned my addr 172.21.15.110:0/1230713720 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae7fff640 10 monclient(hunting): get_auth_request con 0x7f3ae8145980 auth_method 0 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae6ffd640 10 monclient(hunting): get_auth_request con 0x7f3ae806df70 auth_method 0 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.439+0000 7f3ae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.440+0000 7f3ae7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:14.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.440+0000 7f3ae7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:14.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.440+0000 7f3ae7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:14.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.440+0000 7f3ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:14.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.440+0000 7f3ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:14.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.440+0000 7f3ae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:14.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.441+0000 7f3ae6ffd640 10 monclient(hunting): handle_auth_done global_id 5231 payload 274 2024-01-20T11:21:14.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.441+0000 7f3ae6ffd640 1 -- 172.21.15.110:0/1230713720 >> v2:172.21.15.110:3300/0 conn(0x7f3ae8145980 msgr2=0x7f3ae814c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:14.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.441+0000 7f3ae6ffd640 1 --2- 172.21.15.110:0/1230713720 >> v2:172.21.15.110:3300/0 conn(0x7f3ae8145980 0x7f3ae814c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.441+0000 7f3ae6ffd640 1 -- 172.21.15.110:0/1230713720 >> v1:172.21.15.134:6789/0 conn(0x7f3ae8066620 legacy=0x7f3ae813bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:14.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.441+0000 7f3ae6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:14.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.441+0000 7f3ae6ffd640 1 monclient: found mon.noname-b 2024-01-20T11:21:14.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.441+0000 7f3ae6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:14.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.441+0000 7f3ae6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:14.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.441+0000 7f3ae6ffd640 1 -- 172.21.15.110:0/1230713720 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae806f650 con 0x7f3ae806df70 2024-01-20T11:21:14.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.443+0000 7f3ae6ffd640 1 --2- 172.21.15.110:0/1230713720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae806df70 0x7f3ae8145440 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3adc0097a0 tx=0x7f3adc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d9c6b3aba402b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 1 -- 172.21.15.110:0/1230713720 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc02e070 con 0x7f3ae806df70 2024-01-20T11:21:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 10 monclient: dump: 2024-01-20T11:21:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 1 -- 172.21.15.110:0/1230713720 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc0315a0 con 0x7f3ae806df70 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 1 -- 172.21.15.110:0/1230713720 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc031930 con 0x7f3ae806df70 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3aedbd7640 5 monclient: authenticate success, global_id 5231 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3aedbd7640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3aedbd7640 4 set_mon_vals no callback set 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3ae67fc640 10 monclient: dump: 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:14.450 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3aedbd7640 10 monclient: shutdown 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3aedbd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3aedbd7640 1 -- 172.21.15.110:0/1230713720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae806df70 msgr2=0x7f3ae8145440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.447+0000 7f3aedbd7640 1 --2- 172.21.15.110:0/1230713720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae806df70 0x7f3ae8145440 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3adc0097a0 tx=0x7f3adc02b840 comp rx=0 tx=0).stop 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.448+0000 7f3aedbd7640 1 -- 172.21.15.110:0/1230713720 shutdown_connections 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.448+0000 7f3aedbd7640 1 --2- 172.21.15.110:0/1230713720 >> v2:172.21.15.110:3300/0 conn(0x7f3ae8145980 0x7f3ae814c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.448+0000 7f3aedbd7640 1 --2- 172.21.15.110:0/1230713720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae806df70 0x7f3ae8145440 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.448+0000 7f3aedbd7640 1 -- 172.21.15.110:0/1230713720 >> 172.21.15.110:0/1230713720 conn(0x7f3ae80b09e0 msgr2=0x7f3ae80b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.448+0000 7f3aedbd7640 1 -- 172.21.15.110:0/1230713720 shutdown_connections 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.448+0000 7f3aedbd7640 1 -- 172.21.15.110:0/1230713720 wait complete. 2024-01-20T11:21:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.448+0000 7f3aedbd7640 10 monclient: build_initial_monmap 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.448+0000 7f3aedbd7640 10 monclient: monmap: 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 librados: starting msgr at 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 librados: starting objecter 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 Processor -- start 2024-01-20T11:21:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 -- start start 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 librados: setting wanted keys 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 librados: calling monclient init 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient: init 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae8066620 0x7f3ae813dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient(hunting): picked mon.noname-c con 0x7f3ae8066620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae806df70 0x7f3ae81d7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient(hunting): picked mon.noname-b con 0x7f3ae806df70 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ae8145980 0x7f3ae81d7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient(hunting): picked mon.noname-a con 0x7f3ae8145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae813d700 con 0x7f3ae8145980 2024-01-20T11:21:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae81caee0 con 0x7f3ae806df70 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae81cb150 con 0x7f3ae8066620 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.449+0000 7f3aedbd7640 10 monclient(hunting): authenticate will time out at 1717.801102s 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae8066620 0x7f3ae813dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ae8145980 0x7f3ae81d7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae6ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae806df70 0x7f3ae81d7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae8066620 0x7f3ae813dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:33826/0 (socket says 172.21.15.110:33826) 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae77fe640 1 -- 172.21.15.110:0/852155831 learned_addr learned my addr 172.21.15.110:0/852155831 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae77fe640 10 monclient(hunting): get_auth_request con 0x7f3ae8066620 auth_method 0 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae7fff640 10 monclient(hunting): get_auth_request con 0x7f3ae8145980 auth_method 0 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae6ffd640 10 monclient(hunting): get_auth_request con 0x7f3ae806df70 auth_method 0 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.450+0000 7f3ae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.451+0000 7f3ae77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.451+0000 7f3ae77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.451+0000 7f3ae77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.451+0000 7f3ae7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.451+0000 7f3ae7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.451+0000 7f3ae7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.451+0000 7f3ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:14.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.451+0000 7f3ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:14.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.451+0000 7f3ae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:14.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.453+0000 7f3ae77fe640 10 monclient(hunting): handle_auth_done global_id 5234 payload 824 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.454+0000 7f3ae77fe640 1 -- 172.21.15.110:0/852155831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae806df70 msgr2=0x7f3ae81d7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.454+0000 7f3ae77fe640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae806df70 0x7f3ae81d7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.454+0000 7f3ae77fe640 1 -- 172.21.15.110:0/852155831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ae8145980 msgr2=0x7f3ae81d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.454+0000 7f3ae77fe640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ae8145980 0x7f3ae81d7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.454+0000 7f3ae77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.454+0000 7f3ae77fe640 1 monclient: found mon.noname-c 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.454+0000 7f3ae77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.454+0000 7f3ae77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.454+0000 7f3ae77fe640 1 -- 172.21.15.110:0/852155831 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3adc009400 con 0x7f3ae8066620 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.456+0000 7f3ae77fe640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae8066620 0x7f3ae813dff0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800c400 tx=0x7f3ad800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.456+0000 7f3acffff640 10 client.?.objecter ms_handle_connect 0x7f3ae8066620 2024-01-20T11:21:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.456+0000 7f3acffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 1 -- 172.21.15.110:0/852155831 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8005ce0 con 0x7f3ae8066620 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 10 monclient: dump: 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:14.462 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 10 monclient: _finish_auth 0 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 1 -- 172.21.15.110:0/852155831 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad8005e40 con 0x7f3ae8066620 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3aedbd7640 5 monclient: authenticate success, global_id 5234 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 1 -- 172.21.15.110:0/852155831 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad80072d0 con 0x7f3ae8066620 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3aedbd7640 10 monclient: _renew_subs 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3aedbd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae813d300 con 0x7f3ae8066620 2024-01-20T11:21:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3ae5ffb640 4 set_mon_vals no callback set 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3acffff640 10 monclient: dump: 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3aedbd7640 10 client.5234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3aedbd7640 10 monclient: _renew_subs 2024-01-20T11:21:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3aedbd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae81d80a0 con 0x7f3ae8066620 2024-01-20T11:21:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.459+0000 7f3aedbd7640 1 librados: init done 2024-01-20T11:21:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.460+0000 7f3aedbd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.460+0000 7f3aedbd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.460+0000 7f3aedbd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.460+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae8075490 con 0x7f3ae8066620 2024-01-20T11:21:14.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~137s 2024-01-20T11:21:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.703+0000 7f3acffff640 1 -- 172.21.15.110:0/852155831 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3ad80074f0 con 0x7f3ae8066620 2024-01-20T11:21:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.704+0000 7f3acffff640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3ad406c860 0x7f3ad406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.704+0000 7f3ae6ffd640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3ad406c860 0x7f3ad406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.704+0000 7f3ae6ffd640 10 monclient: get_auth_request con 0x7f3ad406c860 auth_method 0 2024-01-20T11:21:14.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.705+0000 7f3ae6ffd640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3ad406c860 0x7f3ad406ed20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3adc036000 tx=0x7f3adc03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:14.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.705+0000 7f3acffff640 10 client.5234.objecter ms_handle_connect 0x7f3ad406c860 2024-01-20T11:21:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.718+0000 7f3acffff640 1 -- 172.21.15.110:0/852155831 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3ad8057c70 con 0x7f3ae8066620 2024-01-20T11:21:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.718+0000 7f3acffff640 10 client.5234.objecter ms_dispatch 0x7f3ae8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.718+0000 7f3acffff640 3 client.5234.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.718+0000 7f3acffff640 3 client.5234.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.718+0000 7f3acffff640 20 client.5234.objecter dump_active .. 0 homeless 2024-01-20T11:21:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.718+0000 7f3acffff640 1 -- 172.21.15.110:0/852155831 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f3ad8053fb0 con 0x7f3ae8066620 2024-01-20T11:21:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.718+0000 7f3acffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.718+0000 7f3acffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:14.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.890+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ae8145e30 con 0x7f3ad406c860 2024-01-20T11:21:14.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.893+0000 7f3acffff640 1 -- 172.21.15.110:0/852155831 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63876 (secure 0 0 0) 0x7f3adc0319f0 con 0x7f3ad406c860 2024-01-20T11:21:14.894 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:14.895 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:14.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 10 librados: watch_flush enter 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 10 librados: watch_flush exit 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 20 client.5234.objecter shutdown clearing up homeless session... 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 10 client.5234.objecter successfully canceled tick 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3ad406c860 msgr2=0x7f3ad406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3ad406c860 0x7f3ad406ed20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3adc036000 tx=0x7f3adc03d040 comp rx=0 tx=0).stop 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 10 monclient: shutdown 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae8066620 msgr2=0x7f3ae813dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae8066620 0x7f3ae813dff0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800c400 tx=0x7f3ad800c8c0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 shutdown_connections 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3ad406c860 0x7f3ad406ed20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ae8145980 0x7f3ae81d7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae806df70 0x7f3ae81d7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 --2- 172.21.15.110:0/852155831 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ae8066620 0x7f3ae813dff0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:14.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 >> 172.21.15.110:0/852155831 conn(0x7f3ae80b09e0 msgr2=0x7f3ae80b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:14.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 shutdown_connections 2024-01-20T11:21:14.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 -- 172.21.15.110:0/852155831 wait complete. 2024-01-20T11:21:14.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:14.895+0000 7f3aedbd7640 1 librados: shutdown 2024-01-20T11:21:14.906 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-01-20T11:21:14.781092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.3719999999999999},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.058999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.1720000000000006},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":5.5709999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":15.138999999999999},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.654999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.5880000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":11.917}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.589},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":11.409000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.5020000000000007},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":14.382999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313292,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.4089999999999998},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.2080000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.016},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.942}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.917999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.628}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.541},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.255000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.9410000000000007},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":13.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.836},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.182}]}]},{"osd":3,"up_from":21,"seq":90194313293,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.3289999999999997},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.2539999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.102},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":8.9190000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":13.205},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.718}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.1929999999999996},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.7370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":10.321999999999999},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":15.679}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":14.013999999999999},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.391999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444107,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.2889999999999997},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":10.298999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.2779999999999996},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":14.737}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":20.829000000000001},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.433}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":17.141999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":18.460999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":19.780000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.582000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.705},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":21.884}]}]},{"osd":1,"up_from":17,"seq":73014444108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.3680000000000003},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":7.3609999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.022},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.0939999999999994}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":15.301},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":12.727}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":9.2420000000000009},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.401}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.019},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":9.843}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.013999999999999},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.428000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444108,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":8.2989999999999995},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":5.2489999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.9009999999999998},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":10.778}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":11.378},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.754}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.693},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.8040000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":12.856},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":14.422000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.178000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":7.4619999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.4729999999999999},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.9299999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.976000000000001},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":6.859}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.824999999999999},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":13.605}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.0109999999999992},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.206}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.431999999999999},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.404999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.452999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.268000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:14.908 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:15.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.006+0000 7fb045ad7640 10 monclient: get_monmap_and_config 2024-01-20T11:21:15.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.006+0000 7fb045ad7640 10 monclient: build_initial_monmap 2024-01-20T11:21:15.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.006+0000 7fb045ad7640 10 monclient: monmap: 2024-01-20T11:21:15.007 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:15.007 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.007 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:15.007683+0000 2024-01-20T11:21:15.007 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:15.007683+0000 2024-01-20T11:21:15.007 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:15.008 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.008 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:15.008 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:15.008 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:15.008 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:15.008 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:15.008 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.006+0000 7fb045ad7640 1 Processor -- start 2024-01-20T11:21:15.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.006+0000 7fb045ad7640 1 -- start start 2024-01-20T11:21:15.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.006+0000 7fb045ad7640 10 monclient: init 2024-01-20T11:21:15.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:15.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-01-20T11:21:15.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb0400a96f0 0x7fb04013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 10 monclient(hunting): picked mon.noname-a con 0x7fb0400a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb04013fa10 0x7fb0401423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 10 monclient(hunting): picked mon.noname-b con 0x7fb04013fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0400a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0400b25c0 con 0x7fb0400a9ad0 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0400aaea0 con 0x7fb0400a96f0 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0400ab0a0 con 0x7fb04013fa10 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.007+0000 7fb045ad7640 10 monclient(hunting): authenticate will time out at 1718.358664s 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb0400a96f0 0x7fb04013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03effd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb04013fa10 0x7fb0401423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03ffff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fb0400a9ad0 0x7fb040142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:41694/0 (socket says 172.21.15.110:41694) 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03ffff640 1 -- 172.21.15.110:0/3140742650 learned_addr learned my addr 172.21.15.110:0/3140742650 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03f7fe640 10 monclient(hunting): get_auth_request con 0x7fb0400a96f0 auth_method 0 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03effd640 10 monclient(hunting): get_auth_request con 0x7fb04013fa10 auth_method 0 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.008+0000 7fb03f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.009+0000 7fb03effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.009+0000 7fb03effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.009+0000 7fb03effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.010+0000 7fb03f7fe640 10 monclient(hunting): handle_auth_done global_id 5493 payload 274 2024-01-20T11:21:15.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.010+0000 7fb03f7fe640 1 -- 172.21.15.110:0/3140742650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb04013fa10 msgr2=0x7fb0401423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:15.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.010+0000 7fb03f7fe640 1 --2- 172.21.15.110:0/3140742650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb04013fa10 0x7fb0401423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.010+0000 7fb03f7fe640 1 -- 172.21.15.110:0/3140742650 >> v1:172.21.15.134:6789/0 conn(0x7fb0400a9ad0 legacy=0x7fb040142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.010+0000 7fb03f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:15.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.010+0000 7fb03f7fe640 1 monclient: found mon.noname-a 2024-01-20T11:21:15.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.010+0000 7fb03f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:15.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.010+0000 7fb03f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:15.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.010+0000 7fb03f7fe640 1 -- 172.21.15.110:0/3140742650 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0400b3fb0 con 0x7fb0400a96f0 2024-01-20T11:21:15.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.012+0000 7fb03f7fe640 1 --2- 172.21.15.110:0/3140742650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a96f0 0x7fb04013f4d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb0300097a0 tx=0x7fb03002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd5b1708f943806 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 1 -- 172.21.15.110:0/3140742650 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03002e070 con 0x7fb0400a96f0 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 10 monclient: dump: 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.017 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 1 -- 172.21.15.110:0/3140742650 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0300315a0 con 0x7fb0400a96f0 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 5 monclient: authenticate success, global_id 5493 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 1 -- 172.21.15.110:0/3140742650 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb030031930 con 0x7fb0400a96f0 2024-01-20T11:21:15.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 4 set_mon_vals no callback set 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb03e7fc640 10 monclient: dump: 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:15.019 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 10 monclient: shutdown 2024-01-20T11:21:15.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:15.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 1 -- 172.21.15.110:0/3140742650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a96f0 msgr2=0x7fb04013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 1 --2- 172.21.15.110:0/3140742650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a96f0 0x7fb04013f4d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb0300097a0 tx=0x7fb03002b840 comp rx=0 tx=0).stop 2024-01-20T11:21:15.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 1 -- 172.21.15.110:0/3140742650 shutdown_connections 2024-01-20T11:21:15.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 1 --2- 172.21.15.110:0/3140742650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb04013fa10 0x7fb0401423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 1 --2- 172.21.15.110:0/3140742650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a96f0 0x7fb04013f4d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 1 -- 172.21.15.110:0/3140742650 >> 172.21.15.110:0/3140742650 conn(0x7fb0400b7bc0 msgr2=0x7fb0400b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:15.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 1 -- 172.21.15.110:0/3140742650 shutdown_connections 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.016+0000 7fb045ad7640 1 -- 172.21.15.110:0/3140742650 wait complete. 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.017+0000 7fb045ad7640 10 monclient: build_initial_monmap 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.017+0000 7fb045ad7640 10 monclient: monmap: 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 1 librados: starting msgr at 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 1 librados: starting objecter 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 1 Processor -- start 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 1 -- start start 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 1 librados: setting wanted keys 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 1 librados: calling monclient init 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 10 monclient: init 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0400a96f0 0x7fb040132730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 10 monclient(hunting): picked mon.noname-b con 0x7fb0400a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a9ad0 0x7fb040138010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.018+0000 7fb045ad7640 10 monclient(hunting): picked mon.noname-a con 0x7fb0400a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb040141a20 0x7fb04012d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 10 monclient(hunting): picked mon.noname-c con 0x7fb040141a20 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0401330e0 con 0x7fb0400a9ad0 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040133dc0 con 0x7fb0400a96f0 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb040134030 con 0x7fb040141a20 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb045ad7640 10 monclient(hunting): authenticate will time out at 1718.370220s 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a9ad0 0x7fb040138010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb040141a20 0x7fb04012d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03f7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0400a96f0 0x7fb040132730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a9ad0 0x7fb040138010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58278/0 (socket says 172.21.15.110:58278) 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03effd640 1 -- 172.21.15.110:0/3545179063 learned_addr learned my addr 172.21.15.110:0/3545179063 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03effd640 10 monclient(hunting): get_auth_request con 0x7fb0400a9ad0 auth_method 0 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03ffff640 10 monclient(hunting): get_auth_request con 0x7fb040141a20 auth_method 0 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.019+0000 7fb03ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03f7fe640 10 monclient(hunting): get_auth_request con 0x7fb0400a96f0 auth_method 0 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.020+0000 7fb03f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 10 monclient(hunting): handle_auth_done global_id 5496 payload 824 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 1 -- 172.21.15.110:0/3545179063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb040141a20 msgr2=0x7fb04012d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb040141a20 0x7fb04012d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 1 -- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0400a96f0 msgr2=0x7fb040132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0400a96f0 0x7fb040132730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 1 monclient: found mon.noname-a 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.023+0000 7fb03effd640 1 -- 172.21.15.110:0/3545179063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030009400 con 0x7fb0400a9ad0 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.025+0000 7fb03effd640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a9ad0 0x7fb040138010 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb03400e870 tx=0x7fb03400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.025+0000 7fb027fff640 10 client.?.objecter ms_handle_connect 0x7fb0400a9ad0 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.025+0000 7fb027fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:15.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 1 -- 172.21.15.110:0/3545179063 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03400ee60 con 0x7fb0400a9ad0 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 10 monclient: dump: 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 10 monclient: _finish_auth 0 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 1 -- 172.21.15.110:0/3545179063 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03400c940 con 0x7fb0400a9ad0 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 1 -- 172.21.15.110:0/3545179063 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03400ccf0 con 0x7fb0400a9ad0 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb027fff640 10 monclient: dump: 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.033 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb045ad7640 5 monclient: authenticate success, global_id 5496 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb045ad7640 10 monclient: _renew_subs 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb045ad7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb03dffb640 4 set_mon_vals no callback set 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb040134310 con 0x7fb0400a9ad0 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb045ad7640 10 client.5496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb045ad7640 10 monclient: _renew_subs 2024-01-20T11:21:15.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb045ad7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:15.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.028+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb040132c70 con 0x7fb0400a9ad0 2024-01-20T11:21:15.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.029+0000 7fb045ad7640 1 librados: init done 2024-01-20T11:21:15.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.029+0000 7fb045ad7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:15.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.029+0000 7fb045ad7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:15.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.029+0000 7fb045ad7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:15.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.029+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0400680c0 con 0x7fb0400a9ad0 2024-01-20T11:21:15.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.048+0000 7fb027fff640 1 -- 172.21.15.110:0/3545179063 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb034013470 con 0x7fb0400a9ad0 2024-01-20T11:21:15.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.049+0000 7fb027fff640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 0x7fb02c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.049+0000 7fb03f7fe640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 0x7fb02c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.049+0000 7fb03f7fe640 10 monclient: get_auth_request con 0x7fb02c06c8b0 auth_method 0 2024-01-20T11:21:15.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.049+0000 7fb03f7fe640 1 -- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 msgr2=0x7fb02c06ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-01-20T11:21:15.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.049+0000 7fb03f7fe640 1 -- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 msgr2=0x7fb02c06ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-20T11:21:15.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.049+0000 7fb03f7fe640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 0x7fb02c06ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:21:15.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.049+0000 7fb03f7fe640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 0x7fb02c06ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:21:15.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.250+0000 7fb03f7fe640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 0x7fb02c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.250+0000 7fb03f7fe640 10 monclient: get_auth_request con 0x7fb02c06c8b0 auth_method 0 2024-01-20T11:21:15.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.251+0000 7fb03f7fe640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 0x7fb02c06ed70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb030036000 tx=0x7fb030008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:15.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.251+0000 7fb027fff640 10 client.5496.objecter ms_handle_connect 0x7fb02c06c8b0 2024-01-20T11:21:15.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.284+0000 7fb027fff640 1 -- 172.21.15.110:0/3545179063 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb0340589b0 con 0x7fb0400a9ad0 2024-01-20T11:21:15.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.284+0000 7fb027fff640 10 client.5496.objecter ms_dispatch 0x7fb040000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:15.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.284+0000 7fb027fff640 3 client.5496.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:15.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.284+0000 7fb027fff640 3 client.5496.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:15.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.284+0000 7fb027fff640 20 client.5496.objecter dump_active .. 0 homeless 2024-01-20T11:21:15.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.284+0000 7fb027fff640 1 -- 172.21.15.110:0/3545179063 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb034054cf0 con 0x7fb0400a9ad0 2024-01-20T11:21:15.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.284+0000 7fb027fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:15.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.284+0000 7fb027fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:15.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.458+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb040132f40 con 0x7fb02c06c8b0 2024-01-20T11:21:15.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.461+0000 7fb027fff640 1 -- 172.21.15.110:0/3545179063 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63876 (secure 0 0 0) 0x7fb030031a90 con 0x7fb02c06c8b0 2024-01-20T11:21:15.463 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:15.463 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 10 librados: watch_flush enter 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 10 librados: watch_flush exit 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 20 client.5496.objecter shutdown clearing up homeless session... 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 10 client.5496.objecter successfully canceled tick 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 msgr2=0x7fb02c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 0x7fb02c06ed70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb030036000 tx=0x7fb030008030 comp rx=0 tx=0).stop 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 10 monclient: shutdown 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a9ad0 msgr2=0x7fb040138010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a9ad0 0x7fb040138010 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb03400e870 tx=0x7fb03400ed30 comp rx=0 tx=0).stop 2024-01-20T11:21:15.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 shutdown_connections 2024-01-20T11:21:15.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb02c06c8b0 0x7fb02c06ed70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb040141a20 0x7fb04012d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0400a9ad0 0x7fb040138010 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 --2- 172.21.15.110:0/3545179063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0400a96f0 0x7fb040132730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 >> 172.21.15.110:0/3545179063 conn(0x7fb0400b7bc0 msgr2=0x7fb040137440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:15.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 shutdown_connections 2024-01-20T11:21:15.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.463+0000 7fb045ad7640 1 -- 172.21.15.110:0/3545179063 wait complete. 2024-01-20T11:21:15.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.464+0000 7fb045ad7640 1 librados: shutdown 2024-01-20T11:21:15.475 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-01-20T11:21:14.781092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.3719999999999999},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.058999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.1720000000000006},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":5.5709999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":15.138999999999999},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.654999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.5880000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":11.917}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.589},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":11.409000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.5020000000000007},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":14.382999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313292,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.4089999999999998},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.2080000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.016},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.942}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.917999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.628}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.541},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.255000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.9410000000000007},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":13.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.836},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.182}]}]},{"osd":3,"up_from":21,"seq":90194313293,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.3289999999999997},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.2539999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.102},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":8.9190000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":13.205},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.718}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.1929999999999996},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.7370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":10.321999999999999},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":15.679}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":14.013999999999999},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.391999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444107,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.2889999999999997},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":10.298999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.2779999999999996},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":14.737}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":20.829000000000001},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.433}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":17.141999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":18.460999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":19.780000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.582000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.705},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":21.884}]}]},{"osd":1,"up_from":17,"seq":73014444108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.3680000000000003},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":7.3609999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.022},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.0939999999999994}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":15.301},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":12.727}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":9.2420000000000009},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.401}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.019},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":9.843}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.013999999999999},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.428000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444108,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":8.2989999999999995},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":5.2489999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.9009999999999998},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":10.778}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":11.378},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.754}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.693},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.8040000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":12.856},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":14.422000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.178000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":7.4619999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.4729999999999999},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.9299999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.976000000000001},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":6.859}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.824999999999999},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":13.605}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.0109999999999992},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.206}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.431999999999999},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.404999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.452999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.268000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:15.476 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:15.476 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:15.477 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:15.477 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:15.477 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:15.477 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:15.477 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:15.477 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 129, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:17:17.929554+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:17:17.929554+0000', 'last_peered': '2024-01-20T11:17:17.929554+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:17:17.929554+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:15.477 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.575+0000 7f7a4ef05640 10 monclient: get_monmap_and_config 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.575+0000 7f7a4ef05640 10 monclient: build_initial_monmap 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.576+0000 7f7a4ef05640 10 monclient: monmap: 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:15.577614+0000 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:15.577614+0000 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:15.577 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:15.578 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:15.578 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:15.578 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:15.578 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.576+0000 7f7a4ef05640 1 Processor -- start 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.576+0000 7f7a4ef05640 1 -- start start 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.576+0000 7f7a4ef05640 10 monclient: init 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a480b4d10 0x7f7a480b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 10 monclient(hunting): picked mon.noname-a con 0x7f7a480b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a480b17c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7a48145980 0x7f7a48145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 10 monclient(hunting): picked mon.noname-c con 0x7f7a48145980 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:15.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a48146370 con 0x7f7a480b17c0 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a480b50f0 con 0x7f7a480b4d10 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a480aade0 con 0x7f7a48145980 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4ef05640 10 monclient(hunting): authenticate will time out at 1718.928576s 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.577+0000 7f7a4cca1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a480b4d10 0x7f7a480b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4d4a2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f7a48145980 0x7f7a48145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4cca1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a480b4d10 0x7f7a480b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58286/0 (socket says 172.21.15.110:58286) 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4cca1640 1 -- 172.21.15.110:0/456876859 learned_addr learned my addr 172.21.15.110:0/456876859 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4cca1640 10 monclient(hunting): get_auth_request con 0x7f7a480b4d10 auth_method 0 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4cca1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4cca1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4cca1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4d4a2640 10 monclient(hunting): get_auth_request con 0x7f7a48145980 auth_method 0 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4d4a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4d4a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.578+0000 7f7a4d4a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.579+0000 7f7a4d4a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.579+0000 7f7a4d4a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.579+0000 7f7a4d4a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.579+0000 7f7a4cca1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.579+0000 7f7a4cca1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.579+0000 7f7a4cca1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.580+0000 7f7a4d4a2640 10 monclient(hunting): handle_auth_done global_id 5578 payload 274 2024-01-20T11:21:15.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.580+0000 7f7a4d4a2640 1 -- 172.21.15.110:0/456876859 >> v2:172.21.15.110:3300/0 conn(0x7f7a480b4d10 msgr2=0x7f7a480b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:15.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.580+0000 7f7a4d4a2640 1 --2- 172.21.15.110:0/456876859 >> v2:172.21.15.110:3300/0 conn(0x7f7a480b4d10 0x7f7a480b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.580+0000 7f7a4d4a2640 1 -- 172.21.15.110:0/456876859 >> v1:172.21.15.110:6789/0 conn(0x7f7a480b17c0 legacy=0x7f7a480b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.580+0000 7f7a4d4a2640 10 monclient: _finish_hunting 0 2024-01-20T11:21:15.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.580+0000 7f7a4d4a2640 1 monclient: found mon.noname-c 2024-01-20T11:21:15.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.580+0000 7f7a4d4a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:15.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.580+0000 7f7a4d4a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:21:15.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.580+0000 7f7a4d4a2640 1 -- 172.21.15.110:0/456876859 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48145460 con 0x7f7a48145980 2024-01-20T11:21:15.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.582+0000 7f7a4d4a2640 1 --2- 172.21.15.110:0/456876859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a48145980 0x7f7a48145e30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7a480aa1c0 tx=0x7f7a3802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35457e09f598d76 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:15.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 1 -- 172.21.15.110:0/456876859 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38031710 con 0x7f7a48145980 2024-01-20T11:21:15.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:15.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:15.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 10 monclient: dump: 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 1 -- 172.21.15.110:0/456876859 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3802f050 con 0x7f7a48145980 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 1 -- 172.21.15.110:0/456876859 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38031a90 con 0x7f7a48145980 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a477fe640 10 monclient: dump: 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.589 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:15.590 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.590 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:15.590 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:15.590 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:15.590 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a4ef05640 5 monclient: authenticate success, global_id 5578 2024-01-20T11:21:15.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a4ef05640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:15.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.586+0000 7f7a4ef05640 4 set_mon_vals no callback set 2024-01-20T11:21:15.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 10 monclient: shutdown 2024-01-20T11:21:15.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 1 -- 172.21.15.110:0/456876859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a48145980 msgr2=0x7f7a48145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 1 --2- 172.21.15.110:0/456876859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a48145980 0x7f7a48145e30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7a480aa1c0 tx=0x7f7a3802b7b0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 1 -- 172.21.15.110:0/456876859 shutdown_connections 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 1 --2- 172.21.15.110:0/456876859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a48145980 0x7f7a48145e30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 1 --2- 172.21.15.110:0/456876859 >> v2:172.21.15.110:3300/0 conn(0x7f7a480b4d10 0x7f7a480b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 1 -- 172.21.15.110:0/456876859 >> 172.21.15.110:0/456876859 conn(0x7f7a48066590 msgr2=0x7f7a480669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 1 -- 172.21.15.110:0/456876859 shutdown_connections 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.587+0000 7f7a4ef05640 1 -- 172.21.15.110:0/456876859 wait complete. 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.588+0000 7f7a4ef05640 10 monclient: build_initial_monmap 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.588+0000 7f7a4ef05640 10 monclient: monmap: 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:15.594 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.588+0000 7f7a4ef05640 1 librados: starting msgr at 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.588+0000 7f7a4ef05640 1 librados: starting objecter 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.588+0000 7f7a4ef05640 1 Processor -- start 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.588+0000 7f7a4ef05640 1 -- start start 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.588+0000 7f7a4ef05640 1 librados: setting wanted keys 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.588+0000 7f7a4ef05640 1 librados: calling monclient init 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.588+0000 7f7a4ef05640 10 monclient: init 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:15.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a480b17c0 0x7f7a48141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient(hunting): picked mon.noname-b con 0x7f7a480b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a480b4d10 0x7f7a48142ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient(hunting): picked mon.noname-a con 0x7f7a480b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a48145980 0x7f7a48142ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient(hunting): picked mon.noname-c con 0x7f7a48145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4813f1a0 con 0x7f7a480b4d10 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a48146c80 con 0x7f7a480b17c0 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a48146f60 con 0x7f7a48145980 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4ef05640 10 monclient(hunting): authenticate will time out at 1718.940521s 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a47fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a480b4d10 0x7f7a48142ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.589+0000 7f7a4d4a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a48145980 0x7f7a48142ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4cca1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a480b17c0 0x7f7a48141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a47fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a480b4d10 0x7f7a48142ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58290/0 (socket says 172.21.15.110:58290) 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a47fff640 1 -- 172.21.15.110:0/3887752298 learned_addr learned my addr 172.21.15.110:0/3887752298 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a47fff640 10 monclient(hunting): get_auth_request con 0x7f7a480b4d10 auth_method 0 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a47fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4d4a2640 10 monclient(hunting): get_auth_request con 0x7f7a48145980 auth_method 0 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4d4a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4d4a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4d4a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4cca1640 10 monclient(hunting): get_auth_request con 0x7f7a480b17c0 auth_method 0 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4cca1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4cca1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4cca1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4d4a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4d4a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4d4a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4cca1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:15.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4cca1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:15.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.590+0000 7f7a4cca1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 10 monclient(hunting): handle_auth_done global_id 5505 payload 824 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 1 -- 172.21.15.110:0/3887752298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a48145980 msgr2=0x7f7a48142ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a48145980 0x7f7a48142ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 1 -- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a480b17c0 msgr2=0x7f7a48141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a480b17c0 0x7f7a48141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 1 monclient: found mon.noname-a 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a47fff640 1 -- 172.21.15.110:0/3887752298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a38009400 con 0x7f7a480b4d10 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.593+0000 7f7a4cca1640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a480b17c0 0x7f7a48141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:15.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.596+0000 7f7a47fff640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a480b4d10 0x7f7a48142ab0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000c440 tx=0x7f7a30001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:15.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.596+0000 7f7a457fa640 10 client.?.objecter ms_handle_connect 0x7f7a480b4d10 2024-01-20T11:21:15.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.596+0000 7f7a457fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:15.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 1 -- 172.21.15.110:0/3887752298 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a300052d0 con 0x7f7a480b4d10 2024-01-20T11:21:15.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:15.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:15.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 10 monclient: dump: 2024-01-20T11:21:15.602 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:15.602 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.602 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 10 monclient: _finish_auth 0 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 10 monclient: _check_auth_tickets 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 1 -- 172.21.15.110:0/3887752298 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30005430 con 0x7f7a480b4d10 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 1 -- 172.21.15.110:0/3887752298 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30007260 con 0x7f7a480b4d10 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a4ef05640 5 monclient: authenticate success, global_id 5505 2024-01-20T11:21:15.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a4ef05640 10 monclient: _renew_subs 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a4ef05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a48147240 con 0x7f7a480b4d10 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a46ffd640 4 set_mon_vals no callback set 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a457fa640 10 monclient: dump: 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:15.604 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a4ef05640 10 client.5505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a4ef05640 10 monclient: _renew_subs 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a4ef05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4813fd00 con 0x7f7a480b4d10 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.600+0000 7f7a4ef05640 1 librados: init done 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.601+0000 7f7a4ef05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.601+0000 7f7a4ef05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.601+0000 7f7a4ef05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:15.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.601+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4813fef0 con 0x7f7a480b4d10 2024-01-20T11:21:15.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.619+0000 7f7a457fa640 1 -- 172.21.15.110:0/3887752298 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7a300073c0 con 0x7f7a480b4d10 2024-01-20T11:21:15.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.620+0000 7f7a457fa640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7a2806c8b0 0x7f7a2806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:15.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.620+0000 7f7a4cca1640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7a2806c8b0 0x7f7a2806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:15.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.620+0000 7f7a4cca1640 10 monclient: get_auth_request con 0x7f7a2806c8b0 auth_method 0 2024-01-20T11:21:15.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.621+0000 7f7a4cca1640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7a2806c8b0 0x7f7a2806ed70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7a400092a0 tx=0x7f7a4000b040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:15.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.621+0000 7f7a457fa640 10 client.5505.objecter ms_handle_connect 0x7f7a2806c8b0 2024-01-20T11:21:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.850+0000 7f7a457fa640 1 -- 172.21.15.110:0/3887752298 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7a30057e40 con 0x7f7a480b4d10 2024-01-20T11:21:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.850+0000 7f7a457fa640 10 client.5505.objecter ms_dispatch 0x7f7a48000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:15.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.850+0000 7f7a457fa640 3 client.5505.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.850+0000 7f7a457fa640 3 client.5505.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.850+0000 7f7a457fa640 20 client.5505.objecter dump_active .. 0 homeless 2024-01-20T11:21:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.850+0000 7f7a457fa640 1 -- 172.21.15.110:0/3887752298 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7a30054180 con 0x7f7a480b4d10 2024-01-20T11:21:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.850+0000 7f7a457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:15.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:15.850+0000 7f7a457fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:16.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.089+0000 7f7a4ef05640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:16.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.089+0000 7f7a4ef05640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:16.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.089+0000 7f7a4ef05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:16.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.089+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7a48075490 con 0x7f7a480b4d10 2024-01-20T11:21:16.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.095+0000 7f7a457fa640 1 -- 172.21.15.110:0/3887752298 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f7a30020520 con 0x7f7a480b4d10 2024-01-20T11:21:16.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.095+0000 7f7a457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:16.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.095+0000 7f7a457fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:16.096 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:16.096 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":410,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:16.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 10 librados: watch_flush enter 2024-01-20T11:21:16.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 10 librados: watch_flush exit 2024-01-20T11:21:16.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 20 client.5505.objecter shutdown clearing up homeless session... 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 10 client.5505.objecter successfully canceled tick 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7a2806c8b0 msgr2=0x7f7a2806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7a2806c8b0 0x7f7a2806ed70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7a400092a0 tx=0x7f7a4000b040 comp rx=0 tx=0).stop 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 10 monclient: shutdown 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a480b4d10 msgr2=0x7f7a48142ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a480b4d10 0x7f7a48142ab0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000c440 tx=0x7f7a30001e70 comp rx=0 tx=0).stop 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 shutdown_connections 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f7a2806c8b0 0x7f7a2806ed70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a48145980 0x7f7a48142ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a480b4d10 0x7f7a48142ab0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 --2- 172.21.15.110:0/3887752298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a480b17c0 0x7f7a48141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 >> 172.21.15.110:0/3887752298 conn(0x7f7a48066590 msgr2=0x7f7a48144c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:16.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 shutdown_connections 2024-01-20T11:21:16.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 -- 172.21.15.110:0/3887752298 wait complete. 2024-01-20T11:21:16.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.097+0000 7f7a4ef05640 1 librados: shutdown 2024-01-20T11:21:16.112 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:16.112 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.210+0000 7f85355f0640 10 monclient: get_monmap_and_config 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.210+0000 7f85355f0640 10 monclient: build_initial_monmap 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.210+0000 7f85355f0640 10 monclient: monmap: 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:16.211977+0000 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:16.211977+0000 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:16.212 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:16.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 1 Processor -- start 2024-01-20T11:21:16.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 1 -- start start 2024-01-20T11:21:16.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 10 monclient: init 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85300b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85300b1ed0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8530146090 0x7f85300b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 10 monclient(hunting): picked mon.noname-a con 0x7f8530146090 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85300aade0 con 0x7f85300b4d10 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8530145460 con 0x7f85300b1ed0 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85300b50f0 con 0x7f8530146090 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.211+0000 7f85355f0640 10 monclient(hunting): authenticate will time out at 1719.563041s 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.212+0000 7f852f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8530146090 0x7f85300b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.212+0000 7f8527fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f85300b1ed0 0x7f8530145980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:41710/0 (socket says 172.21.15.110:41710) 2024-01-20T11:21:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.212+0000 7f8527fff640 1 -- 172.21.15.110:0/3504464668 learned_addr learned my addr 172.21.15.110:0/3504464668 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.213+0000 7f852f7fe640 10 monclient(hunting): get_auth_request con 0x7f8530146090 auth_method 0 2024-01-20T11:21:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.213+0000 7f852f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.213+0000 7f852f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.213+0000 7f852f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.213+0000 7f852f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.213+0000 7f852f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.213+0000 7f852f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852e7fc640 1 -- 172.21.15.110:0/3504464668 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379987300 0 0) 0x7f8530145460 con 0x7f85300b1ed0 2024-01-20T11:21:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852e7fc640 10 monclient(hunting): _negotiate 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852e7fc640 10 monclient(hunting): my global_id is 5584 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852e7fc640 1 -- 172.21.15.110:0/3504464668 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f851c003520 con 0x7f85300b1ed0 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852f7fe640 10 monclient(hunting): handle_auth_done global_id 5508 payload 274 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852f7fe640 1 -- 172.21.15.110:0/3504464668 >> v1:172.21.15.134:6789/0 conn(0x7f85300b1ed0 legacy=0x7f8530145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852f7fe640 1 -- 172.21.15.110:0/3504464668 >> v1:172.21.15.110:6789/0 conn(0x7f85300b4d10 legacy=0x7f85300b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852f7fe640 1 monclient: found mon.noname-a 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.215+0000 7f852f7fe640 1 -- 172.21.15.110:0/3504464668 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85300b2660 con 0x7f8530146090 2024-01-20T11:21:16.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.219+0000 7f852f7fe640 1 --2- 172.21.15.110:0/3504464668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 0x7f85300b3da0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8528009dc0 tx=0x7f852802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef769ac4697f1211 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 1 -- 172.21.15.110:0/3504464668 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852802cc40 con 0x7f8530146090 2024-01-20T11:21:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 10 monclient: dump: 2024-01-20T11:21:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 1 -- 172.21.15.110:0/3504464668 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852802cda0 con 0x7f8530146090 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 1 -- 172.21.15.110:0/3504464668 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528034660 con 0x7f8530146090 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f85355f0640 5 monclient: authenticate success, global_id 5508 2024-01-20T11:21:16.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f85355f0640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f85355f0640 4 set_mon_vals no callback set 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.224+0000 7f852e7fc640 10 monclient: dump: 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:16.227 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:16.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.225+0000 7f85355f0640 10 monclient: shutdown 2024-01-20T11:21:16.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.225+0000 7f85355f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:16.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.225+0000 7f85355f0640 1 -- 172.21.15.110:0/3504464668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 msgr2=0x7f85300b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:16.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.225+0000 7f85355f0640 1 --2- 172.21.15.110:0/3504464668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 0x7f85300b3da0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8528009dc0 tx=0x7f852802be90 comp rx=0 tx=0).stop 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.225+0000 7f85355f0640 1 -- 172.21.15.110:0/3504464668 shutdown_connections 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.225+0000 7f85355f0640 1 --2- 172.21.15.110:0/3504464668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 0x7f85300b3da0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.225+0000 7f85355f0640 1 -- 172.21.15.110:0/3504464668 >> 172.21.15.110:0/3504464668 conn(0x7f8530066590 msgr2=0x7f85300669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.225+0000 7f85355f0640 1 -- 172.21.15.110:0/3504464668 shutdown_connections 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.225+0000 7f85355f0640 1 -- 172.21.15.110:0/3504464668 wait complete. 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.226+0000 7f85355f0640 10 monclient: build_initial_monmap 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.226+0000 7f85355f0640 10 monclient: monmap: 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:16.232 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.226+0000 7f85355f0640 1 librados: starting msgr at 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.226+0000 7f85355f0640 1 librados: starting objecter 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.226+0000 7f85355f0640 1 Processor -- start 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.226+0000 7f85355f0640 1 -- start start 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.226+0000 7f85355f0640 1 librados: setting wanted keys 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.226+0000 7f85355f0640 1 librados: calling monclient init 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.226+0000 7f85355f0640 10 monclient: init 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85300b1ed0 0x7f853013d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient(hunting): picked mon.noname-c con 0x7f85300b1ed0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:16.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85300b4d10 0x7f853013d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient(hunting): picked mon.noname-b con 0x7f85300b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 0x7f8530135c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient(hunting): picked mon.noname-a con 0x7f8530146090 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85300ab300 con 0x7f8530146090 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85300ab5e0 con 0x7f85300b4d10 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85300ab8c0 con 0x7f85300b1ed0 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.227+0000 7f85355f0640 10 monclient(hunting): authenticate will time out at 1719.578813s 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 0x7f8530135c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852effd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85300b1ed0 0x7f853013d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f8527fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85300b4d10 0x7f853013d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:16.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 0x7f8530135c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58312/0 (socket says 172.21.15.110:58312) 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852f7fe640 1 -- 172.21.15.110:0/539384154 learned_addr learned my addr 172.21.15.110:0/539384154 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852f7fe640 10 monclient(hunting): get_auth_request con 0x7f8530146090 auth_method 0 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852effd640 10 monclient(hunting): get_auth_request con 0x7f85300b1ed0 auth_method 0 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f852effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f8527fff640 10 monclient(hunting): get_auth_request con 0x7f85300b4d10 auth_method 0 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f8527fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f8527fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.228+0000 7f8527fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.229+0000 7f852f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.229+0000 7f852f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.229+0000 7f852f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.229+0000 7f852effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.229+0000 7f852effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.229+0000 7f852effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.229+0000 7f8527fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.229+0000 7f8527fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.229+0000 7f8527fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 10 monclient(hunting): handle_auth_done global_id 5514 payload 824 2024-01-20T11:21:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 1 -- 172.21.15.110:0/539384154 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85300b1ed0 msgr2=0x7f853013d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85300b1ed0 0x7f853013d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 1 -- 172.21.15.110:0/539384154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85300b4d10 msgr2=0x7f853013d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85300b4d10 0x7f853013d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 1 monclient: found mon.noname-a 2024-01-20T11:21:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:16.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.232+0000 7f852f7fe640 1 -- 172.21.15.110:0/539384154 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8528009a70 con 0x7f8530146090 2024-01-20T11:21:16.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.234+0000 7f852f7fe640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 0x7f8530135c30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8528037040 tx=0x7f8528004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:16.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.234+0000 7f85277fe640 10 client.?.objecter ms_handle_connect 0x7f8530146090 2024-01-20T11:21:16.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.234+0000 7f85277fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 1 -- 172.21.15.110:0/539384154 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528040650 con 0x7f8530146090 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 10 monclient: dump: 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 1 -- 172.21.15.110:0/539384154 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85280407b0 con 0x7f8530146090 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 1 -- 172.21.15.110:0/539384154 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528040b60 con 0x7f8530146090 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85355f0640 5 monclient: authenticate success, global_id 5514 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85355f0640 10 monclient: _renew_subs 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85355f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853013c1d0 con 0x7f8530146090 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:16.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f852dffb640 4 set_mon_vals no callback set 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85277fe640 10 monclient: dump: 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85355f0640 10 client.5514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85355f0640 10 monclient: _renew_subs 2024-01-20T11:21:16.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85355f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:16.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853013ec70 con 0x7f8530146090 2024-01-20T11:21:16.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.237+0000 7f85355f0640 1 librados: init done 2024-01-20T11:21:16.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.238+0000 7f85355f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:16.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.238+0000 7f85355f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:16.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.238+0000 7f85355f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:16.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.238+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853013c410 con 0x7f8530146090 2024-01-20T11:21:16.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.479+0000 7f85277fe640 1 -- 172.21.15.110:0/539384154 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8528040cc0 con 0x7f8530146090 2024-01-20T11:21:16.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.479+0000 7f85277fe640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f851006c8b0 0x7f851006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:16.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.479+0000 7f852effd640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f851006c8b0 0x7f851006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:16.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.480+0000 7f852effd640 10 monclient: get_auth_request con 0x7f851006c8b0 auth_method 0 2024-01-20T11:21:16.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.480+0000 7f852effd640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f851006c8b0 0x7f851006ed70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8520001780 tx=0x7f8520009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:16.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.480+0000 7f85277fe640 10 client.5514.objecter ms_handle_connect 0x7f851006c8b0 2024-01-20T11:21:16.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.491+0000 7f85277fe640 1 -- 172.21.15.110:0/539384154 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f852807b160 con 0x7f8530146090 2024-01-20T11:21:16.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.491+0000 7f85277fe640 10 client.5514.objecter ms_dispatch 0x7f8530000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:16.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.491+0000 7f85277fe640 3 client.5514.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:16.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.491+0000 7f85277fe640 3 client.5514.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:16.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.491+0000 7f85277fe640 20 client.5514.objecter dump_active .. 0 homeless 2024-01-20T11:21:16.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.491+0000 7f85277fe640 1 -- 172.21.15.110:0/539384154 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f85280774a0 con 0x7f8530146090 2024-01-20T11:21:16.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.491+0000 7f85277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:16.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.491+0000 7f85277fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:16.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.672+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f853013c410 con 0x7f851006c8b0 2024-01-20T11:21:16.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.675+0000 7f85277fe640 1 -- 172.21.15.110:0/539384154 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63876 (secure 0 0 0) 0x7f853013c410 con 0x7f851006c8b0 2024-01-20T11:21:16.676 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:16.677 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:16.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 10 librados: watch_flush enter 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 10 librados: watch_flush exit 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 20 client.5514.objecter shutdown clearing up homeless session... 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 10 client.5514.objecter successfully canceled tick 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f851006c8b0 msgr2=0x7f851006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f851006c8b0 0x7f851006ed70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8520001780 tx=0x7f8520009040 comp rx=0 tx=0).stop 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 10 monclient: shutdown 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 msgr2=0x7f8530135c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 0x7f8530135c30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8528037040 tx=0x7f8528004a40 comp rx=0 tx=0).stop 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 shutdown_connections 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f851006c8b0 0x7f851006ed70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8530146090 0x7f8530135c30 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85300b4d10 0x7f853013d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 --2- 172.21.15.110:0/539384154 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85300b1ed0 0x7f853013d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:16.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 >> 172.21.15.110:0/539384154 conn(0x7f8530066590 msgr2=0x7f8530139ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:16.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 shutdown_connections 2024-01-20T11:21:16.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 -- 172.21.15.110:0/539384154 wait complete. 2024-01-20T11:21:16.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:16.677+0000 7f85355f0640 1 librados: shutdown 2024-01-20T11:21:16.688 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-01-20T11:21:14.781092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":129,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:17:17.929554+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:17:17.929554+0000","last_peered":"2024-01-20T11:17:17.929554+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:17:17.929554+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":7.3719999999999999},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":10.058999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.1720000000000006},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":5.5709999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":15.138999999999999},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":12.654999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.5880000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":11.917}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.589},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":11.409000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.5020000000000007},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":14.382999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313292,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.4089999999999998},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":9.2080000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.016},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":10.942}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":13.917999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.628}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.541},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.255000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":9.9410000000000007},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":13.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":15.836},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":12.182}]}]},{"osd":3,"up_from":21,"seq":90194313293,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.3289999999999997},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.2539999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.102},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":8.9190000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":13.205},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":14.718}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.1929999999999996},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":9.7370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":10.321999999999999},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":15.679}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":14.013999999999999},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.391999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444107,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":8.2889999999999997},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":10.298999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":7.2779999999999996},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":14.737}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":20.829000000000001},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":13.433}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":17.141999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":18.460999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":19.780000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.582000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":15.705},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":21.884}]}]},{"osd":1,"up_from":17,"seq":73014444108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":4.3680000000000003},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":7.3609999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":12.022},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.0939999999999994}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":15.301},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":12.727}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":9.2420000000000009},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.401}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.019},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":9.843}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.013999999999999},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":10.428000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444108,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":8.2989999999999995},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":5.2489999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":8.9009999999999998},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":10.778}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":11.378},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.754}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.693},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.8040000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":12.856},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":14.422000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.178000000000001},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":7.4619999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":5.4729999999999999},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":9.9299999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":12.976000000000001},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":6.859}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":11.824999999999999},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":13.605}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":8.0109999999999992},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":11.206}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":16.431999999999999},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":12.404999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.452999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":10.268000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:19.041 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:10.065 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:19.046 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:10.070 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:19.062 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:10.087 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:19.096 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:10.120 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:19.097 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:07:10.103 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:19.099 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:10.123 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:19.099 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:07:10.105 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:19.114 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:07:10.120 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:19.114 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:10.138 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:19.691 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:19.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.789+0000 7feedec39640 10 monclient: get_monmap_and_config 2024-01-20T11:21:19.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.789+0000 7feedec39640 10 monclient: build_initial_monmap 2024-01-20T11:21:19.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.789+0000 7feedec39640 10 monclient: monmap: 2024-01-20T11:21:19.790 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:19.790 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:19.790 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:19.791194+0000 2024-01-20T11:21:19.790 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:19.791194+0000 2024-01-20T11:21:19.790 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:19.791 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:19.791 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:19.791 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:19.791 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:19.791 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:19.791 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:19.791 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:19.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 1 Processor -- start 2024-01-20T11:21:19.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 1 -- start start 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 10 monclient: init 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feed8066620 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feed806df70 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feed8145b50 0x7feed814d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 10 monclient(hunting): picked mon.noname-a con 0x7feed8145b50 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed80adf80 con 0x7feed8066620 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed806f650 con 0x7feed806df70 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed8072880 con 0x7feed8145b50 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.790+0000 7feedec39640 10 monclient(hunting): authenticate will time out at 1723.142150s 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.791+0000 7feedd1d6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feed8145b50 0x7feed814d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:19.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.791+0000 7feecffff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7feed806df70 0x7feed8145440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:33870/0 (socket says 172.21.15.110:33870) 2024-01-20T11:21:19.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.791+0000 7feecffff640 1 -- 172.21.15.110:0/3853440032 learned_addr learned my addr 172.21.15.110:0/3853440032 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:19.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.792+0000 7feedd1d6640 10 monclient(hunting): get_auth_request con 0x7feed8145b50 auth_method 0 2024-01-20T11:21:19.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.792+0000 7feedd1d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:19.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.792+0000 7feedd1d6640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:19.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.792+0000 7feedd1d6640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:19.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.792+0000 7feedd1d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:19.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.792+0000 7feedd1d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:19.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.792+0000 7feedd1d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:19.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.794+0000 7feecf7fe640 1 -- 172.21.15.110:0/3853440032 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861396470 0 0) 0x7feed806f650 con 0x7feed806df70 2024-01-20T11:21:19.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.794+0000 7feecf7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:21:19.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.794+0000 7feecf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:19.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.794+0000 7feecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:19.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.794+0000 7feecf7fe640 10 monclient(hunting): my global_id is 5590 2024-01-20T11:21:19.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.794+0000 7feecf7fe640 1 -- 172.21.15.110:0/3853440032 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec0003520 con 0x7feed806df70 2024-01-20T11:21:19.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.795+0000 7feedd1d6640 10 monclient(hunting): handle_auth_done global_id 5517 payload 274 2024-01-20T11:21:19.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.795+0000 7feedd1d6640 1 -- 172.21.15.110:0/3853440032 >> v1:172.21.15.134:6789/0 conn(0x7feed806df70 legacy=0x7feed8145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:19.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.795+0000 7feedd1d6640 1 -- 172.21.15.110:0/3853440032 >> v1:172.21.15.110:6789/0 conn(0x7feed8066620 legacy=0x7feed813bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:19.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.795+0000 7feedd1d6640 10 monclient: _finish_hunting 0 2024-01-20T11:21:19.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.795+0000 7feedd1d6640 1 monclient: found mon.noname-a 2024-01-20T11:21:19.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.795+0000 7feedd1d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:19.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.795+0000 7feedd1d6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:19.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.795+0000 7feedd1d6640 1 -- 172.21.15.110:0/3853440032 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed8072be0 con 0x7feed8145b50 2024-01-20T11:21:19.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.799+0000 7feedd1d6640 1 --2- 172.21.15.110:0/3853440032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8145b50 0x7feed814d7e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7feed0009d90 tx=0x7feed002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72421f2a5d117f3b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:19.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 1 -- 172.21.15.110:0/3853440032 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed002cc60 con 0x7feed8145b50 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 10 monclient: dump: 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:19.804 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 1 -- 172.21.15.110:0/3853440032 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed002cdc0 con 0x7feed8145b50 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feedec39640 5 monclient: authenticate success, global_id 5517 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 1 -- 172.21.15.110:0/3853440032 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0034660 con 0x7feed8145b50 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.802+0000 7feecf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feecf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feecf7fe640 10 monclient: dump: 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:19.805 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:19.806 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:19.806 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:19.806 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:19.806 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:19.806 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:19.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:19.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 4 set_mon_vals no callback set 2024-01-20T11:21:19.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 10 monclient: shutdown 2024-01-20T11:21:19.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:19.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 1 -- 172.21.15.110:0/3853440032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8145b50 msgr2=0x7feed814d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:19.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 1 --2- 172.21.15.110:0/3853440032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8145b50 0x7feed814d7e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7feed0009d90 tx=0x7feed002beb0 comp rx=0 tx=0).stop 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 1 -- 172.21.15.110:0/3853440032 shutdown_connections 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 1 --2- 172.21.15.110:0/3853440032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8145b50 0x7feed814d7e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 1 -- 172.21.15.110:0/3853440032 >> 172.21.15.110:0/3853440032 conn(0x7feed80b09e0 msgr2=0x7feed80b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 1 -- 172.21.15.110:0/3853440032 shutdown_connections 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.803+0000 7feedec39640 1 -- 172.21.15.110:0/3853440032 wait complete. 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.804+0000 7feedec39640 10 monclient: build_initial_monmap 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.804+0000 7feedec39640 10 monclient: monmap: 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:19.810 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.804+0000 7feedec39640 1 librados: starting msgr at 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.804+0000 7feedec39640 1 librados: starting objecter 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.804+0000 7feedec39640 1 Processor -- start 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.804+0000 7feedec39640 1 -- start start 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.804+0000 7feedec39640 1 librados: setting wanted keys 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.804+0000 7feedec39640 1 librados: calling monclient init 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.804+0000 7feedec39640 10 monclient: init 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8066620 0x7feed81dae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:19.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient(hunting): picked mon.noname-a con 0x7feed8066620 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feed806df70 0x7feed8140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient(hunting): picked mon.noname-c con 0x7feed806df70 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feed8145b50 0x7feed8141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient(hunting): picked mon.noname-b con 0x7feed8145b50 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed81db520 con 0x7feed8066620 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed8142450 con 0x7feed8145b50 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed8148a40 con 0x7feed806df70 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:19.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.805+0000 7feedec39640 10 monclient(hunting): authenticate will time out at 1723.156757s 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedc9d5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8066620 0x7feed81dae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feecffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feed806df70 0x7feed8140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedd1d6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feed8145b50 0x7feed8141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedc9d5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8066620 0x7feed81dae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47358/0 (socket says 172.21.15.110:47358) 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedc9d5640 1 -- 172.21.15.110:0/1298195356 learned_addr learned my addr 172.21.15.110:0/1298195356 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedc9d5640 10 monclient(hunting): get_auth_request con 0x7feed8066620 auth_method 0 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedc9d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedc9d5640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedc9d5640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feecffff640 10 monclient(hunting): get_auth_request con 0x7feed806df70 auth_method 0 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feecffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feecffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feecffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedd1d6640 10 monclient(hunting): get_auth_request con 0x7feed8145b50 auth_method 0 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedd1d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:19.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedd1d6640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedd1d6640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedc9d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.806+0000 7feedc9d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.807+0000 7feedc9d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.807+0000 7feecffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.807+0000 7feecffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.807+0000 7feecffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.807+0000 7feedd1d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.807+0000 7feedd1d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:19.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.807+0000 7feedd1d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 10 monclient(hunting): handle_auth_done global_id 5593 payload 824 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 1 -- 172.21.15.110:0/1298195356 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feed806df70 msgr2=0x7feed8140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feed806df70 0x7feed8140e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 1 -- 172.21.15.110:0/1298195356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8066620 msgr2=0x7feed81dae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8066620 0x7feed81dae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 10 monclient: _finish_hunting 0 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 1 monclient: found mon.noname-b 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:19.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.809+0000 7feedd1d6640 1 -- 172.21.15.110:0/1298195356 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0009a70 con 0x7feed8145b50 2024-01-20T11:21:19.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.812+0000 7feedd1d6640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feed8145b50 0x7feed8141350 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7feed0033000 tx=0x7feed0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:19.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.812+0000 7feecd7fa640 10 client.?.objecter ms_handle_connect 0x7feed8145b50 2024-01-20T11:21:19.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.812+0000 7feecd7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.815+0000 7feecd7fa640 1 -- 172.21.15.110:0/1298195356 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed003e610 con 0x7feed8145b50 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 10 monclient: dump: 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:19.817 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 10 monclient: _finish_auth 0 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 1 -- 172.21.15.110:0/1298195356 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed003e7b0 con 0x7feed8145b50 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 1 -- 172.21.15.110:0/1298195356 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed003eb60 con 0x7feed8145b50 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feecd7fa640 10 monclient: dump: 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:19.818 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feedec39640 5 monclient: authenticate success, global_id 5593 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feedec39640 10 monclient: _renew_subs 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feedec39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed8148d20 con 0x7feed8145b50 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feeceffd640 4 set_mon_vals no callback set 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feedec39640 10 client.5593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:19.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feedec39640 10 monclient: _renew_subs 2024-01-20T11:21:19.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feedec39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:19.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed8136c70 con 0x7feed8145b50 2024-01-20T11:21:19.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.816+0000 7feedec39640 1 librados: init done 2024-01-20T11:21:19.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.817+0000 7feedec39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:19.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.817+0000 7feedec39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:19.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.817+0000 7feedec39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:19.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:19.817+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed81db6d0 con 0x7feed8145b50 2024-01-20T11:21:20.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.056+0000 7feecd7fa640 1 -- 172.21.15.110:0/1298195356 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7feed003b050 con 0x7feed8145b50 2024-01-20T11:21:20.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.057+0000 7feecd7fa640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feec406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:20.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.057+0000 7feedc9d5640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feec406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:20.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.057+0000 7feedc9d5640 10 monclient: get_auth_request con 0x7feec406c860 auth_method 0 2024-01-20T11:21:20.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.057+0000 7feedc9d5640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feec406ed20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7feed40017a0 tx=0x7feed4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:20.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.057+0000 7feecd7fa640 10 client.5593.objecter ms_handle_connect 0x7feec406c860 2024-01-20T11:21:20.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.060+0000 7feecd7fa640 1 -- 172.21.15.110:0/1298195356 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feed007ad00 con 0x7feed8145b50 2024-01-20T11:21:20.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.060+0000 7feecd7fa640 10 client.5593.objecter ms_dispatch 0x7feed8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:20.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.060+0000 7feecd7fa640 3 client.5593.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:20.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.060+0000 7feecd7fa640 3 client.5593.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:20.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.060+0000 7feecd7fa640 20 client.5593.objecter dump_active .. 0 homeless 2024-01-20T11:21:20.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.070+0000 7feecd7fa640 1 -- 172.21.15.110:0/1298195356 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7feed0077040 con 0x7feed8145b50 2024-01-20T11:21:20.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.070+0000 7feecd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:20.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.070+0000 7feecd7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:20.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.247+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feed8066a00 con 0x7feec406c860 2024-01-20T11:21:20.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.247+0000 7feedc9d5640 1 -- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 msgr2=0x7feec406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-01-20T11:21:20.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.247+0000 7feedc9d5640 1 -- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 msgr2=0x7feec406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-20T11:21:20.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.247+0000 7feedc9d5640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feec406ed20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7feed40017a0 tx=0x7feed4009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:21:20.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.247+0000 7feedc9d5640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feec406ed20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7feed40017a0 tx=0x7feed4009040 comp rx=0 tx=0).stop 2024-01-20T11:21:20.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.248+0000 7feecd7fa640 1 -- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 msgr2=0x7feec406ed20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-01-20T11:21:20.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:20.248+0000 7feecd7fa640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feec406ed20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:20.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~143s 2024-01-20T11:21:21.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.057+0000 7feebf7fe640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feeb0005560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.057+0000 7feebf7fe640 1 -- 172.21.15.110:0/1298195356 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feeb0005ac0 con 0x7feec406c860 2024-01-20T11:21:21.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.057+0000 7feedc9d5640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feeb0005560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.057+0000 7feedc9d5640 10 monclient: get_auth_request con 0x7feec406c860 auth_method 0 2024-01-20T11:21:21.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.058+0000 7feedc9d5640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feeb0005560 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feed4001eb0 tx=0x7feed400a000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:21.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.058+0000 7feecd7fa640 10 client.5593.objecter ms_handle_connect 0x7feec406c860 2024-01-20T11:21:21.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.061+0000 7feecd7fa640 1 -- 172.21.15.110:0/1298195356 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63868 (secure 0 0 0) 0x7feeb0005ac0 con 0x7feec406c860 2024-01-20T11:21:21.062 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:21.062 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:21.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 10 librados: watch_flush enter 2024-01-20T11:21:21.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 10 librados: watch_flush exit 2024-01-20T11:21:21.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 20 client.5593.objecter shutdown clearing up homeless session... 2024-01-20T11:21:21.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 10 client.5593.objecter successfully canceled tick 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 msgr2=0x7feeb0005560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feeb0005560 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feed4001eb0 tx=0x7feed400a000 comp rx=0 tx=0).stop 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 10 monclient: shutdown 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feed8145b50 msgr2=0x7feed8141350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feed8145b50 0x7feed8141350 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7feed0033000 tx=0x7feed0004a40 comp rx=0 tx=0).stop 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 shutdown_connections 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feec406c860 0x7feeb0005560 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feed8145b50 0x7feed8141350 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feed806df70 0x7feed8140e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 --2- 172.21.15.110:0/1298195356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feed8066620 0x7feed81dae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 >> 172.21.15.110:0/1298195356 conn(0x7feed80b09e0 msgr2=0x7feed80b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 shutdown_connections 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 -- 172.21.15.110:0/1298195356 wait complete. 2024-01-20T11:21:21.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.063+0000 7feedec39640 1 librados: shutdown 2024-01-20T11:21:21.075 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-01-20T11:21:20.782485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002808"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280591,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.7859999999999996},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.9880000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.513},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3360000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.912000000000001},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.6940000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.5869999999999997},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":13.935}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.813000000000001},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":10.375}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.15},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":14.85}]}]},{"osd":7,"up_from":21,"seq":90194313293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.5419999999999998},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.8259999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.054},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.0340000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.335000000000001},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":17.547999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.122999999999999},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.702}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.75},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":14.176}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.524000000000001},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.478999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313294,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.6980000000000004},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.5350000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.4130000000000003},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.579000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":16.210000000000001},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.8309999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.207000000000001},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":10.852},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.788}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":15.385},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.58}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.2309999999999999},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":6.25}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.396000000000001},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.9699999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.1890000000000001},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.4619999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.090999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.7750000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.771000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.08}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.372},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.029999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.8630000000000004},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":6.1159999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":7.5759999999999996},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.8510000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":14.378},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.5489999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":8.1630000000000003},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.66}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.723000000000001},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":10.173999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.118},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.93}]}]},{"osd":0,"up_from":17,"seq":73014444109,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":9.0419999999999998},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":10.16}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.609},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.693}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":5.2389999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":6.5419999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.722},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.366}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":14.377000000000001},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":8.2599999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.061999999999999},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":7.2220000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":11.161},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.4729999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.5540000000000003},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":12.929}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":6.8620000000000001},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.565}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.958},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":8.2189999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.8510000000000009},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.397}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.194000000000001},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.289999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:21.076 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.178+0000 7f818bfb6640 10 monclient: get_monmap_and_config 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.178+0000 7f818bfb6640 10 monclient: build_initial_monmap 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.178+0000 7f818bfb6640 10 monclient: monmap: 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:21.179693+0000 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:21.179693+0000 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:21.179 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:21.180 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:21.180 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:21.180 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.178+0000 7f818bfb6640 1 Processor -- start 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.178+0000 7f818bfb6640 1 -- start start 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.178+0000 7f818bfb6640 10 monclient: init 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a96f0 0x7f818413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient(hunting): picked mon.noname-b con 0x7f81840a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f818413fa10 0x7f81841423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient(hunting): picked mon.noname-a con 0x7f818413fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f81840a9ad0 0x7f8184142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient(hunting): picked mon.noname-c con 0x7f81840a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81840aaea0 con 0x7f818413fa10 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81840b25c0 con 0x7f81840a9ad0 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81840ab0a0 con 0x7f81840a96f0 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.179+0000 7f818bfb6640 10 monclient(hunting): authenticate will time out at 1724.530727s 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f8189d52640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a96f0 0x7f818413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f818a553640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f81840a9ad0 0x7f8184142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f8189d52640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a96f0 0x7f818413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43240/0 (socket says 172.21.15.110:43240) 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f8189d52640 1 -- 172.21.15.110:0/4132932103 learned_addr learned my addr 172.21.15.110:0/4132932103 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f8189d52640 10 monclient(hunting): get_auth_request con 0x7f81840a96f0 auth_method 0 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f8189d52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f8189d52640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:21.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f8189d52640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f818a553640 10 monclient(hunting): get_auth_request con 0x7f81840a9ad0 auth_method 0 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f818a553640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f818a553640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.180+0000 7f818a553640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.181+0000 7f8189d52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.181+0000 7f8189d52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.181+0000 7f8189d52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.181+0000 7f818a553640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.181+0000 7f818a553640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:21.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.181+0000 7f818a553640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.182+0000 7f8189d52640 10 monclient(hunting): handle_auth_done global_id 5252 payload 274 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.182+0000 7f8189d52640 1 -- 172.21.15.110:0/4132932103 >> v2:172.21.15.134:3300/0 conn(0x7f81840a9ad0 msgr2=0x7f8184142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.183+0000 7f8189d52640 1 --2- 172.21.15.110:0/4132932103 >> v2:172.21.15.134:3300/0 conn(0x7f81840a9ad0 0x7f8184142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.183+0000 7f8189d52640 1 -- 172.21.15.110:0/4132932103 >> v2:172.21.15.110:3300/0 conn(0x7f818413fa10 msgr2=0x7f81841423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.183+0000 7f8189d52640 1 --2- 172.21.15.110:0/4132932103 >> v2:172.21.15.110:3300/0 conn(0x7f818413fa10 0x7f81841423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.183+0000 7f8189d52640 10 monclient: _finish_hunting 0 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.183+0000 7f8189d52640 1 monclient: found mon.noname-b 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.183+0000 7f8189d52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.183+0000 7f8189d52640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.183+0000 7f8189d52640 1 -- 172.21.15.110:0/4132932103 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81840b3fb0 con 0x7f81840a96f0 2024-01-20T11:21:21.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.183+0000 7f818a553640 1 --2- 172.21.15.110:0/4132932103 >> v2:172.21.15.134:3300/0 conn(0x7f81840a9ad0 0x7f8184142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:21.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.184+0000 7f8189d52640 1 --2- 172.21.15.110:0/4132932103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a96f0 0x7f818413f4d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f81841465c0 tx=0x7f817402b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b947cc6d4c73aa6 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:21.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 1 -- 172.21.15.110:0/4132932103 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81740317e0 con 0x7f81840a96f0 2024-01-20T11:21:21.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:21.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:21.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 10 monclient: dump: 2024-01-20T11:21:21.191 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:21.191 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.191 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.191 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.191 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 10 monclient: _finish_auth 0 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 10 monclient: _check_auth_tickets 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 1 -- 172.21.15.110:0/4132932103 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8174031940 con 0x7f81840a96f0 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f818bfb6640 5 monclient: authenticate success, global_id 5252 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f818bfb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f818bfb6640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:21.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 1 -- 172.21.15.110:0/4132932103 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8174031cd0 con 0x7f81840a96f0 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f818bfb6640 4 set_mon_vals no callback set 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.188+0000 7f8188d50640 10 monclient: dump: 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 10 monclient: shutdown 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 1 -- 172.21.15.110:0/4132932103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a96f0 msgr2=0x7f818413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 1 --2- 172.21.15.110:0/4132932103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a96f0 0x7f818413f4d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f81841465c0 tx=0x7f817402b880 comp rx=0 tx=0).stop 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 1 -- 172.21.15.110:0/4132932103 shutdown_connections 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 1 --2- 172.21.15.110:0/4132932103 >> v2:172.21.15.134:3300/0 conn(0x7f81840a9ad0 0x7f8184142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 1 --2- 172.21.15.110:0/4132932103 >> v2:172.21.15.110:3300/0 conn(0x7f818413fa10 0x7f81841423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 1 --2- 172.21.15.110:0/4132932103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a96f0 0x7f818413f4d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 1 -- 172.21.15.110:0/4132932103 >> 172.21.15.110:0/4132932103 conn(0x7f81840b7bc0 msgr2=0x7f81840b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 1 -- 172.21.15.110:0/4132932103 shutdown_connections 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.189+0000 7f818bfb6640 1 -- 172.21.15.110:0/4132932103 wait complete. 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.190+0000 7f818bfb6640 10 monclient: build_initial_monmap 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.190+0000 7f818bfb6640 10 monclient: monmap: 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:21.194 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.190+0000 7f818bfb6640 1 librados: starting msgr at 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.190+0000 7f818bfb6640 1 librados: starting objecter 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.190+0000 7f818bfb6640 1 Processor -- start 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.190+0000 7f818bfb6640 1 -- start start 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.190+0000 7f818bfb6640 1 librados: setting wanted keys 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.190+0000 7f818bfb6640 1 librados: calling monclient init 2024-01-20T11:21:21.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.190+0000 7f818bfb6640 10 monclient: init 2024-01-20T11:21:21.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:21.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:21:21.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a96f0 0x7f8184139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient(hunting): picked mon.noname-a con 0x7f81840a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:21.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a9ad0 0x7f81841da9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient(hunting): picked mon.noname-c con 0x7f81840a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:21.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81840b09c0 0x7f81840b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient(hunting): picked mon.noname-b con 0x7f81840b09c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:21.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81841db390 con 0x7f81840a96f0 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81841cd710 con 0x7f81840b09c0 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81840b13b0 con 0x7f81840a9ad0 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818bfb6640 10 monclient(hunting): authenticate will time out at 1724.542383s 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f8189551640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a9ad0 0x7f81841da9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.191+0000 7f818a553640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81840b09c0 0x7f81840b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f8189551640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a9ad0 0x7f81841da9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43244/0 (socket says 172.21.15.110:43244) 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f8189551640 1 -- 172.21.15.110:0/494452497 learned_addr learned my addr 172.21.15.110:0/494452497 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:21.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f8189551640 10 monclient(hunting): get_auth_request con 0x7f81840a9ad0 auth_method 0 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f8189551640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f8189551640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f8189551640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f818a553640 10 monclient(hunting): get_auth_request con 0x7f81840b09c0 auth_method 0 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f818a553640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f818a553640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f818a553640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f8189551640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f8189551640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f8189551640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f818a553640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f818a553640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.192+0000 7f818a553640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 10 monclient(hunting): handle_auth_done global_id 5255 payload 824 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 1 -- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81840b09c0 msgr2=0x7f81840b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81840b09c0 0x7f81840b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 1 -- 172.21.15.110:0/494452497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a96f0 msgr2=0x7f8184139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a96f0 0x7f8184139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 10 monclient: _finish_hunting 0 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 1 monclient: found mon.noname-c 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f8189551640 1 -- 172.21.15.110:0/494452497 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c009a70 con 0x7f81840a9ad0 2024-01-20T11:21:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.195+0000 7f818a553640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81840b09c0 0x7f81840b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.197+0000 7f8189551640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a9ad0 0x7f81841da9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f818000b390 tx=0x7f818000b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.197+0000 7f817a7fc640 10 client.?.objecter ms_handle_connect 0x7f81840a9ad0 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.197+0000 7f817a7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 1 -- 172.21.15.110:0/494452497 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180011830 con 0x7f81840a9ad0 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 10 monclient: dump: 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 1 -- 172.21.15.110:0/494452497 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8180011990 con 0x7f81840a9ad0 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 1 -- 172.21.15.110:0/494452497 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180011d20 con 0x7f81840a9ad0 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f818bfb6640 5 monclient: authenticate success, global_id 5255 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f818bfb6640 10 monclient: _renew_subs 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f818bfb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81740094d0 con 0x7f81840a9ad0 2024-01-20T11:21:21.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817bfff640 4 set_mon_vals no callback set 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f817a7fc640 10 monclient: dump: 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:21.206 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f818bfb6640 10 client.5255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f818bfb6640 10 monclient: _renew_subs 2024-01-20T11:21:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f818bfb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81841cb320 con 0x7f81840a9ad0 2024-01-20T11:21:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.201+0000 7f818bfb6640 1 librados: init done 2024-01-20T11:21:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.202+0000 7f818bfb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.202+0000 7f818bfb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.202+0000 7f818bfb6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:21.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.202+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81840b5df0 con 0x7f81840a9ad0 2024-01-20T11:21:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.225+0000 7f817a7fc640 1 -- 172.21.15.110:0/494452497 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f818001a400 con 0x7f81840a9ad0 2024-01-20T11:21:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.226+0000 7f817a7fc640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f816806c8b0 0x7f816806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.226+0000 7f8189d52640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f816806c8b0 0x7f816806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.226+0000 7f8189d52640 10 monclient: get_auth_request con 0x7f816806c8b0 auth_method 0 2024-01-20T11:21:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.226+0000 7f8189d52640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f816806c8b0 0x7f816806ed70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8174036000 tx=0x7f8174030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:21.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.226+0000 7f817a7fc640 10 client.5255.objecter ms_handle_connect 0x7f816806c8b0 2024-01-20T11:21:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.229+0000 7f817a7fc640 1 -- 172.21.15.110:0/494452497 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8180058ed0 con 0x7f81840a9ad0 2024-01-20T11:21:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.229+0000 7f817a7fc640 10 client.5255.objecter ms_dispatch 0x7f8184000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.229+0000 7f817a7fc640 3 client.5255.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.229+0000 7f817a7fc640 3 client.5255.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:21.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.229+0000 7f817a7fc640 20 client.5255.objecter dump_active .. 0 homeless 2024-01-20T11:21:21.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.508+0000 7f817a7fc640 1 -- 172.21.15.110:0/494452497 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f81800b50d0 con 0x7f81840a9ad0 2024-01-20T11:21:21.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.508+0000 7f817a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:21.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.508+0000 7f817a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:21.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.684+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f81840b4400 con 0x7f816806c8b0 2024-01-20T11:21:21.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.688+0000 7f817a7fc640 1 -- 172.21.15.110:0/494452497 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63868 (secure 0 0 0) 0x7f8174031ab0 con 0x7f816806c8b0 2024-01-20T11:21:21.688 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:21.689 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 10 librados: watch_flush enter 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 10 librados: watch_flush exit 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 20 client.5255.objecter shutdown clearing up homeless session... 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 10 client.5255.objecter successfully canceled tick 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f816806c8b0 msgr2=0x7f816806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f816806c8b0 0x7f816806ed70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8174036000 tx=0x7f8174030000 comp rx=0 tx=0).stop 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 10 monclient: shutdown 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a9ad0 msgr2=0x7f81841da9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a9ad0 0x7f81841da9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f818000b390 tx=0x7f818000b790 comp rx=0 tx=0).stop 2024-01-20T11:21:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 shutdown_connections 2024-01-20T11:21:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f816806c8b0 0x7f816806ed70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81840b09c0 0x7f81840b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81840a9ad0 0x7f81841da9c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 --2- 172.21.15.110:0/494452497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a96f0 0x7f8184139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 >> 172.21.15.110:0/494452497 conn(0x7f81840b7bc0 msgr2=0x7f818413bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 shutdown_connections 2024-01-20T11:21:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 -- 172.21.15.110:0/494452497 wait complete. 2024-01-20T11:21:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.689+0000 7f818bfb6640 1 librados: shutdown 2024-01-20T11:21:21.700 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-01-20T11:21:20.782485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002808"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280591,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.7859999999999996},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.9880000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.513},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3360000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.912000000000001},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.6940000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.5869999999999997},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":13.935}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.813000000000001},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":10.375}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.15},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":14.85}]}]},{"osd":7,"up_from":21,"seq":90194313293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":6.5419999999999998},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":7.8259999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":11.054},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.0340000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":10.335000000000001},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":17.547999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":12.122999999999999},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.702}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.75},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":14.176}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.524000000000001},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.478999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313294,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.6980000000000004},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":7.5350000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":8.4130000000000003},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":11.579000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":16.210000000000001},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":9.8309999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.207000000000001},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":13.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":10.852},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.788}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":15.385},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":12.58}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.2309999999999999},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":6.25}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.396000000000001},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.9699999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.1890000000000001},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.4619999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.090999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.7750000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.771000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.08}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.372},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.029999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.8630000000000004},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":6.1159999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":7.5759999999999996},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.8510000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":14.378},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.5489999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":8.1630000000000003},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.66}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.723000000000001},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":10.173999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.118},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.93}]}]},{"osd":0,"up_from":17,"seq":73014444109,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":9.0419999999999998},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":10.16}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.609},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.693}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":5.2389999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":6.5419999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.722},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.366}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":14.377000000000001},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":8.2599999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.061999999999999},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":7.2220000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":11.161},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":5.4729999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":9.5540000000000003},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":12.929}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":6.8620000000000001},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":11.565}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":15.958},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":8.2189999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":8.8510000000000009},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.397}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":15.194000000000001},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.289999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:21.702 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:21.702 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:21.702 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:21.702 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:21.702 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:21.702 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:21.702 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:21.702 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:21.703 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:21.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.800+0000 7f4f19f89640 10 monclient: get_monmap_and_config 2024-01-20T11:21:21.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.800+0000 7f4f19f89640 10 monclient: build_initial_monmap 2024-01-20T11:21:21.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.800+0000 7f4f19f89640 10 monclient: monmap: 2024-01-20T11:21:21.801 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:21.801 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:21.802180+0000 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:21.802180+0000 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:21.802 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 1 Processor -- start 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 1 -- start start 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 10 monclient: init 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f140af3a0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4f140ab6d0 0x7f4f140ad420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 10 monclient(hunting): picked mon.noname-c con 0x7f4f140ab6d0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f140ad960 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f14141930 con 0x7f4f140ad960 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f140ab2b0 con 0x7f4f140af3a0 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f14142920 con 0x7f4f140ab6d0 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.801+0000 7f4f19f89640 10 monclient(hunting): authenticate will time out at 1725.153132s 2024-01-20T11:21:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.802+0000 7f4f12ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4f140ab6d0 0x7f4f140ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.802+0000 7f4f13fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4f140ad960 0x7f4f14148cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50522/0 (socket says 172.21.15.110:50522) 2024-01-20T11:21:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.802+0000 7f4f13fff640 1 -- 172.21.15.110:0/1412611321 learned_addr learned my addr 172.21.15.110:0/1412611321 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.803+0000 7f4f12ffd640 10 monclient(hunting): get_auth_request con 0x7f4f140ab6d0 auth_method 0 2024-01-20T11:21:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.803+0000 7f4f12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.803+0000 7f4f12ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.803+0000 7f4f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.803+0000 7f4f12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:21.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.803+0000 7f4f12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:21.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.803+0000 7f4f12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:21.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.804+0000 7f4f127fc640 1 -- 172.21.15.110:0/1412611321 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780886192 0 0) 0x7f4f14141930 con 0x7f4f140ad960 2024-01-20T11:21:21.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.804+0000 7f4f127fc640 10 monclient(hunting): _negotiate 2024-01-20T11:21:21.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.804+0000 7f4f127fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:21.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.804+0000 7f4f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:21.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.804+0000 7f4f127fc640 10 monclient(hunting): my global_id is 5526 2024-01-20T11:21:21.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.804+0000 7f4f127fc640 1 -- 172.21.15.110:0/1412611321 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f00003520 con 0x7f4f140ad960 2024-01-20T11:21:21.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.805+0000 7f4f12ffd640 10 monclient(hunting): handle_auth_done global_id 5602 payload 274 2024-01-20T11:21:21.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.805+0000 7f4f12ffd640 1 -- 172.21.15.110:0/1412611321 >> v1:172.21.15.134:6789/0 conn(0x7f4f140af3a0 legacy=0x7f4f1413b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.805+0000 7f4f12ffd640 1 -- 172.21.15.110:0/1412611321 >> v1:172.21.15.110:6789/0 conn(0x7f4f140ad960 legacy=0x7f4f14148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.805+0000 7f4f12ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:21.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.805+0000 7f4f12ffd640 1 monclient: found mon.noname-c 2024-01-20T11:21:21.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.805+0000 7f4f12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:21.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.805+0000 7f4f12ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:21:21.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.805+0000 7f4f12ffd640 1 -- 172.21.15.110:0/1412611321 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f140aca40 con 0x7f4f140ab6d0 2024-01-20T11:21:21.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.810+0000 7f4f12ffd640 1 --2- 172.21.15.110:0/1412611321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140ab6d0 0x7f4f140ad420 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4f0400a940 tx=0x7f4f0402ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4156f63663597b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:21.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 1 -- 172.21.15.110:0/1412611321 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0402dbe0 con 0x7f4f140ab6d0 2024-01-20T11:21:21.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 10 monclient: dump: 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:21.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 1 -- 172.21.15.110:0/1412611321 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0402dd40 con 0x7f4f140ab6d0 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 1 -- 172.21.15.110:0/1412611321 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04035660 con 0x7f4f140ab6d0 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f19f89640 5 monclient: authenticate success, global_id 5602 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f19f89640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f19f89640 4 set_mon_vals no callback set 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.813+0000 7f4f127fc640 10 monclient: dump: 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.816 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.817 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:21.817 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.817 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:21.817 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:21.817 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:21.817 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.814+0000 7f4f19f89640 10 monclient: shutdown 2024-01-20T11:21:21.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.814+0000 7f4f19f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:21.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.814+0000 7f4f19f89640 1 -- 172.21.15.110:0/1412611321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140ab6d0 msgr2=0x7f4f140ad420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.814+0000 7f4f19f89640 1 --2- 172.21.15.110:0/1412611321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140ab6d0 0x7f4f140ad420 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4f0400a940 tx=0x7f4f0402ce30 comp rx=0 tx=0).stop 2024-01-20T11:21:21.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.814+0000 7f4f19f89640 1 -- 172.21.15.110:0/1412611321 shutdown_connections 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.814+0000 7f4f19f89640 1 --2- 172.21.15.110:0/1412611321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140ab6d0 0x7f4f140ad420 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.814+0000 7f4f19f89640 1 -- 172.21.15.110:0/1412611321 >> 172.21.15.110:0/1412611321 conn(0x7f4f140b3e60 msgr2=0x7f4f140b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.814+0000 7f4f19f89640 1 -- 172.21.15.110:0/1412611321 shutdown_connections 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.814+0000 7f4f19f89640 1 -- 172.21.15.110:0/1412611321 wait complete. 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.815+0000 7f4f19f89640 10 monclient: build_initial_monmap 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.815+0000 7f4f19f89640 10 monclient: monmap: 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:21.821 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.815+0000 7f4f19f89640 1 librados: starting msgr at 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.815+0000 7f4f19f89640 1 librados: starting objecter 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.815+0000 7f4f19f89640 1 Processor -- start 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 1 -- start start 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 1 librados: setting wanted keys 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 1 librados: calling monclient init 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient: init 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f140ab6d0 0x7f4f1406d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient(hunting): picked mon.noname-a con 0x7f4f140ab6d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f140ad960 0x7f4f141380e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient(hunting): picked mon.noname-c con 0x7f4f140ad960 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140af3a0 0x7f4f14138620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient(hunting): picked mon.noname-b con 0x7f4f140af3a0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f14138b60 con 0x7f4f140ab6d0 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1406e6a0 con 0x7f4f140af3a0 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1406e910 con 0x7f4f140ad960 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.816+0000 7f4f19f89640 10 monclient(hunting): authenticate will time out at 1725.167787s 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f137fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f140ab6d0 0x7f4f1406d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f12ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f140ad960 0x7f4f141380e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140af3a0 0x7f4f14138620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140af3a0 0x7f4f14138620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:54208/0 (socket says 172.21.15.110:54208) 2024-01-20T11:21:21.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 1 -- 172.21.15.110:0/2305840266 learned_addr learned my addr 172.21.15.110:0/2305840266 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 10 monclient(hunting): get_auth_request con 0x7f4f140af3a0 auth_method 0 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f137fe640 10 monclient(hunting): get_auth_request con 0x7f4f140ab6d0 auth_method 0 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f137fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f12ffd640 10 monclient(hunting): get_auth_request con 0x7f4f140ad960 auth_method 0 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f12ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.817+0000 7f4f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.818+0000 7f4f137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.818+0000 7f4f137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.818+0000 7f4f137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.818+0000 7f4f12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.818+0000 7f4f12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.818+0000 7f4f12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 10 monclient(hunting): handle_auth_done global_id 5608 payload 824 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 1 -- 172.21.15.110:0/2305840266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f140ad960 msgr2=0x7f4f141380e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f140ad960 0x7f4f141380e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 1 -- 172.21.15.110:0/2305840266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f140ab6d0 msgr2=0x7f4f1406d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f140ab6d0 0x7f4f1406d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 1 monclient: found mon.noname-b 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.820+0000 7f4f13fff640 1 -- 172.21.15.110:0/2305840266 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0400a5a0 con 0x7f4f140af3a0 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.822+0000 7f4f13fff640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140af3a0 0x7f4f14138620 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00cb40 tx=0x7f4f0c00a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:21.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.822+0000 7f4f0bfff640 10 client.?.objecter ms_handle_connect 0x7f4f140af3a0 2024-01-20T11:21:21.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.822+0000 7f4f0bfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 1 -- 172.21.15.110:0/2305840266 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c00aca0 con 0x7f4f140af3a0 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 10 monclient: dump: 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:21.828 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:21.829 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 10 monclient: _finish_auth 0 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 1 -- 172.21.15.110:0/2305840266 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c00ae00 con 0x7f4f140af3a0 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 1 -- 172.21.15.110:0/2305840266 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c012620 con 0x7f4f140af3a0 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f19f89640 5 monclient: authenticate success, global_id 5608 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f19f89640 10 monclient: _renew_subs 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f19f89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1406ebf0 con 0x7f4f140af3a0 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f11ffb640 4 set_mon_vals no callback set 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:21.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f0bfff640 10 monclient: dump: 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f19f89640 10 client.5608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f19f89640 10 monclient: _renew_subs 2024-01-20T11:21:21.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f19f89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:21.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1413ae60 con 0x7f4f140af3a0 2024-01-20T11:21:21.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.826+0000 7f4f19f89640 1 librados: init done 2024-01-20T11:21:21.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.827+0000 7f4f19f89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:21.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.827+0000 7f4f19f89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:21.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.827+0000 7f4f19f89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:21.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.827+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f14075490 con 0x7f4f140af3a0 2024-01-20T11:21:21.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.845+0000 7f4f0bfff640 1 -- 172.21.15.110:0/2305840266 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4f0c007070 con 0x7f4f140af3a0 2024-01-20T11:21:21.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.846+0000 7f4f0bfff640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c890 0x7f4ef406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:21.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.846+0000 7f4f137fe640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c890 0x7f4ef406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:21.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.846+0000 7f4f137fe640 10 monclient: get_auth_request con 0x7f4ef406c890 auth_method 0 2024-01-20T11:21:21.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.847+0000 7f4f137fe640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c890 0x7f4ef406ed50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4efc001780 tx=0x7f4efc009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:21.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:21.847+0000 7f4f0bfff640 10 client.5608.objecter ms_handle_connect 0x7f4ef406c890 2024-01-20T11:21:22.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.072+0000 7f4f0bfff640 1 -- 172.21.15.110:0/2305840266 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4f0c058620 con 0x7f4f140af3a0 2024-01-20T11:21:22.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.072+0000 7f4f0bfff640 10 client.5608.objecter ms_dispatch 0x7f4f14000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:22.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.072+0000 7f4f0bfff640 3 client.5608.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:22.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.072+0000 7f4f0bfff640 3 client.5608.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:22.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.072+0000 7f4f0bfff640 20 client.5608.objecter dump_active .. 0 homeless 2024-01-20T11:21:22.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.081+0000 7f4f0bfff640 1 -- 172.21.15.110:0/2305840266 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4f0c054960 con 0x7f4f140af3a0 2024-01-20T11:21:22.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.081+0000 7f4f0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:22.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.081+0000 7f4f0bfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:22.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.314+0000 7f4f19f89640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:22.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.314+0000 7f4f19f89640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:22.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.314+0000 7f4f19f89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:22.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.314+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f4f140b3200 con 0x7f4f140af3a0 2024-01-20T11:21:22.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.320+0000 7f4f0bfff640 1 -- 172.21.15.110:0/2305840266 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f4f0c017080 con 0x7f4f140af3a0 2024-01-20T11:21:22.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.320+0000 7f4f0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:22.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.320+0000 7f4f0bfff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:22.321 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:22.321 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":416,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 10 librados: watch_flush enter 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 10 librados: watch_flush exit 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 20 client.5608.objecter shutdown clearing up homeless session... 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 10 client.5608.objecter successfully canceled tick 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c890 msgr2=0x7f4ef406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c890 0x7f4ef406ed50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4efc001780 tx=0x7f4efc009040 comp rx=0 tx=0).stop 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 10 monclient: shutdown 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140af3a0 msgr2=0x7f4f14138620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140af3a0 0x7f4f14138620 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00cb40 tx=0x7f4f0c00a060 comp rx=0 tx=0).stop 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 shutdown_connections 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4ef406c890 0x7f4ef406ed50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f140af3a0 0x7f4f14138620 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f140ad960 0x7f4f141380e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 --2- 172.21.15.110:0/2305840266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f140ab6d0 0x7f4f1406d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 >> 172.21.15.110:0/2305840266 conn(0x7f4f140b3e60 msgr2=0x7f4f140a9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 shutdown_connections 2024-01-20T11:21:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 -- 172.21.15.110:0/2305840266 wait complete. 2024-01-20T11:21:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.322+0000 7f4f19f89640 1 librados: shutdown 2024-01-20T11:21:22.334 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:22.335 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:22.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.435+0000 7f95d05a4640 10 monclient: get_monmap_and_config 2024-01-20T11:21:22.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.435+0000 7f95d05a4640 10 monclient: build_initial_monmap 2024-01-20T11:21:22.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.435+0000 7f95d05a4640 10 monclient: monmap: 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:22.437406+0000 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:22.437406+0000 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:22.437 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:22.438 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:22.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.436+0000 7f95d05a4640 1 Processor -- start 2024-01-20T11:21:22.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.436+0000 7f95d05a4640 1 -- start start 2024-01-20T11:21:22.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.436+0000 7f95d05a4640 10 monclient: init 2024-01-20T11:21:22.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:22.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-20T11:21:22.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95c80b0ba0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:22.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f95c80adc30 0x7f95c80ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:22.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 10 monclient(hunting): picked mon.noname-a con 0x7f95c80adc30 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95c80ac4d0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c80ad530 con 0x7f95c80b0ba0 2024-01-20T11:21:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c80ad810 con 0x7f95c80ac4d0 2024-01-20T11:21:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c80af840 con 0x7f95c80adc30 2024-01-20T11:21:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.437+0000 7f95d05a4640 10 monclient(hunting): authenticate will time out at 1725.788371s 2024-01-20T11:21:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.438+0000 7f95ceb41640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f95c80ac4d0 0x7f95c813bf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:33884/0 (socket says 172.21.15.110:33884) 2024-01-20T11:21:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.438+0000 7f95ceb41640 1 -- 172.21.15.110:0/409475704 learned_addr learned my addr 172.21.15.110:0/409475704 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.439+0000 7f95cdb3f640 1 --2- 172.21.15.110:0/409475704 >> v2:172.21.15.110:3300/0 conn(0x7f95c80adc30 0x7f95c80ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.439+0000 7f95cdb3f640 10 monclient(hunting): get_auth_request con 0x7f95c80adc30 auth_method 0 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.439+0000 7f95cdb3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.439+0000 7f95cdb3f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.439+0000 7f95cdb3f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.440+0000 7f95cdb3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.440+0000 7f95cdb3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.440+0000 7f95cdb3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.440+0000 7f95cd33e640 1 -- 172.21.15.110:0/409475704 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317456480 0 0) 0x7f95c80ad810 con 0x7f95c80ac4d0 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.440+0000 7f95cd33e640 10 monclient(hunting): _negotiate 2024-01-20T11:21:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.440+0000 7f95cd33e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.440+0000 7f95cd33e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.440+0000 7f95cd33e640 10 monclient(hunting): my global_id is 5611 2024-01-20T11:21:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.440+0000 7f95cd33e640 1 -- 172.21.15.110:0/409475704 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b4005730 con 0x7f95c80ac4d0 2024-01-20T11:21:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.441+0000 7f95cdb3f640 10 monclient(hunting): handle_auth_done global_id 5532 payload 274 2024-01-20T11:21:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.441+0000 7f95cdb3f640 1 -- 172.21.15.110:0/409475704 >> v1:172.21.15.134:6789/0 conn(0x7f95c80ac4d0 legacy=0x7f95c813bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.442+0000 7f95cdb3f640 1 -- 172.21.15.110:0/409475704 >> v1:172.21.15.110:6789/0 conn(0x7f95c80b0ba0 legacy=0x7f95c80af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.442+0000 7f95cdb3f640 10 monclient: _finish_hunting 0 2024-01-20T11:21:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.442+0000 7f95cdb3f640 1 monclient: found mon.noname-a 2024-01-20T11:21:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.442+0000 7f95cdb3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.442+0000 7f95cdb3f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.442+0000 7f95cdb3f640 1 -- 172.21.15.110:0/409475704 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c80ab110 con 0x7f95c80adc30 2024-01-20T11:21:22.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.444+0000 7f95cdb3f640 1 --2- 172.21.15.110:0/409475704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80adc30 0x7f95c80ab630 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f95b80097a0 tx=0x7f95b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20487c056c6b0cdb server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 1 -- 172.21.15.110:0/409475704 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8031830 con 0x7f95c80adc30 2024-01-20T11:21:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 10 monclient: dump: 2024-01-20T11:21:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 10 monclient: _finish_auth 0 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 10 monclient: _check_auth_tickets 2024-01-20T11:21:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 1 -- 172.21.15.110:0/409475704 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b8031990 con 0x7f95c80adc30 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95d05a4640 5 monclient: authenticate success, global_id 5532 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95d05a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95d05a4640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95d05a4640 4 set_mon_vals no callback set 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 1 -- 172.21.15.110:0/409475704 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8031d20 con 0x7f95c80adc30 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.448+0000 7f95cd33e640 10 monclient: dump: 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:22.451 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:22.452 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:22.452 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.449+0000 7f95d05a4640 10 monclient: shutdown 2024-01-20T11:21:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.449+0000 7f95d05a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.449+0000 7f95d05a4640 1 -- 172.21.15.110:0/409475704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80adc30 msgr2=0x7f95c80ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.449+0000 7f95d05a4640 1 --2- 172.21.15.110:0/409475704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80adc30 0x7f95c80ab630 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f95b80097a0 tx=0x7f95b802b8d0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.449+0000 7f95d05a4640 1 -- 172.21.15.110:0/409475704 shutdown_connections 2024-01-20T11:21:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.449+0000 7f95d05a4640 1 --2- 172.21.15.110:0/409475704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80adc30 0x7f95c80ab630 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.449+0000 7f95d05a4640 1 -- 172.21.15.110:0/409475704 >> 172.21.15.110:0/409475704 conn(0x7f95c80b3e60 msgr2=0x7f95c80b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.449+0000 7f95d05a4640 1 -- 172.21.15.110:0/409475704 shutdown_connections 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.449+0000 7f95d05a4640 1 -- 172.21.15.110:0/409475704 wait complete. 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.450+0000 7f95d05a4640 10 monclient: build_initial_monmap 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.450+0000 7f95d05a4640 10 monclient: monmap: 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.450+0000 7f95d05a4640 1 librados: starting msgr at 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.450+0000 7f95d05a4640 1 librados: starting objecter 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.450+0000 7f95d05a4640 1 Processor -- start 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.450+0000 7f95d05a4640 1 -- start start 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.450+0000 7f95d05a4640 1 librados: setting wanted keys 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.450+0000 7f95d05a4640 1 librados: calling monclient init 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.450+0000 7f95d05a4640 10 monclient: init 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95c80ac4d0 0x7f95c81d61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:22.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient(hunting): picked mon.noname-b con 0x7f95c80ac4d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80b0ba0 0x7f95c81411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient(hunting): picked mon.noname-a con 0x7f95c80b0ba0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95c8140810 0x7f95c8141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient(hunting): picked mon.noname-c con 0x7f95c8140810 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c8134520 con 0x7f95c80b0ba0 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c8142ee0 con 0x7f95c80ac4d0 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c8140ca0 con 0x7f95c8140810 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:22.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95d05a4640 10 monclient(hunting): authenticate will time out at 1725.802305s 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95cdb3f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80b0ba0 0x7f95c81411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95ce340640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95c80ac4d0 0x7f95c81d61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95ceb41640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95c8140810 0x7f95c8141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95cdb3f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80b0ba0 0x7f95c81411c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47412/0 (socket says 172.21.15.110:47412) 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95cdb3f640 1 -- 172.21.15.110:0/1446294902 learned_addr learned my addr 172.21.15.110:0/1446294902 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95cdb3f640 10 monclient(hunting): get_auth_request con 0x7f95c80b0ba0 auth_method 0 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95cdb3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95cdb3f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.451+0000 7f95cdb3f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ce340640 10 monclient(hunting): get_auth_request con 0x7f95c80ac4d0 auth_method 0 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ce340640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ce340640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:22.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ce340640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ceb41640 10 monclient(hunting): get_auth_request con 0x7f95c8140810 auth_method 0 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ceb41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ceb41640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ceb41640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95cdb3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95cdb3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95cdb3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ce340640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ce340640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ce340640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ceb41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ceb41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:22.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.452+0000 7f95ceb41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 10 monclient(hunting): handle_auth_done global_id 5535 payload 824 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 1 -- 172.21.15.110:0/1446294902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95c8140810 msgr2=0x7f95c8141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95c8140810 0x7f95c8141700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 1 -- 172.21.15.110:0/1446294902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95c80ac4d0 msgr2=0x7f95c81d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95c80ac4d0 0x7f95c81d61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 10 monclient: _finish_hunting 0 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 1 monclient: found mon.noname-a 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:22.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.455+0000 7f95cdb3f640 1 -- 172.21.15.110:0/1446294902 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b8009400 con 0x7f95c80b0ba0 2024-01-20T11:21:22.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.457+0000 7f95cdb3f640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80b0ba0 0x7f95c81411c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f95b802bee0 tx=0x7f95b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:22.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.457+0000 7f95beffd640 10 client.?.objecter ms_handle_connect 0x7f95c80b0ba0 2024-01-20T11:21:22.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.457+0000 7f95beffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:22.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 1 -- 172.21.15.110:0/1446294902 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b803a6f0 con 0x7f95c80b0ba0 2024-01-20T11:21:22.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:22.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:22.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 10 monclient: dump: 2024-01-20T11:21:22.464 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 10 monclient: _finish_auth 0 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 1 -- 172.21.15.110:0/1446294902 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b803a870 con 0x7f95c80b0ba0 2024-01-20T11:21:22.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95d05a4640 5 monclient: authenticate success, global_id 5535 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95d05a4640 10 monclient: _renew_subs 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95d05a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c8141c40 con 0x7f95c80b0ba0 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 1 -- 172.21.15.110:0/1446294902 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b803ac20 con 0x7f95c80b0ba0 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95beffd640 10 monclient: dump: 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:22.466 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95ccb3d640 4 set_mon_vals no callback set 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95d05a4640 10 client.5535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95d05a4640 10 monclient: _renew_subs 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95d05a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c81d6dd0 con 0x7f95c80b0ba0 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.460+0000 7f95d05a4640 1 librados: init done 2024-01-20T11:21:22.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.461+0000 7f95d05a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.461+0000 7f95d05a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.461+0000 7f95d05a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:22.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.461+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c8075490 con 0x7f95c80b0ba0 2024-01-20T11:21:22.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.479+0000 7f95beffd640 1 -- 172.21.15.110:0/1446294902 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f95b804d440 con 0x7f95c80b0ba0 2024-01-20T11:21:22.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.479+0000 7f95beffd640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f95ac06c8b0 0x7f95ac06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:22.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.479+0000 7f95ce340640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f95ac06c8b0 0x7f95ac06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:22.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.479+0000 7f95ce340640 10 monclient: get_auth_request con 0x7f95ac06c8b0 auth_method 0 2024-01-20T11:21:22.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.480+0000 7f95ce340640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f95ac06c8b0 0x7f95ac06ed70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f95c40016e0 tx=0x7f95c4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:22.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.480+0000 7f95beffd640 10 client.5535.objecter ms_handle_connect 0x7f95ac06c8b0 2024-01-20T11:21:22.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.481+0000 7f95beffd640 1 -- 172.21.15.110:0/1446294902 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f95b807a360 con 0x7f95c80b0ba0 2024-01-20T11:21:22.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.481+0000 7f95beffd640 10 client.5535.objecter ms_dispatch 0x7f95c8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:22.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.481+0000 7f95beffd640 3 client.5535.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:22.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.481+0000 7f95beffd640 3 client.5535.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:22.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.481+0000 7f95beffd640 20 client.5535.objecter dump_active .. 0 homeless 2024-01-20T11:21:22.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.710+0000 7f95beffd640 1 -- 172.21.15.110:0/1446294902 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f95b80766a0 con 0x7f95c80b0ba0 2024-01-20T11:21:22.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.710+0000 7f95beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:22.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.710+0000 7f95beffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:22.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.909+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95c80b0f80 con 0x7f95ac06c8b0 2024-01-20T11:21:22.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.913+0000 7f95beffd640 1 -- 172.21.15.110:0/1446294902 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63856 (secure 0 0 0) 0x7f95c4001cc0 con 0x7f95ac06c8b0 2024-01-20T11:21:22.914 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:22.914 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:22.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 10 librados: watch_flush enter 2024-01-20T11:21:22.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 10 librados: watch_flush exit 2024-01-20T11:21:22.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 20 client.5535.objecter shutdown clearing up homeless session... 2024-01-20T11:21:22.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 10 client.5535.objecter successfully canceled tick 2024-01-20T11:21:22.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f95ac06c8b0 msgr2=0x7f95ac06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:22.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f95ac06c8b0 0x7f95ac06ed70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f95c40016e0 tx=0x7f95c4009040 comp rx=0 tx=0).stop 2024-01-20T11:21:22.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 10 monclient: shutdown 2024-01-20T11:21:22.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80b0ba0 msgr2=0x7f95c81411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80b0ba0 0x7f95c81411c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f95b802bee0 tx=0x7f95b80046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 shutdown_connections 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f95ac06c8b0 0x7f95ac06ed70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95c8140810 0x7f95c8141700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95c80b0ba0 0x7f95c81411c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 --2- 172.21.15.110:0/1446294902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95c80ac4d0 0x7f95c81d61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 >> 172.21.15.110:0/1446294902 conn(0x7f95c80b3e60 msgr2=0x7f95c80aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 shutdown_connections 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 -- 172.21.15.110:0/1446294902 wait complete. 2024-01-20T11:21:22.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:22.914+0000 7f95d05a4640 1 librados: shutdown 2024-01-20T11:21:22.925 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-01-20T11:21:22.782802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002602"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280591,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":6.7859999999999996},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.9880000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":12.513},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.3360000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":11.912000000000001},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":9.6940000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":7.5869999999999997},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":13.935}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":15.813000000000001},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":10.375}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":11.15},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":14.85}]}]},{"osd":7,"up_from":21,"seq":90194313294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.2949999999999999},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.0250000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":10.56},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":14.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":16.603999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.4190000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.16},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.478000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.177},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":11.617000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":9.1660000000000004},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.237}]}]},{"osd":3,"up_from":21,"seq":90194313295,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.0110000000000001},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.7840000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.9850000000000003},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.4540000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.069000000000001},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.754}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.256},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":14.087}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":13.134},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":16.966000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":18.491},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.973000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":7.7380000000000004,"5min":8.6430000000000007,"15min":9.8190000000000008},"min":{"1min":4.2640000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.265000000000001,"5min":25.603999999999999,"15min":25.603999999999999},"last":5.2309999999999999},{"interface":"front","average":{"1min":7.3230000000000004,"5min":8.3849999999999998,"15min":8.5389999999999997},"min":{"1min":4.3230000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.246,"5min":16.163,"15min":16.163},"last":6.25}]},{"osd":1,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":8.9499999999999993,"5min":9.8529999999999998,"15min":12.811},"min":{"1min":5.5389999999999997,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":15.747,"5min":65.064999999999998,"15min":65.064999999999998},"last":11.396000000000001},{"interface":"front","average":{"1min":11.500999999999999,"5min":12.505000000000001,"15min":15.385999999999999},"min":{"1min":6.9619999999999997,"5min":5.0460000000000003,"15min":5.0460000000000003},"max":{"1min":19.446000000000002,"5min":32.414999999999999,"15min":32.414999999999999},"last":6.9699999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":8.1890000000000001},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":9.4619999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":10.090999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":8.7750000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:20:22 2024","interfaces":[{"interface":"back","average":{"1min":13.766999999999999,"5min":14.106999999999999,"15min":15.419},"min":{"1min":8.8170000000000002,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":16.760000000000002,"5min":35.186999999999998,"15min":35.186999999999998},"last":13.771000000000001},{"interface":"front","average":{"1min":11.173,"5min":10.819000000000001,"15min":10.904999999999999},"min":{"1min":6.7709999999999999,"5min":4.7850000000000001,"15min":4.7850000000000001},"max":{"1min":15.689,"5min":32.984000000000002,"15min":32.984000000000002},"last":12.08}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":14.372},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.029999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.8630000000000004},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":6.1159999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":7.5759999999999996},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.8510000000000009}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":14.378},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.5489999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":8.1630000000000003},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.66}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.723000000000001},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":10.173999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":15.118},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.93}]}]},{"osd":0,"up_from":17,"seq":73014444109,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":9.0419999999999998},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":10.16}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":9.609},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":12.693}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":5.2389999999999999},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":6.5419999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":10.722},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":11.366}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":14.377000000000001},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":8.2599999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.061999999999999},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":7.2220000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":8.9100000000000001},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.8449999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":10.532},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":15.093}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6159999999999997},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.7270000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.427},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.436}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.509},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.180999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.978999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.679}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:25.928 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:26.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.029+0000 7fb73dd9e640 10 monclient: get_monmap_and_config 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.029+0000 7fb73dd9e640 10 monclient: build_initial_monmap 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.029+0000 7fb73dd9e640 10 monclient: monmap: 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:26.030662+0000 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:26.030662+0000 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:26.030 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.029+0000 7fb73dd9e640 1 Processor -- start 2024-01-20T11:21:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 1 -- start start 2024-01-20T11:21:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient: init 2024-01-20T11:21:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-01-20T11:21:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7380a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 0x7fb7381423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient(hunting): picked mon.noname-b con 0x7fb7380a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb738142920 0x7fb73814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient(hunting): picked mon.noname-a con 0x7fb738142920 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7380b25c0 con 0x7fb7380a96f0 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7380aaea0 con 0x7fb738142920 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73813fbe0 con 0x7fb7380a9ad0 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.030+0000 7fb73dd9e640 10 monclient(hunting): authenticate will time out at 1729.381832s 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.031+0000 7fb72ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 0x7fb7381423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.031+0000 7fb7377fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fb7380a96f0 0x7fb73813f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:33890/0 (socket says 172.21.15.110:33890) 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.031+0000 7fb7377fe640 1 -- 172.21.15.110:0/808512 learned_addr learned my addr 172.21.15.110:0/808512 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.031+0000 7fb72ffff640 10 monclient(hunting): get_auth_request con 0x7fb7380a9ad0 auth_method 0 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.031+0000 7fb72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.031+0000 7fb72ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:26.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.031+0000 7fb72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:26.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.032+0000 7fb72ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:26.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.032+0000 7fb72ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:26.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.032+0000 7fb72ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:26.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.034+0000 7fb72ffff640 10 monclient(hunting): handle_auth_done global_id 5264 payload 274 2024-01-20T11:21:26.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.034+0000 7fb72ffff640 1 -- 172.21.15.110:0/808512 >> v2:172.21.15.110:3300/0 conn(0x7fb738142920 msgr2=0x7fb73814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:26.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.034+0000 7fb72ffff640 1 --2- 172.21.15.110:0/808512 >> v2:172.21.15.110:3300/0 conn(0x7fb738142920 0x7fb73814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.034+0000 7fb72ffff640 1 -- 172.21.15.110:0/808512 >> v1:172.21.15.134:6789/0 conn(0x7fb7380a96f0 legacy=0x7fb73813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.034+0000 7fb72ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:26.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.034+0000 7fb72ffff640 1 monclient: found mon.noname-b 2024-01-20T11:21:26.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.034+0000 7fb72ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:26.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.034+0000 7fb72ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:26.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.034+0000 7fb72ffff640 1 -- 172.21.15.110:0/808512 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380ab0a0 con 0x7fb7380a9ad0 2024-01-20T11:21:26.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.038+0000 7fb72ffff640 1 --2- 172.21.15.110:0/808512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 0x7fb7381423e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb7200097a0 tx=0x7fb72002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=320521619591414a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:26.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 1 -- 172.21.15.110:0/808512 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72002e070 con 0x7fb7380a9ad0 2024-01-20T11:21:26.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:26.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:26.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 10 monclient: dump: 2024-01-20T11:21:26.088 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:26.088 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.088 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 10 monclient: _finish_auth 0 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 1 -- 172.21.15.110:0/808512 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7200315a0 con 0x7fb7380a9ad0 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 1 -- 172.21.15.110:0/808512 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720031930 con 0x7fb7380a9ad0 2024-01-20T11:21:26.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb73dd9e640 5 monclient: authenticate success, global_id 5264 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb73dd9e640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb73dd9e640 4 set_mon_vals no callback set 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.087+0000 7fb736ffd640 10 monclient: dump: 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:26.090 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 10 monclient: shutdown 2024-01-20T11:21:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 1 -- 172.21.15.110:0/808512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 msgr2=0x7fb7381423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 1 --2- 172.21.15.110:0/808512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 0x7fb7381423e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb7200097a0 tx=0x7fb72002b840 comp rx=0 tx=0).stop 2024-01-20T11:21:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 1 -- 172.21.15.110:0/808512 shutdown_connections 2024-01-20T11:21:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 1 --2- 172.21.15.110:0/808512 >> v2:172.21.15.110:3300/0 conn(0x7fb738142920 0x7fb73814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 1 --2- 172.21.15.110:0/808512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 0x7fb7381423e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 1 -- 172.21.15.110:0/808512 >> 172.21.15.110:0/808512 conn(0x7fb7380b7bc0 msgr2=0x7fb7380b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 1 -- 172.21.15.110:0/808512 shutdown_connections 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.088+0000 7fb73dd9e640 1 -- 172.21.15.110:0/808512 wait complete. 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.089+0000 7fb73dd9e640 10 monclient: build_initial_monmap 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.089+0000 7fb73dd9e640 10 monclient: monmap: 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.089+0000 7fb73dd9e640 1 librados: starting msgr at 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.089+0000 7fb73dd9e640 1 librados: starting objecter 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.089+0000 7fb73dd9e640 1 Processor -- start 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.089+0000 7fb73dd9e640 1 -- start start 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.089+0000 7fb73dd9e640 1 librados: setting wanted keys 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.089+0000 7fb73dd9e640 1 librados: calling monclient init 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.089+0000 7fb73dd9e640 10 monclient: init 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7380a96f0 0x7fb7380b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient(hunting): picked mon.noname-a con 0x7fb7380a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 0x7fb7380af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient(hunting): picked mon.noname-c con 0x7fb7380a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb738142920 0x7fb7381d7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient(hunting): picked mon.noname-b con 0x7fb738142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380acd10 con 0x7fb7380a96f0 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380ace80 con 0x7fb738142920 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380b0ad0 con 0x7fb7380a9ad0 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb73dd9e640 10 monclient(hunting): authenticate will time out at 1729.441499s 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.090+0000 7fb737fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb738142920 0x7fb7381d7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb737fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb738142920 0x7fb7381d7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:54230/0 (socket says 172.21.15.110:54230) 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb737fff640 1 -- 172.21.15.110:0/2512149259 learned_addr learned my addr 172.21.15.110:0/2512149259 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb737fff640 10 monclient(hunting): get_auth_request con 0x7fb738142920 auth_method 0 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb737fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb737fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb7377fe640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7380a96f0 0x7fb7380b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb72ffff640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 0x7fb7380af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.091+0000 7fb737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb7377fe640 10 monclient(hunting): get_auth_request con 0x7fb7380a96f0 auth_method 0 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb7377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb7377fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb7377fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb7377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb7377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb7377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb72ffff640 10 monclient(hunting): get_auth_request con 0x7fb7380a9ad0 auth_method 0 2024-01-20T11:21:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:26.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb72ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:26.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.092+0000 7fb72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:26.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.093+0000 7fb72ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:26.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.093+0000 7fb72ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:26.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.093+0000 7fb72ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:26.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 10 monclient(hunting): handle_auth_done global_id 5617 payload 824 2024-01-20T11:21:26.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 1 -- 172.21.15.110:0/2512149259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 msgr2=0x7fb7380af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 0x7fb7380af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 1 -- 172.21.15.110:0/2512149259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7380a96f0 msgr2=0x7fb7380b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7380a96f0 0x7fb7380b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 1 monclient: found mon.noname-b 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.094+0000 7fb737fff640 1 -- 172.21.15.110:0/2512149259 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720009400 con 0x7fb738142920 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.097+0000 7fb737fff640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb738142920 0x7fb7381d7490 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb73000b9e0 tx=0x7fb73000bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.097+0000 7fb734ff9640 10 client.?.objecter ms_handle_connect 0x7fb738142920 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.097+0000 7fb734ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb734ff9640 1 -- 172.21.15.110:0/2512149259 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73000cb20 con 0x7fb738142920 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb734ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:26.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb734ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb734ff9640 10 monclient: dump: 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb734ff9640 10 monclient: _finish_auth 0 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb734ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb734ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb73dd9e640 5 monclient: authenticate success, global_id 5617 2024-01-20T11:21:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb73dd9e640 10 monclient: _renew_subs 2024-01-20T11:21:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb73dd9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7380afd60 con 0x7fb738142920 2024-01-20T11:21:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb73dd9e640 10 client.5617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb73dd9e640 10 monclient: _renew_subs 2024-01-20T11:21:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb73dd9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7381d7bb0 con 0x7fb738142920 2024-01-20T11:21:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.099+0000 7fb73dd9e640 1 librados: init done 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.100+0000 7fb73dd9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.100+0000 7fb73dd9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.100+0000 7fb73dd9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.100+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73811fa60 con 0x7fb738142920 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.101+0000 7fb734ff9640 1 -- 172.21.15.110:0/2512149259 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb730009880 con 0x7fb738142920 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.101+0000 7fb734ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.101+0000 7fb734ff9640 1 -- 172.21.15.110:0/2512149259 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73001e440 con 0x7fb738142920 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.101+0000 7fb734ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.101+0000 7fb734ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.101+0000 7fb734ff9640 10 monclient: dump: 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.101+0000 7fb7367fc640 4 set_mon_vals no callback set 2024-01-20T11:21:26.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.342+0000 7fb734ff9640 1 -- 172.21.15.110:0/2512149259 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb730014440 con 0x7fb738142920 2024-01-20T11:21:26.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.342+0000 7fb734ff9640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb71806c890 0x7fb71806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.343+0000 7fb7377fe640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb71806c890 0x7fb71806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.343+0000 7fb7377fe640 10 monclient: get_auth_request con 0x7fb71806c890 auth_method 0 2024-01-20T11:21:26.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.343+0000 7fb7377fe640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb71806c890 0x7fb71806ed50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb728007fd0 tx=0x7fb728007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:26.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.343+0000 7fb734ff9640 10 client.5617.objecter ms_handle_connect 0x7fb71806c890 2024-01-20T11:21:26.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.345+0000 7fb734ff9640 1 -- 172.21.15.110:0/2512149259 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb730058950 con 0x7fb738142920 2024-01-20T11:21:26.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.345+0000 7fb734ff9640 10 client.5617.objecter ms_dispatch 0x7fb738000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:26.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.345+0000 7fb734ff9640 3 client.5617.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:26.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.345+0000 7fb734ff9640 3 client.5617.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:26.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.345+0000 7fb734ff9640 20 client.5617.objecter dump_active .. 0 homeless 2024-01-20T11:21:26.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.355+0000 7fb734ff9640 1 -- 172.21.15.110:0/2512149259 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb730054c90 con 0x7fb738142920 2024-01-20T11:21:26.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.355+0000 7fb734ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:26.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.355+0000 7fb734ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:26.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.521+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7380b4400 con 0x7fb71806c890 2024-01-20T11:21:26.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.525+0000 7fb734ff9640 1 -- 172.21.15.110:0/2512149259 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63796 (secure 0 0 0) 0x7fb728001460 con 0x7fb71806c890 2024-01-20T11:21:26.526 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:26.526 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 10 librados: watch_flush enter 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 10 librados: watch_flush exit 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 20 client.5617.objecter shutdown clearing up homeless session... 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 10 client.5617.objecter successfully canceled tick 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb71806c890 msgr2=0x7fb71806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb71806c890 0x7fb71806ed50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb728007fd0 tx=0x7fb728007e80 comp rx=0 tx=0).stop 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 10 monclient: shutdown 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb738142920 msgr2=0x7fb7381d7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb738142920 0x7fb7381d7490 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb73000b9e0 tx=0x7fb73000bea0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 shutdown_connections 2024-01-20T11:21:26.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb71806c890 0x7fb71806ed50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb738142920 0x7fb7381d7490 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7380a9ad0 0x7fb7380af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 --2- 172.21.15.110:0/2512149259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7380a96f0 0x7fb7380b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 >> 172.21.15.110:0/2512149259 conn(0x7fb7380b7bc0 msgr2=0x7fb73813b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:26.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 shutdown_connections 2024-01-20T11:21:26.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 -- 172.21.15.110:0/2512149259 wait complete. 2024-01-20T11:21:26.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.526+0000 7fb73dd9e640 1 librados: shutdown 2024-01-20T11:21:26.538 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-01-20T11:21:24.783307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002701"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":3.444},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.835}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.4160000000000004},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.678000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.821},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.670999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.976},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":11.481}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.951000000000001},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":8.9260000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.6890000000000001},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":13.34}]}]},{"osd":7,"up_from":21,"seq":90194313294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":7.2949999999999999},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":5.0250000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":10.56},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":14.547000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":16.603999999999999},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":8.4190000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":13.16},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":17.478000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":14.177},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":11.617000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":9.1660000000000004},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":15.237}]}]},{"osd":3,"up_from":21,"seq":90194313295,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.0110000000000001},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.7840000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.9850000000000003},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.4540000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.069000000000001},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.754}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.256},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":14.087}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":13.134},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":16.966000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":18.491},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.973000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":15.361000000000001},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":10.103999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.452},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":11.499000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":9.0540000000000003},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":16.702999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.338999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":7.5890000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":12.153},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":13.939}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":16.006},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.352}]}]},{"osd":1,"up_from":17,"seq":73014444110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.5860000000000003},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":9.9580000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.9340000000000002},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.6869999999999994}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":10.593},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.3840000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":6.2290000000000001},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.630000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.269},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":7.9779999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.399999999999999},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.99}]}]},{"osd":0,"up_from":17,"seq":73014444110,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":7.6550000000000002},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":10.506}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.5890000000000004},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.6850000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.253},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.076000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.967000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":14.06}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":9.9450000000000003},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":17.943000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.692},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":16.992999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.2530000000000001,"5min":8.4380000000000006,"15min":10.666},"min":{"1min":4.1449999999999996,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":13.182,"5min":106.059,"15min":106.059},"last":8.9100000000000001},{"interface":"front","average":{"1min":7.8479999999999999,"5min":9.4250000000000007,"15min":13.122999999999999},"min":{"1min":5.3479999999999999,"5min":3.96,"15min":3.96},"max":{"1min":10.583,"5min":157.67699999999999,"15min":157.67699999999999},"last":6.8449999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":13.853,"15min":24.992999999999999},"min":{"1min":4.8319999999999999,"5min":4.6319999999999997,"15min":4.6319999999999997},"max":{"1min":14.026999999999999,"5min":189.37700000000001,"15min":189.37700000000001},"last":10.532},{"interface":"front","average":{"1min":11.728,"5min":15.339,"15min":26.268999999999998},"min":{"1min":8.1270000000000007,"5min":5.6749999999999998,"15min":5.6749999999999998},"max":{"1min":17.741,"5min":190.304,"15min":190.304},"last":15.093}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":11.276999999999999,"5min":11.534000000000001,"15min":13.942},"min":{"1min":6.3360000000000003,"5min":5.3300000000000001,"15min":5.3300000000000001},"max":{"1min":14.609999999999999,"5min":128.67099999999999,"15min":128.67099999999999},"last":7.6159999999999997},{"interface":"front","average":{"1min":10.021000000000001,"5min":12.166,"15min":14.834},"min":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.489000000000001,"5min":101.75700000000001,"15min":101.75700000000001},"last":9.7270000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":16.427},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":14.436}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":17.509},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":11.180999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":11.978999999999999},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":13.679}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:26.539 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.638+0000 7f67ed0f7640 10 monclient: get_monmap_and_config 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.638+0000 7f67ed0f7640 10 monclient: build_initial_monmap 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.638+0000 7f67ed0f7640 10 monclient: monmap: 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:26.639834+0000 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:26.639834+0000 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.639 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:26.640 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:26.640 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:26.640 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:26.640 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:26.640 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.638+0000 7f67ed0f7640 1 Processor -- start 2024-01-20T11:21:26.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 1 -- start start 2024-01-20T11:21:26.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 10 monclient: init 2024-01-20T11:21:26.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:26.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f67e80b4d10 0x7f67e80b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 10 monclient(hunting): picked mon.noname-c con 0x7f67e80b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67e80b17c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67e8145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e80aade0 con 0x7f67e80b17c0 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e8145460 con 0x7f67e8145980 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e80b50f0 con 0x7f67e80b4d10 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.639+0000 7f67ed0f7640 10 monclient(hunting): authenticate will time out at 1729.990796s 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.640+0000 7f67dffff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f67e80b17c0 0x7f67e80b1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50542/0 (socket says 172.21.15.110:50542) 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.640+0000 7f67dffff640 1 -- 172.21.15.110:0/4078054253 learned_addr learned my addr 172.21.15.110:0/4078054253 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.640+0000 7f67e6d9d640 1 --2- 172.21.15.110:0/4078054253 >> v2:172.21.15.134:3300/0 conn(0x7f67e80b4d10 0x7f67e80b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.640+0000 7f67e6d9d640 10 monclient(hunting): get_auth_request con 0x7f67e80b4d10 auth_method 0 2024-01-20T11:21:26.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.640+0000 7f67e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:26.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.640+0000 7f67e6d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:26.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.640+0000 7f67e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:26.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.641+0000 7f67e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:26.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.641+0000 7f67e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:26.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.641+0000 7f67e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:26.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.642+0000 7f67e659c640 1 -- 172.21.15.110:0/4078054253 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954176160 0 0) 0x7f67e80aade0 con 0x7f67e80b17c0 2024-01-20T11:21:26.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.642+0000 7f67e659c640 10 monclient(hunting): _negotiate 2024-01-20T11:21:26.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.642+0000 7f67e659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:26.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.642+0000 7f67e659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:26.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.642+0000 7f67e659c640 10 monclient(hunting): my global_id is 5541 2024-01-20T11:21:26.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.642+0000 7f67e659c640 1 -- 172.21.15.110:0/4078054253 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d4003520 con 0x7f67e80b17c0 2024-01-20T11:21:26.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.643+0000 7f67e6d9d640 10 monclient(hunting): handle_auth_done global_id 5620 payload 274 2024-01-20T11:21:26.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.643+0000 7f67e6d9d640 1 -- 172.21.15.110:0/4078054253 >> v1:172.21.15.134:6789/0 conn(0x7f67e8145980 legacy=0x7f67e8145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.643+0000 7f67e6d9d640 1 -- 172.21.15.110:0/4078054253 >> v1:172.21.15.110:6789/0 conn(0x7f67e80b17c0 legacy=0x7f67e80b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.643+0000 7f67e6d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:21:26.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.643+0000 7f67e6d9d640 1 monclient: found mon.noname-c 2024-01-20T11:21:26.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.643+0000 7f67e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:26.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.643+0000 7f67e6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:21:26.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.643+0000 7f67e6d9d640 1 -- 172.21.15.110:0/4078054253 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e80b2660 con 0x7f67e80b4d10 2024-01-20T11:21:26.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.646+0000 7f67e6d9d640 1 --2- 172.21.15.110:0/4078054253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 0x7f67e80b3da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f67d80097a0 tx=0x7f67d802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=965166f69b3a4c7c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:26.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 1 -- 172.21.15.110:0/4078054253 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8031830 con 0x7f67e80b4d10 2024-01-20T11:21:26.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:26.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:26.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 10 monclient: dump: 2024-01-20T11:21:26.651 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:26.651 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 10 monclient: _finish_auth 0 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 10 monclient: _check_auth_tickets 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 1 -- 172.21.15.110:0/4078054253 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d8031990 con 0x7f67e80b4d10 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:26.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 1 -- 172.21.15.110:0/4078054253 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8031d20 con 0x7f67e80b4d10 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67e659c640 10 monclient: dump: 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67ed0f7640 5 monclient: authenticate success, global_id 5620 2024-01-20T11:21:26.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67ed0f7640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:26.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.650+0000 7f67ed0f7640 4 set_mon_vals no callback set 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.651+0000 7f67ed0f7640 10 monclient: shutdown 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.651+0000 7f67ed0f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.651+0000 7f67ed0f7640 1 -- 172.21.15.110:0/4078054253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 msgr2=0x7f67e80b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.651+0000 7f67ed0f7640 1 --2- 172.21.15.110:0/4078054253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 0x7f67e80b3da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f67d80097a0 tx=0x7f67d802b8d0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.651+0000 7f67ed0f7640 1 -- 172.21.15.110:0/4078054253 shutdown_connections 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.651+0000 7f67ed0f7640 1 --2- 172.21.15.110:0/4078054253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 0x7f67e80b3da0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.651+0000 7f67ed0f7640 1 -- 172.21.15.110:0/4078054253 >> 172.21.15.110:0/4078054253 conn(0x7f67e8066590 msgr2=0x7f67e80669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.651+0000 7f67ed0f7640 1 -- 172.21.15.110:0/4078054253 shutdown_connections 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.651+0000 7f67ed0f7640 1 -- 172.21.15.110:0/4078054253 wait complete. 2024-01-20T11:21:26.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.652+0000 7f67ed0f7640 10 monclient: build_initial_monmap 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.652+0000 7f67ed0f7640 10 monclient: monmap: 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.652+0000 7f67ed0f7640 1 librados: starting msgr at 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.652+0000 7f67ed0f7640 1 librados: starting objecter 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.652+0000 7f67ed0f7640 1 Processor -- start 2024-01-20T11:21:26.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.652+0000 7f67ed0f7640 1 -- start start 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.652+0000 7f67ed0f7640 1 librados: setting wanted keys 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.652+0000 7f67ed0f7640 1 librados: calling monclient init 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.652+0000 7f67ed0f7640 10 monclient: init 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67e80b17c0 0x7f67e813bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient(hunting): picked mon.noname-c con 0x7f67e80b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 0x7f67e813df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient(hunting): picked mon.noname-b con 0x7f67e80b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67e8145980 0x7f67e80ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient(hunting): picked mon.noname-a con 0x7f67e8145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e8136cd0 con 0x7f67e8145980 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:26.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e813c500 con 0x7f67e80b4d10 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e80ab9f0 con 0x7f67e80b17c0 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67ed0f7640 10 monclient(hunting): authenticate will time out at 1730.004475s 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67e759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67e8145980 0x7f67e80ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67e6d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67e80b17c0 0x7f67e813bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.653+0000 7f67dffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 0x7f67e813df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67dffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 0x7f67e813df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:54248/0 (socket says 172.21.15.110:54248) 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67dffff640 1 -- 172.21.15.110:0/1598735573 learned_addr learned my addr 172.21.15.110:0/1598735573 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67dffff640 10 monclient(hunting): get_auth_request con 0x7f67e80b4d10 auth_method 0 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67dffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67dffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:26.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e759e640 10 monclient(hunting): get_auth_request con 0x7f67e8145980 auth_method 0 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e759e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e759e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e6d9d640 10 monclient(hunting): get_auth_request con 0x7f67e80b17c0 auth_method 0 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e6d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:26.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:26.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.654+0000 7f67e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:26.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 10 monclient(hunting): handle_auth_done global_id 5623 payload 824 2024-01-20T11:21:26.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 1 -- 172.21.15.110:0/1598735573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67e80b17c0 msgr2=0x7f67e813bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67e80b17c0 0x7f67e813bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 1 -- 172.21.15.110:0/1598735573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67e8145980 msgr2=0x7f67e80ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67e8145980 0x7f67e80ac0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 1 monclient: found mon.noname-b 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.657+0000 7f67dffff640 1 -- 172.21.15.110:0/1598735573 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d8009400 con 0x7f67e80b4d10 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.659+0000 7f67dffff640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 0x7f67e813df90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f67d000c4d0 tx=0x7f67d000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.659+0000 7f67df7fe640 10 client.?.objecter ms_handle_connect 0x7f67e80b4d10 2024-01-20T11:21:26.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.659+0000 7f67df7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:26.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.662+0000 7f67df7fe640 1 -- 172.21.15.110:0/1598735573 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d0005ce0 con 0x7f67e80b4d10 2024-01-20T11:21:26.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.662+0000 7f67df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 10 monclient: dump: 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:26.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 1 -- 172.21.15.110:0/1598735573 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d0005e40 con 0x7f67e80b4d10 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67ed0f7640 5 monclient: authenticate success, global_id 5623 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67ed0f7640 10 monclient: _renew_subs 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67ed0f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e80abd30 con 0x7f67e80b4d10 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 1 -- 172.21.15.110:0/1598735573 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d0007270 con 0x7f67e80b4d10 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67df7fe640 10 monclient: dump: 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:26.666 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67e5d9b640 4 set_mon_vals no callback set 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67ed0f7640 10 client.5623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67ed0f7640 10 monclient: _renew_subs 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67ed0f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e813ec80 con 0x7f67e80b4d10 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.663+0000 7f67ed0f7640 1 librados: init done 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.664+0000 7f67ed0f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.664+0000 7f67ed0f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.664+0000 7f67ed0f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:26.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.664+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e8075490 con 0x7f67e80b4d10 2024-01-20T11:21:26.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.904+0000 7f67df7fe640 1 -- 172.21.15.110:0/1598735573 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f67d0007490 con 0x7f67e80b4d10 2024-01-20T11:21:26.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.905+0000 7f67df7fe640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67c806c8b0 0x7f67c806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:26.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.905+0000 7f67e6d9d640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67c806c8b0 0x7f67c806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:26.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.905+0000 7f67e6d9d640 10 monclient: get_auth_request con 0x7f67c806c8b0 auth_method 0 2024-01-20T11:21:26.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.906+0000 7f67df7fe640 10 client.5623.objecter ms_handle_connect 0x7f67c806c8b0 2024-01-20T11:21:26.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.906+0000 7f67e6d9d640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67c806c8b0 0x7f67c806ed70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f67d8036000 tx=0x7f67d803c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.908+0000 7f67df7fe640 1 -- 172.21.15.110:0/1598735573 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f67d0057fe0 con 0x7f67e80b4d10 2024-01-20T11:21:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.908+0000 7f67df7fe640 10 client.5623.objecter ms_dispatch 0x7f67e8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.908+0000 7f67df7fe640 3 client.5623.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.908+0000 7f67df7fe640 3 client.5623.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.908+0000 7f67df7fe640 20 client.5623.objecter dump_active .. 0 homeless 2024-01-20T11:21:26.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.919+0000 7f67df7fe640 1 -- 172.21.15.110:0/1598735573 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f67d0054320 con 0x7f67e80b4d10 2024-01-20T11:21:26.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.919+0000 7f67df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:26.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:26.919+0000 7f67df7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:27.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~149s 2024-01-20T11:21:27.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.089+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67e80a9cf0 con 0x7f67c806c8b0 2024-01-20T11:21:27.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.092+0000 7f67df7fe640 1 -- 172.21.15.110:0/1598735573 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63815 (secure 0 0 0) 0x7f67d803a420 con 0x7f67c806c8b0 2024-01-20T11:21:27.093 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:27.093 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:27.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 10 librados: watch_flush enter 2024-01-20T11:21:27.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 10 librados: watch_flush exit 2024-01-20T11:21:27.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 20 client.5623.objecter shutdown clearing up homeless session... 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 10 client.5623.objecter successfully canceled tick 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67c806c8b0 msgr2=0x7f67c806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67c806c8b0 0x7f67c806ed70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f67d8036000 tx=0x7f67d803c040 comp rx=0 tx=0).stop 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 10 monclient: shutdown 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 msgr2=0x7f67e813df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 0x7f67e813df90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f67d000c4d0 tx=0x7f67d000c990 comp rx=0 tx=0).stop 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 shutdown_connections 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f67c806c8b0 0x7f67c806ed70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67e8145980 0x7f67e80ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e80b4d10 0x7f67e813df90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 --2- 172.21.15.110:0/1598735573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f67e80b17c0 0x7f67e813bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 >> 172.21.15.110:0/1598735573 conn(0x7f67e8066590 msgr2=0x7f67e813f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:27.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 shutdown_connections 2024-01-20T11:21:27.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 -- 172.21.15.110:0/1598735573 wait complete. 2024-01-20T11:21:27.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.094+0000 7f67ed0f7640 1 librados: shutdown 2024-01-20T11:21:27.105 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-01-20T11:21:26.783705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002612"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":3.444},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.835}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.4160000000000004},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.678000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.821},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.670999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.976},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":11.481}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.951000000000001},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":8.9260000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.6890000000000001},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":13.34}]}]},{"osd":7,"up_from":21,"seq":90194313295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":4.2930000000000001},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":12.066000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.3559999999999999},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.8089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.01},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.039999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.4580000000000002},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.015000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.532},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":13.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.561},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.1880000000000006}]}]},{"osd":3,"up_from":21,"seq":90194313295,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.0110000000000001},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.7840000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.9850000000000003},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.4540000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.069000000000001},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.754}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.256},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":14.087}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":13.134},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":16.966000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":18.491},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.973000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":15.361000000000001},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":10.103999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.452},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":11.499000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":9.0540000000000003},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":16.702999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.338999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":7.5890000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":12.153},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":13.939}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":16.006},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.352}]}]},{"osd":1,"up_from":17,"seq":73014444110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.5860000000000003},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":9.9580000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.9340000000000002},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.6869999999999994}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":10.593},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.3840000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":6.2290000000000001},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.630000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.269},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":7.9779999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.399999999999999},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.99}]}]},{"osd":0,"up_from":17,"seq":73014444110,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":7.6550000000000002},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":10.506}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.5890000000000004},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.6850000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.253},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.076000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.967000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":14.06}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":9.9450000000000003},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":17.943000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.692},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":16.992999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.7830000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.5310000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.2590000000000003},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":7.6319999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":10.821},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":15.432}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.069000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.286}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.478999999999999},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.680999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.193000000000001},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.798999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:27.107 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:27.107 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:27.107 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:27.107 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:27.107 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:27.107 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:27.107 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:27.107 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:27.107 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:27.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.204+0000 7f2f03aec640 10 monclient: get_monmap_and_config 2024-01-20T11:21:27.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.204+0000 7f2f03aec640 10 monclient: build_initial_monmap 2024-01-20T11:21:27.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.204+0000 7f2f03aec640 10 monclient: monmap: 2024-01-20T11:21:27.205 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:27.205 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.205 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:27.206006+0000 2024-01-20T11:21:27.205 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:27.206006+0000 2024-01-20T11:21:27.205 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:27.206 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.206 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:27.206 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:27.206 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:27.206 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:27.206 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:27.206 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.204+0000 7f2f03aec640 1 Processor -- start 2024-01-20T11:21:27.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 1 -- start start 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient: init 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2efc071c60 0x7f2efc142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient(hunting): picked mon.noname-c con 0x7f2efc071c60 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2efc0ac540 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2efc0b0930 0x7f2efc149510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient(hunting): picked mon.noname-a con 0x7f2efc0b0930 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc0ad610 con 0x7f2efc0ac540 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc072040 con 0x7f2efc0b0930 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc1426c0 con 0x7f2efc071c60 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.205+0000 7f2f03aec640 10 monclient(hunting): authenticate will time out at 1730.556952s 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f02089640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2efc0b0930 0x7f2efc149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f01888640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2efc071c60 0x7f2efc142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f01888640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2efc071c60 0x7f2efc142180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:54262/0 (socket says 172.21.15.110:54262) 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f01888640 1 -- 172.21.15.110:0/4068059679 learned_addr learned my addr 172.21.15.110:0/4068059679 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f01888640 10 monclient(hunting): get_auth_request con 0x7f2efc071c60 auth_method 0 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f01888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f01888640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f01888640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f02089640 10 monclient(hunting): get_auth_request con 0x7f2efc0b0930 auth_method 0 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f02089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f02089640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.206+0000 7f2f02089640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.207+0000 7f2f02089640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.207+0000 7f2f02089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.207+0000 7f2f02089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.207+0000 7f2f01888640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.207+0000 7f2f01888640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.207+0000 7f2f01888640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.208+0000 7f2f02089640 10 monclient(hunting): handle_auth_done global_id 5547 payload 274 2024-01-20T11:21:27.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.208+0000 7f2f02089640 1 -- 172.21.15.110:0/4068059679 >> v2:172.21.15.134:3300/0 conn(0x7f2efc071c60 msgr2=0x7f2efc142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:27.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.208+0000 7f2f02089640 1 --2- 172.21.15.110:0/4068059679 >> v2:172.21.15.134:3300/0 conn(0x7f2efc071c60 0x7f2efc142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.208+0000 7f2f02089640 1 -- 172.21.15.110:0/4068059679 >> v1:172.21.15.134:6789/0 conn(0x7f2efc0ac540 legacy=0x7f2efc0ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.208+0000 7f2f02089640 10 monclient: _finish_hunting 0 2024-01-20T11:21:27.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.208+0000 7f2f02089640 1 monclient: found mon.noname-a 2024-01-20T11:21:27.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.208+0000 7f2f02089640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:27.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.208+0000 7f2f02089640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:27.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.208+0000 7f2f02089640 1 -- 172.21.15.110:0/4068059679 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc0721b0 con 0x7f2efc0b0930 2024-01-20T11:21:27.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.210+0000 7f2f02089640 1 --2- 172.21.15.110:0/4068059679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 0x7f2efc149510 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0ada90 tx=0x7f2ef402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5481f03ceb8005ea server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:27.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 1 -- 172.21.15.110:0/4068059679 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef402cc80 con 0x7f2efc0b0930 2024-01-20T11:21:27.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:27.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:27.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 10 monclient: dump: 2024-01-20T11:21:27.215 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:27.215 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.215 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.215 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.215 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 10 monclient: _finish_auth 0 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 10 monclient: _check_auth_tickets 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f03aec640 5 monclient: authenticate success, global_id 5547 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f03aec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 1 -- 172.21.15.110:0/4068059679 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef402cde0 con 0x7f2efc0b0930 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f03aec640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:27.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f03aec640 4 set_mon_vals no callback set 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 1 -- 172.21.15.110:0/4068059679 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4034660 con 0x7f2efc0b0930 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.214+0000 7f2f00886640 10 monclient: dump: 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:27.217 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 10 monclient: shutdown 2024-01-20T11:21:27.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 1 -- 172.21.15.110:0/4068059679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 msgr2=0x7f2efc149510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 1 --2- 172.21.15.110:0/4068059679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 0x7f2efc149510 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0ada90 tx=0x7f2ef402bed0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 1 -- 172.21.15.110:0/4068059679 shutdown_connections 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 1 --2- 172.21.15.110:0/4068059679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 0x7f2efc149510 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 1 --2- 172.21.15.110:0/4068059679 >> v2:172.21.15.134:3300/0 conn(0x7f2efc071c60 0x7f2efc142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 1 -- 172.21.15.110:0/4068059679 >> 172.21.15.110:0/4068059679 conn(0x7f2efc0b3e60 msgr2=0x7f2efc0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 1 -- 172.21.15.110:0/4068059679 shutdown_connections 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 1 -- 172.21.15.110:0/4068059679 wait complete. 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 10 monclient: build_initial_monmap 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.215+0000 7f2f03aec640 10 monclient: monmap: 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:27.222 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.216+0000 7f2f03aec640 1 librados: starting msgr at 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.216+0000 7f2f03aec640 1 librados: starting objecter 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.216+0000 7f2f03aec640 1 Processor -- start 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.216+0000 7f2f03aec640 1 -- start start 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.216+0000 7f2f03aec640 1 librados: setting wanted keys 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.216+0000 7f2f03aec640 1 librados: calling monclient init 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.216+0000 7f2f03aec640 10 monclient: init 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.216+0000 7f2f03aec640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:27.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.216+0000 7f2f03aec640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2efc071c60 0x7f2efc136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 10 monclient(hunting): picked mon.noname-c con 0x7f2efc071c60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2efc0ac540 0x7f2efc13efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 10 monclient(hunting): picked mon.noname-b con 0x7f2efc0ac540 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 0x7f2efc13ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 10 monclient(hunting): picked mon.noname-a con 0x7f2efc0b0930 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc13ebc0 con 0x7f2efc0b0930 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc1ca9a0 con 0x7f2efc0ac540 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc13f520 con 0x7f2efc071c60 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f03aec640 10 monclient(hunting): authenticate will time out at 1730.568243s 2024-01-20T11:21:27.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f01888640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2efc071c60 0x7f2efc136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f02089640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 0x7f2efc13ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f01087640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2efc0ac540 0x7f2efc13efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f02089640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 0x7f2efc13ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47448/0 (socket says 172.21.15.110:47448) 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f02089640 1 -- 172.21.15.110:0/2599237332 learned_addr learned my addr 172.21.15.110:0/2599237332 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f02089640 10 monclient(hunting): get_auth_request con 0x7f2efc0b0930 auth_method 0 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f02089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f02089640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f02089640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f01087640 10 monclient(hunting): get_auth_request con 0x7f2efc0ac540 auth_method 0 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f01087640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f01087640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.217+0000 7f2f01087640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01888640 10 monclient(hunting): get_auth_request con 0x7f2efc071c60 auth_method 0 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01888640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01888640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01087640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01087640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01087640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f02089640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f02089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f02089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01888640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01888640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.218+0000 7f2f01888640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.220+0000 7f2f01087640 10 monclient(hunting): handle_auth_done global_id 5632 payload 824 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.220+0000 7f2f01087640 1 -- 172.21.15.110:0/2599237332 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2efc071c60 msgr2=0x7f2efc136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.220+0000 7f2f01087640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2efc071c60 0x7f2efc136540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.221+0000 7f2f01087640 1 -- 172.21.15.110:0/2599237332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 msgr2=0x7f2efc13ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.221+0000 7f2f01087640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 0x7f2efc13ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.221+0000 7f2f01087640 10 monclient: _finish_hunting 0 2024-01-20T11:21:27.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.221+0000 7f2f01087640 1 monclient: found mon.noname-b 2024-01-20T11:21:27.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.221+0000 7f2f01087640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:27.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.221+0000 7f2f01087640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:27.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.221+0000 7f2f01087640 1 -- 172.21.15.110:0/2599237332 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef4009a70 con 0x7f2efc0ac540 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.223+0000 7f2f01087640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2efc0ac540 0x7f2efc13efe0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00c340 tx=0x7f2eec00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.223+0000 7f2ef27fc640 10 client.?.objecter ms_handle_connect 0x7f2efc0ac540 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.223+0000 7f2ef27fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 1 -- 172.21.15.110:0/2599237332 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0052d0 con 0x7f2efc0ac540 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 10 monclient: dump: 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.229 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 1 -- 172.21.15.110:0/2599237332 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec005450 con 0x7f2efc0ac540 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 1 -- 172.21.15.110:0/2599237332 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec007260 con 0x7f2efc0ac540 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2f03aec640 5 monclient: authenticate success, global_id 5632 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2f03aec640 10 monclient: _renew_subs 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2f03aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:27.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc13cf40 con 0x7f2efc0ac540 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef27fc640 10 monclient: dump: 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2ef3fff640 4 set_mon_vals no callback set 2024-01-20T11:21:27.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2f03aec640 10 client.5632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:27.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2f03aec640 10 monclient: _renew_subs 2024-01-20T11:21:27.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2f03aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:27.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc13fbe0 con 0x7f2efc0ac540 2024-01-20T11:21:27.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.226+0000 7f2f03aec640 1 librados: init done 2024-01-20T11:21:27.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.227+0000 7f2f03aec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:27.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.227+0000 7f2f03aec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:27.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.227+0000 7f2f03aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:27.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.227+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc1cab60 con 0x7f2efc0ac540 2024-01-20T11:21:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.466+0000 7f2ef27fc640 1 -- 172.21.15.110:0/2599237332 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2eec0073c0 con 0x7f2efc0ac540 2024-01-20T11:21:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.466+0000 7f2ef27fc640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ee006c8b0 0x7f2ee006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.466+0000 7f2f01888640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ee006c8b0 0x7f2ee006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.466+0000 7f2f01888640 10 monclient: get_auth_request con 0x7f2ee006c8b0 auth_method 0 2024-01-20T11:21:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.467+0000 7f2f01888640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ee006c8b0 0x7f2ee006ed70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800bdd0 tx=0x7f2ef800b450 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.467+0000 7f2ef27fc640 10 client.5632.objecter ms_handle_connect 0x7f2ee006c8b0 2024-01-20T11:21:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.470+0000 7f2ef27fc640 1 -- 172.21.15.110:0/2599237332 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2eec057a60 con 0x7f2efc0ac540 2024-01-20T11:21:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.470+0000 7f2ef27fc640 10 client.5632.objecter ms_dispatch 0x7f2efc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.470+0000 7f2ef27fc640 3 client.5632.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.470+0000 7f2ef27fc640 3 client.5632.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.470+0000 7f2ef27fc640 20 client.5632.objecter dump_active .. 0 homeless 2024-01-20T11:21:27.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.480+0000 7f2ef27fc640 1 -- 172.21.15.110:0/2599237332 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2eec053da0 con 0x7f2efc0ac540 2024-01-20T11:21:27.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.480+0000 7f2ef27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:27.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.480+0000 7f2ef27fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:27.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.713+0000 7f2f03aec640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:27.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.713+0000 7f2f03aec640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:27.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.713+0000 7f2f03aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:27.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.713+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2efc13e9f0 con 0x7f2efc0ac540 2024-01-20T11:21:27.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.720+0000 7f2ef27fc640 1 -- 172.21.15.110:0/2599237332 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f2eec020080 con 0x7f2efc0ac540 2024-01-20T11:21:27.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.721+0000 7f2ef27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:27.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.721+0000 7f2ef27fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:27.722 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:27.722 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":422,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.722+0000 7f2f03aec640 10 librados: watch_flush enter 2024-01-20T11:21:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.722+0000 7f2f03aec640 10 librados: watch_flush exit 2024-01-20T11:21:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.722+0000 7f2f03aec640 20 client.5632.objecter shutdown clearing up homeless session... 2024-01-20T11:21:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.722+0000 7f2f03aec640 10 client.5632.objecter successfully canceled tick 2024-01-20T11:21:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ee006c8b0 msgr2=0x7f2ee006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ee006c8b0 0x7f2ee006ed70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800bdd0 tx=0x7f2ef800b450 comp rx=0 tx=0).stop 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 10 monclient: shutdown 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2efc0ac540 msgr2=0x7f2efc13efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2efc0ac540 0x7f2efc13efe0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00c340 tx=0x7f2eec00c800 comp rx=0 tx=0).stop 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 shutdown_connections 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2ee006c8b0 0x7f2ee006ed70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2efc0b0930 0x7f2efc13ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2efc0ac540 0x7f2efc13efe0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 --2- 172.21.15.110:0/2599237332 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2efc071c60 0x7f2efc136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 >> 172.21.15.110:0/2599237332 conn(0x7f2efc0b3e60 msgr2=0x7f2efc073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 shutdown_connections 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 -- 172.21.15.110:0/2599237332 wait complete. 2024-01-20T11:21:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.723+0000 7f2f03aec640 1 librados: shutdown 2024-01-20T11:21:27.735 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:27.735 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.833+0000 7ff0397ce640 10 monclient: get_monmap_and_config 2024-01-20T11:21:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.833+0000 7ff0397ce640 10 monclient: build_initial_monmap 2024-01-20T11:21:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.833+0000 7ff0397ce640 10 monclient: monmap: 2024-01-20T11:21:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:27.834831+0000 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:27.834831+0000 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.833+0000 7ff0397ce640 1 Processor -- start 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.833+0000 7ff0397ce640 1 -- start start 2024-01-20T11:21:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.833+0000 7ff0397ce640 10 monclient: init 2024-01-20T11:21:27.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:27.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-01-20T11:21:27.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0340a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:27.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff0340a9ad0 0x7ff0341423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 10 monclient(hunting): picked mon.noname-c con 0x7ff0340a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:27.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff03414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 10 monclient(hunting): picked mon.noname-b con 0x7ff034142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0340b25c0 con 0x7ff0340a96f0 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0340aaea0 con 0x7ff0340a9ad0 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03413fbe0 con 0x7ff034142920 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.834+0000 7ff0397ce640 10 monclient(hunting): authenticate will time out at 1731.185652s 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff03414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0327fc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff0340a9ad0 0x7ff0341423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff03414c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43298/0 (socket says 172.21.15.110:43298) 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 1 -- 172.21.15.110:0/4013561134 learned_addr learned my addr 172.21.15.110:0/4013561134 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 10 monclient(hunting): get_auth_request con 0x7ff034142920 auth_method 0 2024-01-20T11:21:27.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0327fc640 10 monclient(hunting): get_auth_request con 0x7ff0340a9ad0 auth_method 0 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0327fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0327fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.835+0000 7ff0337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.836+0000 7ff0327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.836+0000 7ff0327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.836+0000 7ff0327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.837+0000 7ff0337fe640 10 monclient(hunting): handle_auth_done global_id 5276 payload 274 2024-01-20T11:21:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.837+0000 7ff0337fe640 1 -- 172.21.15.110:0/4013561134 >> v2:172.21.15.134:3300/0 conn(0x7ff0340a9ad0 msgr2=0x7ff0341423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.837+0000 7ff0337fe640 1 --2- 172.21.15.110:0/4013561134 >> v2:172.21.15.134:3300/0 conn(0x7ff0340a9ad0 0x7ff0341423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.837+0000 7ff0337fe640 1 -- 172.21.15.110:0/4013561134 >> v1:172.21.15.134:6789/0 conn(0x7ff0340a96f0 legacy=0x7ff03413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.837+0000 7ff0337fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.837+0000 7ff0337fe640 1 monclient: found mon.noname-b 2024-01-20T11:21:27.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.837+0000 7ff0337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.837+0000 7ff0337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.837+0000 7ff0337fe640 1 -- 172.21.15.110:0/4013561134 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0340ab0a0 con 0x7ff034142920 2024-01-20T11:21:27.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.839+0000 7ff0337fe640 1 --2- 172.21.15.110:0/4013561134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff03414c850 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009870 tx=0x7ff02c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b102da83a0be0a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 1 -- 172.21.15.110:0/4013561134 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0317c0 con 0x7ff034142920 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 10 monclient: dump: 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:27.845 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 10 monclient: _finish_auth 0 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 1 -- 172.21.15.110:0/4013561134 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c031920 con 0x7ff034142920 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 1 -- 172.21.15.110:0/4013561134 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c031cb0 con 0x7ff034142920 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 5 monclient: authenticate success, global_id 5276 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 4 set_mon_vals no callback set 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff031ffb640 10 monclient: dump: 2024-01-20T11:21:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 10 monclient: shutdown 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 1 -- 172.21.15.110:0/4013561134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 msgr2=0x7ff03414c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 1 --2- 172.21.15.110:0/4013561134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff03414c850 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009870 tx=0x7ff02c02b860 comp rx=0 tx=0).stop 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 1 -- 172.21.15.110:0/4013561134 shutdown_connections 2024-01-20T11:21:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 1 --2- 172.21.15.110:0/4013561134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff03414c850 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 1 --2- 172.21.15.110:0/4013561134 >> v2:172.21.15.134:3300/0 conn(0x7ff0340a9ad0 0x7ff0341423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 1 -- 172.21.15.110:0/4013561134 >> 172.21.15.110:0/4013561134 conn(0x7ff0340b7bc0 msgr2=0x7ff0340b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 1 -- 172.21.15.110:0/4013561134 shutdown_connections 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.843+0000 7ff0397ce640 1 -- 172.21.15.110:0/4013561134 wait complete. 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.844+0000 7ff0397ce640 10 monclient: build_initial_monmap 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.844+0000 7ff0397ce640 10 monclient: monmap: 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:27.849 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.844+0000 7ff0397ce640 1 librados: starting msgr at 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.844+0000 7ff0397ce640 1 librados: starting objecter 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.844+0000 7ff0397ce640 1 Processor -- start 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 1 -- start start 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 1 librados: setting wanted keys 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 1 librados: calling monclient init 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient: init 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0340a96f0 0x7ff0340af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient(hunting): picked mon.noname-a con 0x7ff0340a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0340a9ad0 0x7ff0341d7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient(hunting): picked mon.noname-b con 0x7ff0340a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff0340afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient(hunting): picked mon.noname-c con 0x7ff034142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0340b1320 con 0x7ff0340a96f0 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0340b1600 con 0x7ff0340a9ad0 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0341399d0 con 0x7ff034142920 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.845+0000 7ff0397ce640 10 monclient(hunting): authenticate will time out at 1731.196747s 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff0340afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0327fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0340a9ad0 0x7ff0341d7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff032ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0340a96f0 0x7ff0340af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff0340afe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43300/0 (socket says 172.21.15.110:43300) 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 1 -- 172.21.15.110:0/1681719443 learned_addr learned my addr 172.21.15.110:0/1681719443 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 10 monclient(hunting): get_auth_request con 0x7ff034142920 auth_method 0 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0327fc640 10 monclient(hunting): get_auth_request con 0x7ff0340a9ad0 auth_method 0 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0327fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0327fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff032ffd640 10 monclient(hunting): get_auth_request con 0x7ff0340a96f0 auth_method 0 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff032ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff032ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.846+0000 7ff0337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.847+0000 7ff032ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.847+0000 7ff032ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.847+0000 7ff032ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 10 monclient(hunting): handle_auth_done global_id 5638 payload 824 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 1 -- 172.21.15.110:0/1681719443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 msgr2=0x7ff0340afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff0340afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 1 -- 172.21.15.110:0/1681719443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0340a96f0 msgr2=0x7ff0340af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0340a96f0 0x7ff0340af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 10 monclient: _finish_hunting 0 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 1 monclient: found mon.noname-b 2024-01-20T11:21:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.849+0000 7ff0327fc640 1 -- 172.21.15.110:0/1681719443 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02c0094d0 con 0x7ff0340a9ad0 2024-01-20T11:21:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.851+0000 7ff0327fc640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0340a9ad0 0x7ff0341d7480 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff02800e820 tx=0x7ff02800ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.851+0000 7ff01b7fe640 10 client.?.objecter ms_handle_connect 0x7ff0340a9ad0 2024-01-20T11:21:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.851+0000 7ff01b7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 1 -- 172.21.15.110:0/1681719443 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02800c940 con 0x7ff0340a9ad0 2024-01-20T11:21:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 10 monclient: dump: 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 1 -- 172.21.15.110:0/1681719443 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02800caa0 con 0x7ff0340a9ad0 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0397ce640 5 monclient: authenticate success, global_id 5638 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 1 -- 172.21.15.110:0/1681719443 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02800ce30 con 0x7ff0340a9ad0 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff01b7fe640 10 monclient: dump: 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0397ce640 10 monclient: _renew_subs 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0397ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0341d79c0 con 0x7ff0340a9ad0 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0317fa640 4 set_mon_vals no callback set 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0397ce640 10 client.5638.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0397ce640 10 monclient: _renew_subs 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0397ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0340b0570 con 0x7ff0340a9ad0 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.855+0000 7ff0397ce640 1 librados: init done 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.856+0000 7ff0397ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.856+0000 7ff0397ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.856+0000 7ff0397ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.856+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0340b3da0 con 0x7ff0340a9ad0 2024-01-20T11:21:27.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.875+0000 7ff01b7fe640 1 -- 172.21.15.110:0/1681719443 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff028013470 con 0x7ff0340a9ad0 2024-01-20T11:21:27.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.875+0000 7ff01b7fe640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff02006c8b0 0x7ff02006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:27.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.876+0000 7ff032ffd640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff02006c8b0 0x7ff02006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:27.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.876+0000 7ff032ffd640 10 monclient: get_auth_request con 0x7ff02006c8b0 auth_method 0 2024-01-20T11:21:27.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.876+0000 7ff032ffd640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff02006c8b0 0x7ff02006ed70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff024005660 tx=0x7ff024003d90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:27.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:27.876+0000 7ff01b7fe640 10 client.5638.objecter ms_handle_connect 0x7ff02006c8b0 2024-01-20T11:21:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.103+0000 7ff01b7fe640 1 -- 172.21.15.110:0/1681719443 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff028058af0 con 0x7ff0340a9ad0 2024-01-20T11:21:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.103+0000 7ff01b7fe640 10 client.5638.objecter ms_dispatch 0x7ff034000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.103+0000 7ff01b7fe640 3 client.5638.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.103+0000 7ff01b7fe640 3 client.5638.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.103+0000 7ff01b7fe640 20 client.5638.objecter dump_active .. 0 homeless 2024-01-20T11:21:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.112+0000 7ff01b7fe640 1 -- 172.21.15.110:0/1681719443 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff028054e30 con 0x7ff0340a9ad0 2024-01-20T11:21:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.112+0000 7ff01b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.112+0000 7ff01b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:28.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.283+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0340b3bd0 con 0x7ff02006c8b0 2024-01-20T11:21:28.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.287+0000 7ff01b7fe640 1 -- 172.21.15.110:0/1681719443 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63815 (secure 0 0 0) 0x7ff0340b3bd0 con 0x7ff02006c8b0 2024-01-20T11:21:28.287 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:28.288 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 10 librados: watch_flush enter 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 10 librados: watch_flush exit 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 20 client.5638.objecter shutdown clearing up homeless session... 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 10 client.5638.objecter successfully canceled tick 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff02006c8b0 msgr2=0x7ff02006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff02006c8b0 0x7ff02006ed70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff024005660 tx=0x7ff024003d90 comp rx=0 tx=0).stop 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 10 monclient: shutdown 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0340a9ad0 msgr2=0x7ff0341d7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:28.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0340a9ad0 0x7ff0341d7480 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff02800e820 tx=0x7ff02800ece0 comp rx=0 tx=0).stop 2024-01-20T11:21:28.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 shutdown_connections 2024-01-20T11:21:28.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff02006c8b0 0x7ff02006ed70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:28.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff034142920 0x7ff0340afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:28.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0340a9ad0 0x7ff0341d7480 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:28.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 --2- 172.21.15.110:0/1681719443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0340a96f0 0x7ff0340af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:28.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 >> 172.21.15.110:0/1681719443 conn(0x7ff0340b7bc0 msgr2=0x7ff0340b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:28.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 shutdown_connections 2024-01-20T11:21:28.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 -- 172.21.15.110:0/1681719443 wait complete. 2024-01-20T11:21:28.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:28.288+0000 7ff0397ce640 1 librados: shutdown 2024-01-20T11:21:28.300 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-01-20T11:21:26.783705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002612"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":3.444},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":5.835}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.4160000000000004},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":12.678000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":10.821},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":14.670999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":6.976},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":11.481}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":13.951000000000001},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":8.9260000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":9.6890000000000001},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":13.34}]}]},{"osd":7,"up_from":21,"seq":90194313295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":4.2930000000000001},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":12.066000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.3559999999999999},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.8089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.01},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.039999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.4580000000000002},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.015000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.532},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":13.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.561},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.1880000000000006}]}]},{"osd":3,"up_from":21,"seq":90194313295,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":6.0110000000000001},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.7840000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":7.9850000000000003},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":9.4540000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.069000000000001},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":10.754}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":16.256},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":14.087}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":13.134},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":16.966000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":18.491},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":15.973000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444109,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":15.361000000000001},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":10.103999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.452},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":11.499000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":9.0540000000000003},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":16.702999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":18.338999999999999},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":7.5890000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":12.153},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":13.939}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":16.006},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":13.352}]}]},{"osd":1,"up_from":17,"seq":73014444110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":7.5380000000000003,"5min":8.5609999999999999,"15min":9.3460000000000001},"min":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":11.605,"5min":18.547000000000001,"15min":18.547000000000001},"last":5.5860000000000003},{"interface":"front","average":{"1min":9.2919999999999998,"5min":9.1010000000000009,"15min":10.097},"min":{"1min":5.1769999999999996,"5min":5.1150000000000002,"15min":5.1150000000000002},"max":{"1min":15.962999999999999,"5min":30.439,"15min":30.439},"last":9.9580000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":12.08,"5min":11.855,"15min":12.913},"min":{"1min":6.4290000000000003,"5min":6.351,"15min":6.351},"max":{"1min":16.916,"5min":24.725000000000001,"15min":24.725000000000001},"last":6.9340000000000002},{"interface":"front","average":{"1min":10.067,"5min":11.382,"15min":12.859999999999999},"min":{"1min":6.3230000000000004,"5min":6.1100000000000003,"15min":6.1100000000000003},"max":{"1min":15.845000000000001,"5min":23.981999999999999,"15min":23.981999999999999},"last":8.6869999999999994}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":10.593},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":9.3840000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":6.2290000000000001},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.630000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.955,"5min":11.247,"15min":11.936999999999999},"min":{"1min":7.1360000000000001,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":15.561,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.269},{"interface":"front","average":{"1min":14.081,"5min":14.294,"15min":15.285},"min":{"1min":10.722,"5min":7.9009999999999998,"15min":7.9009999999999998},"max":{"1min":18.262,"5min":31.634,"15min":31.634},"last":7.9779999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":16.399999999999999},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":11.99}]}]},{"osd":0,"up_from":17,"seq":73014444110,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":6.726,"5min":9.407,"15min":13.388},"min":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":4.0949999999999998},"max":{"1min":11.095000000000001,"5min":114.843,"15min":114.843},"last":7.6550000000000002},{"interface":"front","average":{"1min":8.8970000000000002,"5min":10.355,"15min":14.329000000000001},"min":{"1min":5.0419999999999998,"5min":4.3040000000000003,"15min":4.3040000000000003},"max":{"1min":16.108000000000001,"5min":121.044,"15min":121.044},"last":10.506}]},{"osd":2,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":8.3870000000000005,"5min":12.728,"15min":25.581},"min":{"1min":4.5949999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":18.198,"5min":379.17099999999999,"15min":379.17099999999999},"last":5.5890000000000004},{"interface":"front","average":{"1min":10.298999999999999,"5min":15.326000000000001,"15min":27.443999999999999},"min":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"max":{"1min":16.462,"5min":373.88299999999998,"15min":373.88299999999998},"last":8.6850000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":13.253},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.076000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":14.967000000000001},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":14.06}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:24 2024","interfaces":[{"interface":"back","average":{"1min":11.728,"5min":17.809000000000001,"15min":30.032},"min":{"1min":7.2800000000000002,"5min":7.2800000000000002,"15min":7.2800000000000002},"max":{"1min":15.666,"5min":375.733,"15min":375.733},"last":9.9450000000000003},{"interface":"front","average":{"1min":14.224,"5min":17.849,"15min":28.274999999999999},"min":{"1min":9.8190000000000008,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":17.341000000000001,"5min":374.654,"15min":374.654},"last":17.943000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":12.692},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":16.992999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.7830000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.5310000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.2590000000000003},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":7.6319999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":10.821},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":15.432}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.069000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.286}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.478999999999999},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.680999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.193000000000001},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.798999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:31.301 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.400+0000 7fc588cf7640 10 monclient: get_monmap_and_config 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.400+0000 7fc588cf7640 10 monclient: build_initial_monmap 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.400+0000 7fc588cf7640 10 monclient: monmap: 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:31.402477+0000 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:31.402477+0000 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:31.402 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.401+0000 7fc588cf7640 1 Processor -- start 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.401+0000 7fc588cf7640 1 -- start start 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.401+0000 7fc588cf7640 10 monclient: init 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 10 monclient(hunting): picked mon.noname-b con 0x7fc5840a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc58413fa10 0x7fc5841423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 10 monclient(hunting): picked mon.noname-c con 0x7fc58413fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5840a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5840b25c0 con 0x7fc5840a9ad0 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5840aaea0 con 0x7fc58413fa10 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5840ab0a0 con 0x7fc5840a96f0 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc588cf7640 10 monclient(hunting): authenticate will time out at 1734.753439s 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.402+0000 7fc58259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc57bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc58413fa10 0x7fc5841423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc58259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57612/0 (socket says 172.21.15.110:57612) 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc58259c640 1 -- 172.21.15.110:0/1645651966 learned_addr learned my addr 172.21.15.110:0/1645651966 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc58259c640 10 monclient(hunting): get_auth_request con 0x7fc5840a96f0 auth_method 0 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc58259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc58259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc58259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc57bfff640 10 monclient(hunting): get_auth_request con 0x7fc58413fa10 auth_method 0 2024-01-20T11:21:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc57bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc57bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc58259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc58259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.403+0000 7fc58259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.404+0000 7fc57bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.404+0000 7fc57bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.404+0000 7fc57bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.406+0000 7fc58259c640 10 monclient(hunting): handle_auth_done global_id 5282 payload 274 2024-01-20T11:21:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.406+0000 7fc58259c640 1 -- 172.21.15.110:0/1645651966 >> v2:172.21.15.134:3300/0 conn(0x7fc58413fa10 msgr2=0x7fc5841423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.406+0000 7fc58259c640 1 --2- 172.21.15.110:0/1645651966 >> v2:172.21.15.134:3300/0 conn(0x7fc58413fa10 0x7fc5841423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.406+0000 7fc58259c640 1 -- 172.21.15.110:0/1645651966 >> v1:172.21.15.134:6789/0 conn(0x7fc5840a9ad0 legacy=0x7fc584142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.406+0000 7fc58259c640 10 monclient: _finish_hunting 0 2024-01-20T11:21:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.406+0000 7fc58259c640 1 monclient: found mon.noname-b 2024-01-20T11:21:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.406+0000 7fc58259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.406+0000 7fc58259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.406+0000 7fc58259c640 1 -- 172.21.15.110:0/1645651966 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5840b3fb0 con 0x7fc5840a96f0 2024-01-20T11:21:31.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.408+0000 7fc58259c640 1 --2- 172.21.15.110:0/1645651966 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413f4d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc574009870 tx=0x7fc57402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4684491213d5ede server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 1 -- 172.21.15.110:0/1645651966 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574031640 con 0x7fc5840a96f0 2024-01-20T11:21:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 10 monclient: dump: 2024-01-20T11:21:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:31.422 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 10 monclient: _finish_auth 0 2024-01-20T11:21:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 1 -- 172.21.15.110:0/1645651966 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5740317a0 con 0x7fc5840a96f0 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 1 -- 172.21.15.110:0/1645651966 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574031b30 con 0x7fc5840a96f0 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc581d9b640 10 monclient: dump: 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc588cf7640 5 monclient: authenticate success, global_id 5282 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc588cf7640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.411+0000 7fc588cf7640 4 set_mon_vals no callback set 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 10 monclient: shutdown 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 1 -- 172.21.15.110:0/1645651966 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 msgr2=0x7fc58413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 1 --2- 172.21.15.110:0/1645651966 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413f4d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc574009870 tx=0x7fc57402b8f0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 1 -- 172.21.15.110:0/1645651966 shutdown_connections 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 1 --2- 172.21.15.110:0/1645651966 >> v2:172.21.15.134:3300/0 conn(0x7fc58413fa10 0x7fc5841423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 1 --2- 172.21.15.110:0/1645651966 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413f4d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 1 -- 172.21.15.110:0/1645651966 >> 172.21.15.110:0/1645651966 conn(0x7fc5840b7bc0 msgr2=0x7fc5840b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:31.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 1 -- 172.21.15.110:0/1645651966 shutdown_connections 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.412+0000 7fc588cf7640 1 -- 172.21.15.110:0/1645651966 wait complete. 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.413+0000 7fc588cf7640 10 monclient: build_initial_monmap 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.413+0000 7fc588cf7640 10 monclient: monmap: 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.413+0000 7fc588cf7640 1 librados: starting msgr at 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.413+0000 7fc588cf7640 1 librados: starting objecter 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.413+0000 7fc588cf7640 1 Processor -- start 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.413+0000 7fc588cf7640 1 -- start start 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.413+0000 7fc588cf7640 1 librados: setting wanted keys 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.413+0000 7fc588cf7640 1 librados: calling monclient init 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.413+0000 7fc588cf7640 10 monclient: init 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient(hunting): picked mon.noname-c con 0x7fc5840a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5840a9ad0 0x7fc584133fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient(hunting): picked mon.noname-b con 0x7fc5840a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc584134510 0x7fc5841da570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient(hunting): picked mon.noname-a con 0x7fc584134510 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc584146050 con 0x7fc584134510 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc584145370 con 0x7fc5840a9ad0 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc584145bc0 con 0x7fc5840a96f0 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc588cf7640 10 monclient(hunting): authenticate will time out at 1734.765547s 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.414+0000 7fc58259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc582d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc584134510 0x7fc5841da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc57bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5840a9ad0 0x7fc584133fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc58259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57616/0 (socket says 172.21.15.110:57616) 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc58259c640 1 -- 172.21.15.110:0/3701196447 learned_addr learned my addr 172.21.15.110:0/3701196447 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:31.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc58259c640 10 monclient(hunting): get_auth_request con 0x7fc5840a96f0 auth_method 0 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc58259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc58259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc58259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc57bfff640 10 monclient(hunting): get_auth_request con 0x7fc5840a9ad0 auth_method 0 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc57bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc57bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc582d9d640 10 monclient(hunting): get_auth_request con 0x7fc584134510 auth_method 0 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc582d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc582d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc582d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc58259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc58259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc58259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc57bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc57bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc57bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc582d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc582d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.415+0000 7fc582d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 10 monclient(hunting): handle_auth_done global_id 5644 payload 824 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 1 -- 172.21.15.110:0/3701196447 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 msgr2=0x7fc58413ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 1 -- 172.21.15.110:0/3701196447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc584134510 msgr2=0x7fc5841da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc584134510 0x7fc5841da570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 1 monclient: found mon.noname-b 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:31.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.418+0000 7fc57bfff640 1 -- 172.21.15.110:0/3701196447 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5740094d0 con 0x7fc5840a9ad0 2024-01-20T11:21:31.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.420+0000 7fc57bfff640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5840a9ad0 0x7fc584133fd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc5840aac70 tx=0x7fc56c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:31.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.420+0000 7fc57affd640 10 client.?.objecter ms_handle_connect 0x7fc5840a9ad0 2024-01-20T11:21:31.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.420+0000 7fc57affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:31.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 1 -- 172.21.15.110:0/3701196447 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56c00c700 con 0x7fc5840a9ad0 2024-01-20T11:21:31.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:31.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:31.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 10 monclient: dump: 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 10 monclient: _finish_auth 0 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 1 -- 172.21.15.110:0/3701196447 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56c00c880 con 0x7fc5840a9ad0 2024-01-20T11:21:31.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc588cf7640 5 monclient: authenticate success, global_id 5644 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc588cf7640 10 monclient: _renew_subs 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc588cf7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc584145d90 con 0x7fc5840a9ad0 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 1 -- 172.21.15.110:0/3701196447 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56c00cc30 con 0x7fc5840a9ad0 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc57affd640 10 monclient: dump: 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc588cf7640 10 client.5644.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc588cf7640 10 monclient: _renew_subs 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc58159a640 4 set_mon_vals no callback set 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.424+0000 7fc588cf7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.425+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5841380f0 con 0x7fc5840a9ad0 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.425+0000 7fc588cf7640 1 librados: init done 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.425+0000 7fc588cf7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.425+0000 7fc588cf7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.425+0000 7fc588cf7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:31.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.425+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5841461c0 con 0x7fc5840a9ad0 2024-01-20T11:21:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.664+0000 7fc57affd640 1 -- 172.21.15.110:0/3701196447 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc56c013470 con 0x7fc5840a9ad0 2024-01-20T11:21:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.664+0000 7fc57affd640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc56406c8b0 0x7fc56406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.665+0000 7fc58259c640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc56406c8b0 0x7fc56406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.665+0000 7fc58259c640 10 monclient: get_auth_request con 0x7fc56406c8b0 auth_method 0 2024-01-20T11:21:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.665+0000 7fc57affd640 10 client.5644.objecter ms_handle_connect 0x7fc56406c8b0 2024-01-20T11:21:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.665+0000 7fc58259c640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc56406c8b0 0x7fc56406ed70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc57402b930 tx=0x7fc57403c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.667+0000 7fc57affd640 1 -- 172.21.15.110:0/3701196447 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc56c058920 con 0x7fc5840a9ad0 2024-01-20T11:21:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.667+0000 7fc57affd640 10 client.5644.objecter ms_dispatch 0x7fc584000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.667+0000 7fc57affd640 3 client.5644.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.667+0000 7fc57affd640 3 client.5644.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.667+0000 7fc57affd640 20 client.5644.objecter dump_active .. 0 homeless 2024-01-20T11:21:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.679+0000 7fc57affd640 1 -- 172.21.15.110:0/3701196447 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc56c054c60 con 0x7fc5840a9ad0 2024-01-20T11:21:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.679+0000 7fc57affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.679+0000 7fc57affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:31.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.850+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc584075490 con 0x7fc56406c8b0 2024-01-20T11:21:31.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.853+0000 7fc57affd640 1 -- 172.21.15.110:0/3701196447 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64031 (secure 0 0 0) 0x7fc584075490 con 0x7fc56406c8b0 2024-01-20T11:21:31.854 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:31.854 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:31.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.854+0000 7fc588cf7640 10 librados: watch_flush enter 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.854+0000 7fc588cf7640 10 librados: watch_flush exit 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.854+0000 7fc588cf7640 20 client.5644.objecter shutdown clearing up homeless session... 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.854+0000 7fc588cf7640 10 client.5644.objecter successfully canceled tick 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.854+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc56406c8b0 msgr2=0x7fc56406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.854+0000 7fc588cf7640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc56406c8b0 0x7fc56406ed70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc57402b930 tx=0x7fc57403c040 comp rx=0 tx=0).stop 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 10 monclient: shutdown 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5840a9ad0 msgr2=0x7fc584133fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5840a9ad0 0x7fc584133fd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc5840aac70 tx=0x7fc56c00ecd0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 shutdown_connections 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc56406c8b0 0x7fc56406ed70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc584134510 0x7fc5841da570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5840a9ad0 0x7fc584133fd0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 --2- 172.21.15.110:0/3701196447 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5840a96f0 0x7fc58413ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 >> 172.21.15.110:0/3701196447 conn(0x7fc5840b7bc0 msgr2=0x7fc58413c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:31.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 shutdown_connections 2024-01-20T11:21:31.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 -- 172.21.15.110:0/3701196447 wait complete. 2024-01-20T11:21:31.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.855+0000 7fc588cf7640 1 librados: shutdown 2024-01-20T11:21:31.866 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-01-20T11:21:30.784567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002590"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.7320000000000002},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.3579999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6720000000000006},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.9019999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.054},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.497}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":12.784000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":10.279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.98},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":12.093}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.6310000000000002},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.757999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":4.2930000000000001},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":12.066000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.3559999999999999},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.8089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.01},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.039999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.4580000000000002},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.015000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.532},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":13.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.561},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.1880000000000006}]}]},{"osd":3,"up_from":21,"seq":90194313296,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.7200000000000006},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4030000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.922000000000001},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.77}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":11.510999999999999},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.135}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.3230000000000004},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.744}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":12.090999999999999},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":13.741}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":14.321},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.818999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.1660000000000004},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":4.2839999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":10.784000000000001},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.8070000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.208},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.5330000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.4499999999999993},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.757999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":15.630000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.206}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.462},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":12.811999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.0890000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":5.7610000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":9.3650000000000002},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.8129999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":14.035},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.7829999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":9.9760000000000009},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.531000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":11.513}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.401},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.129}]}]},{"osd":0,"up_from":17,"seq":73014444111,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":5.3079999999999998},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.7219999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":10.587},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":11.696999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":12.382},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.004}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.093999999999999},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.633}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":14.214},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":15.917}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.789},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":7.3099999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.7830000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.5310000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.2590000000000003},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":7.6319999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":10.821},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":15.432}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.069000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.286}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.478999999999999},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.680999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.193000000000001},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.798999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:31.867 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:31.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.965+0000 7f263f161640 10 monclient: get_monmap_and_config 2024-01-20T11:21:31.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.965+0000 7f263f161640 10 monclient: build_initial_monmap 2024-01-20T11:21:31.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.965+0000 7f263f161640 10 monclient: monmap: 2024-01-20T11:21:31.966 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:31.966 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.966 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:31.966732+0000 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:31.966732+0000 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.965+0000 7f263f161640 1 Processor -- start 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.965+0000 7f263f161640 1 -- start start 2024-01-20T11:21:31.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.965+0000 7f263f161640 10 monclient: init 2024-01-20T11:21:31.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:31.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-01-20T11:21:31.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f26380a96f0 0x7f263813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 10 monclient(hunting): picked mon.noname-a con 0x7f26380a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:31.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f263813fa10 0x7f26381423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 10 monclient(hunting): picked mon.noname-c con 0x7f263813fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26380a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26380b25c0 con 0x7f26380a9ad0 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26380aaea0 con 0x7f26380a96f0 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26380ab0a0 con 0x7f263813fa10 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.966+0000 7f263f161640 10 monclient(hunting): authenticate will time out at 1735.317694s 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f26380a96f0 0x7f263813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f2637fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f263813fa10 0x7f26381423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f26380a96f0 0x7f263813f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47296/0 (socket says 172.21.15.110:47296) 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 1 -- 172.21.15.110:0/2722195776 learned_addr learned my addr 172.21.15.110:0/2722195776 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 10 monclient(hunting): get_auth_request con 0x7f26380a96f0 auth_method 0 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f2637fff640 10 monclient(hunting): get_auth_request con 0x7f263813fa10 auth_method 0 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f2637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f2637fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f2637fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.967+0000 7f263cefd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.968+0000 7f2637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.968+0000 7f2637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.968+0000 7f2637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.969+0000 7f263cefd640 10 monclient(hunting): handle_auth_done global_id 5559 payload 274 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.969+0000 7f263cefd640 1 -- 172.21.15.110:0/2722195776 >> v2:172.21.15.134:3300/0 conn(0x7f263813fa10 msgr2=0x7f26381423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.969+0000 7f263cefd640 1 --2- 172.21.15.110:0/2722195776 >> v2:172.21.15.134:3300/0 conn(0x7f263813fa10 0x7f26381423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.969+0000 7f263cefd640 1 -- 172.21.15.110:0/2722195776 >> v1:172.21.15.134:6789/0 conn(0x7f26380a9ad0 legacy=0x7f2638142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.969+0000 7f263cefd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.969+0000 7f263cefd640 1 monclient: found mon.noname-a 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.969+0000 7f263cefd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.969+0000 7f263cefd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.969+0000 7f263cefd640 1 -- 172.21.15.110:0/2722195776 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26380b3fb0 con 0x7f26380a96f0 2024-01-20T11:21:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.971+0000 7f263cefd640 1 --2- 172.21.15.110:0/2722195776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26380a96f0 0x7f263813f4d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f263806d0f0 tx=0x7f262802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f5ef89d4ae44c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 1 -- 172.21.15.110:0/2722195776 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628031870 con 0x7f26380a96f0 2024-01-20T11:21:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 10 monclient: dump: 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:31.976 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 1 -- 172.21.15.110:0/2722195776 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26280319d0 con 0x7f26380a96f0 2024-01-20T11:21:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 1 -- 172.21.15.110:0/2722195776 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628031d60 con 0x7f26380a96f0 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f26377fe640 10 monclient: dump: 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:31.979 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:31.979 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f263f161640 5 monclient: authenticate success, global_id 5559 2024-01-20T11:21:31.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f263f161640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:31.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.974+0000 7f263f161640 4 set_mon_vals no callback set 2024-01-20T11:21:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 10 monclient: shutdown 2024-01-20T11:21:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 1 -- 172.21.15.110:0/2722195776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26380a96f0 msgr2=0x7f263813f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 1 --2- 172.21.15.110:0/2722195776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26380a96f0 0x7f263813f4d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f263806d0f0 tx=0x7f262802b910 comp rx=0 tx=0).stop 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 1 -- 172.21.15.110:0/2722195776 shutdown_connections 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 1 --2- 172.21.15.110:0/2722195776 >> v2:172.21.15.134:3300/0 conn(0x7f263813fa10 0x7f26381423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 1 --2- 172.21.15.110:0/2722195776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26380a96f0 0x7f263813f4d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 1 -- 172.21.15.110:0/2722195776 >> 172.21.15.110:0/2722195776 conn(0x7f26380b7bc0 msgr2=0x7f26380b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 1 -- 172.21.15.110:0/2722195776 shutdown_connections 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.975+0000 7f263f161640 1 -- 172.21.15.110:0/2722195776 wait complete. 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.976+0000 7f263f161640 10 monclient: build_initial_monmap 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.976+0000 7f263f161640 10 monclient: monmap: 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.976+0000 7f263f161640 1 librados: starting msgr at 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.976+0000 7f263f161640 1 librados: starting objecter 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.976+0000 7f263f161640 1 Processor -- start 2024-01-20T11:21:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.976+0000 7f263f161640 1 -- start start 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.976+0000 7f263f161640 1 librados: setting wanted keys 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.976+0000 7f263f161640 1 librados: calling monclient init 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.976+0000 7f263f161640 10 monclient: init 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26380a96f0 0x7f26381d75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient(hunting): picked mon.noname-c con 0x7f26380a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26380a9ad0 0x7f26381d7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient(hunting): picked mon.noname-b con 0x7f26380a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2638137e50 0x7f263813a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient(hunting): picked mon.noname-a con 0x7f2638137e50 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2638139fa0 con 0x7f2638137e50 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26381d70a0 con 0x7f26380a9ad0 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263813bd40 con 0x7f26380a96f0 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263f161640 10 monclient(hunting): authenticate will time out at 1735.328683s 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.977+0000 7f263d6fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2638137e50 0x7f263813a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f2637fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26380a9ad0 0x7f26381d7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263cefd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26380a96f0 0x7f26381d75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263d6fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2638137e50 0x7f263813a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47302/0 (socket says 172.21.15.110:47302) 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263d6fe640 1 -- 172.21.15.110:0/1769449198 learned_addr learned my addr 172.21.15.110:0/1769449198 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263d6fe640 10 monclient(hunting): get_auth_request con 0x7f2638137e50 auth_method 0 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263d6fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263d6fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263d6fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f2637fff640 10 monclient(hunting): get_auth_request con 0x7f26380a9ad0 auth_method 0 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f2637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f2637fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f2637fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263cefd640 10 monclient(hunting): get_auth_request con 0x7f26380a96f0 auth_method 0 2024-01-20T11:21:31.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263cefd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263cefd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263cefd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263d6fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263d6fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263d6fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f2637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f2637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f2637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263cefd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263cefd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.978+0000 7f263cefd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 10 monclient(hunting): handle_auth_done global_id 5653 payload 824 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 1 -- 172.21.15.110:0/1769449198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26380a96f0 msgr2=0x7f26381d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26380a96f0 0x7f26381d75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 1 -- 172.21.15.110:0/1769449198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2638137e50 msgr2=0x7f263813a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2638137e50 0x7f263813a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 1 monclient: found mon.noname-b 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f2637fff640 1 -- 172.21.15.110:0/1769449198 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26280094d0 con 0x7f26380a9ad0 2024-01-20T11:21:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.981+0000 7f263d6fe640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2638137e50 0x7f263813a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.983+0000 7f2637fff640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26380a9ad0 0x7f26381d7b00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f26380aac70 tx=0x7f262c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.983+0000 7f26357fa640 10 client.?.objecter ms_handle_connect 0x7f26380a9ad0 2024-01-20T11:21:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.983+0000 7f26357fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 1 -- 172.21.15.110:0/1769449198 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c00c700 con 0x7f26380a9ad0 2024-01-20T11:21:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 10 monclient: dump: 2024-01-20T11:21:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 10 monclient: _finish_auth 0 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 10 monclient: _check_auth_tickets 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 1 -- 172.21.15.110:0/1769449198 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c00c880 con 0x7f26380a9ad0 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 1 -- 172.21.15.110:0/1769449198 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c00cc30 con 0x7f26380a9ad0 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 5 monclient: authenticate success, global_id 5653 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 10 monclient: _renew_subs 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263813aec0 con 0x7f26380a9ad0 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f26357fa640 10 monclient: dump: 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f2636ffd640 4 set_mon_vals no callback set 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 10 client.5653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 10 monclient: _renew_subs 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263813b3d0 con 0x7f26380a9ad0 2024-01-20T11:21:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 1 librados: init done 2024-01-20T11:21:31.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:31.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:31.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:31.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:31.988+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2638075490 con 0x7f26380a9ad0 2024-01-20T11:21:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.230+0000 7f26357fa640 1 -- 172.21.15.110:0/1769449198 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f262c013470 con 0x7f26380a9ad0 2024-01-20T11:21:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.230+0000 7f26357fa640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f262406c8e0 0x7f262406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.230+0000 7f263cefd640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f262406c8e0 0x7f262406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.230+0000 7f263cefd640 10 monclient: get_auth_request con 0x7f262406c8e0 auth_method 0 2024-01-20T11:21:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.230+0000 7f26357fa640 10 client.5653.objecter ms_handle_connect 0x7f262406c8e0 2024-01-20T11:21:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.230+0000 7f263cefd640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f262406c8e0 0x7f262406eda0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2628036000 tx=0x7f262803c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:32.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.231+0000 7f26357fa640 1 -- 172.21.15.110:0/1769449198 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f262c058890 con 0x7f26380a9ad0 2024-01-20T11:21:32.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.231+0000 7f26357fa640 10 client.5653.objecter ms_dispatch 0x7f2638000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:32.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.231+0000 7f26357fa640 3 client.5653.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:32.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.231+0000 7f26357fa640 3 client.5653.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:32.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.232+0000 7f26357fa640 20 client.5653.objecter dump_active .. 0 homeless 2024-01-20T11:21:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.242+0000 7f26357fa640 1 -- 172.21.15.110:0/1769449198 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f262c054bd0 con 0x7f26380a9ad0 2024-01-20T11:21:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.242+0000 7f26357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.242+0000 7f26357fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.430+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2638069850 con 0x7f262406c8e0 2024-01-20T11:21:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.434+0000 7f26357fa640 1 -- 172.21.15.110:0/1769449198 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64031 (secure 0 0 0) 0x7f2638069850 con 0x7f262406c8e0 2024-01-20T11:21:32.435 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 10 librados: watch_flush enter 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 10 librados: watch_flush exit 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 20 client.5653.objecter shutdown clearing up homeless session... 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 10 client.5653.objecter successfully canceled tick 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f262406c8e0 msgr2=0x7f262406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f262406c8e0 0x7f262406eda0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2628036000 tx=0x7f262803c040 comp rx=0 tx=0).stop 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 10 monclient: shutdown 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26380a9ad0 msgr2=0x7f26381d7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26380a9ad0 0x7f26381d7b00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f26380aac70 tx=0x7f262c00ecd0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 shutdown_connections 2024-01-20T11:21:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f262406c8e0 0x7f262406eda0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2638137e50 0x7f263813a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26380a9ad0 0x7f26381d7b00 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 --2- 172.21.15.110:0/1769449198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26380a96f0 0x7f26381d75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 >> 172.21.15.110:0/1769449198 conn(0x7f26380b7bc0 msgr2=0x7f26380b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 shutdown_connections 2024-01-20T11:21:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 -- 172.21.15.110:0/1769449198 wait complete. 2024-01-20T11:21:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.436+0000 7f263f161640 1 librados: shutdown 2024-01-20T11:21:32.447 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-01-20T11:21:30.784567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002590"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:31.664257+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:17:31.664257+0000","last_peered":"2024-01-20T11:17:31.664257+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:17:31.664257+0000","last_undegraded":"2024-01-20T11:17:31.664257+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.7320000000000002},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.3579999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6720000000000006},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.9019999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.054},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.497}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":12.784000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":10.279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.98},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":12.093}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.6310000000000002},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.757999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":4.2930000000000001},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":12.066000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":6.3559999999999999},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":9.8089999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.01},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.039999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":7.4580000000000002},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":15.015000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":8.532},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":13.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":16.561},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":9.1880000000000006}]}]},{"osd":3,"up_from":21,"seq":90194313296,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":8.7200000000000006},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":9.4030000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":10.922000000000001},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":16.77}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":11.510999999999999},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":13.135}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":8.3230000000000004},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":15.744}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":12.090999999999999},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":13.741}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":14.321},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":17.818999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.1660000000000004},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":4.2839999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":10.784000000000001},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.8070000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.208},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.5330000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.4499999999999993},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.757999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":15.630000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.206}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.462},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":12.811999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.0890000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":5.7610000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":9.3650000000000002},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.8129999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":14.035},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.7829999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":9.9760000000000009},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.531000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":11.513}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.401},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.129}]}]},{"osd":0,"up_from":17,"seq":73014444111,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":5.3079999999999998},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.7219999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":10.587},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":11.696999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":12.382},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.004}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.093999999999999},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.633}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":14.214},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":15.917}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.789},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":7.3099999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.7830000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.5310000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.2590000000000003},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":7.6319999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":10.821},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":15.432}]},{"osd":3,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":13.018000000000001,"5min":13.521000000000001,"15min":15.364000000000001},"min":{"1min":6.4119999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"max":{"1min":18.550999999999998,"5min":81.373999999999995,"15min":81.373999999999995},"last":12.069000000000001},{"interface":"front","average":{"1min":12.394,"5min":13.481,"15min":15.613},"min":{"1min":8.7240000000000002,"5min":6.6900000000000004,"15min":6.6900000000000004},"max":{"1min":14.65,"5min":84.524000000000001,"15min":84.524000000000001},"last":10.286}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.478999999999999},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.680999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:20:28 2024","interfaces":[{"interface":"back","average":{"1min":12.430999999999999,"5min":14.58,"15min":16.029},"min":{"1min":8.6639999999999997,"5min":8.6639999999999997,"15min":8.6639999999999997},"max":{"1min":16.919,"5min":25.303999999999998,"15min":25.303999999999998},"last":16.193000000000001},{"interface":"front","average":{"1min":13.641,"5min":14.198,"15min":15.547000000000001},"min":{"1min":7.149,"5min":7.149,"15min":7.149},"max":{"1min":19.462,"5min":22.838999999999999,"15min":22.838999999999999},"last":12.798999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:32.448 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:32.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:32.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:32.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:32.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:32.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:31.664257+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:17:31.664257+0000', 'last_peered': '2024-01-20T11:17:31.664257+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:17:31.664257+0000', 'last_undegraded': '2024-01-20T11:17:31.664257+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:32.449 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:32.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:32.449 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.548+0000 7f6d94c34640 10 monclient: get_monmap_and_config 2024-01-20T11:21:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.548+0000 7f6d94c34640 10 monclient: build_initial_monmap 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.548+0000 7f6d94c34640 10 monclient: monmap: 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:32.550397+0000 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:32.550397+0000 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:32.550 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.549+0000 7f6d94c34640 1 Processor -- start 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.549+0000 7f6d94c34640 1 -- start start 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.549+0000 7f6d94c34640 10 monclient: init 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d900af3a0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ab6d0 0x7f6d900ad420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 10 monclient(hunting): picked mon.noname-b con 0x7f6d900ab6d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6d900ad960 0x7f6d90148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 10 monclient(hunting): picked mon.noname-c con 0x7f6d900ad960 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d90141930 con 0x7f6d900af3a0 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d90142920 con 0x7f6d900ad960 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900af780 con 0x7f6d900ab6d0 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d94c34640 10 monclient(hunting): authenticate will time out at 1735.901532s 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.550+0000 7f6d8dd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ab6d0 0x7f6d900ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8ed9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6d900ad960 0x7f6d90148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8dd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ab6d0 0x7f6d900ad420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57646/0 (socket says 172.21.15.110:57646) 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8dd9b640 1 -- 172.21.15.110:0/780066273 learned_addr learned my addr 172.21.15.110:0/780066273 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8dd9b640 10 monclient(hunting): get_auth_request con 0x7f6d900ab6d0 auth_method 0 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8ed9d640 10 monclient(hunting): get_auth_request con 0x7f6d900ad960 auth_method 0 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.551+0000 7f6d8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.552+0000 7f6d8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.552+0000 7f6d8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.552+0000 7f6d8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.554+0000 7f6d8dd9b640 10 monclient(hunting): handle_auth_done global_id 5291 payload 274 2024-01-20T11:21:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.554+0000 7f6d8dd9b640 1 -- 172.21.15.110:0/780066273 >> v2:172.21.15.134:3300/0 conn(0x7f6d900ad960 msgr2=0x7f6d90148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.554+0000 7f6d8dd9b640 1 --2- 172.21.15.110:0/780066273 >> v2:172.21.15.134:3300/0 conn(0x7f6d900ad960 0x7f6d90148cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.554+0000 7f6d8dd9b640 1 -- 172.21.15.110:0/780066273 >> v1:172.21.15.134:6789/0 conn(0x7f6d900af3a0 legacy=0x7f6d9013b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.554+0000 7f6d8dd9b640 10 monclient: _finish_hunting 0 2024-01-20T11:21:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.554+0000 7f6d8dd9b640 1 monclient: found mon.noname-b 2024-01-20T11:21:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.554+0000 7f6d8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.554+0000 7f6d8dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.554+0000 7f6d8dd9b640 1 -- 172.21.15.110:0/780066273 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d900ab2b0 con 0x7f6d900ab6d0 2024-01-20T11:21:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.555+0000 7f6d8dd9b640 1 --2- 172.21.15.110:0/780066273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ab6d0 0x7f6d900ad420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6d84009870 tx=0x7f6d8402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a39909651a0e1b32 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 1 -- 172.21.15.110:0/780066273 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84030840 con 0x7f6d900ab6d0 2024-01-20T11:21:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 10 monclient: dump: 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 10 monclient: _finish_auth 0 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 10 monclient: _check_auth_tickets 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 1 -- 172.21.15.110:0/780066273 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d840309a0 con 0x7f6d900ab6d0 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d94c34640 5 monclient: authenticate success, global_id 5291 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d94c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d94c34640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d94c34640 4 set_mon_vals no callback set 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 1 -- 172.21.15.110:0/780066273 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84030d30 con 0x7f6d900ab6d0 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.558+0000 7f6d8d59a640 10 monclient: dump: 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:32.562 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:32.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 10 monclient: shutdown 2024-01-20T11:21:32.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:32.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 1 -- 172.21.15.110:0/780066273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ab6d0 msgr2=0x7f6d900ad420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:32.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 1 --2- 172.21.15.110:0/780066273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ab6d0 0x7f6d900ad420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6d84009870 tx=0x7f6d8402b980 comp rx=0 tx=0).stop 2024-01-20T11:21:32.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 1 -- 172.21.15.110:0/780066273 shutdown_connections 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 1 --2- 172.21.15.110:0/780066273 >> v2:172.21.15.134:3300/0 conn(0x7f6d900ad960 0x7f6d90148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 1 --2- 172.21.15.110:0/780066273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ab6d0 0x7f6d900ad420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 1 -- 172.21.15.110:0/780066273 >> 172.21.15.110:0/780066273 conn(0x7f6d900b3e60 msgr2=0x7f6d900b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 1 -- 172.21.15.110:0/780066273 shutdown_connections 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.559+0000 7f6d94c34640 1 -- 172.21.15.110:0/780066273 wait complete. 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.560+0000 7f6d94c34640 10 monclient: build_initial_monmap 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.560+0000 7f6d94c34640 10 monclient: monmap: 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.560+0000 7f6d94c34640 1 librados: starting msgr at 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.560+0000 7f6d94c34640 1 librados: starting objecter 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.560+0000 7f6d94c34640 1 Processor -- start 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.560+0000 7f6d94c34640 1 -- start start 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.560+0000 7f6d94c34640 1 librados: setting wanted keys 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.560+0000 7f6d94c34640 1 librados: calling monclient init 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.560+0000 7f6d94c34640 10 monclient: init 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d900ab6d0 0x7f6d9013e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient(hunting): picked mon.noname-b con 0x7f6d900ab6d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ad960 0x7f6d9006f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient(hunting): picked mon.noname-c con 0x7f6d900ad960 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:32.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d900af3a0 0x7f6d9006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient(hunting): picked mon.noname-a con 0x7f6d900af3a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9012f820 con 0x7f6d900af3a0 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d90070210 con 0x7f6d900ab6d0 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9006f7f0 con 0x7f6d900ad960 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d94c34640 10 monclient(hunting): authenticate will time out at 1735.912548s 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d8dd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ad960 0x7f6d9006f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.561+0000 7f6d8ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d900af3a0 0x7f6d9006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8dd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ad960 0x7f6d9006f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57648/0 (socket says 172.21.15.110:57648) 2024-01-20T11:21:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8dd9b640 1 -- 172.21.15.110:0/2693817441 learned_addr learned my addr 172.21.15.110:0/2693817441 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8dd9b640 10 monclient(hunting): get_auth_request con 0x7f6d900ad960 auth_method 0 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8e59c640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d900ab6d0 0x7f6d9013e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8ed9d640 10 monclient(hunting): get_auth_request con 0x7f6d900af3a0 auth_method 0 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8e59c640 10 monclient(hunting): get_auth_request con 0x7f6d900ab6d0 auth_method 0 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:32.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:32.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:32.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:32.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:32.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:32.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:32.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.562+0000 7f6d8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:32.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 10 monclient(hunting): handle_auth_done global_id 5294 payload 824 2024-01-20T11:21:32.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 1 -- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d900ab6d0 msgr2=0x7f6d9013e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:32.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d900ab6d0 0x7f6d9013e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 1 -- 172.21.15.110:0/2693817441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d900af3a0 msgr2=0x7f6d9006fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:32.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d900af3a0 0x7f6d9006fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:32.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 10 monclient: _finish_hunting 0 2024-01-20T11:21:32.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 1 monclient: found mon.noname-c 2024-01-20T11:21:32.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8dd9b640 1 -- 172.21.15.110:0/2693817441 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d840094d0 con 0x7f6d900ad960 2024-01-20T11:21:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.565+0000 7f6d8e59c640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d900ab6d0 0x7f6d9013e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.568+0000 7f6d8dd9b640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ad960 0x7f6d9006f030 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6d84000c00 tx=0x7f6d84004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.568+0000 7f6d7effd640 10 client.?.objecter ms_handle_connect 0x7f6d900ad960 2024-01-20T11:21:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.568+0000 7f6d7effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 1 -- 172.21.15.110:0/2693817441 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84039c90 con 0x7f6d900ad960 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 10 monclient: dump: 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:32.574 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 10 monclient: _finish_auth 0 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d94c34640 5 monclient: authenticate success, global_id 5294 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d94c34640 10 monclient: _renew_subs 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d94c34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d90070f10 con 0x7f6d900ad960 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d94c34640 10 client.5294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d94c34640 10 monclient: _renew_subs 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d94c34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d90071860 con 0x7f6d900ad960 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d94c34640 1 librados: init done 2024-01-20T11:21:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 1 -- 172.21.15.110:0/2693817441 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d84039df0 con 0x7f6d900ad960 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 1 -- 172.21.15.110:0/2693817441 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8404d660 con 0x7f6d900ad960 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d7effd640 10 monclient: dump: 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:32.576 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:32.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.571+0000 7f6d8cd99640 4 set_mon_vals no callback set 2024-01-20T11:21:32.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.572+0000 7f6d94c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:32.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.572+0000 7f6d94c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:32.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.572+0000 7f6d94c34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:32.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.572+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d74005080 con 0x7f6d900ad960 2024-01-20T11:21:32.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.817+0000 7f6d7effd640 1 -- 172.21.15.110:0/2693817441 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6d8404d7c0 con 0x7f6d900ad960 2024-01-20T11:21:32.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.817+0000 7f6d7effd640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d7006c860 0x7f6d7006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:32.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.818+0000 7f6d8e59c640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d7006c860 0x7f6d7006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:32.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.818+0000 7f6d8e59c640 10 monclient: get_auth_request con 0x7f6d7006c860 auth_method 0 2024-01-20T11:21:32.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.818+0000 7f6d7effd640 10 client.5294.objecter ms_handle_connect 0x7f6d7006c860 2024-01-20T11:21:32.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.818+0000 7f6d8e59c640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d7006c860 0x7f6d7006ed20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6d80000f60 tx=0x7f6d80009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:32.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.830+0000 7f6d7effd640 1 -- 172.21.15.110:0/2693817441 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6d8403e070 con 0x7f6d900ad960 2024-01-20T11:21:32.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.830+0000 7f6d7effd640 10 client.5294.objecter ms_dispatch 0x7f6d90000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:32.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.830+0000 7f6d7effd640 3 client.5294.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:32.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.830+0000 7f6d7effd640 3 client.5294.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:32.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.830+0000 7f6d7effd640 20 client.5294.objecter dump_active .. 0 homeless 2024-01-20T11:21:32.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.830+0000 7f6d7effd640 1 -- 172.21.15.110:0/2693817441 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6d84077f80 con 0x7f6d900ad960 2024-01-20T11:21:32.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.830+0000 7f6d7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:32.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:32.830+0000 7f6d7effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:33.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.070+0000 7f6d94c34640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:33.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.070+0000 7f6d94c34640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:33.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.070+0000 7f6d94c34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:33.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.070+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f6d74005600 con 0x7f6d900ad960 2024-01-20T11:21:33.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.076+0000 7f6d7effd640 1 -- 172.21.15.110:0/2693817441 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f6d84034020 con 0x7f6d900ad960 2024-01-20T11:21:33.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.076+0000 7f6d7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:33.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.076+0000 7f6d7effd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:33.077 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:33.077 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":427,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 10 librados: watch_flush enter 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 10 librados: watch_flush exit 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 20 client.5294.objecter shutdown clearing up homeless session... 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 10 client.5294.objecter successfully canceled tick 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d7006c860 msgr2=0x7f6d7006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d7006c860 0x7f6d7006ed20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6d80000f60 tx=0x7f6d80009040 comp rx=0 tx=0).stop 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 10 monclient: shutdown 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ad960 msgr2=0x7f6d9006f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:33.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ad960 0x7f6d9006f030 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6d84000c00 tx=0x7f6d84004730 comp rx=0 tx=0).stop 2024-01-20T11:21:33.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 shutdown_connections 2024-01-20T11:21:33.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6d7006c860 0x7f6d7006ed20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d900af3a0 0x7f6d9006fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d900ad960 0x7f6d9006f030 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 --2- 172.21.15.110:0/2693817441 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d900ab6d0 0x7f6d9013e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.078+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 >> 172.21.15.110:0/2693817441 conn(0x7f6d900b3e60 msgr2=0x7f6d900b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:33.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.079+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 shutdown_connections 2024-01-20T11:21:33.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.079+0000 7f6d94c34640 1 -- 172.21.15.110:0/2693817441 wait complete. 2024-01-20T11:21:33.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.079+0000 7f6d94c34640 1 librados: shutdown 2024-01-20T11:21:33.090 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:33.091 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.188+0000 7f6ab8153640 10 monclient: get_monmap_and_config 2024-01-20T11:21:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.188+0000 7f6ab8153640 10 monclient: build_initial_monmap 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.188+0000 7f6ab8153640 10 monclient: monmap: 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:33.190528+0000 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:33.190528+0000 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:33.190 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:33.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.189+0000 7f6ab8153640 1 Processor -- start 2024-01-20T11:21:33.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.189+0000 7f6ab8153640 1 -- start start 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.189+0000 7f6ab8153640 10 monclient: init 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6ab0066620 0x7f6ab013bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 10 monclient(hunting): picked mon.noname-a con 0x7f6ab0066620 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab006df70 0x7f6ab0145440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 10 monclient(hunting): picked mon.noname-b con 0x7f6ab006df70 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ab0145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab00adf80 con 0x7f6ab0145980 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab006e690 con 0x7f6ab0066620 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0066a00 con 0x7f6ab006df70 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab8153640 10 monclient(hunting): authenticate will time out at 1736.541465s 2024-01-20T11:21:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab5eef640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6ab0066620 0x7f6ab013bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.190+0000 7f6ab56ee640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab006df70 0x7f6ab0145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab5eef640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6ab0066620 0x7f6ab013bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47326/0 (socket says 172.21.15.110:47326) 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab5eef640 1 -- 172.21.15.110:0/2648414328 learned_addr learned my addr 172.21.15.110:0/2648414328 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab5eef640 10 monclient(hunting): get_auth_request con 0x7f6ab0066620 auth_method 0 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab5eef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab5eef640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab5eef640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab56ee640 10 monclient(hunting): get_auth_request con 0x7f6ab006df70 auth_method 0 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab56ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab56ee640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab56ee640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab56ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab56ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.191+0000 7f6ab56ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.192+0000 7f6ab5eef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.192+0000 7f6ab5eef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.192+0000 7f6ab5eef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.193+0000 7f6ab56ee640 10 monclient(hunting): handle_auth_done global_id 5297 payload 274 2024-01-20T11:21:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.193+0000 7f6ab56ee640 1 -- 172.21.15.110:0/2648414328 >> v2:172.21.15.110:3300/0 conn(0x7f6ab0066620 msgr2=0x7f6ab013bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.193+0000 7f6ab56ee640 1 --2- 172.21.15.110:0/2648414328 >> v2:172.21.15.110:3300/0 conn(0x7f6ab0066620 0x7f6ab013bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.193+0000 7f6ab56ee640 1 -- 172.21.15.110:0/2648414328 >> v1:172.21.15.110:6789/0 conn(0x7f6ab0145980 legacy=0x7f6ab00af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.193+0000 7f6ab56ee640 10 monclient: _finish_hunting 0 2024-01-20T11:21:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.193+0000 7f6ab56ee640 1 monclient: found mon.noname-b 2024-01-20T11:21:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.193+0000 7f6ab56ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.193+0000 7f6ab56ee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.193+0000 7f6ab56ee640 1 -- 172.21.15.110:0/2648414328 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab006f650 con 0x7f6ab006df70 2024-01-20T11:21:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.196+0000 7f6ab56ee640 1 --2- 172.21.15.110:0/2648414328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab006df70 0x7f6ab0145440 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00097a0 tx=0x7f6aa002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=652de1cf39d1e2dd server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 1 -- 172.21.15.110:0/2648414328 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa002e070 con 0x7f6ab006df70 2024-01-20T11:21:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 10 monclient: dump: 2024-01-20T11:21:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:33.201 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 10 monclient: _finish_auth 0 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 10 monclient: _check_auth_tickets 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 1 -- 172.21.15.110:0/2648414328 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa00315a0 con 0x7f6ab006df70 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 1 -- 172.21.15.110:0/2648414328 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0031930 con 0x7f6ab006df70 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:33.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab4eed640 10 monclient: dump: 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab8153640 5 monclient: authenticate success, global_id 5297 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab8153640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.199+0000 7f6ab8153640 4 set_mon_vals no callback set 2024-01-20T11:21:33.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 10 monclient: shutdown 2024-01-20T11:21:33.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:33.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 1 -- 172.21.15.110:0/2648414328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab006df70 msgr2=0x7f6ab0145440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:33.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 1 --2- 172.21.15.110:0/2648414328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab006df70 0x7f6ab0145440 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00097a0 tx=0x7f6aa002b840 comp rx=0 tx=0).stop 2024-01-20T11:21:33.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 1 -- 172.21.15.110:0/2648414328 shutdown_connections 2024-01-20T11:21:33.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 1 --2- 172.21.15.110:0/2648414328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab006df70 0x7f6ab0145440 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 1 --2- 172.21.15.110:0/2648414328 >> v2:172.21.15.110:3300/0 conn(0x7f6ab0066620 0x7f6ab013bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 1 -- 172.21.15.110:0/2648414328 >> 172.21.15.110:0/2648414328 conn(0x7f6ab00b09e0 msgr2=0x7f6ab00b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:33.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 1 -- 172.21.15.110:0/2648414328 shutdown_connections 2024-01-20T11:21:33.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.200+0000 7f6ab8153640 1 -- 172.21.15.110:0/2648414328 wait complete. 2024-01-20T11:21:33.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.201+0000 7f6ab8153640 10 monclient: build_initial_monmap 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.201+0000 7f6ab8153640 10 monclient: monmap: 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.201+0000 7f6ab8153640 1 librados: starting msgr at 2024-01-20T11:21:33.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.201+0000 7f6ab8153640 1 librados: starting objecter 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.201+0000 7f6ab8153640 1 Processor -- start 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.201+0000 7f6ab8153640 1 -- start start 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.201+0000 7f6ab8153640 1 librados: setting wanted keys 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.201+0000 7f6ab8153640 1 librados: calling monclient init 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.201+0000 7f6ab8153640 10 monclient: init 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab0066620 0x7f6ab0141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient(hunting): picked mon.noname-c con 0x7f6ab0066620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ab006df70 0x7f6ab013d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient(hunting): picked mon.noname-b con 0x7f6ab006df70 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab0145980 0x7f6ab0138520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient(hunting): picked mon.noname-a con 0x7f6ab0145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:33.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab013dbf0 con 0x7f6ab0145980 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab013dea0 con 0x7f6ab006df70 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab01cb240 con 0x7f6ab0066620 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab8153640 10 monclient(hunting): authenticate will time out at 1736.553494s 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab5eef640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab0066620 0x7f6ab0141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.202+0000 7f6ab66f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab0145980 0x7f6ab0138520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab56ee640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ab006df70 0x7f6ab013d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab5eef640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab0066620 0x7f6ab0141a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57670/0 (socket says 172.21.15.110:57670) 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab5eef640 1 -- 172.21.15.110:0/1126778787 learned_addr learned my addr 172.21.15.110:0/1126778787 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab66f0640 10 monclient(hunting): get_auth_request con 0x7f6ab0145980 auth_method 0 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab66f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab66f0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:33.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab66f0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab5eef640 10 monclient(hunting): get_auth_request con 0x7f6ab0066620 auth_method 0 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab5eef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab5eef640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab5eef640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab56ee640 10 monclient(hunting): get_auth_request con 0x7f6ab006df70 auth_method 0 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab56ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab56ee640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab56ee640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab5eef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab5eef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab5eef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab66f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab66f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:33.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.203+0000 7f6ab66f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.204+0000 7f6ab56ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.204+0000 7f6ab56ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.204+0000 7f6ab56ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 10 monclient(hunting): handle_auth_done global_id 5571 payload 824 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 1 -- 172.21.15.110:0/1126778787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab0066620 msgr2=0x7f6ab0141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab0066620 0x7f6ab0141a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 1 -- 172.21.15.110:0/1126778787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ab006df70 msgr2=0x7f6ab013d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ab006df70 0x7f6ab013d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 10 monclient: _finish_hunting 0 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 1 monclient: found mon.noname-a 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:33.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab66f0640 1 -- 172.21.15.110:0/1126778787 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa0009400 con 0x7f6ab0145980 2024-01-20T11:21:33.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.206+0000 7f6ab5eef640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab0066620 0x7f6ab0141a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:33.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.208+0000 7f6ab66f0640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab0145980 0x7f6ab0138520 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00c3c0 tx=0x7f6aac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:33.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.208+0000 7f6aa67fc640 10 client.?.objecter ms_handle_connect 0x7f6ab0145980 2024-01-20T11:21:33.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.208+0000 7f6aa67fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:33.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 1 -- 172.21.15.110:0/1126778787 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0052d0 con 0x7f6ab0145980 2024-01-20T11:21:33.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:33.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:33.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 10 monclient: dump: 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:33.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 1 -- 172.21.15.110:0/1126778787 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac005430 con 0x7f6ab0145980 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 1 -- 172.21.15.110:0/1126778787 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac003830 con 0x7f6ab0145980 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6ab8153640 5 monclient: authenticate success, global_id 5571 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6ab8153640 10 monclient: _renew_subs 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6ab8153640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab013c380 con 0x7f6ab0145980 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa67fc640 10 monclient: dump: 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:33.216 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6aa7fff640 4 set_mon_vals no callback set 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6ab8153640 10 client.5571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6ab8153640 10 monclient: _renew_subs 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6ab8153640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab01d7190 con 0x7f6ab0145980 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.212+0000 7f6ab8153640 1 librados: init done 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.213+0000 7f6ab8153640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.213+0000 7f6ab8153640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.213+0000 7f6ab8153640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:33.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.213+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab01cb0b0 con 0x7f6ab0145980 2024-01-20T11:21:33.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~155s 2024-01-20T11:21:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.456+0000 7f6aa67fc640 1 -- 172.21.15.110:0/1126778787 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6aac003990 con 0x7f6ab0145980 2024-01-20T11:21:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.456+0000 7f6aa67fc640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6a9406c8b0 0x7f6a9406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.457+0000 7f6ab5eef640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6a9406c8b0 0x7f6a9406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.457+0000 7f6ab5eef640 10 monclient: get_auth_request con 0x7f6a9406c8b0 auth_method 0 2024-01-20T11:21:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.457+0000 7f6aa67fc640 10 client.5571.objecter ms_handle_connect 0x7f6a9406c8b0 2024-01-20T11:21:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.457+0000 7f6ab5eef640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6a9406c8b0 0x7f6a9406ed70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009360 tx=0x7f6aa800b040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:33.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.469+0000 7f6aa67fc640 1 -- 172.21.15.110:0/1126778787 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6aac057ae0 con 0x7f6ab0145980 2024-01-20T11:21:33.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.469+0000 7f6aa67fc640 10 client.5571.objecter ms_dispatch 0x7f6ab0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:33.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.469+0000 7f6aa67fc640 3 client.5571.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.469+0000 7f6aa67fc640 3 client.5571.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.469+0000 7f6aa67fc640 20 client.5571.objecter dump_active .. 0 homeless 2024-01-20T11:21:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.469+0000 7f6aa67fc640 1 -- 172.21.15.110:0/1126778787 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6aac053e20 con 0x7f6ab0145980 2024-01-20T11:21:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.469+0000 7f6aa67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.469+0000 7f6aa67fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:33.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.647+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ab00b1500 con 0x7f6a9406c8b0 2024-01-20T11:21:33.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.650+0000 7f6aa67fc640 1 -- 172.21.15.110:0/1126778787 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64007 (secure 0 0 0) 0x7f6aa800f040 con 0x7f6a9406c8b0 2024-01-20T11:21:33.651 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:33.651 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:33.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 10 librados: watch_flush enter 2024-01-20T11:21:33.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 10 librados: watch_flush exit 2024-01-20T11:21:33.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 20 client.5571.objecter shutdown clearing up homeless session... 2024-01-20T11:21:33.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 10 client.5571.objecter successfully canceled tick 2024-01-20T11:21:33.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6a9406c8b0 msgr2=0x7f6a9406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6a9406c8b0 0x7f6a9406ed70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009360 tx=0x7f6aa800b040 comp rx=0 tx=0).stop 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 10 monclient: shutdown 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab0145980 msgr2=0x7f6ab0138520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab0145980 0x7f6ab0138520 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00c3c0 tx=0x7f6aac00c880 comp rx=0 tx=0).stop 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 shutdown_connections 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6a9406c8b0 0x7f6a9406ed70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab0145980 0x7f6ab0138520 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ab006df70 0x7f6ab013d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 --2- 172.21.15.110:0/1126778787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ab0066620 0x7f6ab0141a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 >> 172.21.15.110:0/1126778787 conn(0x7f6ab00b09e0 msgr2=0x7f6ab00afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 shutdown_connections 2024-01-20T11:21:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 -- 172.21.15.110:0/1126778787 wait complete. 2024-01-20T11:21:33.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:33.652+0000 7f6ab8153640 1 librados: shutdown 2024-01-20T11:21:33.664 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-01-20T11:21:32.785056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002570"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":7.4450000000000003,"5min":9.2550000000000008,"15min":13.324999999999999},"min":{"1min":4.8440000000000003,"5min":4.3220000000000001,"15min":4.3220000000000001},"max":{"1min":13.589,"5min":100.05800000000001,"15min":100.05800000000001},"last":5.7320000000000002},{"interface":"front","average":{"1min":8.7729999999999997,"5min":9.9809999999999999,"15min":14.504},"min":{"1min":4.4969999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.018000000000001,"5min":91.789000000000001,"15min":91.789000000000001},"last":6.3579999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.577,"5min":11.884,"15min":16.199000000000002},"min":{"1min":7.3899999999999997,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":13.903,"5min":98.700000000000003,"15min":98.700000000000003},"last":9.6720000000000006},{"interface":"front","average":{"1min":10.343999999999999,"5min":11.958,"15min":15.356999999999999},"min":{"1min":5.7910000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"max":{"1min":17.707999999999998,"5min":95.603999999999999,"15min":95.603999999999999},"last":8.9019999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":11.824,"5min":13.266999999999999,"15min":17.303000000000001},"min":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":15.699999999999999,"5min":131.09700000000001,"15min":131.09700000000001},"last":14.054},{"interface":"front","average":{"1min":11.898999999999999,"5min":13.811999999999999,"15min":16.754000000000001},"min":{"1min":7.5960000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"max":{"1min":16.082999999999998,"5min":105.43899999999999,"15min":105.43899999999999},"last":11.497}]},{"osd":3,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":9.4420000000000002,"5min":13.632,"15min":19.149999999999999},"min":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"max":{"1min":15.443,"5min":94.911000000000001,"15min":94.911000000000001},"last":12.784000000000001},{"interface":"front","average":{"1min":12.56,"5min":15.948,"15min":21.048999999999999},"min":{"1min":8.6359999999999992,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":15.823,"5min":100.633,"15min":100.633},"last":10.279999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":14.127000000000001,"5min":16.562999999999999,"15min":20.863},"min":{"1min":11.73,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.920999999999999,"5min":131.70599999999999,"15min":131.70599999999999},"last":14.98},{"interface":"front","average":{"1min":11.83,"5min":14.897,"15min":19.475000000000001},"min":{"1min":6.9939999999999998,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":17.809999999999999,"5min":107.962,"15min":107.962},"last":12.093}]},{"osd":7,"last update":"Sat Jan 20 11:20:29 2024","interfaces":[{"interface":"back","average":{"1min":10.475,"5min":13.804,"15min":18.122},"min":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.039000000000001,"5min":96.911000000000001,"15min":96.911000000000001},"last":8.6310000000000002},{"interface":"front","average":{"1min":14.247,"5min":17.617000000000001,"15min":21.736000000000001},"min":{"1min":9.766,"5min":9.6739999999999995,"15min":9.6739999999999995},"max":{"1min":16.706,"5min":129.828,"15min":129.828},"last":15.757999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313296,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":8.2970000000000006,"15min":9.6880000000000006},"min":{"1min":4.524,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":13.144,"5min":38.005000000000003,"15min":38.005000000000003},"last":5.8470000000000004},{"interface":"front","average":{"1min":8.2070000000000007,"5min":10.077,"15min":12.731},"min":{"1min":4.0730000000000004,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":15.167,"5min":76.718999999999994,"15min":76.718999999999994},"last":6.7919999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":8.3170000000000002,"5min":9.327,"15min":10.210000000000001},"min":{"1min":5.6120000000000001,"5min":5.1440000000000001,"15min":5.1440000000000001},"max":{"1min":12.765000000000001,"5min":18.859000000000002,"15min":18.859000000000002},"last":9.0380000000000003},{"interface":"front","average":{"1min":10.612,"5min":11.499000000000001,"15min":12.829000000000001},"min":{"1min":6.2240000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.484,"5min":43.774000000000001,"15min":43.774000000000001},"last":7.915}]},{"osd":2,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":11.651999999999999,"5min":13.237,"15min":16.076000000000001},"min":{"1min":7.0570000000000004,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.428000000000001,"5min":49.210000000000001,"15min":49.210000000000001},"last":11.496},{"interface":"front","average":{"1min":9.9440000000000008,"5min":12.291,"15min":15.537000000000001},"min":{"1min":5.7510000000000003,"5min":5.7510000000000003,"15min":5.7510000000000003},"max":{"1min":15.755000000000001,"5min":78.548000000000002,"15min":78.548000000000002},"last":14.978999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":10.725,"5min":12.265000000000001,"15min":14.398999999999999},"min":{"1min":6.5270000000000001,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":15.611000000000001,"5min":55.771000000000001,"15min":55.771000000000001},"last":10.148999999999999},{"interface":"front","average":{"1min":13.539999999999999,"5min":14.837,"15min":16.568999999999999},"min":{"1min":8.7799999999999994,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.385000000000002,"5min":49.829999999999998,"15min":49.829999999999998},"last":10.866}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":12.15,"5min":14.099,"15min":16.856999999999999},"min":{"1min":9.2279999999999998,"5min":7.4080000000000004,"15min":7.4080000000000004},"max":{"1min":15.438000000000001,"5min":91.667000000000002,"15min":91.667000000000002},"last":12.221},{"interface":"front","average":{"1min":11.084,"5min":13.510999999999999,"15min":16.715},"min":{"1min":6.5039999999999996,"5min":6.5039999999999996,"15min":6.5039999999999996},"max":{"1min":15.997999999999999,"5min":92.613,"15min":92.613},"last":12.856999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:20:31 2024","interfaces":[{"interface":"back","average":{"1min":12.93,"5min":15.109999999999999,"15min":17.879000000000001},"min":{"1min":9.548,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":16.408999999999999,"5min":80.102000000000004,"15min":80.102000000000004},"last":14.228999999999999},{"interface":"front","average":{"1min":13.510999999999999,"5min":15.808,"15min":18.274999999999999},"min":{"1min":7.6790000000000003,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.550999999999998,"5min":53.398000000000003,"15min":53.398000000000003},"last":13.52}]}]},{"osd":3,"up_from":21,"seq":90194313297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.617},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.484}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.932},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.029}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.278},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":7.0119999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.738},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":7.8630000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":9.1899999999999995},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.000999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":15.488},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.186}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.1660000000000004},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":4.2839999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":10.784000000000001},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.8070000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.212999999999999,"5min":13.593,"15min":16.375},"min":{"1min":8.6799999999999997,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":15.856,"5min":49.237000000000002,"15min":49.237000000000002},"last":12.208},{"interface":"front","average":{"1min":10.585000000000001,"5min":11.468999999999999,"15min":13.85},"min":{"1min":7.2149999999999999,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":15.077,"5min":52.905999999999999,"15min":52.905999999999999},"last":6.5330000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":9.4499999999999993},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":14.757999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":15.630000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.206}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":12.573,"5min":12.672000000000001,"15min":13.452},"min":{"1min":9.0809999999999995,"5min":7.6580000000000004,"15min":7.6580000000000004},"max":{"1min":16.238,"5min":24.266999999999999,"15min":24.266999999999999},"last":11.462},{"interface":"front","average":{"1min":14.433,"5min":15.532999999999999,"15min":17.238},"min":{"1min":11.493,"5min":10.297000000000001,"15min":10.297000000000001},"max":{"1min":16.664000000000001,"5min":28.148,"15min":28.148},"last":12.811999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.0890000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":5.7610000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":9.3650000000000002},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.8129999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":11.819000000000001,"5min":15.890000000000001,"15min":23.317},"min":{"1min":9.0960000000000001,"5min":7.9240000000000004,"15min":7.9240000000000004},"max":{"1min":16.783000000000001,"5min":259.483,"15min":259.483},"last":14.035},{"interface":"front","average":{"1min":9.5990000000000002,"5min":12.212,"15min":18.497},"min":{"1min":6.1639999999999997,"5min":5.8840000000000003,"15min":5.8840000000000003},"max":{"1min":15.446999999999999,"5min":243.24199999999999,"15min":243.24199999999999},"last":8.7829999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":9.9760000000000009},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.727}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.531000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":11.513}]},{"osd":7,"last update":"Sat Jan 20 11:20:26 2024","interfaces":[{"interface":"back","average":{"1min":13.114000000000001,"5min":16.474,"15min":27.009},"min":{"1min":9.6980000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":20.875,"5min":344.98099999999999,"15min":344.98099999999999},"last":13.401},{"interface":"front","average":{"1min":15.055,"5min":18.616,"15min":27.896000000000001},"min":{"1min":10.977,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":19.672000000000001,"5min":313.45999999999998,"15min":313.45999999999998},"last":12.129}]}]},{"osd":0,"up_from":17,"seq":73014444111,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":5.3079999999999998},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.7219999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":10.587},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":11.696999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":12.382},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.004}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":13.093999999999999},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":6.633}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":14.214},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":15.917}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":14.789},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":7.3099999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509520,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":7.569},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":5.8010000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":6.9059999999999997},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":10.468999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":15.603},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":11.253}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":12.233000000000001},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":9.4329999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":13.743},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":13.124000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":14.388},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":8.2769999999999992}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:36.667 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.775+0000 7f4df03f5640 10 monclient: get_monmap_and_config 2024-01-20T11:21:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.775+0000 7f4df03f5640 10 monclient: build_initial_monmap 2024-01-20T11:21:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.775+0000 7f4df03f5640 10 monclient: monmap: 2024-01-20T11:21:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:36.776350+0000 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:36.776350+0000 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:36.776 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.776+0000 7f4df03f5640 1 Processor -- start 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.776+0000 7f4df03f5640 1 -- start start 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.776+0000 7f4df03f5640 10 monclient: init 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4de806b380 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4de806e430 0x7f4de81435d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 10 monclient(hunting): picked mon.noname-a con 0x7f4de806e430 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4de813d600 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de8071de0 con 0x7f4de806b380 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de8070ef0 con 0x7f4de813d600 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de8144220 con 0x7f4de806e430 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4df03f5640 10 monclient(hunting): authenticate will time out at 1740.127504s 2024-01-20T11:21:36.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.777+0000 7f4ded990640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4de806e430 0x7f4de81435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:36.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.778+0000 7f4dee992640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4de813d600 0x7f4de8143b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:40302/0 (socket says 172.21.15.110:40302) 2024-01-20T11:21:36.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.778+0000 7f4dee992640 1 -- 172.21.15.110:0/3244216723 learned_addr learned my addr 172.21.15.110:0/3244216723 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:36.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.778+0000 7f4ded990640 10 monclient(hunting): get_auth_request con 0x7f4de806e430 auth_method 0 2024-01-20T11:21:36.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.778+0000 7f4ded990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:36.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.778+0000 7f4ded990640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:36.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.778+0000 7f4ded990640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:36.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.779+0000 7f4ded990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:36.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.779+0000 7f4ded990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:36.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.779+0000 7f4ded990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded990640 10 monclient(hunting): handle_auth_done global_id 5574 payload 274 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded990640 1 -- 172.21.15.110:0/3244216723 >> v1:172.21.15.134:6789/0 conn(0x7f4de813d600 legacy=0x7f4de8143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded990640 1 -- 172.21.15.110:0/3244216723 >> v1:172.21.15.110:6789/0 conn(0x7f4de806b380 legacy=0x7f4de813cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded990640 10 monclient: _finish_hunting 0 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded990640 1 monclient: found mon.noname-a 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded18f640 1 -- 172.21.15.110:0/3244216723 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15513325 0 0) 0x7f4de8070ef0 con 0x7f4de813d600 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded990640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded990640 1 -- 172.21.15.110:0/3244216723 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de806dc70 con 0x7f4de806e430 2024-01-20T11:21:36.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.781+0000 7f4ded18f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:21:36.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.785+0000 7f4ded990640 1 --2- 172.21.15.110:0/3244216723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de806e430 0x7f4de81435d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80097a0 tx=0x7f4dd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d25d9e4cb8444 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:36.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 1 -- 172.21.15.110:0/3244216723 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8031830 con 0x7f4de806e430 2024-01-20T11:21:36.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:36.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:36.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 10 monclient: dump: 2024-01-20T11:21:36.789 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:36.789 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 10 monclient: _finish_auth 0 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 10 monclient: _check_auth_tickets 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 1 -- 172.21.15.110:0/3244216723 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd8031990 con 0x7f4de806e430 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4df03f5640 5 monclient: authenticate success, global_id 5574 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4df03f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:36.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4df03f5640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 1 -- 172.21.15.110:0/3244216723 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8031d20 con 0x7f4de806e430 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4df03f5640 4 set_mon_vals no callback set 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.789+0000 7f4ded18f640 10 monclient: dump: 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:36.791 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:36.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.790+0000 7f4df03f5640 10 monclient: shutdown 2024-01-20T11:21:36.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.790+0000 7f4df03f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:36.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.790+0000 7f4df03f5640 1 -- 172.21.15.110:0/3244216723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de806e430 msgr2=0x7f4de81435d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:36.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.790+0000 7f4df03f5640 1 --2- 172.21.15.110:0/3244216723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de806e430 0x7f4de81435d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80097a0 tx=0x7f4dd802b8d0 comp rx=0 tx=0).stop 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.790+0000 7f4df03f5640 1 -- 172.21.15.110:0/3244216723 shutdown_connections 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.790+0000 7f4df03f5640 1 --2- 172.21.15.110:0/3244216723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de806e430 0x7f4de81435d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.790+0000 7f4df03f5640 1 -- 172.21.15.110:0/3244216723 >> 172.21.15.110:0/3244216723 conn(0x7f4de806a5f0 msgr2=0x7f4de806aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.790+0000 7f4df03f5640 1 -- 172.21.15.110:0/3244216723 shutdown_connections 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.790+0000 7f4df03f5640 1 -- 172.21.15.110:0/3244216723 wait complete. 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.791+0000 7f4df03f5640 10 monclient: build_initial_monmap 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.791+0000 7f4df03f5640 10 monclient: monmap: 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.791+0000 7f4df03f5640 1 librados: starting msgr at 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.791+0000 7f4df03f5640 1 librados: starting objecter 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 Processor -- start 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 -- start start 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 librados: setting wanted keys 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 librados: calling monclient init 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient: init 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4de806b380 0x7f4de8139880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient(hunting): picked mon.noname-b con 0x7f4de806b380 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4de806e430 0x7f4de80af200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient(hunting): picked mon.noname-c con 0x7f4de806e430 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de813d600 0x7f4de813a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient(hunting): picked mon.noname-a con 0x7f4de813d600 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de8139e80 con 0x7f4de813d600 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de813b990 con 0x7f4de806b380 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80af740 con 0x7f4de806e430 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.792+0000 7f4df03f5640 10 monclient(hunting): authenticate will time out at 1740.142889s 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.793+0000 7f4ded990640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4de806e430 0x7f4de80af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.793+0000 7f4dee191640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4de806b380 0x7f4de8139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.793+0000 7f4ded990640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4de806e430 0x7f4de80af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57684/0 (socket says 172.21.15.110:57684) 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.793+0000 7f4ded990640 1 -- 172.21.15.110:0/3479630692 learned_addr learned my addr 172.21.15.110:0/3479630692 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.793+0000 7f4ded990640 10 monclient(hunting): get_auth_request con 0x7f4de806e430 auth_method 0 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.793+0000 7f4ded990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.793+0000 7f4ded990640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.793+0000 7f4ded990640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4dee191640 10 monclient(hunting): get_auth_request con 0x7f4de806b380 auth_method 0 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4dee191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4dee191640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4dee191640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4ded990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4ded990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4ded990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4dee191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4dee191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.794+0000 7f4dee191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:36.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 10 monclient(hunting): handle_auth_done global_id 5303 payload 824 2024-01-20T11:21:36.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 1 -- 172.21.15.110:0/3479630692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4de806b380 msgr2=0x7f4de8139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:36.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4de806b380 0x7f4de8139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:36.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 1 -- 172.21.15.110:0/3479630692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de813d600 msgr2=0x7f4de813a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de813d600 0x7f4de813a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 10 monclient: _finish_hunting 0 2024-01-20T11:21:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 1 monclient: found mon.noname-c 2024-01-20T11:21:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.797+0000 7f4ded990640 1 -- 172.21.15.110:0/3479630692 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd8009400 con 0x7f4de806e430 2024-01-20T11:21:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.799+0000 7f4ded990640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4de806e430 0x7f4de80af200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80098c0 tx=0x7f4dd80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.799+0000 7f4ddeffd640 10 client.?.objecter ms_handle_connect 0x7f4de806e430 2024-01-20T11:21:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.799+0000 7f4ddeffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 1 -- 172.21.15.110:0/3479630692 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd803ba70 con 0x7f4de806e430 2024-01-20T11:21:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 10 monclient: dump: 2024-01-20T11:21:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 10 monclient: _finish_auth 0 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 1 -- 172.21.15.110:0/3479630692 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd803bbf0 con 0x7f4de806e430 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4df03f5640 5 monclient: authenticate success, global_id 5303 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4df03f5640 10 monclient: _renew_subs 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 1 -- 172.21.15.110:0/3479630692 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd80453f0 con 0x7f4de806e430 2024-01-20T11:21:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4df03f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de813bc70 con 0x7f4de806e430 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4dec98e640 4 set_mon_vals no callback set 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4ddeffd640 10 monclient: dump: 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:36.805 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4df03f5640 10 client.5303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4df03f5640 10 monclient: _renew_subs 2024-01-20T11:21:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4df03f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de81381d0 con 0x7f4de806e430 2024-01-20T11:21:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.803+0000 7f4df03f5640 1 librados: init done 2024-01-20T11:21:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.804+0000 7f4df03f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.804+0000 7f4df03f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.804+0000 7f4df03f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:36.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.804+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de8139ff0 con 0x7f4de806e430 2024-01-20T11:21:36.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.823+0000 7f4ddeffd640 1 -- 172.21.15.110:0/3479630692 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4dd804e450 con 0x7f4de806e430 2024-01-20T11:21:36.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.824+0000 7f4ddeffd640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dcc06c890 0x7f4dcc06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:36.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.824+0000 7f4dee191640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dcc06c890 0x7f4dcc06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:36.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.824+0000 7f4dee191640 10 monclient: get_auth_request con 0x7f4dcc06c890 auth_method 0 2024-01-20T11:21:36.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.825+0000 7f4ddeffd640 10 client.5303.objecter ms_handle_connect 0x7f4dcc06c890 2024-01-20T11:21:36.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.825+0000 7f4dee191640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dcc06c890 0x7f4dcc06ed50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4de4000f60 tx=0x7f4de4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.827+0000 7f4ddeffd640 1 -- 172.21.15.110:0/3479630692 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4dd807af80 con 0x7f4de806e430 2024-01-20T11:21:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.827+0000 7f4ddeffd640 10 client.5303.objecter ms_dispatch 0x7f4de8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.827+0000 7f4ddeffd640 3 client.5303.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.827+0000 7f4ddeffd640 3 client.5303.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:36.827+0000 7f4ddeffd640 20 client.5303.objecter dump_active .. 0 homeless 2024-01-20T11:21:37.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.063+0000 7f4ddeffd640 1 -- 172.21.15.110:0/3479630692 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4dd8076bc0 con 0x7f4de806e430 2024-01-20T11:21:37.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.063+0000 7f4ddeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:37.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.063+0000 7f4ddeffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:37.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.242+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4de806e0a0 con 0x7f4dcc06c890 2024-01-20T11:21:37.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.245+0000 7f4ddeffd640 1 -- 172.21.15.110:0/3479630692 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64099 (secure 0 0 0) 0x7f4de40014a0 con 0x7f4dcc06c890 2024-01-20T11:21:37.245 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:37.246 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:37.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 10 librados: watch_flush enter 2024-01-20T11:21:37.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 10 librados: watch_flush exit 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 20 client.5303.objecter shutdown clearing up homeless session... 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 10 client.5303.objecter successfully canceled tick 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dcc06c890 msgr2=0x7f4dcc06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dcc06c890 0x7f4dcc06ed50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4de4000f60 tx=0x7f4de4009040 comp rx=0 tx=0).stop 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 10 monclient: shutdown 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4de806e430 msgr2=0x7f4de80af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4de806e430 0x7f4de80af200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80098c0 tx=0x7f4dd80046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 shutdown_connections 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4dcc06c890 0x7f4dcc06ed50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de813d600 0x7f4de813a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4de806e430 0x7f4de80af200 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 --2- 172.21.15.110:0/3479630692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4de806b380 0x7f4de8139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 >> 172.21.15.110:0/3479630692 conn(0x7f4de806a5f0 msgr2=0x7f4de8134dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:37.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 shutdown_connections 2024-01-20T11:21:37.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 -- 172.21.15.110:0/3479630692 wait complete. 2024-01-20T11:21:37.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.248+0000 7f4df03f5640 1 librados: shutdown 2024-01-20T11:21:37.260 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-01-20T11:21:36.785983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002676"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":8.1150000000000002},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":12.589}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":9.1219999999999999},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":11.061999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":10.407999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":18.013000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":8.8499999999999996},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":17.672000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":9.7189999999999994},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":16.481000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":19.302},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":13.259}]}]},{"osd":7,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":9.1869999999999994},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":6.391}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.3460000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.722}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":10.59},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.6769999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":9.9629999999999992},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":15.114000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":16.260000000000002},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.340999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":12.997},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.105}]}]},{"osd":3,"up_from":21,"seq":90194313297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.617},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.484}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.932},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.029}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.278},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":7.0119999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.738},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":7.8630000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":9.1899999999999995},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.000999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":15.488},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.186}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.1660000000000004},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":4.2839999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":7.6150000000000002},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":11.677}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.641999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":11.106999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":19.533000000000001},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":21.629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":16.367000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":8.2149999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":9.8469999999999995},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":17.047000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444112,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.024},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":5.8680000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":10.297000000000001},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":8.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":17.565999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":11.266}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":13.276},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.047000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.0640000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":13.920999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":16.285},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.542999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444112,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":4.5890000000000004},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.2620000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":9.2119999999999997},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":10.185}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":7.4059999999999997},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.994}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.423},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":13.313000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":13.936999999999999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":14.939}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.914},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":10.829000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.6580000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.2469999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.0860000000000003},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":7.4009999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":9.9830000000000005},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.9770000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.583},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":11.821}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.205},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.095000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":14.952},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.404999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:37.261 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.362+0000 7fd9ef6a9640 10 monclient: get_monmap_and_config 2024-01-20T11:21:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.362+0000 7fd9ef6a9640 10 monclient: build_initial_monmap 2024-01-20T11:21:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.362+0000 7fd9ef6a9640 10 monclient: monmap: 2024-01-20T11:21:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:37.363087+0000 2024-01-20T11:21:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:37.363087+0000 2024-01-20T11:21:37.363 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:37.363 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.363 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:37.363 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:37.363 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:37.363 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:37.363 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:37.363 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 1 Processor -- start 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 1 -- start start 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient: init 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd9e806bb00 0x7fd9e80b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient(hunting): picked mon.noname-c con 0x7fd9e806bb00 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9e80af790 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd9e80b0900 0x7fd9e8145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient(hunting): picked mon.noname-a con 0x7fd9e80b0900 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e806c220 con 0x7fd9e80af790 2024-01-20T11:21:37.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e806bee0 con 0x7fd9e80b0900 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e80ada30 con 0x7fd9e806bb00 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.363+0000 7fd9ef6a9640 10 monclient(hunting): authenticate will time out at 1740.714022s 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.364+0000 7fd9edc46640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd9e80b0900 0x7fd9e8145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.364+0000 7fd9ed445640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd9e806bb00 0x7fd9e80b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.364+0000 7fd9edc46640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd9e80b0900 0x7fd9e8145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47368/0 (socket says 172.21.15.110:47368) 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.364+0000 7fd9edc46640 1 -- 172.21.15.110:0/2760276380 learned_addr learned my addr 172.21.15.110:0/2760276380 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.364+0000 7fd9edc46640 10 monclient(hunting): get_auth_request con 0x7fd9e80b0900 auth_method 0 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.364+0000 7fd9edc46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.364+0000 7fd9edc46640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.364+0000 7fd9edc46640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9ed445640 10 monclient(hunting): get_auth_request con 0x7fd9e806bb00 auth_method 0 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9ed445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9ed445640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9ed445640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9edc46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9edc46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9edc46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9ed445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9ed445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.365+0000 7fd9ed445640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.366+0000 7fd9edc46640 10 monclient(hunting): handle_auth_done global_id 5580 payload 274 2024-01-20T11:21:37.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.366+0000 7fd9edc46640 1 -- 172.21.15.110:0/2760276380 >> v2:172.21.15.134:3300/0 conn(0x7fd9e806bb00 msgr2=0x7fd9e80b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:37.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.366+0000 7fd9edc46640 1 --2- 172.21.15.110:0/2760276380 >> v2:172.21.15.134:3300/0 conn(0x7fd9e806bb00 0x7fd9e80b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.366+0000 7fd9edc46640 1 -- 172.21.15.110:0/2760276380 >> v1:172.21.15.134:6789/0 conn(0x7fd9e80af790 legacy=0x7fd9e8145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.366+0000 7fd9edc46640 10 monclient: _finish_hunting 0 2024-01-20T11:21:37.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.366+0000 7fd9edc46640 1 monclient: found mon.noname-a 2024-01-20T11:21:37.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.366+0000 7fd9edc46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:37.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.366+0000 7fd9edc46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.366+0000 7fd9edc46640 1 -- 172.21.15.110:0/2760276380 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e80add10 con 0x7fd9e80b0900 2024-01-20T11:21:37.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.368+0000 7fd9edc46640 1 --2- 172.21.15.110:0/2760276380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 0x7fd9e8145e80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd9e80b6630 tx=0x7fd9e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab25525f11f32934 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:37.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 1 -- 172.21.15.110:0/2760276380 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e002ca80 con 0x7fd9e80b0900 2024-01-20T11:21:37.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:37.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:37.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 10 monclient: dump: 2024-01-20T11:21:37.372 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:37.372 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.372 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 10 monclient: _finish_auth 0 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 1 -- 172.21.15.110:0/2760276380 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e002cbe0 con 0x7fd9e80b0900 2024-01-20T11:21:37.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 1 -- 172.21.15.110:0/2760276380 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e00343f0 con 0x7fd9e80b0900 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9dffff640 10 monclient: dump: 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:37.374 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9ef6a9640 5 monclient: authenticate success, global_id 5580 2024-01-20T11:21:37.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9ef6a9640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:37.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.372+0000 7fd9ef6a9640 4 set_mon_vals no callback set 2024-01-20T11:21:37.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 10 monclient: shutdown 2024-01-20T11:21:37.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:37.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/2760276380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 msgr2=0x7fd9e8145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 1 --2- 172.21.15.110:0/2760276380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 0x7fd9e8145e80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd9e80b6630 tx=0x7fd9e002bee0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/2760276380 shutdown_connections 2024-01-20T11:21:37.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 1 --2- 172.21.15.110:0/2760276380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 0x7fd9e8145e80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 1 --2- 172.21.15.110:0/2760276380 >> v2:172.21.15.134:3300/0 conn(0x7fd9e806bb00 0x7fd9e80b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/2760276380 >> 172.21.15.110:0/2760276380 conn(0x7fd9e8066d60 msgr2=0x7fd9e80b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/2760276380 shutdown_connections 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/2760276380 wait complete. 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 10 monclient: build_initial_monmap 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.373+0000 7fd9ef6a9640 10 monclient: monmap: 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:37.380 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 librados: starting msgr at 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 librados: starting objecter 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 Processor -- start 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 -- start start 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 librados: setting wanted keys 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 librados: calling monclient init 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 10 monclient: init 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9e806bb00 0x7fd9e80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 10 monclient(hunting): picked mon.noname-c con 0x7fd9e806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:37.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e80af790 0x7fd9e811dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 10 monclient(hunting): picked mon.noname-b con 0x7fd9e80af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 0x7fd9e81da600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 10 monclient(hunting): picked mon.noname-a con 0x7fd9e80b0900 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.374+0000 7fd9ef6a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e811ee50 con 0x7fd9e80b0900 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ef6a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ef6a9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e811d720 con 0x7fd9e80af790 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ef6a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ef6a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e8142370 con 0x7fd9e806bb00 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ef6a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ef6a9640 10 monclient(hunting): authenticate will time out at 1740.725198s 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9edc46640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 0x7fd9e81da600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ed445640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9e806bb00 0x7fd9e80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ecc44640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e80af790 0x7fd9e811dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9edc46640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 0x7fd9e81da600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47370/0 (socket says 172.21.15.110:47370) 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9edc46640 1 -- 172.21.15.110:0/264484396 learned_addr learned my addr 172.21.15.110:0/264484396 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9edc46640 10 monclient(hunting): get_auth_request con 0x7fd9e80b0900 auth_method 0 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9edc46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9edc46640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9edc46640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ed445640 10 monclient(hunting): get_auth_request con 0x7fd9e806bb00 auth_method 0 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ed445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ed445640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.375+0000 7fd9ed445640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ecc44640 10 monclient(hunting): get_auth_request con 0x7fd9e80af790 auth_method 0 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ecc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ecc44640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ecc44640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9edc46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9edc46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9edc46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ed445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ed445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ed445640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ecc44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ecc44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.376+0000 7fd9ecc44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 10 monclient(hunting): handle_auth_done global_id 5583 payload 824 2024-01-20T11:21:37.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 1 -- 172.21.15.110:0/264484396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9e806bb00 msgr2=0x7fd9e80b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9e806bb00 0x7fd9e80b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 1 -- 172.21.15.110:0/264484396 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e80af790 msgr2=0x7fd9e811dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e80af790 0x7fd9e811dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 10 monclient: _finish_hunting 0 2024-01-20T11:21:37.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 1 monclient: found mon.noname-a 2024-01-20T11:21:37.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:37.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.379+0000 7fd9edc46640 1 -- 172.21.15.110:0/264484396 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e0009b30 con 0x7fd9e80b0900 2024-01-20T11:21:37.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.381+0000 7fd9edc46640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 0x7fd9e81da600 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0036040 tx=0x7fd9e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.381+0000 7fd9ddffb640 10 client.?.objecter ms_handle_connect 0x7fd9e80b0900 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.381+0000 7fd9ddffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 1 -- 172.21.15.110:0/264484396 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e003e450 con 0x7fd9e80b0900 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 10 monclient: dump: 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:37.387 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 10 monclient: _finish_auth 0 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 10 monclient: _check_auth_tickets 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 1 -- 172.21.15.110:0/264484396 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e003e5b0 con 0x7fd9e80b0900 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 1 -- 172.21.15.110:0/264484396 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e003e960 con 0x7fd9e80b0900 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ddffb640 10 monclient: dump: 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.388 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ef6a9640 5 monclient: authenticate success, global_id 5583 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ef6a9640 10 monclient: _renew_subs 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ef6a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e813c650 con 0x7fd9e80b0900 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9df7fe640 4 set_mon_vals no callback set 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ef6a9640 10 client.5583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ef6a9640 10 monclient: _renew_subs 2024-01-20T11:21:37.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ef6a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e81dad20 con 0x7fd9e80b0900 2024-01-20T11:21:37.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.385+0000 7fd9ef6a9640 1 librados: init done 2024-01-20T11:21:37.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.386+0000 7fd9ef6a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:37.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.386+0000 7fd9ef6a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:37.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.386+0000 7fd9ef6a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.386+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e811d8e0 con 0x7fd9e80b0900 2024-01-20T11:21:37.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.405+0000 7fd9ddffb640 1 -- 172.21.15.110:0/264484396 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd9e002cd50 con 0x7fd9e80b0900 2024-01-20T11:21:37.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.405+0000 7fd9ddffb640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9cc06c8b0 0x7fd9cc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.406+0000 7fd9ed445640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9cc06c8b0 0x7fd9cc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.406+0000 7fd9ed445640 10 monclient: get_auth_request con 0x7fd9cc06c8b0 auth_method 0 2024-01-20T11:21:37.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.406+0000 7fd9ed445640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9cc06c8b0 0x7fd9cc06ed70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009b40 tx=0x7fd9e4009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:37.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.406+0000 7fd9ddffb640 10 client.5583.objecter ms_handle_connect 0x7fd9cc06c8b0 2024-01-20T11:21:37.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.640+0000 7fd9ddffb640 1 -- 172.21.15.110:0/264484396 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd9e007b280 con 0x7fd9e80b0900 2024-01-20T11:21:37.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.640+0000 7fd9ddffb640 10 client.5583.objecter ms_dispatch 0x7fd9e8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:37.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.640+0000 7fd9ddffb640 3 client.5583.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:37.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.640+0000 7fd9ddffb640 3 client.5583.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:37.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.641+0000 7fd9ddffb640 20 client.5583.objecter dump_active .. 0 homeless 2024-01-20T11:21:37.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.641+0000 7fd9ddffb640 1 -- 172.21.15.110:0/264484396 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd9e00775c0 con 0x7fd9e80b0900 2024-01-20T11:21:37.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.641+0000 7fd9ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:37.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.641+0000 7fd9ddffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:37.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.823+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9e806c910 con 0x7fd9cc06c8b0 2024-01-20T11:21:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.826+0000 7fd9ddffb640 1 -- 172.21.15.110:0/264484396 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64099 (secure 0 0 0) 0x7fd9e4011a60 con 0x7fd9cc06c8b0 2024-01-20T11:21:37.826 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:37.827 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:37.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 10 librados: watch_flush enter 2024-01-20T11:21:37.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 10 librados: watch_flush exit 2024-01-20T11:21:37.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 20 client.5583.objecter shutdown clearing up homeless session... 2024-01-20T11:21:37.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 10 client.5583.objecter successfully canceled tick 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9cc06c8b0 msgr2=0x7fd9cc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9cc06c8b0 0x7fd9cc06ed70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009b40 tx=0x7fd9e4009190 comp rx=0 tx=0).stop 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 10 monclient: shutdown 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 msgr2=0x7fd9e81da600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 0x7fd9e81da600 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0036040 tx=0x7fd9e0004a40 comp rx=0 tx=0).stop 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 shutdown_connections 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd9cc06c8b0 0x7fd9cc06ed70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9e80b0900 0x7fd9e81da600 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e80af790 0x7fd9e811dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 --2- 172.21.15.110:0/264484396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9e806bb00 0x7fd9e80b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 >> 172.21.15.110:0/264484396 conn(0x7fd9e8066d60 msgr2=0x7fd9e8142970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 shutdown_connections 2024-01-20T11:21:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 -- 172.21.15.110:0/264484396 wait complete. 2024-01-20T11:21:37.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.828+0000 7fd9ef6a9640 1 librados: shutdown 2024-01-20T11:21:37.839 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-01-20T11:21:36.785983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002676"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":8.1150000000000002},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":12.589}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":9.1219999999999999},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":11.061999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":10.407999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":18.013000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":8.8499999999999996},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":17.672000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":9.7189999999999994},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":16.481000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":19.302},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":13.259}]}]},{"osd":7,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":9.1869999999999994},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":6.391}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.3460000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.722}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":10.59},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.6769999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":9.9629999999999992},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":15.114000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":16.260000000000002},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.340999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":12.997},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.105}]}]},{"osd":3,"up_from":21,"seq":90194313297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":6.9720000000000004,"5min":9.7989999999999995,"15min":14.882999999999999},"min":{"1min":4.3200000000000003,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":9.4410000000000007,"5min":155.27799999999999,"15min":155.27799999999999},"last":5.617},{"interface":"front","average":{"1min":7.6989999999999998,"5min":9.1959999999999997,"15min":13.254},"min":{"1min":5.3220000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.077999999999999,"5min":144.16499999999999,"15min":144.16499999999999},"last":8.484}]},{"osd":1,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":9.7270000000000003,"5min":11.372999999999999,"15min":14.872999999999999},"min":{"1min":6.6059999999999999,"5min":6.6059999999999999,"15min":6.6059999999999999},"max":{"1min":14.398,"5min":149.55000000000001,"15min":149.55000000000001},"last":12.932},{"interface":"front","average":{"1min":11.555999999999999,"5min":13.356999999999999,"15min":18.126999999999999},"min":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.869,"5min":155.91200000000001,"15min":155.91200000000001},"last":12.029}]},{"osd":2,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.1,"5min":12.276999999999999,"15min":16.888000000000002},"min":{"1min":7.4710000000000001,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":14.903,"5min":145.553,"15min":145.553},"last":10.278},{"interface":"front","average":{"1min":11.959,"5min":13.079000000000001,"15min":17.042999999999999},"min":{"1min":8.5630000000000006,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.100999999999999,"5min":147.66300000000001,"15min":147.66300000000001},"last":7.0119999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":12.919,"5min":16.076000000000001,"15min":23.536000000000001},"min":{"1min":8.7530000000000001,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.998999999999999,"5min":284.00700000000001,"15min":284.00700000000001},"last":14.738},{"interface":"front","average":{"1min":12.92,"5min":15.981,"15min":24.140999999999998},"min":{"1min":9.1479999999999997,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":16.048999999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":7.8630000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":10.984,"5min":13.311999999999999,"15min":18.872},"min":{"1min":7.117,"5min":6.2000000000000002,"15min":6.2000000000000002},"max":{"1min":15.039,"5min":146.744,"15min":146.744},"last":9.1899999999999995},{"interface":"front","average":{"1min":13.381,"5min":15.333,"15min":19.829000000000001},"min":{"1min":10.106999999999999,"5min":7.8099999999999996,"15min":7.8099999999999996},"max":{"1min":17.954999999999998,"5min":150.542,"15min":150.542},"last":14.000999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:20:32 2024","interfaces":[{"interface":"back","average":{"1min":15.587,"5min":17.282,"15min":21.867999999999999},"min":{"1min":10.516,"5min":10.516,"15min":10.516},"max":{"1min":19.460999999999999,"5min":159.179,"15min":159.179},"last":15.488},{"interface":"front","average":{"1min":14.334,"5min":16.155999999999999,"15min":21.672000000000001},"min":{"1min":11.002000000000001,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17,"5min":157.35599999999999,"15min":157.35599999999999},"last":11.186}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444111,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.1660000000000004},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":4.2839999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":7.6150000000000002},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":11.677}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.641999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":11.106999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:20:34 2024","interfaces":[{"interface":"back","average":{"1min":13.766,"5min":14.617000000000001,"15min":16.887},"min":{"1min":11.247,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.253,"5min":51.000999999999998,"15min":51.000999999999998},"last":19.533000000000001},{"interface":"front","average":{"1min":12.039999999999999,"5min":12.757,"15min":14.153},"min":{"1min":8.8209999999999997,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.096,"5min":48.299999999999997,"15min":48.299999999999997},"last":21.629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":16.367000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":8.2149999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":9.8469999999999995},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":17.047000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444112,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.024},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":5.8680000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":10.297000000000001},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":8.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":17.565999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":11.266}]},{"osd":4,"last update":"Sat Jan 20 11:20:33 2024","interfaces":[{"interface":"back","average":{"1min":12.551,"5min":12.776999999999999,"15min":13.788},"min":{"1min":6.8109999999999999,"5min":6.5800000000000001,"15min":6.5800000000000001},"max":{"1min":18.899999999999999,"5min":25.541,"15min":25.541},"last":13.276},{"interface":"front","average":{"1min":12.539999999999999,"5min":14.247999999999999,"15min":14.974},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":17.196000000000002,"5min":30.204999999999998,"15min":30.204999999999998},"last":12.047000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":9.0640000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":13.920999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":16.285},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.542999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444112,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":4.5890000000000004},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.2620000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":9.2119999999999997},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":10.185}]},{"osd":3,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":11.891,"5min":15.659000000000001,"15min":25.219000000000001},"min":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"max":{"1min":16.576000000000001,"5min":358.738,"15min":358.738},"last":7.4059999999999997},{"interface":"front","average":{"1min":10.909000000000001,"5min":11.364000000000001,"15min":12.624000000000001},"min":{"1min":6.609,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":17.945,"5min":21.004999999999999,"15min":21.004999999999999},"last":11.994}]},{"osd":4,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":13.098000000000001,"5min":16.684000000000001,"15min":27.236999999999998},"min":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"max":{"1min":16.015000000000001,"5min":372.02300000000002,"15min":372.02300000000002},"last":11.423},{"interface":"front","average":{"1min":10.012,"5min":14.058,"15min":24.456},"min":{"1min":6.7320000000000002,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":17.117000000000001,"5min":372.70800000000003,"15min":372.70800000000003},"last":13.313000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":13.936999999999999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":14.939}]},{"osd":7,"last update":"Sat Jan 20 11:20:36 2024","interfaces":[{"interface":"back","average":{"1min":15.222,"5min":15.502000000000001,"15min":16.855},"min":{"1min":12.554,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.507999999999999,"5min":27.385999999999999,"15min":27.385999999999999},"last":15.914},{"interface":"front","average":{"1min":11.324,"5min":11.946,"15min":12.291},"min":{"1min":7.7720000000000002,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":16.960000000000001,"5min":21.529,"15min":21.529},"last":10.829000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.6580000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.2469999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.0860000000000003},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":7.4009999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":9.9830000000000005},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.9770000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.583},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":11.821}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.205},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.095000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":14.952},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.404999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:37.840 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:37.840 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:37.840 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:37.840 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:37.841 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:37.841 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:37.841 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:37.841 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:37.841 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.941+0000 7fe0ea9d0640 10 monclient: get_monmap_and_config 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.941+0000 7fe0ea9d0640 10 monclient: build_initial_monmap 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.941+0000 7fe0ea9d0640 10 monclient: monmap: 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:37.941645+0000 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:37.941645+0000 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.941 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:37.942 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:37.942 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:37.942 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:37.942 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:37.942 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.941+0000 7fe0ea9d0640 1 Processor -- start 2024-01-20T11:21:37.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.941+0000 7fe0ea9d0640 1 -- start start 2024-01-20T11:21:37.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.941+0000 7fe0ea9d0640 10 monclient: init 2024-01-20T11:21:37.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:37.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-01-20T11:21:37.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0e40a96f0 0x7fe0e413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe0e40a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:37.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0e413fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe0e40a9ad0 0x7fe0e4142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 10 monclient(hunting): picked mon.noname-c con 0x7fe0e40a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e40b25c0 con 0x7fe0e413fa10 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e40aaea0 con 0x7fe0e40a96f0 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e40ab0a0 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.942+0000 7fe0ea9d0640 10 monclient(hunting): authenticate will time out at 1741.292661s 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0e40a96f0 0x7fe0e413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e8f6d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe0e40a9ad0 0x7fe0e4142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0e40a96f0 0x7fe0e413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47380/0 (socket says 172.21.15.110:47380) 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 1 -- 172.21.15.110:0/3032174619 learned_addr learned my addr 172.21.15.110:0/3032174619 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 10 monclient(hunting): get_auth_request con 0x7fe0e40a96f0 auth_method 0 2024-01-20T11:21:37.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e8f6d640 10 monclient(hunting): get_auth_request con 0x7fe0e40a9ad0 auth_method 0 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e8f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e8f6d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e8f6d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.943+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.944+0000 7fe0e8f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.944+0000 7fe0e8f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.944+0000 7fe0e8f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.945+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_done global_id 5586 payload 274 2024-01-20T11:21:37.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.945+0000 7fe0e3fff640 1 -- 172.21.15.110:0/3032174619 >> v2:172.21.15.134:3300/0 conn(0x7fe0e40a9ad0 msgr2=0x7fe0e4142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:37.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.945+0000 7fe0e3fff640 1 --2- 172.21.15.110:0/3032174619 >> v2:172.21.15.134:3300/0 conn(0x7fe0e40a9ad0 0x7fe0e4142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.945+0000 7fe0e3fff640 1 -- 172.21.15.110:0/3032174619 >> v1:172.21.15.134:6789/0 conn(0x7fe0e413fa10 legacy=0x7fe0e41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.945+0000 7fe0e3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:37.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.945+0000 7fe0e3fff640 1 monclient: found mon.noname-a 2024-01-20T11:21:37.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.945+0000 7fe0e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:37.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.945+0000 7fe0e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.945+0000 7fe0e3fff640 1 -- 172.21.15.110:0/3032174619 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e40b3fb0 con 0x7fe0e40a96f0 2024-01-20T11:21:37.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.947+0000 7fe0e3fff640 1 --2- 172.21.15.110:0/3032174619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a96f0 0x7fe0e413f4d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4074d70 tx=0x7fe0d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90b712a3839eeba server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 1 -- 172.21.15.110:0/3032174619 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4031870 con 0x7fe0e40a96f0 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 10 monclient: dump: 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:37.951 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 1 -- 172.21.15.110:0/3032174619 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d40319d0 con 0x7fe0e40a96f0 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0ea9d0640 5 monclient: authenticate success, global_id 5586 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0ea9d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0ea9d0640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0ea9d0640 4 set_mon_vals no callback set 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 1 -- 172.21.15.110:0/3032174619 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4031d60 con 0x7fe0e40a96f0 2024-01-20T11:21:37.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0e2ffd640 10 monclient: dump: 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:37.953 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0ea9d0640 10 monclient: shutdown 2024-01-20T11:21:37.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0ea9d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:37.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3032174619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a96f0 msgr2=0x7fe0e413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.951+0000 7fe0ea9d0640 1 --2- 172.21.15.110:0/3032174619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a96f0 0x7fe0e413f4d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4074d70 tx=0x7fe0d402b910 comp rx=0 tx=0).stop 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3032174619 shutdown_connections 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 1 --2- 172.21.15.110:0/3032174619 >> v2:172.21.15.134:3300/0 conn(0x7fe0e40a9ad0 0x7fe0e4142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 1 --2- 172.21.15.110:0/3032174619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a96f0 0x7fe0e413f4d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3032174619 >> 172.21.15.110:0/3032174619 conn(0x7fe0e40b7bc0 msgr2=0x7fe0e40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3032174619 shutdown_connections 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3032174619 wait complete. 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 10 monclient: build_initial_monmap 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 10 monclient: monmap: 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:37.957 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 1 librados: starting msgr at 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.952+0000 7fe0ea9d0640 1 librados: starting objecter 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 Processor -- start 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 -- start start 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 librados: setting wanted keys 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 librados: calling monclient init 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient: init 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:37.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e40a96f0 0x7fe0e413acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient(hunting): picked mon.noname-b con 0x7fe0e40a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a9ad0 0x7fe0e4138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe0e40a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0e413b2f0 0x7fe0e4138740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient(hunting): picked mon.noname-c con 0x7fe0e413b2f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40abf80 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4138c80 con 0x7fe0e40a96f0 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e413b860 con 0x7fe0e413b2f0 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:37.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.953+0000 7fe0ea9d0640 10 monclient(hunting): authenticate will time out at 1741.304036s 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a9ad0 0x7fe0e4138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e3fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e40a96f0 0x7fe0e413acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e8f6d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0e413b2f0 0x7fe0e4138740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a9ad0 0x7fe0e4138200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47394/0 (socket says 172.21.15.110:47394) 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e37fe640 1 -- 172.21.15.110:0/3547342057 learned_addr learned my addr 172.21.15.110:0/3547342057 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e37fe640 10 monclient(hunting): get_auth_request con 0x7fe0e40a9ad0 auth_method 0 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e3fff640 10 monclient(hunting): get_auth_request con 0x7fe0e40a96f0 auth_method 0 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e8f6d640 10 monclient(hunting): get_auth_request con 0x7fe0e413b2f0 auth_method 0 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e8f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e8f6d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.954+0000 7fe0e8f6d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.955+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.955+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.955+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.955+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.955+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.955+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.955+0000 7fe0e8f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.955+0000 7fe0e8f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:37.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.955+0000 7fe0e8f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:37.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_done global_id 5589 payload 824 2024-01-20T11:21:37.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 1 -- 172.21.15.110:0/3547342057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0e413b2f0 msgr2=0x7fe0e4138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0e413b2f0 0x7fe0e4138740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 1 -- 172.21.15.110:0/3547342057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e40a96f0 msgr2=0x7fe0e413acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:37.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e40a96f0 0x7fe0e413acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:37.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:21:37.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 1 monclient: found mon.noname-a 2024-01-20T11:21:37.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:37.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.958+0000 7fe0e37fe640 1 -- 172.21.15.110:0/3547342057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d40094d0 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.960+0000 7fe0e37fe640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a9ad0 0x7fe0e4138200 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800c3c0 tx=0x7fe0d800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:37.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.960+0000 7fe0e0ff9640 10 client.?.objecter ms_handle_connect 0x7fe0e40a9ad0 2024-01-20T11:21:37.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.960+0000 7fe0e0ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:37.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 1 -- 172.21.15.110:0/3547342057 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d80052d0 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:37.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:37.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 10 monclient: dump: 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 10 monclient: _finish_auth 0 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 1 -- 172.21.15.110:0/3547342057 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d8005430 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 1 -- 172.21.15.110:0/3547342057 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8007240 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0ea9d0640 5 monclient: authenticate success, global_id 5589 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0ea9d0640 10 monclient: _renew_subs 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0ea9d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e40ac160 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e0ff9640 10 monclient: dump: 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:37.966 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:37.967 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:37.967 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:37.967 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:37.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0e27fc640 4 set_mon_vals no callback set 2024-01-20T11:21:37.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0ea9d0640 10 client.5589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:37.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0ea9d0640 10 monclient: _renew_subs 2024-01-20T11:21:37.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0ea9d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e40ac4f0 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.964+0000 7fe0ea9d0640 1 librados: init done 2024-01-20T11:21:37.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.965+0000 7fe0ea9d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:37.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.965+0000 7fe0ea9d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:37.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.965+0000 7fe0ea9d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:37.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.965+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e413b9f0 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.984+0000 7fe0e0ff9640 1 -- 172.21.15.110:0/3547342057 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe0d80073a0 con 0x7fe0e40a9ad0 2024-01-20T11:21:37.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.985+0000 7fe0e0ff9640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe0d006c860 0x7fe0d006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:37.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.985+0000 7fe0e3fff640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe0d006c860 0x7fe0d006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:37.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.985+0000 7fe0e3fff640 10 monclient: get_auth_request con 0x7fe0d006c860 auth_method 0 2024-01-20T11:21:37.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.986+0000 7fe0e3fff640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe0d006c860 0x7fe0d006ed20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4036000 tx=0x7fe0d403c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:37.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:37.986+0000 7fe0e0ff9640 10 client.5589.objecter ms_handle_connect 0x7fe0d006c860 2024-01-20T11:21:38.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.223+0000 7fe0e0ff9640 1 -- 172.21.15.110:0/3547342057 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe0d8007650 con 0x7fe0e40a9ad0 2024-01-20T11:21:38.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.223+0000 7fe0e0ff9640 10 client.5589.objecter ms_dispatch 0x7fe0e4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:38.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.223+0000 7fe0e0ff9640 3 client.5589.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:38.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.223+0000 7fe0e0ff9640 3 client.5589.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:38.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.223+0000 7fe0e0ff9640 20 client.5589.objecter dump_active .. 0 homeless 2024-01-20T11:21:38.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.223+0000 7fe0e0ff9640 1 -- 172.21.15.110:0/3547342057 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe0d8053e00 con 0x7fe0e40a9ad0 2024-01-20T11:21:38.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.224+0000 7fe0e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:38.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.224+0000 7fe0e0ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:38.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.472+0000 7fe0ea9d0640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:38.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.472+0000 7fe0ea9d0640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:38.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.472+0000 7fe0ea9d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:38.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.472+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fe0e40b4220 con 0x7fe0e40a9ad0 2024-01-20T11:21:38.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.478+0000 7fe0e0ff9640 1 -- 172.21.15.110:0/3547342057 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fe0d8020080 con 0x7fe0e40a9ad0 2024-01-20T11:21:38.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.478+0000 7fe0e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:38.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.478+0000 7fe0e0ff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:38.478 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:38.478 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":433,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:38.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.479+0000 7fe0ea9d0640 10 librados: watch_flush enter 2024-01-20T11:21:38.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.479+0000 7fe0ea9d0640 10 librados: watch_flush exit 2024-01-20T11:21:38.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.479+0000 7fe0ea9d0640 20 client.5589.objecter shutdown clearing up homeless session... 2024-01-20T11:21:38.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.479+0000 7fe0ea9d0640 10 client.5589.objecter successfully canceled tick 2024-01-20T11:21:38.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe0d006c860 msgr2=0x7fe0d006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:38.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe0d006c860 0x7fe0d006ed20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4036000 tx=0x7fe0d403c040 comp rx=0 tx=0).stop 2024-01-20T11:21:38.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 10 monclient: shutdown 2024-01-20T11:21:38.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a9ad0 msgr2=0x7fe0e4138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a9ad0 0x7fe0e4138200 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800c3c0 tx=0x7fe0d800c7c0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 shutdown_connections 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe0d006c860 0x7fe0d006ed20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0e413b2f0 0x7fe0e4138740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0e40a9ad0 0x7fe0e4138200 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 --2- 172.21.15.110:0/3547342057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e40a96f0 0x7fe0e413acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 >> 172.21.15.110:0/3547342057 conn(0x7fe0e40b7bc0 msgr2=0x7fe0e4137310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 shutdown_connections 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 -- 172.21.15.110:0/3547342057 wait complete. 2024-01-20T11:21:38.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.480+0000 7fe0ea9d0640 1 librados: shutdown 2024-01-20T11:21:38.490 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:38.490 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.590+0000 7f9224fdc640 10 monclient: get_monmap_and_config 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.590+0000 7f9224fdc640 10 monclient: build_initial_monmap 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.590+0000 7f9224fdc640 10 monclient: monmap: 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:38.590850+0000 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:38.590850+0000 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:38.590 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:38.591 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:38.591 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:38.591 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:38.591 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:38.591 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:38.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.590+0000 7f9224fdc640 1 Processor -- start 2024-01-20T11:21:38.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.590+0000 7f9224fdc640 1 -- start start 2024-01-20T11:21:38.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.590+0000 7f9224fdc640 10 monclient: init 2024-01-20T11:21:38.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:38.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-01-20T11:21:38.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92200b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:38.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92200b1ed0 0x7f92200b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:38.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 10 monclient(hunting): picked mon.noname-a con 0x7f92200b1ed0 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:38.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 10 monclient(hunting): picked mon.noname-b con 0x7f9220145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9220146370 con 0x7f92200b4d10 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92200b50f0 con 0x7f92200b1ed0 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92200aade0 con 0x7f9220145980 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.591+0000 7f9224fdc640 10 monclient(hunting): authenticate will time out at 1741.941776s 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921dd9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92200b1ed0 0x7f92200b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921dd9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92200b1ed0 0x7f92200b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47396/0 (socket says 172.21.15.110:47396) 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921dd9b640 1 -- 172.21.15.110:0/3476833490 learned_addr learned my addr 172.21.15.110:0/3476833490 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921dd9b640 10 monclient(hunting): get_auth_request con 0x7f92200b1ed0 auth_method 0 2024-01-20T11:21:38.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921ed9d640 10 monclient(hunting): get_auth_request con 0x7f9220145980 auth_method 0 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.592+0000 7f921ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.593+0000 7f921ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.593+0000 7f921ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.593+0000 7f921ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.593+0000 7f921dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.593+0000 7f921dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:38.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.593+0000 7f921dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.594+0000 7f921ed9d640 10 monclient(hunting): handle_auth_done global_id 5312 payload 274 2024-01-20T11:21:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.594+0000 7f921ed9d640 1 -- 172.21.15.110:0/3476833490 >> v2:172.21.15.110:3300/0 conn(0x7f92200b1ed0 msgr2=0x7f92200b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.594+0000 7f921ed9d640 1 --2- 172.21.15.110:0/3476833490 >> v2:172.21.15.110:3300/0 conn(0x7f92200b1ed0 0x7f92200b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.594+0000 7f921ed9d640 1 -- 172.21.15.110:0/3476833490 >> v1:172.21.15.110:6789/0 conn(0x7f92200b4d10 legacy=0x7f92200b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.594+0000 7f921ed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:21:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.594+0000 7f921ed9d640 1 monclient: found mon.noname-b 2024-01-20T11:21:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.594+0000 7f921ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.594+0000 7f921ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.594+0000 7f921ed9d640 1 -- 172.21.15.110:0/3476833490 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220145460 con 0x7f9220145980 2024-01-20T11:21:38.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.596+0000 7f921ed9d640 1 --2- 172.21.15.110:0/3476833490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220145e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f92100097a0 tx=0x7f921002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20312052f1be33aa server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 1 -- 172.21.15.110:0/3476833490 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921002e070 con 0x7f9220145980 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 10 monclient: dump: 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:38.600 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 10 monclient: _finish_auth 0 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 10 monclient: _check_auth_tickets 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 1 -- 172.21.15.110:0/3476833490 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92100315a0 con 0x7f9220145980 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f9224fdc640 5 monclient: authenticate success, global_id 5312 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f9224fdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 1 -- 172.21.15.110:0/3476833490 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210031930 con 0x7f9220145980 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f9224fdc640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f9224fdc640 4 set_mon_vals no callback set 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:38.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f921d59a640 10 monclient: dump: 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:38.602 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:38.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f9224fdc640 10 monclient: shutdown 2024-01-20T11:21:38.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f9224fdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f9224fdc640 1 -- 172.21.15.110:0/3476833490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 msgr2=0x7f9220145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.600+0000 7f9224fdc640 1 --2- 172.21.15.110:0/3476833490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220145e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f92100097a0 tx=0x7f921002b840 comp rx=0 tx=0).stop 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.601+0000 7f9224fdc640 1 -- 172.21.15.110:0/3476833490 shutdown_connections 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.601+0000 7f9224fdc640 1 --2- 172.21.15.110:0/3476833490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220145e30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.601+0000 7f9224fdc640 1 --2- 172.21.15.110:0/3476833490 >> v2:172.21.15.110:3300/0 conn(0x7f92200b1ed0 0x7f92200b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.601+0000 7f9224fdc640 1 -- 172.21.15.110:0/3476833490 >> 172.21.15.110:0/3476833490 conn(0x7f9220066590 msgr2=0x7f92200669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.601+0000 7f9224fdc640 1 -- 172.21.15.110:0/3476833490 shutdown_connections 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.601+0000 7f9224fdc640 1 -- 172.21.15.110:0/3476833490 wait complete. 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.601+0000 7f9224fdc640 10 monclient: build_initial_monmap 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.601+0000 7f9224fdc640 10 monclient: monmap: 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 librados: starting msgr at 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 librados: starting objecter 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 Processor -- start 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 -- start start 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 librados: setting wanted keys 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 librados: calling monclient init 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient: init 2024-01-20T11:21:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92200b1ed0 0x7f92201daaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient(hunting): picked mon.noname-b con 0x7f92200b1ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92200b4d10 0x7f922013fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient(hunting): picked mon.noname-a con 0x7f92200b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient(hunting): picked mon.noname-c con 0x7f9220145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92201db210 con 0x7f92200b4d10 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9220140780 con 0x7f92200b1ed0 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922013f110 con 0x7f9220145980 2024-01-20T11:21:38.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.602+0000 7f9224fdc640 10 monclient(hunting): authenticate will time out at 1741.953093s 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921dd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92200b4d10 0x7f922013fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220140240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57714/0 (socket says 172.21.15.110:57714) 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921ed9d640 1 -- 172.21.15.110:0/205840618 learned_addr learned my addr 172.21.15.110:0/205840618 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921ed9d640 10 monclient(hunting): get_auth_request con 0x7f9220145980 auth_method 0 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921dd9b640 10 monclient(hunting): get_auth_request con 0x7f92200b4d10 auth_method 0 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.603+0000 7f921e59c640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92200b1ed0 0x7f92201daaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921e59c640 10 monclient(hunting): get_auth_request con 0x7f92200b1ed0 auth_method 0 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.604+0000 7f921e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:38.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 10 monclient(hunting): handle_auth_done global_id 5315 payload 824 2024-01-20T11:21:38.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 1 -- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92200b1ed0 msgr2=0x7f92201daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:38.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92200b1ed0 0x7f92201daaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 1 -- 172.21.15.110:0/205840618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92200b4d10 msgr2=0x7f922013fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92200b4d10 0x7f922013fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 1 monclient: found mon.noname-c 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921ed9d640 1 -- 172.21.15.110:0/205840618 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9210009400 con 0x7f9220145980 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.607+0000 7f921dd9b640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92200b4d10 0x7f922013fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.609+0000 7f921ed9d640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220140240 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f92100097a0 tx=0x7f92100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.609+0000 7f920effd640 10 client.?.objecter ms_handle_connect 0x7f9220145980 2024-01-20T11:21:38.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.609+0000 7f920effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 1 -- 172.21.15.110:0/205840618 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210032840 con 0x7f9220145980 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 10 monclient: dump: 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:38.614 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 10 monclient: _finish_auth 0 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 1 -- 172.21.15.110:0/205840618 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92100329a0 con 0x7f9220145980 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 1 -- 172.21.15.110:0/205840618 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210032d50 con 0x7f9220145980 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f9224fdc640 5 monclient: authenticate success, global_id 5315 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f9224fdc640 10 monclient: _renew_subs 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f9224fdc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922013f3f0 con 0x7f9220145980 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f921cd99640 4 set_mon_vals no callback set 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f920effd640 10 monclient: dump: 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:38.615 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f9224fdc640 10 client.5315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f9224fdc640 10 monclient: _renew_subs 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f9224fdc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92201db6a0 con 0x7f9220145980 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.613+0000 7f9224fdc640 1 librados: init done 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.614+0000 7f9224fdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:38.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.614+0000 7f9224fdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.614+0000 7f9224fdc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.614+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92201db3c0 con 0x7f9220145980 2024-01-20T11:21:38.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.631+0000 7f920effd640 1 -- 172.21.15.110:0/205840618 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f921004e430 con 0x7f9220145980 2024-01-20T11:21:38.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.632+0000 7f920effd640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f920006c8b0 0x7f920006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:38.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.632+0000 7f921e59c640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f920006c8b0 0x7f920006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:38.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.632+0000 7f921e59c640 10 monclient: get_auth_request con 0x7f920006c8b0 auth_method 0 2024-01-20T11:21:38.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.633+0000 7f920effd640 10 client.5315.objecter ms_handle_connect 0x7f920006c8b0 2024-01-20T11:21:38.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.633+0000 7f921e59c640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f920006c8b0 0x7f920006ed70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9218001780 tx=0x7f9218009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.635+0000 7f920effd640 1 -- 172.21.15.110:0/205840618 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9210079cc0 con 0x7f9220145980 2024-01-20T11:21:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.635+0000 7f920effd640 10 client.5315.objecter ms_dispatch 0x7f9220000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.635+0000 7f920effd640 3 client.5315.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.635+0000 7f920effd640 3 client.5315.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:38.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.635+0000 7f920effd640 20 client.5315.objecter dump_active .. 0 homeless 2024-01-20T11:21:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.864+0000 7f920effd640 1 -- 172.21.15.110:0/205840618 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9210076000 con 0x7f9220145980 2024-01-20T11:21:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.864+0000 7f920effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:38.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:38.864+0000 7f920effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:39.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.058+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f922014c7f0 con 0x7f920006c8b0 2024-01-20T11:21:39.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.061+0000 7f920effd640 1 -- 172.21.15.110:0/205840618 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63983 (secure 0 0 0) 0x7f9218001420 con 0x7f920006c8b0 2024-01-20T11:21:39.061 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:39.062 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:39.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 10 librados: watch_flush enter 2024-01-20T11:21:39.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 10 librados: watch_flush exit 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 20 client.5315.objecter shutdown clearing up homeless session... 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 10 client.5315.objecter successfully canceled tick 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f920006c8b0 msgr2=0x7f920006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f920006c8b0 0x7f920006ed70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9218001780 tx=0x7f9218009040 comp rx=0 tx=0).stop 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 10 monclient: shutdown 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 msgr2=0x7f9220140240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220140240 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f92100097a0 tx=0x7f92100046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 shutdown_connections 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f920006c8b0 0x7f920006ed70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9220145980 0x7f9220140240 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92200b4d10 0x7f922013fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:39.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 --2- 172.21.15.110:0/205840618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92200b1ed0 0x7f92201daaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:39.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 >> 172.21.15.110:0/205840618 conn(0x7f9220066590 msgr2=0x7f92200b83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:39.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 shutdown_connections 2024-01-20T11:21:39.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 -- 172.21.15.110:0/205840618 wait complete. 2024-01-20T11:21:39.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:39.063+0000 7f9224fdc640 1 librados: shutdown 2024-01-20T11:21:39.074 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-01-20T11:21:38.786414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002709"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":8.6910000000000007},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":4.9349999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":12.468},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":9.4529999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":13.183999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":10.884}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":11.613},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":8.3230000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":10.119},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":15.416}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":13.878},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.622999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":9.1869999999999994},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":6.391}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.3460000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.722}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":10.59},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.6769999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":9.9629999999999992},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":15.114000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":16.260000000000002},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.340999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":12.997},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.105}]}]},{"osd":3,"up_from":21,"seq":90194313298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":13.314},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":12.031000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1359999999999992},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":9.6050000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":8.0660000000000007},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":11.403}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":15.459},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.702999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":10.305999999999999},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":16.289999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":17.117999999999999},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":13.925000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444112,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":6.3639999999999999},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.6280000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":7.3849999999999998},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":16.759}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":17.931999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.577}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":20.393000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.433}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":15.122},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.499000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.0239999999999991},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":21.297999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444113,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.1980000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.3220000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":12.391999999999999},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":10.23}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":8.2710000000000008},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":13.090999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.4979999999999993},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.206},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":10.972}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":16.699999999999999},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.365}]}]},{"osd":0,"up_from":17,"seq":73014444113,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":6.2869999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":10.375999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.4100000000000001},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.2050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.968},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":8.6430000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.48},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":13.815}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":9.8350000000000009},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":14.946999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":15.773999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":16.600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.6580000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.2469999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.0860000000000003},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":7.4009999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":9.9830000000000005},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.9770000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.583},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":11.821}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.205},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.095000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":14.952},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.404999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:39.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~161s 2024-01-20T11:21:42.077 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:42.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.176+0000 7fb582df5640 10 monclient: get_monmap_and_config 2024-01-20T11:21:42.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.176+0000 7fb582df5640 10 monclient: build_initial_monmap 2024-01-20T11:21:42.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.176+0000 7fb582df5640 10 monclient: monmap: 2024-01-20T11:21:42.176 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:42.176 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.176 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:42.177018+0000 2024-01-20T11:21:42.176 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:42.177018+0000 2024-01-20T11:21:42.177 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:42.177 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.177 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:42.177 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:42.177 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:42.177 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:42.177 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:42.177 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 1 Processor -- start 2024-01-20T11:21:42.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 1 -- start start 2024-01-20T11:21:42.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient: init 2024-01-20T11:21:42.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:42.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-01-20T11:21:42.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb57c06f920 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:42.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb57c06fd00 0x7fb57c143950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient(hunting): picked mon.noname-a con 0x7fb57c06fd00 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb57c143e90 0x7fb57c14a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient(hunting): picked mon.noname-c con 0x7fb57c143e90 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c144290 con 0x7fb57c06f920 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb57c0715b0 con 0x7fb57c06fd00 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb57c071890 con 0x7fb57c143e90 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.177+0000 7fb582df5640 10 monclient(hunting): authenticate will time out at 1745.527971s 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.178+0000 7fb57bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb57c06fd00 0x7fb57c143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.178+0000 7fb581392640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb57c143e90 0x7fb57c14a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.178+0000 7fb57bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb57c06fd00 0x7fb57c143950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45194/0 (socket says 172.21.15.110:45194) 2024-01-20T11:21:42.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.178+0000 7fb57bfff640 1 -- 172.21.15.110:0/639194527 learned_addr learned my addr 172.21.15.110:0/639194527 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.178+0000 7fb57bfff640 10 monclient(hunting): get_auth_request con 0x7fb57c06fd00 auth_method 0 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.178+0000 7fb57bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.178+0000 7fb57bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.178+0000 7fb57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.179+0000 7fb57bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.179+0000 7fb57bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.179+0000 7fb57bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.179+0000 7fb581392640 10 monclient(hunting): get_auth_request con 0x7fb57c143e90 auth_method 0 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.179+0000 7fb581392640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.179+0000 7fb581392640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.179+0000 7fb581392640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.180+0000 7fb581392640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.180+0000 7fb581392640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.180+0000 7fb581392640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.181+0000 7fb57bfff640 10 monclient(hunting): handle_auth_done global_id 5601 payload 274 2024-01-20T11:21:42.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.181+0000 7fb57bfff640 1 -- 172.21.15.110:0/639194527 >> v2:172.21.15.134:3300/0 conn(0x7fb57c143e90 msgr2=0x7fb57c14a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:42.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.181+0000 7fb57bfff640 1 --2- 172.21.15.110:0/639194527 >> v2:172.21.15.134:3300/0 conn(0x7fb57c143e90 0x7fb57c14a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.181+0000 7fb57bfff640 1 -- 172.21.15.110:0/639194527 >> v1:172.21.15.134:6789/0 conn(0x7fb57c06f920 legacy=0x7fb57c13d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.181+0000 7fb57bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:42.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.181+0000 7fb57bfff640 1 monclient: found mon.noname-a 2024-01-20T11:21:42.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.181+0000 7fb57bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:42.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.181+0000 7fb57bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:42.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.181+0000 7fb57bfff640 1 -- 172.21.15.110:0/639194527 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c071b70 con 0x7fb57c06fd00 2024-01-20T11:21:42.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.182+0000 7fb57bfff640 1 --2- 172.21.15.110:0/639194527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06fd00 0x7fb57c143950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0b6760 tx=0x7fb56802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afe7d5727b57624 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:42.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 1 -- 172.21.15.110:0/639194527 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568004230 con 0x7fb57c06fd00 2024-01-20T11:21:42.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:42.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:42.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 10 monclient: dump: 2024-01-20T11:21:42.186 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:42.186 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.186 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.186 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.186 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 1 -- 172.21.15.110:0/639194527 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb568004390 con 0x7fb57c06fd00 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 1 -- 172.21.15.110:0/639194527 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568003810 con 0x7fb57c06fd00 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb582df5640 5 monclient: authenticate success, global_id 5601 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb582df5640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb582df5640 4 set_mon_vals no callback set 2024-01-20T11:21:42.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.186+0000 7fb57b7fe640 10 monclient: dump: 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:42.188 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 10 monclient: shutdown 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 1 -- 172.21.15.110:0/639194527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06fd00 msgr2=0x7fb57c143950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 1 --2- 172.21.15.110:0/639194527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06fd00 0x7fb57c143950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0b6760 tx=0x7fb56802b8f0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 1 -- 172.21.15.110:0/639194527 shutdown_connections 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 1 --2- 172.21.15.110:0/639194527 >> v2:172.21.15.134:3300/0 conn(0x7fb57c143e90 0x7fb57c14a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 1 --2- 172.21.15.110:0/639194527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06fd00 0x7fb57c143950 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 1 -- 172.21.15.110:0/639194527 >> 172.21.15.110:0/639194527 conn(0x7fb57c066d60 msgr2=0x7fb57c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 1 -- 172.21.15.110:0/639194527 shutdown_connections 2024-01-20T11:21:42.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.187+0000 7fb582df5640 1 -- 172.21.15.110:0/639194527 wait complete. 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.188+0000 7fb582df5640 10 monclient: build_initial_monmap 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.188+0000 7fb582df5640 10 monclient: monmap: 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.188+0000 7fb582df5640 1 librados: starting msgr at 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.188+0000 7fb582df5640 1 librados: starting objecter 2024-01-20T11:21:42.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.188+0000 7fb582df5640 1 Processor -- start 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.188+0000 7fb582df5640 1 -- start start 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.188+0000 7fb582df5640 1 librados: setting wanted keys 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.188+0000 7fb582df5640 1 librados: calling monclient init 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.188+0000 7fb582df5640 10 monclient: init 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06f920 0x7fb57c1cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient(hunting): picked mon.noname-a con 0x7fb57c06f920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c06fd00 0x7fb57c1da750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient(hunting): picked mon.noname-b con 0x7fb57c06fd00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb57c143e90 0x7fb57c1dac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient(hunting): picked mon.noname-c con 0x7fb57c143e90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0a9fa0 con 0x7fb57c06f920 2024-01-20T11:21:42.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c06cf40 con 0x7fb57c06fd00 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0aa1a0 con 0x7fb57c143e90 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb582df5640 10 monclient(hunting): authenticate will time out at 1745.539553s 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.189+0000 7fb580b91640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06f920 0x7fb57c1cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb581392640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb57c143e90 0x7fb57c1dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb57bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c06fd00 0x7fb57c1da750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb580b91640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06f920 0x7fb57c1cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45206/0 (socket says 172.21.15.110:45206) 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb580b91640 1 -- 172.21.15.110:0/1889361862 learned_addr learned my addr 172.21.15.110:0/1889361862 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb580b91640 10 monclient(hunting): get_auth_request con 0x7fb57c06f920 auth_method 0 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb580b91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb580b91640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb580b91640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb581392640 10 monclient(hunting): get_auth_request con 0x7fb57c143e90 auth_method 0 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb581392640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb581392640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb581392640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb57bfff640 10 monclient(hunting): get_auth_request con 0x7fb57c06fd00 auth_method 0 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb57bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb57bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb580b91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb580b91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.190+0000 7fb580b91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.191+0000 7fb581392640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.191+0000 7fb581392640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.191+0000 7fb581392640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.191+0000 7fb57bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.191+0000 7fb57bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.191+0000 7fb57bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.193+0000 7fb580b91640 10 monclient(hunting): handle_auth_done global_id 5604 payload 824 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.193+0000 7fb580b91640 1 -- 172.21.15.110:0/1889361862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb57c143e90 msgr2=0x7fb57c1dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.193+0000 7fb580b91640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb57c143e90 0x7fb57c1dac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.193+0000 7fb580b91640 1 -- 172.21.15.110:0/1889361862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c06fd00 msgr2=0x7fb57c1da750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.193+0000 7fb580b91640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c06fd00 0x7fb57c1da750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.193+0000 7fb580b91640 10 monclient: _finish_hunting 0 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.193+0000 7fb580b91640 1 monclient: found mon.noname-a 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.194+0000 7fb580b91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.194+0000 7fb580b91640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:42.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.194+0000 7fb580b91640 1 -- 172.21.15.110:0/1889361862 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568009460 con 0x7fb57c06f920 2024-01-20T11:21:42.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.194+0000 7fb581392640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb57c143e90 0x7fb57c1dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:42.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.196+0000 7fb580b91640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06f920 0x7fb57c1cc5c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb57000c3f0 tx=0x7fb57000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:42.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.196+0000 7fb5797fa640 10 client.?.objecter ms_handle_connect 0x7fb57c06f920 2024-01-20T11:21:42.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.196+0000 7fb5797fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 1 -- 172.21.15.110:0/1889361862 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700052d0 con 0x7fb57c06f920 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 10 monclient: dump: 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:42.200 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 10 monclient: _finish_auth 0 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 10 monclient: _check_auth_tickets 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 1 -- 172.21.15.110:0/1889361862 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb570005450 con 0x7fb57c06f920 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 1 -- 172.21.15.110:0/1889361862 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570007260 con 0x7fb57c06f920 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb5797fa640 10 monclient: dump: 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.201 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb582df5640 5 monclient: authenticate success, global_id 5604 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb582df5640 10 monclient: _renew_subs 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb582df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb57affd640 4 set_mon_vals no callback set 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.199+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c06cab0 con 0x7fb57c06f920 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.200+0000 7fb582df5640 10 client.5604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.200+0000 7fb582df5640 10 monclient: _renew_subs 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.200+0000 7fb582df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:42.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.200+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c06d5b0 con 0x7fb57c06f920 2024-01-20T11:21:42.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.200+0000 7fb582df5640 1 librados: init done 2024-01-20T11:21:42.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.201+0000 7fb582df5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:42.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.201+0000 7fb582df5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:42.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.201+0000 7fb582df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:42.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.201+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c075490 con 0x7fb57c06f920 2024-01-20T11:21:42.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.219+0000 7fb5797fa640 1 -- 172.21.15.110:0/1889361862 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb5700073c0 con 0x7fb57c06f920 2024-01-20T11:21:42.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.220+0000 7fb5797fa640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb56006c8b0 0x7fb56006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.220+0000 7fb57bfff640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb56006c8b0 0x7fb56006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.220+0000 7fb57bfff640 10 monclient: get_auth_request con 0x7fb56006c8b0 auth_method 0 2024-01-20T11:21:42.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.220+0000 7fb57bfff640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb56006c8b0 0x7fb56006ed70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb568035000 tx=0x7fb568008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:42.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.220+0000 7fb5797fa640 10 client.5604.objecter ms_handle_connect 0x7fb56006c8b0 2024-01-20T11:21:42.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.223+0000 7fb5797fa640 1 -- 172.21.15.110:0/1889361862 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb570057a50 con 0x7fb57c06f920 2024-01-20T11:21:42.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.223+0000 7fb5797fa640 10 client.5604.objecter ms_dispatch 0x7fb57c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:42.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.223+0000 7fb5797fa640 3 client.5604.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:42.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.223+0000 7fb5797fa640 3 client.5604.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:42.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.223+0000 7fb5797fa640 20 client.5604.objecter dump_active .. 0 homeless 2024-01-20T11:21:42.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.454+0000 7fb5797fa640 1 -- 172.21.15.110:0/1889361862 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb570053d90 con 0x7fb57c06f920 2024-01-20T11:21:42.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.454+0000 7fb5797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:42.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.454+0000 7fb5797fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:42.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.626+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb57c0b03c0 con 0x7fb56006c8b0 2024-01-20T11:21:42.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.629+0000 7fb5797fa640 1 -- 172.21.15.110:0/1889361862 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63983 (secure 0 0 0) 0x7fb568005990 con 0x7fb56006c8b0 2024-01-20T11:21:42.629 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:42.630 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:42.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 10 librados: watch_flush enter 2024-01-20T11:21:42.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 10 librados: watch_flush exit 2024-01-20T11:21:42.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 20 client.5604.objecter shutdown clearing up homeless session... 2024-01-20T11:21:42.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 10 client.5604.objecter successfully canceled tick 2024-01-20T11:21:42.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb56006c8b0 msgr2=0x7fb56006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb56006c8b0 0x7fb56006ed70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb568035000 tx=0x7fb568008030 comp rx=0 tx=0).stop 2024-01-20T11:21:42.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 10 monclient: shutdown 2024-01-20T11:21:42.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:42.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06f920 msgr2=0x7fb57c1cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06f920 0x7fb57c1cc5c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb57000c3f0 tx=0x7fb57000c8b0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 shutdown_connections 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb56006c8b0 0x7fb56006ed70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb57c143e90 0x7fb57c1dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c06fd00 0x7fb57c1da750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 --2- 172.21.15.110:0/1889361862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb57c06f920 0x7fb57c1cc5c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 >> 172.21.15.110:0/1889361862 conn(0x7fb57c066d60 msgr2=0x7fb57c13adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 shutdown_connections 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 -- 172.21.15.110:0/1889361862 wait complete. 2024-01-20T11:21:42.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.631+0000 7fb582df5640 1 librados: shutdown 2024-01-20T11:21:42.642 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-01-20T11:21:40.786822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002688"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":8.6910000000000007},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":4.9349999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":12.468},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":9.4529999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":13.183999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":10.884}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":11.613},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":8.3230000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":10.119},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":15.416}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":13.878},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.622999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":9.1869999999999994},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":6.391}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.3460000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.722}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":10.59},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.6769999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":9.9629999999999992},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":15.114000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":16.260000000000002},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.340999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":12.997},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.105}]}]},{"osd":3,"up_from":21,"seq":90194313298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":13.314},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":12.031000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1359999999999992},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":9.6050000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":8.0660000000000007},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":11.403}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":15.459},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.702999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":10.305999999999999},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":16.289999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":17.117999999999999},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":13.925000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444112,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":6.3639999999999999},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.6280000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":7.3849999999999998},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":16.759}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":17.931999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.577}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":20.393000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.433}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":15.122},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.499000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.0239999999999991},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":21.297999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444113,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.1980000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.3220000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":12.391999999999999},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":10.23}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":8.2710000000000008},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":13.090999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.4979999999999993},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.206},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":10.972}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":16.699999999999999},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.365}]}]},{"osd":0,"up_from":17,"seq":73014444113,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":6.2869999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":10.375999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.4100000000000001},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.2050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.968},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":8.6430000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.48},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":13.815}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":9.8350000000000009},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":14.946999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":15.773999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":16.600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.6580000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.2469999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.0860000000000003},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":7.4009999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":9.9830000000000005},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.9770000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.583},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":11.821}]},{"osd":4,"last update":"Sat Jan 20 11:20:35 2024","interfaces":[{"interface":"back","average":{"1min":13.397,"5min":14.324999999999999,"15min":16.420999999999999},"min":{"1min":9.2759999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":16.869,"5min":70.254999999999995,"15min":70.254999999999995},"last":11.205},{"interface":"front","average":{"1min":12.387,"5min":13.702,"15min":16.242000000000001},"min":{"1min":8.032,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":16.827000000000002,"5min":72.670000000000002,"15min":72.670000000000002},"last":14.095000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":14.952},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.404999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:42.644 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:42.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.744+0000 7f8176905640 10 monclient: get_monmap_and_config 2024-01-20T11:21:42.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.744+0000 7f8176905640 10 monclient: build_initial_monmap 2024-01-20T11:21:42.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.744+0000 7f8176905640 10 monclient: monmap: 2024-01-20T11:21:42.744 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:42.744 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:42.745130+0000 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:42.745130+0000 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:42.745 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 1 Processor -- start 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 1 -- start start 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient: init 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f817006a860 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f817013d800 0x7f8170143390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient(hunting): picked mon.noname-a con 0x7f817013d800 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81700b0620 0x7f81701438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient(hunting): picked mon.noname-b con 0x7f81700b0620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81700b10f0 con 0x7f817006a860 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817006ac40 con 0x7f817013d800 2024-01-20T11:21:42.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8170072150 con 0x7f81700b0620 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.745+0000 7f8176905640 10 monclient(hunting): authenticate will time out at 1746.096151s 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.746+0000 7f816f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f817013d800 0x7f8170143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.746+0000 7f8174ea2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81700b0620 0x7f81701438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.746+0000 7f8174ea2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81700b0620 0x7f81701438d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54776/0 (socket says 172.21.15.110:54776) 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.746+0000 7f8174ea2640 1 -- 172.21.15.110:0/1336697784 learned_addr learned my addr 172.21.15.110:0/1336697784 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.746+0000 7f8174ea2640 10 monclient(hunting): get_auth_request con 0x7f81700b0620 auth_method 0 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.746+0000 7f8174ea2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.746+0000 7f8174ea2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.746+0000 7f8174ea2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f816f7fe640 10 monclient(hunting): get_auth_request con 0x7f817013d800 auth_method 0 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f816f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f816f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f816f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f8174ea2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f8174ea2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f8174ea2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f816f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f816f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.747+0000 7f816f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.749+0000 7f8174ea2640 10 monclient(hunting): handle_auth_done global_id 5321 payload 274 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.749+0000 7f8174ea2640 1 -- 172.21.15.110:0/1336697784 >> v2:172.21.15.110:3300/0 conn(0x7f817013d800 msgr2=0x7f8170143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.749+0000 7f8174ea2640 1 --2- 172.21.15.110:0/1336697784 >> v2:172.21.15.110:3300/0 conn(0x7f817013d800 0x7f8170143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.749+0000 7f8174ea2640 1 -- 172.21.15.110:0/1336697784 >> v1:172.21.15.110:6789/0 conn(0x7f817006a860 legacy=0x7f817013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.749+0000 7f8174ea2640 10 monclient: _finish_hunting 0 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.749+0000 7f8174ea2640 1 monclient: found mon.noname-b 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.749+0000 7f8174ea2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.749+0000 7f8174ea2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.749+0000 7f8174ea2640 1 -- 172.21.15.110:0/1336697784 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8170071e70 con 0x7f81700b0620 2024-01-20T11:21:42.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.751+0000 7f8174ea2640 1 --2- 172.21.15.110:0/1336697784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81700b0620 0x7f81701438d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8168009e50 tx=0x7f816802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f110945faabb8b83 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:42.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 1 -- 172.21.15.110:0/1336697784 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816802cb00 con 0x7f81700b0620 2024-01-20T11:21:42.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:42.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:42.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 10 monclient: dump: 2024-01-20T11:21:42.755 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:42.755 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.755 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 10 monclient: _finish_auth 0 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 1 -- 172.21.15.110:0/1336697784 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816802cc60 con 0x7f81700b0620 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f8176905640 5 monclient: authenticate success, global_id 5321 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f8176905640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:42.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f8176905640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f8176905640 4 set_mon_vals no callback set 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 1 -- 172.21.15.110:0/1336697784 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81680343f0 con 0x7f81700b0620 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.754+0000 7f816effd640 10 monclient: dump: 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:42.757 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 10 monclient: shutdown 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 1 -- 172.21.15.110:0/1336697784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81700b0620 msgr2=0x7f81701438d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 1 --2- 172.21.15.110:0/1336697784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81700b0620 0x7f81701438d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8168009e50 tx=0x7f816802bf60 comp rx=0 tx=0).stop 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 1 -- 172.21.15.110:0/1336697784 shutdown_connections 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 1 --2- 172.21.15.110:0/1336697784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81700b0620 0x7f81701438d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 1 --2- 172.21.15.110:0/1336697784 >> v2:172.21.15.110:3300/0 conn(0x7f817013d800 0x7f8170143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 1 -- 172.21.15.110:0/1336697784 >> 172.21.15.110:0/1336697784 conn(0x7f81700b47d0 msgr2=0x7f81700b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 1 -- 172.21.15.110:0/1336697784 shutdown_connections 2024-01-20T11:21:42.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.755+0000 7f8176905640 1 -- 172.21.15.110:0/1336697784 wait complete. 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.756+0000 7f8176905640 10 monclient: build_initial_monmap 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.756+0000 7f8176905640 10 monclient: monmap: 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:42.762 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.756+0000 7f8176905640 1 librados: starting msgr at 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.756+0000 7f8176905640 1 librados: starting objecter 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.756+0000 7f8176905640 1 Processor -- start 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.756+0000 7f8176905640 1 -- start start 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.756+0000 7f8176905640 1 librados: setting wanted keys 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.756+0000 7f8176905640 1 librados: calling monclient init 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.756+0000 7f8176905640 10 monclient: init 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f817006a860 0x7f81700adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient(hunting): picked mon.noname-a con 0x7f817006a860 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81700b0620 0x7f817006f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient(hunting): picked mon.noname-b con 0x7f81700b0620 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:42.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f817013d800 0x7f817006ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient(hunting): picked mon.noname-c con 0x7f817013d800 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817006ef70 con 0x7f817006a860 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817006f1c0 con 0x7f81700b0620 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81701375c0 con 0x7f817013d800 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8176905640 10 monclient(hunting): authenticate will time out at 1746.107513s 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f816ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f817006a860 0x7f81700adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f8174ea2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f817013d800 0x7f817006ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.757+0000 7f816f7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81700b0620 0x7f817006f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f817006a860 0x7f81700adb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45230/0 (socket says 172.21.15.110:45230) 2024-01-20T11:21:42.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816ffff640 1 -- 172.21.15.110:0/3055068438 learned_addr learned my addr 172.21.15.110:0/3055068438 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816ffff640 10 monclient(hunting): get_auth_request con 0x7f817006a860 auth_method 0 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f8174ea2640 10 monclient(hunting): get_auth_request con 0x7f817013d800 auth_method 0 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f8174ea2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f8174ea2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f8174ea2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816f7fe640 10 monclient(hunting): get_auth_request con 0x7f81700b0620 auth_method 0 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f8174ea2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f8174ea2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f8174ea2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.758+0000 7f816f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 10 monclient(hunting): handle_auth_done global_id 5610 payload 824 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 1 -- 172.21.15.110:0/3055068438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f817013d800 msgr2=0x7f817006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f817013d800 0x7f817006ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 1 -- 172.21.15.110:0/3055068438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81700b0620 msgr2=0x7f817006f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81700b0620 0x7f817006f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 1 monclient: found mon.noname-a 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:42.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:42.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f816ffff640 1 -- 172.21.15.110:0/3055068438 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8168009b30 con 0x7f817006a860 2024-01-20T11:21:42.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.761+0000 7f8174ea2640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f817013d800 0x7f817006ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:42.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.764+0000 7f816ffff640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f817006a860 0x7f81700adb00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f816000c400 tx=0x7f816000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:42.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.764+0000 7f816cff9640 10 client.?.objecter ms_handle_connect 0x7f817006a860 2024-01-20T11:21:42.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.764+0000 7f816cff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:42.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 1 -- 172.21.15.110:0/3055068438 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81600052d0 con 0x7f817006a860 2024-01-20T11:21:42.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:42.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 10 monclient: dump: 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 10 monclient: _finish_auth 0 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 10 monclient: _check_auth_tickets 2024-01-20T11:21:42.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 1 -- 172.21.15.110:0/3055068438 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8160005450 con 0x7f817006a860 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f8176905640 5 monclient: authenticate success, global_id 5610 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f8176905640 10 monclient: _renew_subs 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 1 -- 172.21.15.110:0/3055068438 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160007260 con 0x7f817006a860 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f8176905640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8170070330 con 0x7f817006a860 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816cff9640 10 monclient: dump: 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:42.770 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.767+0000 7f816e7fc640 4 set_mon_vals no callback set 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.768+0000 7f8176905640 10 client.5610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.768+0000 7f8176905640 10 monclient: _renew_subs 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.768+0000 7f8176905640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.768+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8170137bc0 con 0x7f817006a860 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.768+0000 7f8176905640 1 librados: init done 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.769+0000 7f8176905640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.769+0000 7f8176905640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.769+0000 7f8176905640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:42.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.769+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817006f3e0 con 0x7f817006a860 2024-01-20T11:21:42.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.787+0000 7f816cff9640 1 -- 172.21.15.110:0/3055068438 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f81600073c0 con 0x7f817006a860 2024-01-20T11:21:42.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.787+0000 7f816cff9640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f815c06c8b0 0x7f815c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:42.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.788+0000 7f816f7fe640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f815c06c8b0 0x7f815c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:42.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.788+0000 7f816f7fe640 10 monclient: get_auth_request con 0x7f815c06c8b0 auth_method 0 2024-01-20T11:21:42.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.788+0000 7f816cff9640 10 client.5610.objecter ms_handle_connect 0x7f815c06c8b0 2024-01-20T11:21:42.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.788+0000 7f816f7fe640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f815c06c8b0 0x7f815c06ed70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8170139050 tx=0x7f8164009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:42.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.791+0000 7f816cff9640 1 -- 172.21.15.110:0/3055068438 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8160007670 con 0x7f817006a860 2024-01-20T11:21:42.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.791+0000 7f816cff9640 10 client.5610.objecter ms_dispatch 0x7f8170000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:42.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.791+0000 7f816cff9640 3 client.5610.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:42.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.791+0000 7f816cff9640 3 client.5610.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:42.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:42.791+0000 7f816cff9640 20 client.5610.objecter dump_active .. 0 homeless 2024-01-20T11:21:43.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.062+0000 7f816cff9640 1 -- 172.21.15.110:0/3055068438 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8160054000 con 0x7f817006a860 2024-01-20T11:21:43.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.062+0000 7f816cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:43.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.062+0000 7f816cff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:43.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.231+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8170143e10 con 0x7f815c06c8b0 2024-01-20T11:21:43.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.234+0000 7f816cff9640 1 -- 172.21.15.110:0/3055068438 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63956 (secure 0 0 0) 0x7f81640119d0 con 0x7f815c06c8b0 2024-01-20T11:21:43.234 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:43.235 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 10 librados: watch_flush enter 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 10 librados: watch_flush exit 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 20 client.5610.objecter shutdown clearing up homeless session... 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 10 client.5610.objecter successfully canceled tick 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f815c06c8b0 msgr2=0x7f815c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f815c06c8b0 0x7f815c06ed70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8170139050 tx=0x7f8164009190 comp rx=0 tx=0).stop 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 10 monclient: shutdown 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f817006a860 msgr2=0x7f81700adb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f817006a860 0x7f81700adb00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f816000c400 tx=0x7f816000c8c0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 shutdown_connections 2024-01-20T11:21:43.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f815c06c8b0 0x7f815c06ed70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f817013d800 0x7f817006ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81700b0620 0x7f817006f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 --2- 172.21.15.110:0/3055068438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f817006a860 0x7f81700adb00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 >> 172.21.15.110:0/3055068438 conn(0x7f81700b47d0 msgr2=0x7f817013b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:43.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 shutdown_connections 2024-01-20T11:21:43.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 -- 172.21.15.110:0/3055068438 wait complete. 2024-01-20T11:21:43.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.236+0000 7f8176905640 1 librados: shutdown 2024-01-20T11:21:43.247 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-01-20T11:21:42.787301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002733"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":8.6910000000000007},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":4.9349999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":12.468},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":9.4529999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":13.183999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":10.884}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":11.613},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":8.3230000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":10.119},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":15.416}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":13.878},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.622999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.1310000000000002},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":11.654999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":14.005000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":7.5069999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":12.499000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":9.6630000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":14.669},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":8.7929999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":18.501999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":13.23}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":15.34},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.667999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313299,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.8099999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":7.5830000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1129999999999995},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":8.4209999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":10.000999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":12.946}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":11.377000000000001},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":14.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":12.119},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":15.129}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.594999999999999},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.74}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444112,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":6.3639999999999999},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.6280000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":7.3849999999999998},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":16.759}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":17.931999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.577}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":20.393000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.433}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":15.122},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.499000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.0239999999999991},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":21.297999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444113,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.1980000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.3220000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":12.391999999999999},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":10.23}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":8.2710000000000008},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":13.090999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.4979999999999993},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.206},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":10.972}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":16.699999999999999},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.365}]}]},{"osd":0,"up_from":17,"seq":73014444113,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":6.2869999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":10.375999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.4100000000000001},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.2050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.968},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":8.6430000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.48},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":13.815}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":9.8350000000000009},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":14.946999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":15.773999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":16.600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509522,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":8.6839999999999993},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.0099999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.1270000000000007},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":9.4030000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":14.680999999999999},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":5.8760000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.9979999999999993},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":13.917999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":15.49},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.27}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":12.881},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":10.565}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:43.249 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:43.249 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:43.249 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:43.249 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 14, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:17:36.602178+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:17:36.602178+0000', 'last_peered': '2024-01-20T11:17:36.602178+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:17:36.602178+0000', 'last_undegraded': '2024-01-20T11:17:36.602178+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:43.249 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:43.249 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:43.249 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:43.250 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:43.250 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:43.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.348+0000 7fbd2b24f640 10 monclient: get_monmap_and_config 2024-01-20T11:21:43.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.348+0000 7fbd2b24f640 10 monclient: build_initial_monmap 2024-01-20T11:21:43.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.348+0000 7fbd2b24f640 10 monclient: monmap: 2024-01-20T11:21:43.348 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:43.348 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:43.348 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:43.348843+0000 2024-01-20T11:21:43.349 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:43.348843+0000 2024-01-20T11:21:43.349 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:43.349 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:43.349 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:43.349 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:43.349 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:43.349 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:43.349 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:43.349 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:43.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 1 Processor -- start 2024-01-20T11:21:43.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 1 -- start start 2024-01-20T11:21:43.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient: init 2024-01-20T11:21:43.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:43.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-01-20T11:21:43.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbd240b0c00 0x7fbd2413f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:43.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient(hunting): picked mon.noname-a con 0x7fbd240b0c00 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:43.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240adbc0 0x7fbd2413f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:43.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient(hunting): picked mon.noname-b con 0x7fbd240adbc0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd241420b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd240aaa40 con 0x7fbd241420b0 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd24072680 con 0x7fbd240b0c00 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240aac90 con 0x7fbd240adbc0 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.349+0000 7fbd2b24f640 10 monclient(hunting): authenticate will time out at 1746.699903s 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd28feb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbd240b0c00 0x7fbd2413f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd23fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240adbc0 0x7fbd2413f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd28feb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbd240b0c00 0x7fbd2413f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45234/0 (socket says 172.21.15.110:45234) 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd23fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240adbc0 0x7fbd2413f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54804/0 (socket says 172.21.15.110:54804) 2024-01-20T11:21:43.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd28feb640 1 -- 172.21.15.110:0/1756770224 learned_addr learned my addr 172.21.15.110:0/1756770224 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd28feb640 10 monclient(hunting): get_auth_request con 0x7fbd240b0c00 auth_method 0 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd28feb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd28feb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd28feb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd23fff640 10 monclient(hunting): get_auth_request con 0x7fbd240adbc0 auth_method 0 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd23fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.350+0000 7fbd23fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.351+0000 7fbd28feb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.351+0000 7fbd28feb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.351+0000 7fbd28feb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.351+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.351+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.351+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.352+0000 7fbd28feb640 10 monclient(hunting): handle_auth_done global_id 5613 payload 274 2024-01-20T11:21:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.352+0000 7fbd28feb640 1 -- 172.21.15.110:0/1756770224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240adbc0 msgr2=0x7fbd2413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:43.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.352+0000 7fbd28feb640 1 --2- 172.21.15.110:0/1756770224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240adbc0 0x7fbd2413f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.352+0000 7fbd28feb640 1 -- 172.21.15.110:0/1756770224 >> v1:172.21.15.134:6789/0 conn(0x7fbd241420b0 legacy=0x7fbd24142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:43.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.352+0000 7fbd28feb640 10 monclient: _finish_hunting 0 2024-01-20T11:21:43.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.352+0000 7fbd28feb640 1 monclient: found mon.noname-a 2024-01-20T11:21:43.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.352+0000 7fbd28feb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:43.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.352+0000 7fbd28feb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:43.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.352+0000 7fbd28feb640 1 -- 172.21.15.110:0/1756770224 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd240aaf70 con 0x7fbd240b0c00 2024-01-20T11:21:43.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.354+0000 7fbd28feb640 1 --2- 172.21.15.110:0/1756770224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd240b0c00 0x7fbd2413f190 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009e50 tx=0x7fbd1c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72096cdfc99ba2b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:43.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 1 -- 172.21.15.110:0/1756770224 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c02ca70 con 0x7fbd240b0c00 2024-01-20T11:21:43.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 10 monclient: dump: 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:43.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 1 -- 172.21.15.110:0/1756770224 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c02cbd0 con 0x7fbd240b0c00 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 1 -- 172.21.15.110:0/1756770224 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0343f0 con 0x7fbd240b0c00 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd237fe640 10 monclient: dump: 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:43.360 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd2b24f640 5 monclient: authenticate success, global_id 5613 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd2b24f640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd2b24f640 4 set_mon_vals no callback set 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd2b24f640 10 monclient: shutdown 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd2b24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd2b24f640 1 -- 172.21.15.110:0/1756770224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd240b0c00 msgr2=0x7fbd2413f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.358+0000 7fbd2b24f640 1 --2- 172.21.15.110:0/1756770224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd240b0c00 0x7fbd2413f190 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009e50 tx=0x7fbd1c02bed0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 1 -- 172.21.15.110:0/1756770224 shutdown_connections 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 1 --2- 172.21.15.110:0/1756770224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240adbc0 0x7fbd2413f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 1 --2- 172.21.15.110:0/1756770224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd240b0c00 0x7fbd2413f190 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 1 -- 172.21.15.110:0/1756770224 >> 172.21.15.110:0/1756770224 conn(0x7fbd240b3e60 msgr2=0x7fbd240b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:43.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 1 -- 172.21.15.110:0/1756770224 shutdown_connections 2024-01-20T11:21:43.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 1 -- 172.21.15.110:0/1756770224 wait complete. 2024-01-20T11:21:43.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 10 monclient: build_initial_monmap 2024-01-20T11:21:43.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 10 monclient: monmap: 2024-01-20T11:21:43.365 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:43.365 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:43.365 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:43.365 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 1 librados: starting msgr at 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.359+0000 7fbd2b24f640 1 librados: starting objecter 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 Processor -- start 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 -- start start 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 librados: setting wanted keys 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 librados: calling monclient init 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient: init 2024-01-20T11:21:43.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240adbc0 0x7fbd241dab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient(hunting): picked mon.noname-b con 0x7fbd240adbc0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240b0c00 0x7fbd241db0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient(hunting): picked mon.noname-c con 0x7fbd240b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd241420b0 0x7fbd241318b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient(hunting): picked mon.noname-a con 0x7fbd241420b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24138570 con 0x7fbd241420b0 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241387c0 con 0x7fbd240adbc0 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241da650 con 0x7fbd240b0c00 2024-01-20T11:21:43.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.360+0000 7fbd2b24f640 10 monclient(hunting): authenticate will time out at 1746.710998s 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd297ec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd241420b0 0x7fbd241318b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd23fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240b0c00 0x7fbd241db0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd28feb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240adbc0 0x7fbd241dab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd297ec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd241420b0 0x7fbd241318b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45246/0 (socket says 172.21.15.110:45246) 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd297ec640 1 -- 172.21.15.110:0/3376075103 learned_addr learned my addr 172.21.15.110:0/3376075103 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd297ec640 10 monclient(hunting): get_auth_request con 0x7fbd241420b0 auth_method 0 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd297ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd297ec640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd297ec640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd23fff640 10 monclient(hunting): get_auth_request con 0x7fbd240b0c00 auth_method 0 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd23fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:43.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd23fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd28feb640 10 monclient(hunting): get_auth_request con 0x7fbd240adbc0 auth_method 0 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd28feb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd28feb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.361+0000 7fbd28feb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.362+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.362+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.362+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.362+0000 7fbd297ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.362+0000 7fbd297ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.362+0000 7fbd297ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.362+0000 7fbd28feb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.362+0000 7fbd28feb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.362+0000 7fbd28feb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:43.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.364+0000 7fbd23fff640 10 monclient(hunting): handle_auth_done global_id 5330 payload 824 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd23fff640 1 -- 172.21.15.110:0/3376075103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240adbc0 msgr2=0x7fbd241dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd23fff640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240adbc0 0x7fbd241dab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd23fff640 1 -- 172.21.15.110:0/3376075103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd241420b0 msgr2=0x7fbd241318b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd23fff640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd241420b0 0x7fbd241318b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd23fff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd23fff640 1 monclient: found mon.noname-c 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd23fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd23fff640 1 -- 172.21.15.110:0/3376075103 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c009b30 con 0x7fbd240b0c00 2024-01-20T11:21:43.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.365+0000 7fbd297ec640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd241420b0 0x7fbd241318b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:43.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.367+0000 7fbd23fff640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240b0c00 0x7fbd241db0b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00e850 tx=0x7fbd0c00ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:43.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.367+0000 7fbd217fa640 10 client.?.objecter ms_handle_connect 0x7fbd240b0c00 2024-01-20T11:21:43.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.367+0000 7fbd217fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:43.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 1 -- 172.21.15.110:0/3376075103 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c0095b0 con 0x7fbd240b0c00 2024-01-20T11:21:43.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:43.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:43.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 10 monclient: dump: 2024-01-20T11:21:43.372 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:43.372 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 10 monclient: _finish_auth 0 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 10 monclient: _check_auth_tickets 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 1 -- 172.21.15.110:0/3376075103 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0c00c940 con 0x7fbd240b0c00 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 1 -- 172.21.15.110:0/3376075103 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c00ccd0 con 0x7fbd240b0c00 2024-01-20T11:21:43.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd217fa640 10 monclient: dump: 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd2b24f640 5 monclient: authenticate success, global_id 5330 2024-01-20T11:21:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd2b24f640 10 monclient: _renew_subs 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd22ffd640 4 set_mon_vals no callback set 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd2b24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd240b12b0 con 0x7fbd240b0c00 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd2b24f640 10 client.5330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd2b24f640 10 monclient: _renew_subs 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd2b24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd24131fd0 con 0x7fbd240b0c00 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.370+0000 7fbd2b24f640 1 librados: init done 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.371+0000 7fbd2b24f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.371+0000 7fbd2b24f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.371+0000 7fbd2b24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.371+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd240b1470 con 0x7fbd240b0c00 2024-01-20T11:21:43.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.617+0000 7fbd217fa640 1 -- 172.21.15.110:0/3376075103 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fbd0c013470 con 0x7fbd240b0c00 2024-01-20T11:21:43.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.618+0000 7fbd217fa640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd0406c8b0 0x7fbd0406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:43.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.618+0000 7fbd28feb640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd0406c8b0 0x7fbd0406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:43.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.618+0000 7fbd28feb640 10 monclient: get_auth_request con 0x7fbd0406c8b0 auth_method 0 2024-01-20T11:21:43.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.619+0000 7fbd217fa640 10 client.5330.objecter ms_handle_connect 0x7fbd0406c8b0 2024-01-20T11:21:43.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.619+0000 7fbd28feb640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd0406c8b0 0x7fbd0406ed70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c035000 tx=0x7fbd1c030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:43.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.630+0000 7fbd217fa640 1 -- 172.21.15.110:0/3376075103 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbd0c00ce30 con 0x7fbd240b0c00 2024-01-20T11:21:43.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.630+0000 7fbd217fa640 10 client.5330.objecter ms_dispatch 0x7fbd24000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:43.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.630+0000 7fbd217fa640 3 client.5330.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:43.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.630+0000 7fbd217fa640 3 client.5330.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:43.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.630+0000 7fbd217fa640 20 client.5330.objecter dump_active .. 0 homeless 2024-01-20T11:21:43.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.630+0000 7fbd217fa640 1 -- 172.21.15.110:0/3376075103 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fbd0c054e70 con 0x7fbd240b0c00 2024-01-20T11:21:43.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.630+0000 7fbd217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:43.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.630+0000 7fbd217fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:43.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.880+0000 7fbd2b24f640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:43.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.880+0000 7fbd2b24f640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:43.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.880+0000 7fbd2b24f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:43.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.880+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbd241da7e0 con 0x7fbd240b0c00 2024-01-20T11:21:43.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.886+0000 7fbd217fa640 1 -- 172.21.15.110:0/3376075103 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fbd0c021520 con 0x7fbd240b0c00 2024-01-20T11:21:43.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.886+0000 7fbd217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:43.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.886+0000 7fbd217fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:43.886 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:43.886 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":438,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:43.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 10 librados: watch_flush enter 2024-01-20T11:21:43.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 10 librados: watch_flush exit 2024-01-20T11:21:43.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 20 client.5330.objecter shutdown clearing up homeless session... 2024-01-20T11:21:43.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 10 client.5330.objecter successfully canceled tick 2024-01-20T11:21:43.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd0406c8b0 msgr2=0x7fbd0406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:43.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd0406c8b0 0x7fbd0406ed70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c035000 tx=0x7fbd1c030000 comp rx=0 tx=0).stop 2024-01-20T11:21:43.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 10 monclient: shutdown 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240b0c00 msgr2=0x7fbd241db0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240b0c00 0x7fbd241db0b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00e850 tx=0x7fbd0c00ed10 comp rx=0 tx=0).stop 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 shutdown_connections 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbd0406c8b0 0x7fbd0406ed70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd241420b0 0x7fbd241318b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd240b0c00 0x7fbd241db0b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 --2- 172.21.15.110:0/3376075103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240adbc0 0x7fbd241dab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 >> 172.21.15.110:0/3376075103 conn(0x7fbd240b3e60 msgr2=0x7fbd2413ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 shutdown_connections 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 -- 172.21.15.110:0/3376075103 wait complete. 2024-01-20T11:21:43.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.888+0000 7fbd2b24f640 1 librados: shutdown 2024-01-20T11:21:43.900 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:43.900 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:43.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.999+0000 7f16f0275640 10 monclient: get_monmap_and_config 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:43.999+0000 7f16f0275640 10 monclient: build_initial_monmap 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.000+0000 7f16f0275640 10 monclient: monmap: 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:44.000599+0000 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:44.000599+0000 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:44.000 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:44.001 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:44.001 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:44.001 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:44.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.000+0000 7f16f0275640 1 Processor -- start 2024-01-20T11:21:44.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.000+0000 7f16f0275640 1 -- start start 2024-01-20T11:21:44.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.000+0000 7f16f0275640 10 monclient: init 2024-01-20T11:21:44.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:44.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-01-20T11:21:44.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f16e8145760 0x7f16e8145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:44.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 10 monclient(hunting): picked mon.noname-a con 0x7f16e8145760 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:44.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16e80a9d80 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:44.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16e813f8b0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16e8146250 con 0x7f16e813f8b0 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16e80b20c0 con 0x7f16e80a9d80 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16e80ab050 con 0x7f16e8145760 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16f0275640 10 monclient(hunting): authenticate will time out at 1747.351558s 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.001+0000 7f16ee011640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f16e8145760 0x7f16e8145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.002+0000 7f16ed810640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f16e80a9d80 0x7f16e813f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:58934/0 (socket says 172.21.15.110:58934) 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.002+0000 7f16ed810640 1 -- 172.21.15.110:0/561519772 learned_addr learned my addr 172.21.15.110:0/561519772 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.002+0000 7f16ee011640 10 monclient(hunting): get_auth_request con 0x7f16e8145760 auth_method 0 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.002+0000 7f16ee011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.002+0000 7f16ee011640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.002+0000 7f16ee011640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:44.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.003+0000 7f16ee011640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.003+0000 7f16ee011640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:44.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.003+0000 7f16ee011640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ed00f640 1 -- 172.21.15.110:0/561519772 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675653210 0 0) 0x7f16e80b20c0 con 0x7f16e80a9d80 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ed00f640 10 monclient(hunting): _negotiate 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ed00f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ed00f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ed00f640 10 monclient(hunting): my global_id is 5704 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ed00f640 1 -- 172.21.15.110:0/561519772 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d4003520 con 0x7f16e80a9d80 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ee011640 10 monclient(hunting): handle_auth_done global_id 5619 payload 274 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ee011640 1 -- 172.21.15.110:0/561519772 >> v1:172.21.15.134:6789/0 conn(0x7f16e80a9d80 legacy=0x7f16e813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ee011640 1 -- 172.21.15.110:0/561519772 >> v1:172.21.15.110:6789/0 conn(0x7f16e813f8b0 legacy=0x7f16e80b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ee011640 10 monclient: _finish_hunting 0 2024-01-20T11:21:44.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ee011640 1 monclient: found mon.noname-a 2024-01-20T11:21:44.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ee011640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:44.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ee011640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:44.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.005+0000 7f16ee011640 1 -- 172.21.15.110:0/561519772 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e80b2d80 con 0x7f16e8145760 2024-01-20T11:21:44.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.007+0000 7f16ee011640 1 --2- 172.21.15.110:0/561519772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e8145760 0x7f16e8145b40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009dc0 tx=0x7f16e402bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63392dd870fe7675 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 1 -- 172.21.15.110:0/561519772 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e402ccd0 con 0x7f16e8145760 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 10 monclient: dump: 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:44.012 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 10 monclient: _finish_auth 0 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 10 monclient: _check_auth_tickets 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 1 -- 172.21.15.110:0/561519772 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e402ce30 con 0x7f16e8145760 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 1 -- 172.21.15.110:0/561519772 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4034660 con 0x7f16e8145760 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 5 monclient: authenticate success, global_id 5619 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 4 set_mon_vals no callback set 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:44.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16ed00f640 10 monclient: dump: 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:44.014 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 10 monclient: shutdown 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 1 -- 172.21.15.110:0/561519772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e8145760 msgr2=0x7f16e8145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 1 --2- 172.21.15.110:0/561519772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e8145760 0x7f16e8145b40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009dc0 tx=0x7f16e402bf20 comp rx=0 tx=0).stop 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 1 -- 172.21.15.110:0/561519772 shutdown_connections 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 1 --2- 172.21.15.110:0/561519772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e8145760 0x7f16e8145b40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.012+0000 7f16f0275640 1 -- 172.21.15.110:0/561519772 >> 172.21.15.110:0/561519772 conn(0x7f16e80663b0 msgr2=0x7f16e80667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.013+0000 7f16f0275640 1 -- 172.21.15.110:0/561519772 shutdown_connections 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.013+0000 7f16f0275640 1 -- 172.21.15.110:0/561519772 wait complete. 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.013+0000 7f16f0275640 10 monclient: build_initial_monmap 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.013+0000 7f16f0275640 10 monclient: monmap: 2024-01-20T11:21:44.018 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 librados: starting msgr at 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 librados: starting objecter 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 Processor -- start 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 -- start start 2024-01-20T11:21:44.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 librados: setting wanted keys 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 librados: calling monclient init 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient: init 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16e80a9d80 0x7f16e8142920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient(hunting): picked mon.noname-b con 0x7f16e80a9d80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e813f8b0 0x7f16e8141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient(hunting): picked mon.noname-a con 0x7f16e813f8b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16e8145760 0x7f16e81370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient(hunting): picked mon.noname-c con 0x7f16e8145760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e8138520 con 0x7f16e813f8b0 2024-01-20T11:21:44.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e8142030 con 0x7f16e80a9d80 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e81421f0 con 0x7f16e8145760 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.014+0000 7f16f0275640 10 monclient(hunting): authenticate will time out at 1747.365074s 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ed810640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e813f8b0 0x7f16e8141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ee812640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16e8145760 0x7f16e81370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ee011640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16e80a9d80 0x7f16e8142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ed810640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e813f8b0 0x7f16e8141260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45274/0 (socket says 172.21.15.110:45274) 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ed810640 1 -- 172.21.15.110:0/1422637249 learned_addr learned my addr 172.21.15.110:0/1422637249 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ed810640 10 monclient(hunting): get_auth_request con 0x7f16e813f8b0 auth_method 0 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ed810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ed810640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:44.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ed810640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ee812640 10 monclient(hunting): get_auth_request con 0x7f16e8145760 auth_method 0 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ee812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ee812640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.015+0000 7f16ee812640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee011640 10 monclient(hunting): get_auth_request con 0x7f16e80a9d80 auth_method 0 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee011640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee011640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ed810640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ed810640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ed810640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee812640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:44.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee011640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee011640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.016+0000 7f16ee011640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 10 monclient(hunting): handle_auth_done global_id 5622 payload 824 2024-01-20T11:21:44.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 1 -- 172.21.15.110:0/1422637249 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16e8145760 msgr2=0x7f16e81370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16e8145760 0x7f16e81370c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 1 -- 172.21.15.110:0/1422637249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16e80a9d80 msgr2=0x7f16e8142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16e80a9d80 0x7f16e8142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 10 monclient: _finish_hunting 0 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 1 monclient: found mon.noname-a 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.019+0000 7f16ed810640 1 -- 172.21.15.110:0/1422637249 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e4009a70 con 0x7f16e813f8b0 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.021+0000 7f16ed810640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e813f8b0 0x7f16e8141260 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f16d800c410 tx=0x7f16d800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.021+0000 7f16deffd640 10 client.?.objecter ms_handle_connect 0x7f16e813f8b0 2024-01-20T11:21:44.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.021+0000 7f16deffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 1 -- 172.21.15.110:0/1422637249 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d8005ce0 con 0x7f16e813f8b0 2024-01-20T11:21:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 10 monclient: dump: 2024-01-20T11:21:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:44.026 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 10 monclient: _finish_auth 0 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 1 -- 172.21.15.110:0/1422637249 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d8005e40 con 0x7f16e813f8b0 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16deffd640 1 -- 172.21.15.110:0/1422637249 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80072d0 con 0x7f16e813f8b0 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16f0275640 5 monclient: authenticate success, global_id 5622 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16f0275640 10 monclient: _renew_subs 2024-01-20T11:21:44.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16f0275640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.025+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e8138940 con 0x7f16e813f8b0 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.026+0000 7f16deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.026+0000 7f16deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.026+0000 7f16deffd640 10 monclient: dump: 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:44.028 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.026+0000 7f16ec80e640 4 set_mon_vals no callback set 2024-01-20T11:21:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.026+0000 7f16f0275640 10 client.5622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.026+0000 7f16f0275640 10 monclient: _renew_subs 2024-01-20T11:21:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.026+0000 7f16f0275640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.026+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e8138e00 con 0x7f16e813f8b0 2024-01-20T11:21:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.026+0000 7f16f0275640 1 librados: init done 2024-01-20T11:21:44.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.027+0000 7f16f0275640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.027+0000 7f16f0275640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.027+0000 7f16f0275640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:44.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.027+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e8075490 con 0x7f16e813f8b0 2024-01-20T11:21:44.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.046+0000 7f16deffd640 1 -- 172.21.15.110:0/1422637249 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f16d80074f0 con 0x7f16e813f8b0 2024-01-20T11:21:44.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.047+0000 7f16deffd640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f16cc06c8b0 0x7f16cc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:44.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.047+0000 7f16ee011640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f16cc06c8b0 0x7f16cc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:44.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.047+0000 7f16ee011640 10 monclient: get_auth_request con 0x7f16cc06c8b0 auth_method 0 2024-01-20T11:21:44.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.048+0000 7f16deffd640 10 client.5622.objecter ms_handle_connect 0x7f16cc06c8b0 2024-01-20T11:21:44.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.048+0000 7f16ee011640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f16cc06c8b0 0x7f16cc06ed70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f16e4036000 tx=0x7f16e4030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:44.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.049+0000 7f16deffd640 1 -- 172.21.15.110:0/1422637249 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f16d8058020 con 0x7f16e813f8b0 2024-01-20T11:21:44.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.049+0000 7f16deffd640 10 client.5622.objecter ms_dispatch 0x7f16e8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:44.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.049+0000 7f16deffd640 3 client.5622.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:44.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.049+0000 7f16deffd640 3 client.5622.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:44.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.049+0000 7f16deffd640 20 client.5622.objecter dump_active .. 0 homeless 2024-01-20T11:21:44.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.303+0000 7f16deffd640 1 -- 172.21.15.110:0/1422637249 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f16d8054360 con 0x7f16e813f8b0 2024-01-20T11:21:44.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.303+0000 7f16deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:44.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.303+0000 7f16deffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:44.307 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:35.331 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:44.355 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:35.379 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:44.358 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:07:35.365 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:44.362 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:07:35.368 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:44.378 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:07:35.384 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:44.384 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:35.408 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:44.426 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:35.450 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:44.431 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:35.456 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:44.454 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:35.479 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:21:44.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.484+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16e80b3340 con 0x7f16cc06c8b0 2024-01-20T11:21:44.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.487+0000 7f16deffd640 1 -- 172.21.15.110:0/1422637249 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63956 (secure 0 0 0) 0x7f16e4042070 con 0x7f16cc06c8b0 2024-01-20T11:21:44.487 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:44.488 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 10 librados: watch_flush enter 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 10 librados: watch_flush exit 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 20 client.5622.objecter shutdown clearing up homeless session... 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 10 client.5622.objecter successfully canceled tick 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f16cc06c8b0 msgr2=0x7f16cc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f16cc06c8b0 0x7f16cc06ed70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f16e4036000 tx=0x7f16e4030000 comp rx=0 tx=0).stop 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 10 monclient: shutdown 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e813f8b0 msgr2=0x7f16e8141260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e813f8b0 0x7f16e8141260 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f16d800c410 tx=0x7f16d800c8d0 comp rx=0 tx=0).stop 2024-01-20T11:21:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 shutdown_connections 2024-01-20T11:21:44.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f16cc06c8b0 0x7f16cc06ed70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:44.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16e8145760 0x7f16e81370c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:44.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16e813f8b0 0x7f16e8141260 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:44.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 --2- 172.21.15.110:0/1422637249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16e80a9d80 0x7f16e8142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:44.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 >> 172.21.15.110:0/1422637249 conn(0x7f16e80663b0 msgr2=0x7f16e81359f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:44.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 shutdown_connections 2024-01-20T11:21:44.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 -- 172.21.15.110:0/1422637249 wait complete. 2024-01-20T11:21:44.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:44.490+0000 7f16f0275640 1 librados: shutdown 2024-01-20T11:21:44.504 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-01-20T11:21:42.787301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002733"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:17:36.602178+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:17:36.602178+0000","last_peered":"2024-01-20T11:17:36.602178+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:17:36.602178+0000","last_undegraded":"2024-01-20T11:17:36.602178+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":8.6910000000000007},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":4.9349999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":12.468},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":9.4529999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":13.183999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":10.884}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":11.613},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":8.3230000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":10.119},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":15.416}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":13.878},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.622999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.1310000000000002},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":11.654999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":14.005000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":7.5069999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":12.499000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":9.6630000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":14.669},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":8.7929999999999993}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":18.501999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":13.23}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":15.34},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.667999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313299,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.8099999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":7.5830000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1129999999999995},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":8.4209999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":10.000999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":12.946}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":11.377000000000001},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":14.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":12.119},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":15.129}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.594999999999999},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.74}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444112,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":6.3639999999999999},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.6280000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":7.3849999999999998},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":16.759}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":17.931999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.577}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":20.393000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.433}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":15.122},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.499000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.0239999999999991},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":21.297999999999998}]}]},{"osd":1,"up_from":17,"seq":73014444113,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.1980000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.3220000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":12.391999999999999},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":10.23}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":8.2710000000000008},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":13.090999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.4979999999999993},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":15.206},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":10.972}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":16.699999999999999},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.365}]}]},{"osd":0,"up_from":17,"seq":73014444113,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":6.2869999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":10.375999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.4100000000000001},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.2050000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.968},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":8.6430000000000007}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.48},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":13.815}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":9.8350000000000009},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":14.946999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":15.773999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":16.600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509522,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":8.6839999999999993},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.0099999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.1270000000000007},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":9.4030000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":14.680999999999999},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":5.8760000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.9979999999999993},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":13.917999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":15.49},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.27}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":12.881},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":10.565}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:45.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~168s 2024-01-20T11:21:47.507 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.607+0000 7f41c3ca9640 10 monclient: get_monmap_and_config 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.607+0000 7f41c3ca9640 10 monclient: build_initial_monmap 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.607+0000 7f41c3ca9640 10 monclient: monmap: 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:47.607690+0000 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:47.607690+0000 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:47.607 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:47.608 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:47.608 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:47.608 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:47.608 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:47.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.607+0000 7f41c3ca9640 1 Processor -- start 2024-01-20T11:21:47.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.607+0000 7f41c3ca9640 1 -- start start 2024-01-20T11:21:47.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.607+0000 7f41c3ca9640 10 monclient: init 2024-01-20T11:21:47.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:47.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-01-20T11:21:47.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41bc0b4d10 0x7f41bc0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f41bc0b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f41bc0b17c0 0x7f41bc0b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 10 monclient(hunting): picked mon.noname-c con 0x7f41bc0b17c0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41bc145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41bc0b2150 con 0x7f41bc145980 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41bc0b50f0 con 0x7f41bc0b4d10 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41bc0aade0 con 0x7f41bc0b17c0 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.608+0000 7f41c3ca9640 10 monclient(hunting): authenticate will time out at 1750.958683s 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.609+0000 7f41c1a45640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41bc0b4d10 0x7f41bc0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.609+0000 7f41c1244640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f41bc0b17c0 0x7f41bc0b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.609+0000 7f41c1a45640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41bc0b4d10 0x7f41bc0b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45276/0 (socket says 172.21.15.110:45276) 2024-01-20T11:21:47.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.609+0000 7f41c1a45640 1 -- 172.21.15.110:0/474338482 learned_addr learned my addr 172.21.15.110:0/474338482 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.609+0000 7f41c1a45640 10 monclient(hunting): get_auth_request con 0x7f41bc0b4d10 auth_method 0 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.609+0000 7f41c1a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.609+0000 7f41c1a45640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.609+0000 7f41c1a45640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1244640 10 monclient(hunting): get_auth_request con 0x7f41bc0b17c0 auth_method 0 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1244640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1244640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1244640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:47.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.610+0000 7f41c1244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:47.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.612+0000 7f41c1a45640 10 monclient(hunting): handle_auth_done global_id 5625 payload 274 2024-01-20T11:21:47.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.612+0000 7f41c1a45640 1 -- 172.21.15.110:0/474338482 >> v2:172.21.15.134:3300/0 conn(0x7f41bc0b17c0 msgr2=0x7f41bc0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:47.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.612+0000 7f41c1a45640 1 --2- 172.21.15.110:0/474338482 >> v2:172.21.15.134:3300/0 conn(0x7f41bc0b17c0 0x7f41bc0b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:47.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.612+0000 7f41c1a45640 1 -- 172.21.15.110:0/474338482 >> v1:172.21.15.110:6789/0 conn(0x7f41bc145980 legacy=0x7f41bc145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:47.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.612+0000 7f41c1a45640 10 monclient: _finish_hunting 0 2024-01-20T11:21:47.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.612+0000 7f41c1a45640 1 monclient: found mon.noname-a 2024-01-20T11:21:47.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.612+0000 7f41c1a45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:47.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.612+0000 7f41c1a45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:47.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.612+0000 7f41c1a45640 1 -- 172.21.15.110:0/474338482 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc145460 con 0x7f41bc0b4d10 2024-01-20T11:21:47.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.616+0000 7f41c1a45640 1 --2- 172.21.15.110:0/474338482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 0x7f41bc0b3da0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009e50 tx=0x7f41b802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199efebbc4503247 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:47.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 1 -- 172.21.15.110:0/474338482 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b802c9f0 con 0x7f41bc0b4d10 2024-01-20T11:21:47.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:47.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:47.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 10 monclient: dump: 2024-01-20T11:21:47.620 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:47.620 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:47.620 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 10 monclient: _finish_auth 0 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 10 monclient: _check_auth_tickets 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 1 -- 172.21.15.110:0/474338482 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b802cb50 con 0x7f41bc0b4d10 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c3ca9640 5 monclient: authenticate success, global_id 5625 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c3ca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:47.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c3ca9640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c3ca9640 4 set_mon_vals no callback set 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 1 -- 172.21.15.110:0/474338482 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b80343f0 con 0x7f41bc0b4d10 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.620+0000 7f41c0a43640 10 monclient: dump: 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:47.622 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:47.623 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 10 monclient: shutdown 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 1 -- 172.21.15.110:0/474338482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 msgr2=0x7f41bc0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 1 --2- 172.21.15.110:0/474338482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 0x7f41bc0b3da0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009e50 tx=0x7f41b802be50 comp rx=0 tx=0).stop 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 1 -- 172.21.15.110:0/474338482 shutdown_connections 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 1 --2- 172.21.15.110:0/474338482 >> v2:172.21.15.134:3300/0 conn(0x7f41bc0b17c0 0x7f41bc0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 1 --2- 172.21.15.110:0/474338482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 0x7f41bc0b3da0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 1 -- 172.21.15.110:0/474338482 >> 172.21.15.110:0/474338482 conn(0x7f41bc066590 msgr2=0x7f41bc0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 1 -- 172.21.15.110:0/474338482 shutdown_connections 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.621+0000 7f41c3ca9640 1 -- 172.21.15.110:0/474338482 wait complete. 2024-01-20T11:21:47.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.622+0000 7f41c3ca9640 10 monclient: build_initial_monmap 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.622+0000 7f41c3ca9640 10 monclient: monmap: 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.622+0000 7f41c3ca9640 1 librados: starting msgr at 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.622+0000 7f41c3ca9640 1 librados: starting objecter 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.622+0000 7f41c3ca9640 1 Processor -- start 2024-01-20T11:21:47.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.622+0000 7f41c3ca9640 1 -- start start 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.622+0000 7f41c3ca9640 1 librados: setting wanted keys 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.622+0000 7f41c3ca9640 1 librados: calling monclient init 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.622+0000 7f41c3ca9640 10 monclient: init 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc0b17c0 0x7f41bc13c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient(hunting): picked mon.noname-b con 0x7f41bc0b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 0x7f41bc1da800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f41bc0b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41bc145980 0x7f41bc1dad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient(hunting): picked mon.noname-c con 0x7f41bc145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:47.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0ac5f0 con 0x7f41bc0b4d10 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0ac7f0 con 0x7f41bc0b17c0 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc13eeb0 con 0x7f41bc145980 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c3ca9640 10 monclient(hunting): authenticate will time out at 1750.973387s 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c1244640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 0x7f41bc1da800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c2246640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41bc145980 0x7f41bc1dad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.623+0000 7f41c1a45640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc0b17c0 0x7f41bc13c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1244640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 0x7f41bc1da800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45286/0 (socket says 172.21.15.110:45286) 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1244640 1 -- 172.21.15.110:0/1643915156 learned_addr learned my addr 172.21.15.110:0/1643915156 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1244640 10 monclient(hunting): get_auth_request con 0x7f41bc0b4d10 auth_method 0 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:47.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1244640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1244640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c2246640 10 monclient(hunting): get_auth_request con 0x7f41bc145980 auth_method 0 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c2246640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c2246640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c2246640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1a45640 10 monclient(hunting): get_auth_request con 0x7f41bc0b17c0 auth_method 0 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1a45640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1a45640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1244640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c2246640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c2246640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c2246640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:47.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:47.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:47.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.624+0000 7f41c1a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:47.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 10 monclient(hunting): handle_auth_done global_id 5631 payload 824 2024-01-20T11:21:47.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 1 -- 172.21.15.110:0/1643915156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41bc145980 msgr2=0x7f41bc1dad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:47.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41bc145980 0x7f41bc1dad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 1 -- 172.21.15.110:0/1643915156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc0b17c0 msgr2=0x7f41bc13c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc0b17c0 0x7f41bc13c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 10 monclient: _finish_hunting 0 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 1 monclient: found mon.noname-a 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.627+0000 7f41c1244640 1 -- 172.21.15.110:0/1643915156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b8009b30 con 0x7f41bc0b4d10 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.629+0000 7f41c1244640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 0x7f41bc1da800 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00e9b0 tx=0x7f41ac00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.629+0000 7f41b27fc640 10 client.?.objecter ms_handle_connect 0x7f41bc0b4d10 2024-01-20T11:21:47.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.629+0000 7f41b27fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 1 -- 172.21.15.110:0/1643915156 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac009670 con 0x7f41bc0b4d10 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 10 monclient: dump: 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:47.634 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 1 -- 172.21.15.110:0/1643915156 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac00c9a0 con 0x7f41bc0b4d10 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 1 -- 172.21.15.110:0/1643915156 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac00cd50 con 0x7f41bc0b4d10 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41c3ca9640 5 monclient: authenticate success, global_id 5631 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41c3ca9640 10 monclient: _renew_subs 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41c3ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc0ab290 con 0x7f41bc0b4d10 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b3fff640 4 set_mon_vals no callback set 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41b27fc640 10 monclient: dump: 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:47.635 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41c3ca9640 10 client.5631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41c3ca9640 10 monclient: _renew_subs 2024-01-20T11:21:47.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41c3ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:47.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.632+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc1db460 con 0x7f41bc0b4d10 2024-01-20T11:21:47.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.633+0000 7f41c3ca9640 1 librados: init done 2024-01-20T11:21:47.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.633+0000 7f41c3ca9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:47.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.633+0000 7f41c3ca9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:47.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.633+0000 7f41c3ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:47.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.633+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc075490 con 0x7f41bc0b4d10 2024-01-20T11:21:47.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.652+0000 7f41b27fc640 1 -- 172.21.15.110:0/1643915156 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f41ac013470 con 0x7f41bc0b4d10 2024-01-20T11:21:47.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.652+0000 7f41b27fc640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41a006c8b0 0x7f41a006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:47.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.653+0000 7f41c1a45640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41a006c8b0 0x7f41a006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:47.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.653+0000 7f41c1a45640 10 monclient: get_auth_request con 0x7f41a006c8b0 auth_method 0 2024-01-20T11:21:47.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.653+0000 7f41c1a45640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41a006c8b0 0x7f41a006ed70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f41b8035000 tx=0x7f41b8030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:47.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.653+0000 7f41b27fc640 10 client.5631.objecter ms_handle_connect 0x7f41a006c8b0 2024-01-20T11:21:47.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.656+0000 7f41b27fc640 1 -- 172.21.15.110:0/1643915156 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f41ac058bd0 con 0x7f41bc0b4d10 2024-01-20T11:21:47.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.657+0000 7f41b27fc640 10 client.5631.objecter ms_dispatch 0x7f41bc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:47.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.657+0000 7f41b27fc640 3 client.5631.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:47.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.657+0000 7f41b27fc640 3 client.5631.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:47.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.657+0000 7f41b27fc640 20 client.5631.objecter dump_active .. 0 homeless 2024-01-20T11:21:47.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.903+0000 7f41b27fc640 1 -- 172.21.15.110:0/1643915156 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f41ac054f10 con 0x7f41bc0b4d10 2024-01-20T11:21:47.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.903+0000 7f41b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:47.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:47.903+0000 7f41b27fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:48.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.098+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41bc14c7f0 con 0x7f41a006c8b0 2024-01-20T11:21:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.101+0000 7f41b27fc640 1 -- 172.21.15.110:0/1643915156 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63905 (secure 0 0 0) 0x7f41b803f440 con 0x7f41a006c8b0 2024-01-20T11:21:48.101 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 10 librados: watch_flush enter 2024-01-20T11:21:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 10 librados: watch_flush exit 2024-01-20T11:21:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 20 client.5631.objecter shutdown clearing up homeless session... 2024-01-20T11:21:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 10 client.5631.objecter successfully canceled tick 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41a006c8b0 msgr2=0x7f41a006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41a006c8b0 0x7f41a006ed70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f41b8035000 tx=0x7f41b8030000 comp rx=0 tx=0).stop 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 10 monclient: shutdown 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 msgr2=0x7f41bc1da800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 0x7f41bc1da800 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00e9b0 tx=0x7f41ac00ee70 comp rx=0 tx=0).stop 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 shutdown_connections 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f41a006c8b0 0x7f41a006ed70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41bc145980 0x7f41bc1dad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41bc0b4d10 0x7f41bc1da800 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 --2- 172.21.15.110:0/1643915156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc0b17c0 0x7f41bc13c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 >> 172.21.15.110:0/1643915156 conn(0x7f41bc066590 msgr2=0x7f41bc0b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 shutdown_connections 2024-01-20T11:21:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 -- 172.21.15.110:0/1643915156 wait complete. 2024-01-20T11:21:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.103+0000 7f41c3ca9640 1 librados: shutdown 2024-01-20T11:21:48.113 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-01-20T11:21:46.788230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002654"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":8.4689999999999994},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.766}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":7.5039999999999996},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":9.7420000000000009}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":10.955},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":10.343}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":13.643000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":8.1760000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":14.305999999999999},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":11.532999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":12.147},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":15.208}]}]},{"osd":7,"up_from":21,"seq":90194313299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":7.351},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":12.493}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":6.5670000000000002},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":11.112}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":13.228999999999999},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":16.440000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":8.9600000000000009},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":11.788}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":9.5890000000000004},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":8.3729999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":10.394}]}]},{"osd":3,"up_from":21,"seq":90194313299,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.8099999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":7.5830000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1129999999999995},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":8.4209999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":10.000999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":12.946}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":11.377000000000001},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":14.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":12.119},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":15.129}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.594999999999999},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.74}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444113,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":7.9020000000000001},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":7.2080000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.75},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":14.375999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.571999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":11.808999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.308},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.914999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":9.7959999999999994},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":9.2140000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":17.094000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":13.497}]}]},{"osd":1,"up_from":17,"seq":73014444114,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.3849999999999998},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":10.053000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":11.99},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":9.1229999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":14.827},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":10.644}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":5.9669999999999996},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":17.724}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.036},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":13.156000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":16.869},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":15.936999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444114,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":6.3079999999999998},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":11.837999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":5.7240000000000002},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":7.7350000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":8.4269999999999996},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.817}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":14.422000000000001},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":9.7759999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":9.093},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":13.422000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":15.314},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":11.082000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.8579999999999997},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.7129999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":10.914999999999999},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":13.506}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":14.173},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":5.9459999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.301},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":15.612}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":12.804},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":9.4290000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":10.002000000000001},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.827}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:48.115 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:48.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.214+0000 7f29583dc640 10 monclient: get_monmap_and_config 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.214+0000 7f29583dc640 10 monclient: build_initial_monmap 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.214+0000 7f29583dc640 10 monclient: monmap: 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:48.215151+0000 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:48.215151+0000 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 1 Processor -- start 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 1 -- start start 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient: init 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29500b0ba0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500adc30 0x7f29500ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient(hunting): picked mon.noname-b con 0x7f29500adc30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f29500ac4d0 0x7f295013bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient(hunting): picked mon.noname-c con 0x7f29500ac4d0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29500ad530 con 0x7f29500b0ba0 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29500af840 con 0x7f29500ac4d0 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29500b0f80 con 0x7f29500adc30 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.215+0000 7f29583dc640 10 monclient(hunting): authenticate will time out at 1751.566176s 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.216+0000 7f2955977640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500adc30 0x7f29500ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.216+0000 7f2956178640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f29500b0ba0 0x7f29500af130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51816/0 (socket says 172.21.15.110:51816) 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.216+0000 7f2956178640 1 -- 172.21.15.110:0/363394861 learned_addr learned my addr 172.21.15.110:0/363394861 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.216+0000 7f2956979640 1 --2- 172.21.15.110:0/363394861 >> v2:172.21.15.134:3300/0 conn(0x7f29500ac4d0 0x7f295013bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.216+0000 7f2955977640 10 monclient(hunting): get_auth_request con 0x7f29500adc30 auth_method 0 2024-01-20T11:21:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.216+0000 7f2955977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.216+0000 7f2955977640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.216+0000 7f2955977640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2956979640 10 monclient(hunting): get_auth_request con 0x7f29500ac4d0 auth_method 0 2024-01-20T11:21:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2956979640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2956979640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2956979640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2955977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2955977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2955977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2956979640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2956979640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.217+0000 7f2956979640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955176640 1 -- 172.21.15.110:0/363394861 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125927354 0 0) 0x7f29500ad530 con 0x7f29500b0ba0 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955176640 10 monclient(hunting): _negotiate 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955176640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955176640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955176640 10 monclient(hunting): my global_id is 5634 2024-01-20T11:21:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955176640 1 -- 172.21.15.110:0/363394861 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f293c003520 con 0x7f29500b0ba0 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955977640 10 monclient(hunting): handle_auth_done global_id 5339 payload 274 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955977640 1 -- 172.21.15.110:0/363394861 >> v2:172.21.15.134:3300/0 conn(0x7f29500ac4d0 msgr2=0x7f295013bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955977640 1 --2- 172.21.15.110:0/363394861 >> v2:172.21.15.134:3300/0 conn(0x7f29500ac4d0 0x7f295013bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955977640 1 -- 172.21.15.110:0/363394861 >> v1:172.21.15.110:6789/0 conn(0x7f29500b0ba0 legacy=0x7f29500af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955977640 10 monclient: _finish_hunting 0 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955977640 1 monclient: found mon.noname-b 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955977640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955977640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.219+0000 7f2955977640 1 -- 172.21.15.110:0/363394861 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29500ad810 con 0x7f29500adc30 2024-01-20T11:21:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.220+0000 7f2955977640 1 --2- 172.21.15.110:0/363394861 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500adc30 0x7f29500ab630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f29400097a0 tx=0x7f294002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3befe8e0978534d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 1 -- 172.21.15.110:0/363394861 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294002e070 con 0x7f29500adc30 2024-01-20T11:21:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 10 monclient: dump: 2024-01-20T11:21:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 10 monclient: _finish_auth 0 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 10 monclient: _check_auth_tickets 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 1 -- 172.21.15.110:0/363394861 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29400315a0 con 0x7f29500adc30 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f29583dc640 5 monclient: authenticate success, global_id 5339 2024-01-20T11:21:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f29583dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f29583dc640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 1 -- 172.21.15.110:0/363394861 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940031930 con 0x7f29500adc30 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f29583dc640 4 set_mon_vals no callback set 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.224+0000 7f2955176640 10 monclient: dump: 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 10 monclient: shutdown 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 1 -- 172.21.15.110:0/363394861 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500adc30 msgr2=0x7f29500ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 1 --2- 172.21.15.110:0/363394861 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500adc30 0x7f29500ab630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f29400097a0 tx=0x7f294002b840 comp rx=0 tx=0).stop 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 1 -- 172.21.15.110:0/363394861 shutdown_connections 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 1 --2- 172.21.15.110:0/363394861 >> v2:172.21.15.134:3300/0 conn(0x7f29500ac4d0 0x7f295013bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 1 --2- 172.21.15.110:0/363394861 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500adc30 0x7f29500ab630 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 1 -- 172.21.15.110:0/363394861 >> 172.21.15.110:0/363394861 conn(0x7f29500b3e60 msgr2=0x7f29500b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 1 -- 172.21.15.110:0/363394861 shutdown_connections 2024-01-20T11:21:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.225+0000 7f29583dc640 1 -- 172.21.15.110:0/363394861 wait complete. 2024-01-20T11:21:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.226+0000 7f29583dc640 10 monclient: build_initial_monmap 2024-01-20T11:21:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.226+0000 7f29583dc640 10 monclient: monmap: 2024-01-20T11:21:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.226+0000 7f29583dc640 1 librados: starting msgr at 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.226+0000 7f29583dc640 1 librados: starting objecter 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.226+0000 7f29583dc640 1 Processor -- start 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.226+0000 7f29583dc640 1 -- start start 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.226+0000 7f29583dc640 1 librados: setting wanted keys 2024-01-20T11:21:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.226+0000 7f29583dc640 1 librados: calling monclient init 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.226+0000 7f29583dc640 10 monclient: init 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29500ac4d0 0x7f2950071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient(hunting): picked mon.noname-b con 0x7f29500ac4d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500b0ba0 0x7f29501399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient(hunting): picked mon.noname-c con 0x7f29500b0ba0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f295012e920 0x7f295012edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient(hunting): picked mon.noname-a con 0x7f295012e920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2950139f00 con 0x7f295012e920 2024-01-20T11:21:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295012f310 con 0x7f29500ac4d0 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295013a170 con 0x7f29500b0ba0 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f29583dc640 10 monclient(hunting): authenticate will time out at 1751.577408s 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f2955977640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500b0ba0 0x7f29501399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f2956979640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f295012e920 0x7f295012edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f2955977640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500b0ba0 0x7f29501399c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54842/0 (socket says 172.21.15.110:54842) 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f2955977640 1 -- 172.21.15.110:0/492796862 learned_addr learned my addr 172.21.15.110:0/492796862 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f2955977640 10 monclient(hunting): get_auth_request con 0x7f29500b0ba0 auth_method 0 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f2955977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f2955977640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.227+0000 7f2955977640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956178640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29500ac4d0 0x7f2950071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956979640 10 monclient(hunting): get_auth_request con 0x7f295012e920 auth_method 0 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956979640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956979640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956979640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956178640 10 monclient(hunting): get_auth_request con 0x7f29500ac4d0 auth_method 0 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956178640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956178640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956178640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2955977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2955977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2955977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956979640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956979640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956979640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956178640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956178640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.228+0000 7f2956178640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 10 monclient(hunting): handle_auth_done global_id 5342 payload 824 2024-01-20T11:21:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 1 -- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29500ac4d0 msgr2=0x7f2950071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29500ac4d0 0x7f2950071850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 1 -- 172.21.15.110:0/492796862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f295012e920 msgr2=0x7f295012edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f295012e920 0x7f295012edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 10 monclient: _finish_hunting 0 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 1 monclient: found mon.noname-c 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2955977640 1 -- 172.21.15.110:0/492796862 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2940009400 con 0x7f29500b0ba0 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.231+0000 7f2956979640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f295012e920 0x7f295012edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.233+0000 7f2955977640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500b0ba0 0x7f29501399c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f29400098c0 tx=0x7f29400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.233+0000 7f2946ffd640 10 client.?.objecter ms_handle_connect 0x7f29500b0ba0 2024-01-20T11:21:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.233+0000 7f2946ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.236+0000 7f2946ffd640 1 -- 172.21.15.110:0/492796862 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940032840 con 0x7f29500b0ba0 2024-01-20T11:21:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.236+0000 7f2946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.236+0000 7f2946ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.236+0000 7f2946ffd640 10 monclient: dump: 2024-01-20T11:21:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.236+0000 7f2946ffd640 10 monclient: _finish_auth 0 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.236+0000 7f2946ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.236+0000 7f2946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f2946ffd640 1 -- 172.21.15.110:0/492796862 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940031f00 con 0x7f29500b0ba0 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f2946ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f2946ffd640 1 -- 172.21.15.110:0/492796862 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940032c00 con 0x7f29500b0ba0 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f2946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f2946ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f2946ffd640 10 monclient: dump: 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f29583dc640 5 monclient: authenticate success, global_id 5342 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f29583dc640 10 monclient: _renew_subs 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f2954975640 4 set_mon_vals no callback set 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f29583dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29500b1440 con 0x7f29500b0ba0 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f29583dc640 10 client.5342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f29583dc640 10 monclient: _renew_subs 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f29583dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295013a600 con 0x7f29500b0ba0 2024-01-20T11:21:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.237+0000 7f29583dc640 1 librados: init done 2024-01-20T11:21:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.238+0000 7f29583dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.238+0000 7f29583dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.238+0000 7f29583dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.238+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295013a790 con 0x7f29500b0ba0 2024-01-20T11:21:48.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.258+0000 7f2946ffd640 1 -- 172.21.15.110:0/492796862 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f294004e430 con 0x7f29500b0ba0 2024-01-20T11:21:48.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.258+0000 7f2946ffd640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f293406c860 0x7f293406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.259+0000 7f2956178640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f293406c860 0x7f293406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.259+0000 7f2956178640 10 monclient: get_auth_request con 0x7f293406c860 auth_method 0 2024-01-20T11:21:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.259+0000 7f2956178640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f293406c860 0x7f293406ed20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f294c005880 tx=0x7f294c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:48.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.259+0000 7f2946ffd640 10 client.5342.objecter ms_handle_connect 0x7f293406c860 2024-01-20T11:21:48.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.486+0000 7f2946ffd640 1 -- 172.21.15.110:0/492796862 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f294007a490 con 0x7f29500b0ba0 2024-01-20T11:21:48.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.486+0000 7f2946ffd640 10 client.5342.objecter ms_dispatch 0x7f2950000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:48.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.486+0000 7f2946ffd640 3 client.5342.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:48.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.486+0000 7f2946ffd640 3 client.5342.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:48.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.486+0000 7f2946ffd640 20 client.5342.objecter dump_active .. 0 homeless 2024-01-20T11:21:48.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.486+0000 7f2946ffd640 1 -- 172.21.15.110:0/492796862 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f29400767d0 con 0x7f29500b0ba0 2024-01-20T11:21:48.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.486+0000 7f2946ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:48.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.486+0000 7f2946ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.655+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f295013a790 con 0x7f293406c860 2024-01-20T11:21:48.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.658+0000 7f2946ffd640 1 -- 172.21.15.110:0/492796862 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63905 (secure 0 0 0) 0x7f295013a790 con 0x7f293406c860 2024-01-20T11:21:48.659 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 10 librados: watch_flush enter 2024-01-20T11:21:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 10 librados: watch_flush exit 2024-01-20T11:21:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 20 client.5342.objecter shutdown clearing up homeless session... 2024-01-20T11:21:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 10 client.5342.objecter successfully canceled tick 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f293406c860 msgr2=0x7f293406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f293406c860 0x7f293406ed20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f294c005880 tx=0x7f294c009040 comp rx=0 tx=0).stop 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 10 monclient: shutdown 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500b0ba0 msgr2=0x7f29501399c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.660+0000 7f29583dc640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500b0ba0 0x7f29501399c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f29400098c0 tx=0x7f29400046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.661+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 shutdown_connections 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.661+0000 7f29583dc640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f293406c860 0x7f293406ed20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.661+0000 7f29583dc640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f295012e920 0x7f295012edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.661+0000 7f29583dc640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29500b0ba0 0x7f29501399c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.661+0000 7f29583dc640 1 --2- 172.21.15.110:0/492796862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29500ac4d0 0x7f2950071850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.661+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 >> 172.21.15.110:0/492796862 conn(0x7f29500b3e60 msgr2=0x7f29500ae020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.661+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 shutdown_connections 2024-01-20T11:21:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.661+0000 7f29583dc640 1 -- 172.21.15.110:0/492796862 wait complete. 2024-01-20T11:21:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.661+0000 7f29583dc640 1 librados: shutdown 2024-01-20T11:21:48.671 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-01-20T11:21:46.788230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002654"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668943+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:17:36.668943+0000","last_peered":"2024-01-20T11:17:36.668943+0000","last_clean":"2024-01-20T11:17:36.668943+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:17:36.668943+0000","last_undegraded":"2024-01-20T11:17:36.668943+0000","last_fullsized":"2024-01-20T11:17:36.668943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668390+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:17:36.668390+0000","last_peered":"2024-01-20T11:17:36.668390+0000","last_clean":"2024-01-20T11:17:36.668390+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:17:36.668390+0000","last_undegraded":"2024-01-20T11:17:36.668390+0000","last_fullsized":"2024-01-20T11:17:36.668390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:36.668683+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:17:36.668683+0000","last_peered":"2024-01-20T11:17:36.668683+0000","last_clean":"2024-01-20T11:17:36.668683+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:17:36.668683+0000","last_undegraded":"2024-01-20T11:17:36.668683+0000","last_fullsized":"2024-01-20T11:17:36.668683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":8.4689999999999994},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.766}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":7.5039999999999996},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":9.7420000000000009}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":10.955},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":10.343}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":13.643000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":8.1760000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":14.305999999999999},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":11.532999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":12.147},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":15.208}]}]},{"osd":7,"up_from":21,"seq":90194313299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":7.351},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":12.493}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":6.5670000000000002},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":11.112}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":13.228999999999999},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":16.440000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":8.9600000000000009},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":11.788}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":9.5890000000000004},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":8.3729999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":10.394}]}]},{"osd":3,"up_from":21,"seq":90194313299,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.8099999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":7.5830000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1129999999999995},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":8.4209999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":10.000999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":12.946}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":11.377000000000001},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":14.484999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":12.119},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":15.129}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.594999999999999},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.74}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444113,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":7.9020000000000001},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":7.2080000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.75},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":14.375999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.571999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":11.808999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.308},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.914999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":9.7959999999999994},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":9.2140000000000004}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":17.094000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":13.497}]}]},{"osd":1,"up_from":17,"seq":73014444114,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.3849999999999998},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":10.053000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":11.99},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":9.1229999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":14.827},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":10.644}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":5.9669999999999996},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":17.724}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":14.036},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":13.156000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":16.869},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":15.936999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444114,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":6.3079999999999998},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":11.837999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":5.7240000000000002},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":7.7350000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":8.4269999999999996},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.817}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":14.422000000000001},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":9.7759999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":9.093},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":13.422000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":15.314},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":11.082000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.8579999999999997},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.7129999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":10.914999999999999},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":13.506}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":14.173},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":5.9459999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.301},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":15.612}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":12.804},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":9.4290000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":10.002000000000001},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.827}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:48.672 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:48.672 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:48.673 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:48.673 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:48.673 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:48.673 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:48.673 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:48.673 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:48.673 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:48.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.772+0000 7fa8f86b0640 10 monclient: get_monmap_and_config 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.772+0000 7fa8f86b0640 10 monclient: build_initial_monmap 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.772+0000 7fa8f86b0640 10 monclient: monmap: 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:48.773152+0000 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:48.773152+0000 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:48.774 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:48.774 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:48.774 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:48.774 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 1 Processor -- start 2024-01-20T11:21:48.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 1 -- start start 2024-01-20T11:21:48.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 10 monclient: init 2024-01-20T11:21:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-01-20T11:21:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f0145760 0x7fa8f0145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f0145760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8f00a9d80 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8f013f8b0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f0146250 con 0x7fa8f013f8b0 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f00b20c0 con 0x7fa8f00a9d80 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00ab050 con 0x7fa8f0145760 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.773+0000 7fa8f86b0640 10 monclient(hunting): authenticate will time out at 1752.124116s 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.774+0000 7fa8f6c4d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa8f013f8b0 0x7fa8f00b2670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51826/0 (socket says 172.21.15.110:51826) 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.774+0000 7fa8f6c4d640 1 -- 172.21.15.110:0/3970831858 learned_addr learned my addr 172.21.15.110:0/3970831858 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.774+0000 7fa8f644c640 1 --2- 172.21.15.110:0/3970831858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f0145760 0x7fa8f0145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.774+0000 7fa8f644c640 10 monclient(hunting): get_auth_request con 0x7fa8f0145760 auth_method 0 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.774+0000 7fa8f644c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.774+0000 7fa8f644c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.774+0000 7fa8f644c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.775+0000 7fa8f644c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:48.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.775+0000 7fa8f644c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:48.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.775+0000 7fa8f644c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f544a640 1 -- 172.21.15.110:0/3970831858 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488488593 0 0) 0x7fa8f00b20c0 con 0x7fa8f00a9d80 2024-01-20T11:21:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f544a640 10 monclient(hunting): _negotiate 2024-01-20T11:21:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f544a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f544a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f544a640 10 monclient(hunting): my global_id is 5722 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f544a640 1 -- 172.21.15.110:0/3970831858 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8dc003520 con 0x7fa8f00a9d80 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f644c640 10 monclient(hunting): handle_auth_done global_id 5345 payload 274 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f644c640 1 -- 172.21.15.110:0/3970831858 >> v1:172.21.15.134:6789/0 conn(0x7fa8f00a9d80 legacy=0x7fa8f013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f644c640 1 -- 172.21.15.110:0/3970831858 >> v1:172.21.15.110:6789/0 conn(0x7fa8f013f8b0 legacy=0x7fa8f00b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f644c640 10 monclient: _finish_hunting 0 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f644c640 1 monclient: found mon.noname-b 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f644c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f644c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.777+0000 7fa8f644c640 1 -- 172.21.15.110:0/3970831858 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f00b2d80 con 0x7fa8f0145760 2024-01-20T11:21:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.779+0000 7fa8f644c640 1 --2- 172.21.15.110:0/3970831858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f0145760 0x7fa8f0145b40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0097d0 tx=0x7fa8ec02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b039bde7fce15f04 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.781+0000 7fa8f544a640 1 -- 172.21.15.110:0/3970831858 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec031890 con 0x7fa8f0145760 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.781+0000 7fa8f544a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.781+0000 7fa8f544a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.781+0000 7fa8f544a640 10 monclient: dump: 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:48.782 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.781+0000 7fa8f544a640 10 monclient: _finish_auth 0 2024-01-20T11:21:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.781+0000 7fa8f544a640 10 monclient: _check_auth_tickets 2024-01-20T11:21:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.781+0000 7fa8f544a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.781+0000 7fa8f86b0640 5 monclient: authenticate success, global_id 5345 2024-01-20T11:21:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.781+0000 7fa8f86b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f544a640 1 -- 172.21.15.110:0/3970831858 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ec0319f0 con 0x7fa8f0145760 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f544a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f544a640 1 -- 172.21.15.110:0/3970831858 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec031d60 con 0x7fa8f0145760 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f544a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f544a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f544a640 10 monclient: dump: 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.784 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:48.784 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:48.784 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:48.784 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:48.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 4 set_mon_vals no callback set 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 10 monclient: shutdown 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 1 -- 172.21.15.110:0/3970831858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f0145760 msgr2=0x7fa8f0145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 1 --2- 172.21.15.110:0/3970831858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f0145760 0x7fa8f0145b40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0097d0 tx=0x7fa8ec02b930 comp rx=0 tx=0).stop 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 1 -- 172.21.15.110:0/3970831858 shutdown_connections 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 1 --2- 172.21.15.110:0/3970831858 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f0145760 0x7fa8f0145b40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 1 -- 172.21.15.110:0/3970831858 >> 172.21.15.110:0/3970831858 conn(0x7fa8f00663b0 msgr2=0x7fa8f00667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 1 -- 172.21.15.110:0/3970831858 shutdown_connections 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.783+0000 7fa8f86b0640 1 -- 172.21.15.110:0/3970831858 wait complete. 2024-01-20T11:21:48.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.784+0000 7fa8f86b0640 10 monclient: build_initial_monmap 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.784+0000 7fa8f86b0640 10 monclient: monmap: 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.784+0000 7fa8f86b0640 1 librados: starting msgr at 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.784+0000 7fa8f86b0640 1 librados: starting objecter 2024-01-20T11:21:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 Processor -- start 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 -- start start 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 librados: setting wanted keys 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 librados: calling monclient init 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient: init 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f00a9d80 0x7fa8f0142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f00a9d80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f013f8b0 0x7fa8f0137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f013f8b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8f0145760 0x7fa8f00b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:48.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f0145760 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f0138150 con 0x7fa8f0145760 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f0137230 con 0x7fa8f00a9d80 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f0137510 con 0x7fa8f013f8b0 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.785+0000 7fa8f86b0640 10 monclient(hunting): authenticate will time out at 1752.135773s 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f013f8b0 0x7fa8f0137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f6c4d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8f0145760 0x7fa8f00b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f644c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f00a9d80 0x7fa8f0142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:48.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f013f8b0 0x7fa8f0137c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54854/0 (socket says 172.21.15.110:54854) 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 1 -- 172.21.15.110:0/2961656048 learned_addr learned my addr 172.21.15.110:0/2961656048 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 10 monclient(hunting): get_auth_request con 0x7fa8f013f8b0 auth_method 0 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f6c4d640 10 monclient(hunting): get_auth_request con 0x7fa8f0145760 auth_method 0 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f6c4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f6c4d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f6c4d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f644c640 10 monclient(hunting): get_auth_request con 0x7fa8f00a9d80 auth_method 0 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f644c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f644c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:48.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f644c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:48.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:48.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:48.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.786+0000 7fa8f5c4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:48.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.787+0000 7fa8f6c4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:48.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.787+0000 7fa8f6c4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:48.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.787+0000 7fa8f6c4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:48.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.787+0000 7fa8f644c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:48.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.787+0000 7fa8f644c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:48.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.787+0000 7fa8f644c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:48.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 10 monclient(hunting): handle_auth_done global_id 5348 payload 824 2024-01-20T11:21:48.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 1 -- 172.21.15.110:0/2961656048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f00a9d80 msgr2=0x7fa8f0142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f00a9d80 0x7fa8f0142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 1 -- 172.21.15.110:0/2961656048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8f0145760 msgr2=0x7fa8f00b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:48.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8f0145760 0x7fa8f00b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:48.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 10 monclient: _finish_hunting 0 2024-01-20T11:21:48.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 1 monclient: found mon.noname-c 2024-01-20T11:21:48.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:48.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.789+0000 7fa8f5c4b640 1 -- 172.21.15.110:0/2961656048 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec009400 con 0x7fa8f013f8b0 2024-01-20T11:21:48.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.792+0000 7fa8f5c4b640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f013f8b0 0x7fa8f0137c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c450 tx=0x7fa8e000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:48.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.792+0000 7fa8e6ffd640 10 client.?.objecter ms_handle_connect 0x7fa8f013f8b0 2024-01-20T11:21:48.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.792+0000 7fa8e6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:48.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 1 -- 172.21.15.110:0/2961656048 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00052d0 con 0x7fa8f013f8b0 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 10 monclient: dump: 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:48.798 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 1 -- 172.21.15.110:0/2961656048 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e0005430 con 0x7fa8f013f8b0 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f86b0640 5 monclient: authenticate success, global_id 5348 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 1 -- 172.21.15.110:0/2961656048 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0012870 con 0x7fa8f013f8b0 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:48.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8e6ffd640 10 monclient: dump: 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f86b0640 10 monclient: _renew_subs 2024-01-20T11:21:48.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f86b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f4c49640 4 set_mon_vals no callback set 2024-01-20T11:21:48.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f01377f0 con 0x7fa8f013f8b0 2024-01-20T11:21:48.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f86b0640 10 client.5348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:48.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f86b0640 10 monclient: _renew_subs 2024-01-20T11:21:48.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f86b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f00b8360 con 0x7fa8f013f8b0 2024-01-20T11:21:48.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.797+0000 7fa8f86b0640 1 librados: init done 2024-01-20T11:21:48.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.798+0000 7fa8f86b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:48.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.798+0000 7fa8f86b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:48.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.798+0000 7fa8f86b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:48.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:48.798+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f00b8550 con 0x7fa8f013f8b0 2024-01-20T11:21:49.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.044+0000 7fa8e6ffd640 1 -- 172.21.15.110:0/2961656048 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa8e00129d0 con 0x7fa8f013f8b0 2024-01-20T11:21:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.045+0000 7fa8e6ffd640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa8d406c8b0 0x7fa8d406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.045+0000 7fa8f644c640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa8d406c8b0 0x7fa8d406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.045+0000 7fa8f644c640 10 monclient: get_auth_request con 0x7fa8d406c8b0 auth_method 0 2024-01-20T11:21:49.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.046+0000 7fa8e6ffd640 10 client.5348.objecter ms_handle_connect 0x7fa8d406c8b0 2024-01-20T11:21:49.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.046+0000 7fa8f644c640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa8d406c8b0 0x7fa8d406ed70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec036000 tx=0x7fa8ec03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:49.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.057+0000 7fa8e6ffd640 1 -- 172.21.15.110:0/2961656048 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa8e00586c0 con 0x7fa8f013f8b0 2024-01-20T11:21:49.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.057+0000 7fa8e6ffd640 10 client.5348.objecter ms_dispatch 0x7fa8f0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:49.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.057+0000 7fa8e6ffd640 3 client.5348.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.057+0000 7fa8e6ffd640 3 client.5348.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.057+0000 7fa8e6ffd640 20 client.5348.objecter dump_active .. 0 homeless 2024-01-20T11:21:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.057+0000 7fa8e6ffd640 1 -- 172.21.15.110:0/2961656048 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa8e0054a00 con 0x7fa8f013f8b0 2024-01-20T11:21:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.057+0000 7fa8e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:49.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.057+0000 7fa8e6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:49.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.319+0000 7fa8f86b0640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:49.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.319+0000 7fa8f86b0640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:49.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.319+0000 7fa8f86b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:49.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.319+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa8f00b8550 con 0x7fa8f013f8b0 2024-01-20T11:21:49.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.324+0000 7fa8e6ffd640 1 -- 172.21.15.110:0/2961656048 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fa8f00b8550 con 0x7fa8f013f8b0 2024-01-20T11:21:49.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.324+0000 7fa8e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:49.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.324+0000 7fa8e6ffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:49.324 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:49.324 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":443,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:49.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 10 librados: watch_flush enter 2024-01-20T11:21:49.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 10 librados: watch_flush exit 2024-01-20T11:21:49.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 20 client.5348.objecter shutdown clearing up homeless session... 2024-01-20T11:21:49.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 10 client.5348.objecter successfully canceled tick 2024-01-20T11:21:49.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa8d406c8b0 msgr2=0x7fa8d406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa8d406c8b0 0x7fa8d406ed70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec036000 tx=0x7fa8ec03d040 comp rx=0 tx=0).stop 2024-01-20T11:21:49.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 10 monclient: shutdown 2024-01-20T11:21:49.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:49.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f013f8b0 msgr2=0x7fa8f0137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f013f8b0 0x7fa8f0137c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c450 tx=0x7fa8e000c910 comp rx=0 tx=0).stop 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 shutdown_connections 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa8d406c8b0 0x7fa8d406ed70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8f0145760 0x7fa8f00b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8f013f8b0 0x7fa8f0137c10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 --2- 172.21.15.110:0/2961656048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f00a9d80 0x7fa8f0142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 >> 172.21.15.110:0/2961656048 conn(0x7fa8f00663b0 msgr2=0x7fa8f0073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 shutdown_connections 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 -- 172.21.15.110:0/2961656048 wait complete. 2024-01-20T11:21:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.326+0000 7fa8f86b0640 1 librados: shutdown 2024-01-20T11:21:49.337 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:49.337 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.437+0000 7ffacf705640 10 monclient: get_monmap_and_config 2024-01-20T11:21:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.437+0000 7ffacf705640 10 monclient: build_initial_monmap 2024-01-20T11:21:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.437+0000 7ffacf705640 10 monclient: monmap: 2024-01-20T11:21:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:49.438177+0000 2024-01-20T11:21:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:49.438177+0000 2024-01-20T11:21:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 1 Processor -- start 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 1 -- start start 2024-01-20T11:21:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 10 monclient: init 2024-01-20T11:21:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ffac806bb00 0x7ffac80b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 10 monclient(hunting): picked mon.noname-c con 0x7ffac806bb00 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffac80af790 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffac80b0900 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac806c220 con 0x7ffac80b0900 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac8146590 con 0x7ffac80af790 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac806bee0 con 0x7ffac806bb00 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.438+0000 7ffacf705640 10 monclient(hunting): authenticate will time out at 1752.789152s 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.439+0000 7ffacdca2640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ffac80b0900 0x7ffac8145e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60490/0 (socket says 172.21.15.110:60490) 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.439+0000 7ffacdca2640 1 -- 172.21.15.110:0/1596806186 learned_addr learned my addr 172.21.15.110:0/1596806186 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.439+0000 7ffacd4a1640 1 --2- 172.21.15.110:0/1596806186 >> v2:172.21.15.134:3300/0 conn(0x7ffac806bb00 0x7ffac80b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.440+0000 7ffacd4a1640 10 monclient(hunting): get_auth_request con 0x7ffac806bb00 auth_method 0 2024-01-20T11:21:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.440+0000 7ffacd4a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.440+0000 7ffacd4a1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.440+0000 7ffacd4a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.440+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.440+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.440+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.442+0000 7ffabffff640 1 -- 172.21.15.110:0/1596806186 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964436165 0 0) 0x7ffac806c220 con 0x7ffac80b0900 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.442+0000 7ffabffff640 10 monclient(hunting): _negotiate 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.442+0000 7ffabffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.442+0000 7ffabffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.442+0000 7ffabffff640 10 monclient(hunting): my global_id is 5646 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.442+0000 7ffabffff640 1 -- 172.21.15.110:0/1596806186 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab4003520 con 0x7ffac80b0900 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.443+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_done global_id 5728 payload 274 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.443+0000 7ffacd4a1640 1 -- 172.21.15.110:0/1596806186 >> v1:172.21.15.134:6789/0 conn(0x7ffac80af790 legacy=0x7ffac8145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.443+0000 7ffacd4a1640 1 -- 172.21.15.110:0/1596806186 >> v1:172.21.15.110:6789/0 conn(0x7ffac80b0900 legacy=0x7ffac8145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.443+0000 7ffacd4a1640 10 monclient: _finish_hunting 0 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.443+0000 7ffacd4a1640 1 monclient: found mon.noname-c 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.443+0000 7ffacd4a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.443+0000 7ffacd4a1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:21:49.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.443+0000 7ffacd4a1640 1 -- 172.21.15.110:0/1596806186 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac80ada30 con 0x7ffac806bb00 2024-01-20T11:21:49.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.447+0000 7ffacd4a1640 1 --2- 172.21.15.110:0/1596806186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac806bb00 0x7ffac80b03c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffac40097a0 tx=0x7ffac402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=370f9f41e9038adb server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.450+0000 7ffabffff640 1 -- 172.21.15.110:0/1596806186 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac4031830 con 0x7ffac806bb00 2024-01-20T11:21:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.450+0000 7ffabffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:49.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.450+0000 7ffabffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.450+0000 7ffabffff640 10 monclient: dump: 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffabffff640 10 monclient: _finish_auth 0 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffabffff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffabffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffabffff640 1 -- 172.21.15.110:0/1596806186 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac4031990 con 0x7ffac806bb00 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 5 monclient: authenticate success, global_id 5728 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffabffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffabffff640 1 -- 172.21.15.110:0/1596806186 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac4031d20 con 0x7ffac806bb00 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 4 set_mon_vals no callback set 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffabffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffabffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffabffff640 10 monclient: dump: 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 10 monclient: shutdown 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 1 -- 172.21.15.110:0/1596806186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac806bb00 msgr2=0x7ffac80b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 1 --2- 172.21.15.110:0/1596806186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac806bb00 0x7ffac80b03c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffac40097a0 tx=0x7ffac402b8d0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 1 -- 172.21.15.110:0/1596806186 shutdown_connections 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 1 --2- 172.21.15.110:0/1596806186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac806bb00 0x7ffac80b03c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 1 -- 172.21.15.110:0/1596806186 >> 172.21.15.110:0/1596806186 conn(0x7ffac8066d60 msgr2=0x7ffac80b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 1 -- 172.21.15.110:0/1596806186 shutdown_connections 2024-01-20T11:21:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.451+0000 7ffacf705640 1 -- 172.21.15.110:0/1596806186 wait complete. 2024-01-20T11:21:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.452+0000 7ffacf705640 10 monclient: build_initial_monmap 2024-01-20T11:21:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.452+0000 7ffacf705640 10 monclient: monmap: 2024-01-20T11:21:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.452+0000 7ffacf705640 1 librados: starting msgr at 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.452+0000 7ffacf705640 1 librados: starting objecter 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.452+0000 7ffacf705640 1 Processor -- start 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.452+0000 7ffacf705640 1 -- start start 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.452+0000 7ffacf705640 1 librados: setting wanted keys 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.452+0000 7ffacf705640 1 librados: calling monclient init 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.452+0000 7ffacf705640 10 monclient: init 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:21:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 0x7ffac8141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient(hunting): picked mon.noname-c con 0x7ffac806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 0x7ffac80713d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient(hunting): picked mon.noname-a con 0x7ffac80af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac80b0900 0x7ffac806f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient(hunting): picked mon.noname-b con 0x7ffac80b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8070eb0 con 0x7ffac80af790 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8070ad0 con 0x7ffac80b0900 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8070c90 con 0x7ffac806bb00 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.453+0000 7ffacf705640 10 monclient(hunting): authenticate will time out at 1752.803842s 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffaccca0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 0x7ffac80713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:49.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacd4a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 0x7ffac8141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacdca2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac80b0900 0x7ffac806f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffaccca0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 0x7ffac80713d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40808/0 (socket says 172.21.15.110:40808) 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffaccca0640 1 -- 172.21.15.110:0/713711841 learned_addr learned my addr 172.21.15.110:0/713711841 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffaccca0640 10 monclient(hunting): get_auth_request con 0x7ffac80af790 auth_method 0 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffaccca0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffaccca0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffaccca0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacd4a1640 10 monclient(hunting): get_auth_request con 0x7ffac806bb00 auth_method 0 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacd4a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacd4a1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacd4a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacdca2640 10 monclient(hunting): get_auth_request con 0x7ffac80b0900 auth_method 0 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacdca2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacdca2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.454+0000 7ffacdca2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.455+0000 7ffaccca0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.455+0000 7ffaccca0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.455+0000 7ffaccca0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.455+0000 7ffacdca2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.455+0000 7ffacdca2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.455+0000 7ffacdca2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.455+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.455+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.455+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 10 monclient(hunting): handle_auth_done global_id 5731 payload 824 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 1 -- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 msgr2=0x7ffac8141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 0x7ffac8141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 1 -- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 msgr2=0x7ffac80713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 0x7ffac80713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 10 monclient: _finish_hunting 0 2024-01-20T11:21:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 1 monclient: found mon.noname-b 2024-01-20T11:21:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.458+0000 7ffacdca2640 1 -- 172.21.15.110:0/713711841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac4009400 con 0x7ffac80b0900 2024-01-20T11:21:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.460+0000 7ffacdca2640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac80b0900 0x7ffac806f520 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ffac000cad0 tx=0x7ffac000ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.460+0000 7ffabdffb640 10 client.?.objecter ms_handle_connect 0x7ffac80b0900 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.460+0000 7ffabdffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffacdca2640 1 -- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac80b0900 msgr2=0x7ffac806f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffacdca2640 1 -- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac80b0900 msgr2=0x7ffac806f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffacdca2640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac80b0900 0x7ffac806f520 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ffac000cad0 tx=0x7ffac000ced0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffacdca2640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac80b0900 0x7ffac806f520 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ffac000cad0 tx=0x7ffac000ced0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7ffac000ab70 con 0x7ffac80b0900 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient: dump: 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:49.466 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient: _finish_auth 0 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient: _check_auth_tickets 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac80b0900 msgr2=0x7ffac806f520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffac80b0900 0x7ffac806f520 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 0x7ffaac0070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient(hunting): picked mon.c con 0x7ffac806bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 0x7ffaac009b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient(hunting): picked mon.a con 0x7ffac80af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffaac00a560 0x7ffaac00c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient(hunting): picked mon.b con 0x7ffaac00a560 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac00d4a0 con 0x7ffac80af790 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac00d5e0 con 0x7ffaac00a560 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac00d720 con 0x7ffac806bb00 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffabdffb640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffaccca0640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 0x7ffaac009b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.464+0000 7ffacd4a1640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 0x7ffaac0070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffacd4a1640 10 monclient(hunting): get_auth_request con 0x7ffac806bb00 auth_method 0 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffacd4a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffacd4a1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffacd4a1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffaccca0640 10 monclient(hunting): get_auth_request con 0x7ffac80af790 auth_method 0 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffaccca0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffaccca0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffaccca0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffaccca0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffaccca0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.465+0000 7ffaccca0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.466+0000 7ffacdca2640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffaac00a560 0x7ffaac00c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.466+0000 7ffacdca2640 10 monclient(hunting): get_auth_request con 0x7ffaac00a560 auth_method 0 2024-01-20T11:21:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.466+0000 7ffacdca2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.466+0000 7ffacdca2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.466+0000 7ffacdca2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:21:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.466+0000 7ffacdca2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.466+0000 7ffacdca2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:49.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.466+0000 7ffacdca2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 10 monclient(hunting): handle_auth_done global_id 5731 payload 293 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 1 -- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffaac00a560 msgr2=0x7ffaac00c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffaac00a560 0x7ffaac00c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 1 -- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 msgr2=0x7ffaac009b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 0x7ffaac009b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 10 monclient: _finish_hunting 0 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 1 monclient: found mon.c 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 1 -- 172.21.15.110:0/713711841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7ffaac00d950 con 0x7ffac806bb00 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 10 monclient: _finish_auth 0 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 10 monclient: _check_auth_tickets 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacd4a1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacf705640 5 monclient: authenticate success, global_id 5731 2024-01-20T11:21:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacf705640 10 monclient: _renew_subs 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacf705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac813c540 con 0x7ffac806bb00 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacf705640 10 client.5731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacf705640 10 monclient: _renew_subs 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacf705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac811dcd0 con 0x7ffac806bb00 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.467+0000 7ffacf705640 1 librados: init done 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.468+0000 7ffacf705640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.468+0000 7ffacf705640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.468+0000 7ffacf705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.468+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac80710f0 con 0x7ffac806bb00 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.469+0000 7ffabdffb640 10 client.5731.objecter ms_handle_connect 0x7ffac806bb00 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.469+0000 7ffabdffb640 10 client.5731.objecter resend_mon_ops 2024-01-20T11:21:49.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.469+0000 7ffacd4a1640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 0x7ffaac0070d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffac4033000 tx=0x7ffac0001970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:49.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.715+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac403f040 con 0x7ffac806bb00 2024-01-20T11:21:49.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.715+0000 7ffabdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:49.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.715+0000 7ffabdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:49.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.715+0000 7ffabdffb640 10 monclient: dump: 2024-01-20T11:21:49.715 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:49.715 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:49.715 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.715+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac4031d00 con 0x7ffac806bb00 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.715+0000 7ffabdffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:49.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.715+0000 7ffabf7fe640 4 set_mon_vals no callback set 2024-01-20T11:21:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.716+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ffac404e430 con 0x7ffac806bb00 2024-01-20T11:21:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.716+0000 7ffabdffb640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffac80b0900 0x7ffaac076ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.716+0000 7ffaccca0640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffac80b0900 0x7ffaac076ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.716+0000 7ffaccca0640 10 monclient: get_auth_request con 0x7ffac80b0900 auth_method 0 2024-01-20T11:21:49.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.717+0000 7ffaccca0640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffac80b0900 0x7ffaac076ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ffac8142280 tx=0x7ffab8009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:49.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.717+0000 7ffabdffb640 10 client.5731.objecter ms_handle_connect 0x7ffac80b0900 2024-01-20T11:21:49.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.728+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 <== mon.2 v2:172.21.15.110:3301/0 4 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffac40794e0 con 0x7ffac806bb00 2024-01-20T11:21:49.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.728+0000 7ffabdffb640 10 client.5731.objecter ms_dispatch 0x7ffac8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:49.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.728+0000 7ffabdffb640 3 client.5731.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:49.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.728+0000 7ffabdffb640 3 client.5731.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:49.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.728+0000 7ffabdffb640 20 client.5731.objecter dump_active .. 0 homeless 2024-01-20T11:21:49.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.728+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 <== mon.2 v2:172.21.15.110:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ffac4075820 con 0x7ffac806bb00 2024-01-20T11:21:49.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.728+0000 7ffabdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:49.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.728+0000 7ffabdffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.910+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffac806d380 con 0x7ffac80b0900 2024-01-20T11:21:49.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.913+0000 7ffabdffb640 1 -- 172.21.15.110:0/713711841 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63897 (secure 0 0 0) 0x7ffab8001560 con 0x7ffac80b0900 2024-01-20T11:21:49.914 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:49.914 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 10 librados: watch_flush enter 2024-01-20T11:21:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 10 librados: watch_flush exit 2024-01-20T11:21:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 20 client.5731.objecter shutdown clearing up homeless session... 2024-01-20T11:21:49.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 10 client.5731.objecter successfully canceled tick 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffac80b0900 msgr2=0x7ffaac076ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffac80b0900 0x7ffaac076ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ffac8142280 tx=0x7ffab8009040 comp rx=0 tx=0).stop 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 10 monclient: shutdown 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 msgr2=0x7ffaac0070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 0x7ffaac0070d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffac4033000 tx=0x7ffac0001970 comp rx=0 tx=0).stop 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 shutdown_connections 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffac80b0900 0x7ffaac076ec0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffaac00a560 0x7ffaac00c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffac80af790 0x7ffaac009b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 --2- 172.21.15.110:0/713711841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffac806bb00 0x7ffaac0070d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.915+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 >> 172.21.15.110:0/713711841 conn(0x7ffac8066d60 msgr2=0x7ffac8145e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:49.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.916+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 shutdown_connections 2024-01-20T11:21:49.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.916+0000 7ffacf705640 1 -- 172.21.15.110:0/713711841 wait complete. 2024-01-20T11:21:49.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:49.916+0000 7ffacf705640 1 librados: shutdown 2024-01-20T11:21:49.926 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-01-20T11:21:48.788735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280597,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":7.0369999999999999},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.1479999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":9.3710000000000004},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":15.375999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.826000000000001},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":12.186}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":8.3539999999999992},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":10.144}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":16.167000000000002},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":14.753}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":10.949},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.026}]}]},{"osd":7,"up_from":21,"seq":90194313299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":7.351},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":12.493}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":6.5670000000000002},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":11.112}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":13.228999999999999},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":16.440000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":8.9600000000000009},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":11.788}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":9.5890000000000004},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":8.3729999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":10.394}]}]},{"osd":3,"up_from":21,"seq":90194313300,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.2889999999999997},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":9.2780000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.909000000000001},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":14.319000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":8.3200000000000003},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":13.250999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":15.097},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":10.000999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":7.5990000000000002},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":11.526}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.023},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":12.663}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444114,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":4.9349999999999996},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":8.1839999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.851},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.071}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.827},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":10.555}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.683},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.228999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":14.364000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":12.997999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":13.640000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":15.138999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444115,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":15.151999999999999},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":12.741}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":6.5220000000000002},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":7.2830000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":15.961},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":8.6020000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":7.7130000000000001},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.707000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.771000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":9.2710000000000008}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":10.733000000000001},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":13.628}]}]},{"osd":0,"up_from":17,"seq":73014444115,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":10.146000000000001},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.359}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.2599999999999998},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":11.964}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":15.686999999999999},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":10.802}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.430999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":9.2300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":18.495999999999999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.395}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":17.239999999999998},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":14.756}]}]},{"osd":6,"up_from":15,"seq":64424509523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.8579999999999997},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.7129999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":10.914999999999999},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":13.506}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":14.173},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":5.9459999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.301},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":15.612}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":12.804},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":9.4290000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":10.002000000000001},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.827}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:51.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~174s 2024-01-20T11:21:52.928 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.027+0000 7fad22df0640 10 monclient: get_monmap_and_config 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.027+0000 7fad22df0640 10 monclient: build_initial_monmap 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.027+0000 7fad22df0640 10 monclient: monmap: 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:53.027786+0000 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:53.027786+0000 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:53.027 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:53.028 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:53.028 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:53.028 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.027+0000 7fad22df0640 1 Processor -- start 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.027+0000 7fad22df0640 1 -- start start 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.027+0000 7fad22df0640 10 monclient: init 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad1c0b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad1c0b1ed0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fad1c146090 0x7fad1c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 10 monclient(hunting): picked mon.noname-a con 0x7fad1c146090 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0aade0 con 0x7fad1c0b4d10 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c145460 con 0x7fad1c0b1ed0 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:53.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1c0b50f0 con 0x7fad1c146090 2024-01-20T11:21:53.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:53.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.028+0000 7fad22df0640 10 monclient(hunting): authenticate will time out at 1756.378708s 2024-01-20T11:21:53.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.029+0000 7fad2138d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fad1c146090 0x7fad1c0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:53.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.029+0000 7fad1bfff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fad1c0b1ed0 0x7fad1c145980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:49426/0 (socket says 172.21.15.110:49426) 2024-01-20T11:21:53.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.029+0000 7fad1bfff640 1 -- 172.21.15.110:0/2040753652 learned_addr learned my addr 172.21.15.110:0/2040753652 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:53.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.030+0000 7fad2138d640 10 monclient(hunting): get_auth_request con 0x7fad1c146090 auth_method 0 2024-01-20T11:21:53.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.030+0000 7fad2138d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:53.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.030+0000 7fad2138d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.030+0000 7fad2138d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.030+0000 7fad2138d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:53.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.030+0000 7fad2138d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:53.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.030+0000 7fad2138d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:53.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.032+0000 7fad1b7fe640 1 -- 172.21.15.110:0/2040753652 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170674482 0 0) 0x7fad1c145460 con 0x7fad1c0b1ed0 2024-01-20T11:21:53.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.032+0000 7fad1b7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:21:53.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.032+0000 7fad1b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.032+0000 7fad1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.032+0000 7fad1b7fe640 10 monclient(hunting): my global_id is 5734 2024-01-20T11:21:53.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.032+0000 7fad1b7fe640 1 -- 172.21.15.110:0/2040753652 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad0c003520 con 0x7fad1c0b1ed0 2024-01-20T11:21:53.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.033+0000 7fad1b7fe640 1 -- 172.21.15.110:0/2040753652 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623799486 0 0) 0x7fad1c0aade0 con 0x7fad1c0b4d10 2024-01-20T11:21:53.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.033+0000 7fad1b7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:21:53.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.033+0000 7fad1b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.033+0000 7fad1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.033+0000 7fad1b7fe640 10 monclient(hunting): my global_id is 5655 2024-01-20T11:21:53.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.033+0000 7fad1b7fe640 1 -- 172.21.15.110:0/2040753652 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad1c145460 con 0x7fad1c0b4d10 2024-01-20T11:21:53.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.035+0000 7fad2138d640 10 monclient(hunting): handle_auth_done global_id 5652 payload 274 2024-01-20T11:21:53.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.035+0000 7fad2138d640 1 -- 172.21.15.110:0/2040753652 >> v1:172.21.15.134:6789/0 conn(0x7fad1c0b1ed0 legacy=0x7fad1c145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.035+0000 7fad2138d640 1 -- 172.21.15.110:0/2040753652 >> v1:172.21.15.110:6789/0 conn(0x7fad1c0b4d10 legacy=0x7fad1c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.035+0000 7fad2138d640 10 monclient: _finish_hunting 0 2024-01-20T11:21:53.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.035+0000 7fad2138d640 1 monclient: found mon.noname-a 2024-01-20T11:21:53.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.035+0000 7fad2138d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:53.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.035+0000 7fad2138d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:53.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.035+0000 7fad2138d640 1 -- 172.21.15.110:0/2040753652 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0b2660 con 0x7fad1c146090 2024-01-20T11:21:53.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.038+0000 7fad2138d640 1 --2- 172.21.15.110:0/2040753652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad1c146090 0x7fad1c0b3da0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fad14009dc0 tx=0x7fad1402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af7e395651e0980 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 1 -- 172.21.15.110:0/2040753652 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1402cc40 con 0x7fad1c146090 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 10 monclient: dump: 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:53.043 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 1 -- 172.21.15.110:0/2040753652 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1402cda0 con 0x7fad1c146090 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 5 monclient: authenticate success, global_id 5652 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 1 -- 172.21.15.110:0/2040753652 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14034660 con 0x7fad1c146090 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 4 set_mon_vals no callback set 2024-01-20T11:21:53.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad1b7fe640 10 monclient: dump: 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:53.045 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 10 monclient: shutdown 2024-01-20T11:21:53.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:53.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 1 -- 172.21.15.110:0/2040753652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad1c146090 msgr2=0x7fad1c0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 1 --2- 172.21.15.110:0/2040753652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad1c146090 0x7fad1c0b3da0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fad14009dc0 tx=0x7fad1402be90 comp rx=0 tx=0).stop 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 1 -- 172.21.15.110:0/2040753652 shutdown_connections 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 1 --2- 172.21.15.110:0/2040753652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad1c146090 0x7fad1c0b3da0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 1 -- 172.21.15.110:0/2040753652 >> 172.21.15.110:0/2040753652 conn(0x7fad1c066590 msgr2=0x7fad1c0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 1 -- 172.21.15.110:0/2040753652 shutdown_connections 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.043+0000 7fad22df0640 1 -- 172.21.15.110:0/2040753652 wait complete. 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.044+0000 7fad22df0640 10 monclient: build_initial_monmap 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.044+0000 7fad22df0640 10 monclient: monmap: 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:53.049 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.044+0000 7fad22df0640 1 librados: starting msgr at 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.044+0000 7fad22df0640 1 librados: starting objecter 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.044+0000 7fad22df0640 1 Processor -- start 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 1 -- start start 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 1 librados: setting wanted keys 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 1 librados: calling monclient init 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient: init 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:21:53.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad1c0b1ed0 0x7fad1c1daea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient(hunting): picked mon.noname-b con 0x7fad1c0b1ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad1c0b4d10 0x7fad1c1db3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient(hunting): picked mon.noname-a con 0x7fad1c0b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad1c146090 0x7fad1c13b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient(hunting): picked mon.noname-c con 0x7fad1c146090 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c137350 con 0x7fad1c0b4d10 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c1db920 con 0x7fad1c0b1ed0 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0abd40 con 0x7fad1c146090 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.045+0000 7fad22df0640 10 monclient(hunting): authenticate will time out at 1756.395958s 2024-01-20T11:21:53.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad2138d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad1c146090 0x7fad1c13b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad20b8c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad1c0b1ed0 0x7fad1c1daea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad2138d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad1c146090 0x7fad1c13b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:39406/0 (socket says 172.21.15.110:39406) 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad2138d640 1 -- 172.21.15.110:0/750117963 learned_addr learned my addr 172.21.15.110:0/750117963 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad2138d640 10 monclient(hunting): get_auth_request con 0x7fad1c146090 auth_method 0 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad2138d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad2138d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad2138d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad20b8c640 10 monclient(hunting): get_auth_request con 0x7fad1c0b1ed0 auth_method 0 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad20b8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad20b8c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.046+0000 7fad20b8c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.047+0000 7fad2138d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:53.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.047+0000 7fad2138d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:53.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.047+0000 7fad2138d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:53.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.047+0000 7fad20b8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:53.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.047+0000 7fad20b8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:53.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.047+0000 7fad20b8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 10 monclient(hunting): handle_auth_done global_id 5354 payload 824 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 1 -- 172.21.15.110:0/750117963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad1c0b1ed0 msgr2=0x7fad1c1daea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad1c0b1ed0 0x7fad1c1daea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 1 -- 172.21.15.110:0/750117963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad1c0b4d10 msgr2=0x7fad1c1db3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad1c0b4d10 0x7fad1c1db3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 10 monclient: _finish_hunting 0 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 1 monclient: found mon.noname-c 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.050+0000 7fad2138d640 1 -- 172.21.15.110:0/750117963 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad14009a70 con 0x7fad1c146090 2024-01-20T11:21:53.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.052+0000 7fad2138d640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad1c146090 0x7fad1c13b2b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fad14037040 tx=0x7fad14004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:53.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.052+0000 7fad197fa640 10 client.?.objecter ms_handle_connect 0x7fad1c146090 2024-01-20T11:21:53.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.052+0000 7fad197fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 1 -- 172.21.15.110:0/750117963 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14040650 con 0x7fad1c146090 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 10 monclient: dump: 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.056 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 10 monclient: _finish_auth 0 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 10 monclient: _check_auth_tickets 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 1 -- 172.21.15.110:0/750117963 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad140407b0 con 0x7fad1c146090 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad22df0640 5 monclient: authenticate success, global_id 5354 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad22df0640 10 monclient: _renew_subs 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad22df0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1c13ed40 con 0x7fad1c146090 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 1 -- 172.21.15.110:0/750117963 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14040b60 con 0x7fad1c146090 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:53.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad197fa640 10 monclient: dump: 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad22df0640 10 client.5354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad22df0640 10 monclient: _renew_subs 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad22df0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:53.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c13b9d0 con 0x7fad1c146090 2024-01-20T11:21:53.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad22df0640 1 librados: init done 2024-01-20T11:21:53.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.055+0000 7fad1affd640 4 set_mon_vals no callback set 2024-01-20T11:21:53.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.057+0000 7fad22df0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:53.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.057+0000 7fad22df0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:53.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.057+0000 7fad22df0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:53.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.057+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1c13af70 con 0x7fad1c146090 2024-01-20T11:21:53.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.078+0000 7fad197fa640 1 -- 172.21.15.110:0/750117963 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fad14040cc0 con 0x7fad1c146090 2024-01-20T11:21:53.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.078+0000 7fad197fa640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fad0006c8b0 0x7fad0006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.079+0000 7fad20b8c640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fad0006c8b0 0x7fad0006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:53.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.079+0000 7fad20b8c640 10 monclient: get_auth_request con 0x7fad0006c8b0 auth_method 0 2024-01-20T11:21:53.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.079+0000 7fad20b8c640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fad0006c8b0 0x7fad0006ed70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fad10007fd0 tx=0x7fad10009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:53.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.079+0000 7fad197fa640 10 client.5354.objecter ms_handle_connect 0x7fad0006c8b0 2024-01-20T11:21:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.319+0000 7fad197fa640 1 -- 172.21.15.110:0/750117963 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fad1407b220 con 0x7fad1c146090 2024-01-20T11:21:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.319+0000 7fad197fa640 10 client.5354.objecter ms_dispatch 0x7fad1c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.319+0000 7fad197fa640 3 client.5354.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.319+0000 7fad197fa640 3 client.5354.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.319+0000 7fad197fa640 20 client.5354.objecter dump_active .. 0 homeless 2024-01-20T11:21:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.319+0000 7fad197fa640 1 -- 172.21.15.110:0/750117963 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fad14077560 con 0x7fad1c146090 2024-01-20T11:21:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.319+0000 7fad197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.319+0000 7fad197fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:53.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.523+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad1c1dba60 con 0x7fad0006c8b0 2024-01-20T11:21:53.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.526+0000 7fad197fa640 1 -- 172.21.15.110:0/750117963 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63974 (secure 0 0 0) 0x7fad1c1dba60 con 0x7fad0006c8b0 2024-01-20T11:21:53.527 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:53.527 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:53.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 10 librados: watch_flush enter 2024-01-20T11:21:53.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 10 librados: watch_flush exit 2024-01-20T11:21:53.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 20 client.5354.objecter shutdown clearing up homeless session... 2024-01-20T11:21:53.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 10 client.5354.objecter successfully canceled tick 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fad0006c8b0 msgr2=0x7fad0006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fad0006c8b0 0x7fad0006ed70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fad10007fd0 tx=0x7fad10009040 comp rx=0 tx=0).stop 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 10 monclient: shutdown 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad1c146090 msgr2=0x7fad1c13b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad1c146090 0x7fad1c13b2b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fad14037040 tx=0x7fad14004a40 comp rx=0 tx=0).stop 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 shutdown_connections 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fad0006c8b0 0x7fad0006ed70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad1c146090 0x7fad1c13b2b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad1c0b4d10 0x7fad1c1db3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 --2- 172.21.15.110:0/750117963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad1c0b1ed0 0x7fad1c1daea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 >> 172.21.15.110:0/750117963 conn(0x7fad1c066590 msgr2=0x7fad1c143260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:53.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 shutdown_connections 2024-01-20T11:21:53.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 -- 172.21.15.110:0/750117963 wait complete. 2024-01-20T11:21:53.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.528+0000 7fad22df0640 1 librados: shutdown 2024-01-20T11:21:53.539 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-01-20T11:21:52.789730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002908"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280597,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":7.0369999999999999},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.1479999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":9.3710000000000004},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":15.375999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.826000000000001},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":12.186}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":8.3539999999999992},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":10.144}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":16.167000000000002},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":14.753}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":10.949},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.026}]}]},{"osd":7,"up_from":21,"seq":90194313300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.3220000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":9.8490000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":12.797000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":7.6310000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":9.4849999999999994},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":10.516}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":16.427},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":13.475}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":14.984999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":11.941000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14.204000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.245000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":8.7569999999999997},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":6.5979999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.212},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":11.513999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":7.2869999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":10.880000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":14.138999999999999},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":14.779999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":8.0129999999999999},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":12.246}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.625},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.961}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444114,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":4.9349999999999996},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":8.1839999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.851},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.071}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.827},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":10.555}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.683},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.228999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":14.364000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":12.997999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":13.640000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":15.138999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444115,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":15.151999999999999},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":12.741}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":6.5220000000000002},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":7.2830000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":15.961},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":8.6020000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":7.7130000000000001},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.707000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.771000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":9.2710000000000008}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":10.733000000000001},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":13.628}]}]},{"osd":0,"up_from":17,"seq":73014444115,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":10.146000000000001},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.359}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.2599999999999998},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":11.964}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":15.686999999999999},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":10.802}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.430999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":9.2300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":18.495999999999999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.395}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":17.239999999999998},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":14.756}]}]},{"osd":6,"up_from":15,"seq":64424509524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":10.481999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.4589999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.2770000000000001},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":12.387}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":11.199999999999999},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.6340000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":14.675000000000001},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":10.146000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":13.196},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.689}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.907999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.652000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:53.541 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:53.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.639+0000 7f3792261640 10 monclient: get_monmap_and_config 2024-01-20T11:21:53.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.639+0000 7f3792261640 10 monclient: build_initial_monmap 2024-01-20T11:21:53.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.639+0000 7f3792261640 10 monclient: monmap: 2024-01-20T11:21:53.639 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:53.639 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:53.640175+0000 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:53.640175+0000 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:53.640 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 1 Processor -- start 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 1 -- start start 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 10 monclient: init 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f378c0b4d10 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f378c0b1ed0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c146090 0x7f378c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 10 monclient(hunting): picked mon.noname-b con 0x7f378c146090 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:53.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c0aade0 con 0x7f378c0b1ed0 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c145460 con 0x7f378c0b4d10 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378c0b50f0 con 0x7f378c146090 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.640+0000 7f3792261640 10 monclient(hunting): authenticate will time out at 1756.991115s 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.641+0000 7f378affd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f378c0b1ed0 0x7f378c145980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60508/0 (socket says 172.21.15.110:60508) 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.641+0000 7f378affd640 1 -- 172.21.15.110:0/944412252 learned_addr learned my addr 172.21.15.110:0/944412252 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.641+0000 7f378bfff640 1 --2- 172.21.15.110:0/944412252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c146090 0x7f378c0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.641+0000 7f378bfff640 10 monclient(hunting): get_auth_request con 0x7f378c146090 auth_method 0 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.642+0000 7f378bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.642+0000 7f378bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.642+0000 7f378bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.642+0000 7f378bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:53.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.642+0000 7f378bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:53.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.642+0000 7f378bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:53.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378a7fc640 1 -- 172.21.15.110:0/944412252 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790703185 0 0) 0x7f378c0aade0 con 0x7f378c0b1ed0 2024-01-20T11:21:53.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378a7fc640 10 monclient(hunting): _negotiate 2024-01-20T11:21:53.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378a7fc640 10 monclient(hunting): my global_id is 5658 2024-01-20T11:21:53.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378a7fc640 1 -- 172.21.15.110:0/944412252 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3774003520 con 0x7f378c0b1ed0 2024-01-20T11:21:53.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378bfff640 10 monclient(hunting): handle_auth_done global_id 5357 payload 274 2024-01-20T11:21:53.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378bfff640 1 -- 172.21.15.110:0/944412252 >> v1:172.21.15.134:6789/0 conn(0x7f378c0b4d10 legacy=0x7f378c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378bfff640 1 -- 172.21.15.110:0/944412252 >> v1:172.21.15.110:6789/0 conn(0x7f378c0b1ed0 legacy=0x7f378c145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:53.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378bfff640 1 monclient: found mon.noname-b 2024-01-20T11:21:53.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:53.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:53.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.644+0000 7f378bfff640 1 -- 172.21.15.110:0/944412252 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c0b2660 con 0x7f378c146090 2024-01-20T11:21:53.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.646+0000 7f378bfff640 1 --2- 172.21.15.110:0/944412252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c146090 0x7f378c0b3da0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3784009df0 tx=0x7f378402be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21f5dd5f215a4098 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:53.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.649+0000 7f378a7fc640 1 -- 172.21.15.110:0/944412252 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378402cc20 con 0x7f378c146090 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.649+0000 7f378a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.649+0000 7f378a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.649+0000 7f378a7fc640 10 monclient: dump: 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f378a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f378a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f378a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f378a7fc640 1 -- 172.21.15.110:0/944412252 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378402cd80 con 0x7f378c146090 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 5 monclient: authenticate success, global_id 5357 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f378a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f378a7fc640 1 -- 172.21.15.110:0/944412252 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3784034660 con 0x7f378c146090 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f378a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f378a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f378a7fc640 10 monclient: dump: 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.651 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:53.652 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.652 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:53.652 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:53.652 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:53.652 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:53.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 4 set_mon_vals no callback set 2024-01-20T11:21:53.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 10 monclient: shutdown 2024-01-20T11:21:53.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:53.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 1 -- 172.21.15.110:0/944412252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c146090 msgr2=0x7f378c0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 1 --2- 172.21.15.110:0/944412252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c146090 0x7f378c0b3da0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3784009df0 tx=0x7f378402be70 comp rx=0 tx=0).stop 2024-01-20T11:21:53.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 1 -- 172.21.15.110:0/944412252 shutdown_connections 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 1 --2- 172.21.15.110:0/944412252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c146090 0x7f378c0b3da0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 1 -- 172.21.15.110:0/944412252 >> 172.21.15.110:0/944412252 conn(0x7f378c066590 msgr2=0x7f378c0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 1 -- 172.21.15.110:0/944412252 shutdown_connections 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.650+0000 7f3792261640 1 -- 172.21.15.110:0/944412252 wait complete. 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.651+0000 7f3792261640 10 monclient: build_initial_monmap 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.651+0000 7f3792261640 10 monclient: monmap: 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:53.657 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.651+0000 7f3792261640 1 librados: starting msgr at 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.651+0000 7f3792261640 1 librados: starting objecter 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.651+0000 7f3792261640 1 Processor -- start 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 1 -- start start 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 1 librados: setting wanted keys 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 1 librados: calling monclient init 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient: init 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f378c0b1ed0 0x7f378c13ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient(hunting): picked mon.noname-b con 0x7f378c0b1ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:53.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c0b4d10 0x7f378c13b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient(hunting): picked mon.noname-c con 0x7f378c0b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f378c146090 0x7f378c13b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient(hunting): picked mon.noname-a con 0x7f378c146090 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c13eb20 con 0x7f378c146090 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c0abb60 con 0x7f378c0b1ed0 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378c0abe40 con 0x7f378c0b4d10 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.652+0000 7f3792261640 10 monclient(hunting): authenticate will time out at 1757.002834s 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c0b4d10 0x7f378c13b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:53.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f378c146090 0x7f378c13b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f378c0b1ed0 0x7f378c13ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c0b4d10 0x7f378c13b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:39412/0 (socket says 172.21.15.110:39412) 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 1 -- 172.21.15.110:0/958177721 learned_addr learned my addr 172.21.15.110:0/958177721 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 10 monclient(hunting): get_auth_request con 0x7f378c0b4d10 auth_method 0 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378bfff640 10 monclient(hunting): get_auth_request con 0x7f378c146090 auth_method 0 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378b7fe640 10 monclient(hunting): get_auth_request con 0x7f378c0b1ed0 auth_method 0 2024-01-20T11:21:53.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.653+0000 7f378affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.654+0000 7f378bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.654+0000 7f378bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.654+0000 7f378bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.654+0000 7f378b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.654+0000 7f378b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.654+0000 7f378b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 10 monclient(hunting): handle_auth_done global_id 5360 payload 824 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 1 -- 172.21.15.110:0/958177721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f378c0b1ed0 msgr2=0x7f378c13ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f378c0b1ed0 0x7f378c13ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 1 -- 172.21.15.110:0/958177721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f378c146090 msgr2=0x7f378c13b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:53.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f378c146090 0x7f378c13b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:53.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:53.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 1 monclient: found mon.noname-c 2024-01-20T11:21:53.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:53.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:53.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.656+0000 7f378affd640 1 -- 172.21.15.110:0/958177721 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3784009a70 con 0x7f378c0b4d10 2024-01-20T11:21:53.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.658+0000 7f378affd640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c0b4d10 0x7f378c13b180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f378000c4d0 tx=0x7f378000c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.658+0000 7f3773fff640 10 client.?.objecter ms_handle_connect 0x7f378c0b4d10 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.658+0000 7f3773fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 1 -- 172.21.15.110:0/958177721 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780005ce0 con 0x7f378c0b4d10 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 10 monclient: dump: 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.664 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 10 monclient: _finish_auth 0 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 1 -- 172.21.15.110:0/958177721 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3780005e40 con 0x7f378c0b4d10 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 1 -- 172.21.15.110:0/958177721 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780007270 con 0x7f378c0b4d10 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3773fff640 10 monclient: dump: 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:53.665 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3792261640 5 monclient: authenticate success, global_id 5360 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3792261640 10 monclient: _renew_subs 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3792261640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378c13e310 con 0x7f378c0b4d10 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3789ffb640 4 set_mon_vals no callback set 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3792261640 10 client.5360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3792261640 10 monclient: _renew_subs 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3792261640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:53.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378c13f190 con 0x7f378c0b4d10 2024-01-20T11:21:53.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.662+0000 7f3792261640 1 librados: init done 2024-01-20T11:21:53.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.663+0000 7f3792261640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:53.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.663+0000 7f3792261640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:53.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.663+0000 7f3792261640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:53.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.663+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378c0aaad0 con 0x7f378c0b4d10 2024-01-20T11:21:53.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.680+0000 7f3773fff640 1 -- 172.21.15.110:0/958177721 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3780007490 con 0x7f378c0b4d10 2024-01-20T11:21:53.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.681+0000 7f3773fff640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377806c860 0x7f377806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:53.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.681+0000 7f378b7fe640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377806c860 0x7f377806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:53.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.681+0000 7f378b7fe640 10 monclient: get_auth_request con 0x7f377806c860 auth_method 0 2024-01-20T11:21:53.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.682+0000 7f378b7fe640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377806c860 0x7f377806ed20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f377c001780 tx=0x7f377c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:53.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.682+0000 7f3773fff640 10 client.5360.objecter ms_handle_connect 0x7f377806c860 2024-01-20T11:21:53.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.683+0000 7f3773fff640 1 -- 172.21.15.110:0/958177721 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3780057fd0 con 0x7f378c0b4d10 2024-01-20T11:21:53.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.683+0000 7f3773fff640 10 client.5360.objecter ms_dispatch 0x7f378c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:53.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.683+0000 7f3773fff640 3 client.5360.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:53.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.683+0000 7f3773fff640 3 client.5360.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:53.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.683+0000 7f3773fff640 20 client.5360.objecter dump_active .. 0 homeless 2024-01-20T11:21:53.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.921+0000 7f3773fff640 1 -- 172.21.15.110:0/958177721 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f3780054310 con 0x7f378c0b4d10 2024-01-20T11:21:53.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.921+0000 7f3773fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:53.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:53.921+0000 7f3773fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:54.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.106+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f378c0a9cf0 con 0x7f377806c860 2024-01-20T11:21:54.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.109+0000 7f3773fff640 1 -- 172.21.15.110:0/958177721 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63974 (secure 0 0 0) 0x7f377c001420 con 0x7f377806c860 2024-01-20T11:21:54.109 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:54.109 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:54.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 10 librados: watch_flush enter 2024-01-20T11:21:54.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 10 librados: watch_flush exit 2024-01-20T11:21:54.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 20 client.5360.objecter shutdown clearing up homeless session... 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 10 client.5360.objecter successfully canceled tick 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377806c860 msgr2=0x7f377806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377806c860 0x7f377806ed20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f377c001780 tx=0x7f377c009040 comp rx=0 tx=0).stop 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 10 monclient: shutdown 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c0b4d10 msgr2=0x7f378c13b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c0b4d10 0x7f378c13b180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f378000c4d0 tx=0x7f378000c990 comp rx=0 tx=0).stop 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 shutdown_connections 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f377806c860 0x7f377806ed20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f378c146090 0x7f378c13b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f378c0b4d10 0x7f378c13b180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 --2- 172.21.15.110:0/958177721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f378c0b1ed0 0x7f378c13ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 >> 172.21.15.110:0/958177721 conn(0x7f378c066590 msgr2=0x7f378c0b6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 shutdown_connections 2024-01-20T11:21:54.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 -- 172.21.15.110:0/958177721 wait complete. 2024-01-20T11:21:54.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.111+0000 7f3792261640 1 librados: shutdown 2024-01-20T11:21:54.122 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-01-20T11:21:52.789730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002908"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280597,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":7.0369999999999999},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.1479999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":9.3710000000000004},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":15.375999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.826000000000001},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":12.186}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":8.3539999999999992},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":10.144}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":16.167000000000002},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":14.753}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":10.949},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.026}]}]},{"osd":7,"up_from":21,"seq":90194313300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.3220000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":9.8490000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":12.797000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":7.6310000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":9.4849999999999994},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":10.516}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":16.427},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":13.475}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":14.984999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":11.941000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14.204000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.245000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":8.7569999999999997},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":6.5979999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.212},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":11.513999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":7.2869999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":10.880000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":14.138999999999999},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":14.779999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":8.0129999999999999},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":12.246}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.625},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.961}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444114,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":4.9349999999999996},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":8.1839999999999993}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.851},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.071}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.827},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":10.555}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.683},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.228999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":14.364000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":12.997999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":13.640000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":15.138999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444115,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":15.151999999999999},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":12.741}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":6.5220000000000002},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":7.2830000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":15.961},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":8.6020000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":7.7130000000000001},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.707000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.771000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":9.2710000000000008}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":10.733000000000001},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":13.628}]}]},{"osd":0,"up_from":17,"seq":73014444115,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":10.146000000000001},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.359}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.2599999999999998},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":11.964}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":15.686999999999999},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":10.802}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.430999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":9.2300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":18.495999999999999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.395}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":17.239999999999998},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":14.756}]}]},{"osd":6,"up_from":15,"seq":64424509524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":10.481999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.4589999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.2770000000000001},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":12.387}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":11.199999999999999},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.6340000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":14.675000000000001},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":10.146000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":13.196},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.689}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.907999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.652000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:54.123 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:54.123 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:54.123 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:54.124 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:54.124 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:54.124 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:54.124 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:54.124 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:54.125 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.223+0000 7fc8e1e07640 10 monclient: get_monmap_and_config 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.223+0000 7fc8e1e07640 10 monclient: build_initial_monmap 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.223+0000 7fc8e1e07640 10 monclient: monmap: 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:54.223729+0000 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:54.223729+0000 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:54.223 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:54.224 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:54.224 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:54.224 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:54.224 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.223+0000 7fc8e1e07640 1 Processor -- start 2024-01-20T11:21:54.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.223+0000 7fc8e1e07640 1 -- start start 2024-01-20T11:21:54.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.223+0000 7fc8e1e07640 10 monclient: init 2024-01-20T11:21:54.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:54.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8dc0b4d10 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc8dc0b1ed0 0x7fc8dc0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 10 monclient(hunting): picked mon.noname-c con 0x7fc8dc0b1ed0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc8dc145980 0x7fc8dc145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 10 monclient(hunting): picked mon.noname-a con 0x7fc8dc145980 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc146370 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc0b50f0 con 0x7fc8dc145980 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc0aade0 con 0x7fc8dc0b1ed0 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:54.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.224+0000 7fc8e1e07640 10 monclient(hunting): authenticate will time out at 1757.574664s 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8dbfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc8dc145980 0x7fc8dc145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8daffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc8dc0b1ed0 0x7fc8dc0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8dbfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc8dc145980 0x7fc8dc145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40852/0 (socket says 172.21.15.110:40852) 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8dbfff640 1 -- 172.21.15.110:0/3218222754 learned_addr learned my addr 172.21.15.110:0/3218222754 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8dbfff640 10 monclient(hunting): get_auth_request con 0x7fc8dc145980 auth_method 0 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8dbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8daffd640 10 monclient(hunting): get_auth_request con 0x7fc8dc0b1ed0 auth_method 0 2024-01-20T11:21:54.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8daffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.225+0000 7fc8daffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.226+0000 7fc8dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.226+0000 7fc8dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.226+0000 7fc8dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.226+0000 7fc8daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.226+0000 7fc8daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.226+0000 7fc8daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.227+0000 7fc8dbfff640 10 monclient(hunting): handle_auth_done global_id 5664 payload 274 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.227+0000 7fc8dbfff640 1 -- 172.21.15.110:0/3218222754 >> v2:172.21.15.134:3300/0 conn(0x7fc8dc0b1ed0 msgr2=0x7fc8dc0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.227+0000 7fc8dbfff640 1 --2- 172.21.15.110:0/3218222754 >> v2:172.21.15.134:3300/0 conn(0x7fc8dc0b1ed0 0x7fc8dc0b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.227+0000 7fc8dbfff640 1 -- 172.21.15.110:0/3218222754 >> v1:172.21.15.134:6789/0 conn(0x7fc8dc0b4d10 legacy=0x7fc8dc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:54.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.227+0000 7fc8dbfff640 10 monclient: _finish_hunting 0 2024-01-20T11:21:54.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.227+0000 7fc8dbfff640 1 monclient: found mon.noname-a 2024-01-20T11:21:54.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.227+0000 7fc8dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:54.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.227+0000 7fc8dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:54.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.227+0000 7fc8dbfff640 1 -- 172.21.15.110:0/3218222754 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc145460 con 0x7fc8dc145980 2024-01-20T11:21:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.229+0000 7fc8dbfff640 1 --2- 172.21.15.110:0/3218222754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc145980 0x7fc8dc145e30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0b53c0 tx=0x7fc8d402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678af043071d6ed2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 1 -- 172.21.15.110:0/3218222754 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4031750 con 0x7fc8dc145980 2024-01-20T11:21:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 10 monclient: dump: 2024-01-20T11:21:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 1 -- 172.21.15.110:0/3218222754 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d40318b0 con 0x7fc8dc145980 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 1 -- 172.21.15.110:0/3218222754 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4031c40 con 0x7fc8dc145980 2024-01-20T11:21:54.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8e1e07640 5 monclient: authenticate success, global_id 5664 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8e1e07640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8e1e07640 4 set_mon_vals no callback set 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.232+0000 7fc8da7fc640 10 monclient: dump: 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 10 monclient: shutdown 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 1 -- 172.21.15.110:0/3218222754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc145980 msgr2=0x7fc8dc145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 1 --2- 172.21.15.110:0/3218222754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc145980 0x7fc8dc145e30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0b53c0 tx=0x7fc8d402b7f0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 1 -- 172.21.15.110:0/3218222754 shutdown_connections 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 1 --2- 172.21.15.110:0/3218222754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc145980 0x7fc8dc145e30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 1 --2- 172.21.15.110:0/3218222754 >> v2:172.21.15.134:3300/0 conn(0x7fc8dc0b1ed0 0x7fc8dc0b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 1 -- 172.21.15.110:0/3218222754 >> 172.21.15.110:0/3218222754 conn(0x7fc8dc066590 msgr2=0x7fc8dc0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 1 -- 172.21.15.110:0/3218222754 shutdown_connections 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.233+0000 7fc8e1e07640 1 -- 172.21.15.110:0/3218222754 wait complete. 2024-01-20T11:21:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.234+0000 7fc8e1e07640 10 monclient: build_initial_monmap 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.234+0000 7fc8e1e07640 10 monclient: monmap: 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:54.239 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.234+0000 7fc8e1e07640 1 librados: starting msgr at 2024-01-20T11:21:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.234+0000 7fc8e1e07640 1 librados: starting objecter 2024-01-20T11:21:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.234+0000 7fc8e1e07640 1 Processor -- start 2024-01-20T11:21:54.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.234+0000 7fc8e1e07640 1 -- start start 2024-01-20T11:21:54.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.234+0000 7fc8e1e07640 1 librados: setting wanted keys 2024-01-20T11:21:54.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.234+0000 7fc8e1e07640 1 librados: calling monclient init 2024-01-20T11:21:54.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.234+0000 7fc8e1e07640 10 monclient: init 2024-01-20T11:21:54.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:54.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:21:54.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc0b1ed0 0x7fc8dc140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient(hunting): picked mon.noname-a con 0x7fc8dc0b1ed0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:54.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8dc0b4d10 0x7fc8dc140200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient(hunting): picked mon.noname-b con 0x7fc8dc0b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:54.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8dc145980 0x7fc8dc135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient(hunting): picked mon.noname-c con 0x7fc8dc145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:54.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1415d0 con 0x7fc8dc0b1ed0 2024-01-20T11:21:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc0ab580 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc13fd00 con 0x7fc8dc145980 2024-01-20T11:21:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.235+0000 7fc8e1e07640 10 monclient(hunting): authenticate will time out at 1757.585642s 2024-01-20T11:21:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8daffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8dc0b4d10 0x7fc8dc140200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8db7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc0b1ed0 0x7fc8dc140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8dbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8dc145980 0x7fc8dc135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8db7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc0b1ed0 0x7fc8dc140f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40856/0 (socket says 172.21.15.110:40856) 2024-01-20T11:21:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8daffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8dc0b4d10 0x7fc8dc140200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:36664/0 (socket says 172.21.15.110:36664) 2024-01-20T11:21:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8db7fe640 1 -- 172.21.15.110:0/4099838073 learned_addr learned my addr 172.21.15.110:0/4099838073 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8db7fe640 10 monclient(hunting): get_auth_request con 0x7fc8dc0b1ed0 auth_method 0 2024-01-20T11:21:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8db7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8daffd640 10 monclient(hunting): get_auth_request con 0x7fc8dc0b4d10 auth_method 0 2024-01-20T11:21:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8daffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8daffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8dbfff640 10 monclient(hunting): get_auth_request con 0x7fc8dc145980 auth_method 0 2024-01-20T11:21:54.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8dbfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.236+0000 7fc8daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.237+0000 7fc8db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.237+0000 7fc8db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.237+0000 7fc8db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.237+0000 7fc8dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.237+0000 7fc8dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.237+0000 7fc8dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 10 monclient(hunting): handle_auth_done global_id 5749 payload 824 2024-01-20T11:21:54.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 1 -- 172.21.15.110:0/4099838073 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8dc145980 msgr2=0x7fc8dc135fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8dc145980 0x7fc8dc135fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 1 -- 172.21.15.110:0/4099838073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc0b1ed0 msgr2=0x7fc8dc140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc0b1ed0 0x7fc8dc140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:54.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 1 monclient: found mon.noname-b 2024-01-20T11:21:54.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:54.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:21:54.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.239+0000 7fc8daffd640 1 -- 172.21.15.110:0/4099838073 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d40094d0 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.241+0000 7fc8daffd640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8dc0b4d10 0x7fc8dc140200 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e820 tx=0x7fc8d000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:54.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.241+0000 7fc8c3fff640 10 client.?.objecter ms_handle_connect 0x7fc8dc0b4d10 2024-01-20T11:21:54.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.241+0000 7fc8c3fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:54.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 1 -- 172.21.15.110:0/4099838073 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d00095b0 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:54.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 10 monclient: dump: 2024-01-20T11:21:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.256 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:54.257 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.257 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:54.257 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:54.257 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:54.257 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 10 monclient: _finish_auth 0 2024-01-20T11:21:54.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:54.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:54.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 1 -- 172.21.15.110:0/4099838073 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d000c940 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8e1e07640 5 monclient: authenticate success, global_id 5749 2024-01-20T11:21:54.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8e1e07640 10 monclient: _renew_subs 2024-01-20T11:21:54.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8e1e07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:54.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc140740 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:54.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 1 -- 172.21.15.110:0/4099838073 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000ccf0 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:54.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:54.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8c3fff640 10 monclient: dump: 2024-01-20T11:21:54.259 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:54.260 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.260 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.260 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.260 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:54.260 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.260 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:54.261 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:54.261 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:54.261 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8e1e07640 10 client.5749.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:54.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8e1e07640 10 monclient: _renew_subs 2024-01-20T11:21:54.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8d9ffb640 4 set_mon_vals no callback set 2024-01-20T11:21:54.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8e1e07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:54.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc1366d0 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.245+0000 7fc8e1e07640 1 librados: init done 2024-01-20T11:21:54.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.246+0000 7fc8e1e07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:54.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.246+0000 7fc8e1e07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:54.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.246+0000 7fc8e1e07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:54.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.246+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc1dad90 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.485+0000 7fc8c3fff640 1 -- 172.21.15.110:0/4099838073 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc8d0013470 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.486+0000 7fc8c3fff640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc8c806c8b0 0x7fc8c806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.486+0000 7fc8db7fe640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc8c806c8b0 0x7fc8c806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.486+0000 7fc8db7fe640 10 monclient: get_auth_request con 0x7fc8c806c8b0 auth_method 0 2024-01-20T11:21:54.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.487+0000 7fc8c3fff640 10 client.5749.objecter ms_handle_connect 0x7fc8c806c8b0 2024-01-20T11:21:54.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.487+0000 7fc8db7fe640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc8c806c8b0 0x7fc8c806ed70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc007fd0 tx=0x7fc8cc003e60 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:54.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.491+0000 7fc8c3fff640 1 -- 172.21.15.110:0/4099838073 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc8d00589b0 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.491+0000 7fc8c3fff640 10 client.5749.objecter ms_dispatch 0x7fc8dc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:54.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.491+0000 7fc8c3fff640 3 client.5749.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:54.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.491+0000 7fc8c3fff640 3 client.5749.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:54.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.491+0000 7fc8c3fff640 20 client.5749.objecter dump_active .. 0 homeless 2024-01-20T11:21:54.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.500+0000 7fc8c3fff640 1 -- 172.21.15.110:0/4099838073 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc8d0054cf0 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.500+0000 7fc8c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:54.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.500+0000 7fc8c3fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.750+0000 7fc8e1e07640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:21:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.750+0000 7fc8e1e07640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.750+0000 7fc8e1e07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:21:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.750+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc8dc1dad90 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.757+0000 7fc8c3fff640 1 -- 172.21.15.110:0/4099838073 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fc8d0021520 con 0x7fc8dc0b4d10 2024-01-20T11:21:54.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.757+0000 7fc8c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:21:54.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.757+0000 7fc8c3fff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:21:54.759 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:54.759 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":449,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 10 librados: watch_flush enter 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 10 librados: watch_flush exit 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 20 client.5749.objecter shutdown clearing up homeless session... 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 10 client.5749.objecter successfully canceled tick 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc8c806c8b0 msgr2=0x7fc8c806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc8c806c8b0 0x7fc8c806ed70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc007fd0 tx=0x7fc8cc003e60 comp rx=0 tx=0).stop 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 10 monclient: shutdown 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8dc0b4d10 msgr2=0x7fc8dc140200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8dc0b4d10 0x7fc8dc140200 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e820 tx=0x7fc8d000ece0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 shutdown_connections 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc8c806c8b0 0x7fc8c806ed70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8dc145980 0x7fc8dc135fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8dc0b4d10 0x7fc8dc140200 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 --2- 172.21.15.110:0/4099838073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc0b1ed0 0x7fc8dc140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 >> 172.21.15.110:0/4099838073 conn(0x7fc8dc066590 msgr2=0x7fc8dc0b83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:54.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 shutdown_connections 2024-01-20T11:21:54.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 -- 172.21.15.110:0/4099838073 wait complete. 2024-01-20T11:21:54.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.759+0000 7fc8e1e07640 1 librados: shutdown 2024-01-20T11:21:54.770 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:21:54.770 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:54.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.871+0000 7f4b1d583640 10 monclient: get_monmap_and_config 2024-01-20T11:21:54.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.871+0000 7f4b1d583640 10 monclient: build_initial_monmap 2024-01-20T11:21:54.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.871+0000 7f4b1d583640 10 monclient: monmap: 2024-01-20T11:21:54.871 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:54.872324+0000 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:54.872324+0000 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:54.872 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 1 Processor -- start 2024-01-20T11:21:54.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 1 -- start start 2024-01-20T11:21:54.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient: init 2024-01-20T11:21:54.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:54.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4b180b4d10 0x7f4b180b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient(hunting): picked mon.noname-c con 0x7f4b180b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4b180b17c0 0x7f4b180b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient(hunting): picked mon.noname-a con 0x7f4b180b17c0 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b18145980 0x7f4b18145e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient(hunting): picked mon.noname-b con 0x7f4b18145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b180b50f0 con 0x7f4b180b17c0 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b180b2150 con 0x7f4b180b4d10 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180aade0 con 0x7f4b18145980 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.872+0000 7f4b1d583640 10 monclient(hunting): authenticate will time out at 1758.223124s 2024-01-20T11:21:54.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.873+0000 7f4b16ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4b180b4d10 0x7f4b180b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.873+0000 7f4b0e7fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4b180b17c0 0x7f4b180b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.873+0000 7f4b177fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b18145980 0x7f4b18145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.873+0000 7f4b16ffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f4b180b4d10 0x7f4b180b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:36680/0 (socket says 172.21.15.110:36680) 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.873+0000 7f4b16ffd640 1 -- 172.21.15.110:0/1037410477 learned_addr learned my addr 172.21.15.110:0/1037410477 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.873+0000 7f4b16ffd640 10 monclient(hunting): get_auth_request con 0x7f4b180b4d10 auth_method 0 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.873+0000 7f4b16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.873+0000 7f4b16ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.873+0000 7f4b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b177fe640 10 monclient(hunting): get_auth_request con 0x7f4b18145980 auth_method 0 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b177fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b0e7fc640 10 monclient(hunting): get_auth_request con 0x7f4b180b17c0 auth_method 0 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b0e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.874+0000 7f4b0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 10 monclient(hunting): handle_auth_done global_id 5752 payload 274 2024-01-20T11:21:54.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 1 -- 172.21.15.110:0/1037410477 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b18145980 msgr2=0x7f4b18145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:54.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 1 --2- 172.21.15.110:0/1037410477 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b18145980 0x7f4b18145e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 1 -- 172.21.15.110:0/1037410477 >> v2:172.21.15.110:3300/0 conn(0x7f4b180b17c0 msgr2=0x7f4b180b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:54.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 1 --2- 172.21.15.110:0/1037410477 >> v2:172.21.15.110:3300/0 conn(0x7f4b180b17c0 0x7f4b180b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:54.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 1 monclient: found mon.noname-c 2024-01-20T11:21:54.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:54.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:21:54.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.875+0000 7f4b16ffd640 1 -- 172.21.15.110:0/1037410477 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18145460 con 0x7f4b180b4d10 2024-01-20T11:21:54.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.878+0000 7f4b16ffd640 1 --2- 172.21.15.110:0/1037410477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b180b4d10 0x7f4b180b3da0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009870 tx=0x7f4b0802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6af60a46d29dbd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:54.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 1 -- 172.21.15.110:0/1037410477 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b080317c0 con 0x7f4b180b4d10 2024-01-20T11:21:54.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:54.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:54.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 10 monclient: dump: 2024-01-20T11:21:54.881 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:54.881 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 1 -- 172.21.15.110:0/1037410477 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b08031920 con 0x7f4b180b4d10 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b1d583640 5 monclient: authenticate success, global_id 5752 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 1 -- 172.21.15.110:0/1037410477 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08031cb0 con 0x7f4b180b4d10 2024-01-20T11:21:54.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b167fc640 10 monclient: dump: 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b1d583640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:54.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.881+0000 7f4b1d583640 4 set_mon_vals no callback set 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 10 monclient: shutdown 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 1 -- 172.21.15.110:0/1037410477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b180b4d10 msgr2=0x7f4b180b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 1 --2- 172.21.15.110:0/1037410477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b180b4d10 0x7f4b180b3da0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009870 tx=0x7f4b0802b860 comp rx=0 tx=0).stop 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 1 -- 172.21.15.110:0/1037410477 shutdown_connections 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 1 --2- 172.21.15.110:0/1037410477 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b18145980 0x7f4b18145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 1 --2- 172.21.15.110:0/1037410477 >> v2:172.21.15.110:3300/0 conn(0x7f4b180b17c0 0x7f4b180b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 1 --2- 172.21.15.110:0/1037410477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b180b4d10 0x7f4b180b3da0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 1 -- 172.21.15.110:0/1037410477 >> 172.21.15.110:0/1037410477 conn(0x7f4b18066590 msgr2=0x7f4b180669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 1 -- 172.21.15.110:0/1037410477 shutdown_connections 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.882+0000 7f4b1d583640 1 -- 172.21.15.110:0/1037410477 wait complete. 2024-01-20T11:21:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.883+0000 7f4b1d583640 10 monclient: build_initial_monmap 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.883+0000 7f4b1d583640 10 monclient: monmap: 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.883+0000 7f4b1d583640 1 librados: starting msgr at 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.883+0000 7f4b1d583640 1 librados: starting objecter 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.883+0000 7f4b1d583640 1 Processor -- start 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.883+0000 7f4b1d583640 1 -- start start 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.883+0000 7f4b1d583640 1 librados: setting wanted keys 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.883+0000 7f4b1d583640 1 librados: calling monclient init 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.883+0000 7f4b1d583640 10 monclient: init 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b180b17c0 0x7f4b181d7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient(hunting): picked mon.noname-c con 0x7f4b180b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b180b4d10 0x7f4b181ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient(hunting): picked mon.noname-b con 0x7f4b180b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b18145980 0x7f4b181cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient(hunting): picked mon.noname-a con 0x7f4b18145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b18138540 con 0x7f4b18145980 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b18138820 con 0x7f4b180b4d10 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b181365b0 con 0x7f4b180b17c0 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b1d583640 10 monclient(hunting): authenticate will time out at 1758.234361s 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.884+0000 7f4b16ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b180b17c0 0x7f4b181d7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b16ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b180b17c0 0x7f4b181d7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:39450/0 (socket says 172.21.15.110:39450) 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b16ffd640 1 -- 172.21.15.110:0/2075505983 learned_addr learned my addr 172.21.15.110:0/2075505983 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b16ffd640 10 monclient(hunting): get_auth_request con 0x7f4b180b17c0 auth_method 0 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b16ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b177fe640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b18145980 0x7f4b181cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b0e7fc640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b180b4d10 0x7f4b181ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b177fe640 10 monclient(hunting): get_auth_request con 0x7f4b18145980 auth_method 0 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b177fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b0e7fc640 10 monclient(hunting): get_auth_request con 0x7f4b180b4d10 auth_method 0 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b0e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.885+0000 7f4b0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.886+0000 7f4b177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.886+0000 7f4b177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.886+0000 7f4b177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.886+0000 7f4b0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.886+0000 7f4b0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:54.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.886+0000 7f4b0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:54.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 10 monclient(hunting): handle_auth_done global_id 5369 payload 824 2024-01-20T11:21:54.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 1 -- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b180b4d10 msgr2=0x7f4b181ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b180b4d10 0x7f4b181ce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 1 -- 172.21.15.110:0/2075505983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b18145980 msgr2=0x7f4b181cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b18145980 0x7f4b181cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 1 monclient: found mon.noname-c 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.889+0000 7f4b16ffd640 1 -- 172.21.15.110:0/2075505983 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b080094d0 con 0x7f4b180b17c0 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.891+0000 7f4b16ffd640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b180b17c0 0x7f4b181d7e00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4b0802b8a0 tx=0x7f4b080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.891+0000 7f4b0ffff640 10 client.?.objecter ms_handle_connect 0x7f4b180b17c0 2024-01-20T11:21:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.891+0000 7f4b0ffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 1 -- 172.21.15.110:0/2075505983 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08032860 con 0x7f4b180b17c0 2024-01-20T11:21:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 10 monclient: dump: 2024-01-20T11:21:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:54.896 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 10 monclient: _finish_auth 0 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 1 -- 172.21.15.110:0/2075505983 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b080329e0 con 0x7f4b180b17c0 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b1d583640 5 monclient: authenticate success, global_id 5369 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b1d583640 10 monclient: _renew_subs 2024-01-20T11:21:54.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b1d583640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b181cf090 con 0x7f4b180b17c0 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 1 -- 172.21.15.110:0/2075505983 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08032d90 con 0x7f4b180b17c0 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b0ffff640 10 monclient: dump: 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b15ffb640 4 set_mon_vals no callback set 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b1d583640 10 client.5369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b1d583640 10 monclient: _renew_subs 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b1d583640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b181d8a00 con 0x7f4b180b17c0 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.895+0000 7f4b1d583640 1 librados: init done 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.896+0000 7f4b1d583640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.896+0000 7f4b1d583640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.896+0000 7f4b1d583640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:54.896+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b18075490 con 0x7f4b180b17c0 2024-01-20T11:21:55.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.139+0000 7f4b0ffff640 1 -- 172.21.15.110:0/2075505983 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4b0804e450 con 0x7f4b180b17c0 2024-01-20T11:21:55.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.139+0000 7f4b0ffff640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4af806c860 0x7f4af806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:55.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.139+0000 7f4b0e7fc640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4af806c860 0x7f4af806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:55.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.139+0000 7f4b0e7fc640 10 monclient: get_auth_request con 0x7f4af806c860 auth_method 0 2024-01-20T11:21:55.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.140+0000 7f4b0e7fc640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4af806c860 0x7f4af806ed20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4b180b4370 tx=0x7f4b00009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:55.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.140+0000 7f4b0ffff640 10 client.5369.objecter ms_handle_connect 0x7f4af806c860 2024-01-20T11:21:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.152+0000 7f4b0ffff640 1 -- 172.21.15.110:0/2075505983 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4b0807a460 con 0x7f4b180b17c0 2024-01-20T11:21:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.152+0000 7f4b0ffff640 10 client.5369.objecter ms_dispatch 0x7f4b18000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.152+0000 7f4b0ffff640 3 client.5369.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.152+0000 7f4b0ffff640 3 client.5369.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.152+0000 7f4b0ffff640 20 client.5369.objecter dump_active .. 0 homeless 2024-01-20T11:21:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.152+0000 7f4b0ffff640 1 -- 172.21.15.110:0/2075505983 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4b080767a0 con 0x7f4b180b17c0 2024-01-20T11:21:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.152+0000 7f4b0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.152+0000 7f4b0ffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:55.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.330+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b180b2660 con 0x7f4af806c860 2024-01-20T11:21:55.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.332+0000 7f4b0ffff640 1 -- 172.21.15.110:0/2075505983 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63961 (secure 0 0 0) 0x7f4b000119d0 con 0x7f4af806c860 2024-01-20T11:21:55.333 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 10 librados: watch_flush enter 2024-01-20T11:21:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 10 librados: watch_flush exit 2024-01-20T11:21:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 20 client.5369.objecter shutdown clearing up homeless session... 2024-01-20T11:21:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 10 client.5369.objecter successfully canceled tick 2024-01-20T11:21:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4af806c860 msgr2=0x7f4af806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4af806c860 0x7f4af806ed20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4b180b4370 tx=0x7f4b00009190 comp rx=0 tx=0).stop 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 10 monclient: shutdown 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b180b17c0 msgr2=0x7f4b181d7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.334+0000 7f4b1d583640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b180b17c0 0x7f4b181d7e00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4b0802b8a0 tx=0x7f4b080046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.335+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 shutdown_connections 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.335+0000 7f4b1d583640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4af806c860 0x7f4af806ed20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.335+0000 7f4b1d583640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b18145980 0x7f4b181cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.335+0000 7f4b1d583640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b180b4d10 0x7f4b181ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.335+0000 7f4b1d583640 1 --2- 172.21.15.110:0/2075505983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b180b17c0 0x7f4b181d7e00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.335+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 >> 172.21.15.110:0/2075505983 conn(0x7f4b18066590 msgr2=0x7f4b1813ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.335+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 shutdown_connections 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.335+0000 7f4b1d583640 1 -- 172.21.15.110:0/2075505983 wait complete. 2024-01-20T11:21:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:55.335+0000 7f4b1d583640 1 librados: shutdown 2024-01-20T11:21:55.345 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-01-20T11:21:54.790203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002902"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:38.298320+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:17:38.298320+0000","last_peered":"2024-01-20T11:17:38.298320+0000","last_clean":"2024-01-20T11:17:38.298320+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:17:38.298320+0000","last_undegraded":"2024-01-20T11:17:38.298320+0000","last_fullsized":"2024-01-20T11:17:38.298320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280598,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":14.537000000000001},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.6769999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":5.9649999999999999},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":11.781000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":10.071},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":13.74}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.439},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":8.3819999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":11.097},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":16.257000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":9.3780000000000001},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":12.484999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.3220000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":9.8490000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":12.797000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":7.6310000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":9.4849999999999994},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":10.516}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":16.427},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":13.475}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":14.984999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":11.941000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14.204000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.245000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":8.7569999999999997},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":6.5979999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.212},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":11.513999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":7.2869999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":10.880000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":14.138999999999999},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":14.779999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":8.0129999999999999},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":12.246}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.625},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.961}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444115,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":6.6219999999999999},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":10.877000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":5.6699999999999999},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.5069999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":10.311},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":12.006}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":11.449},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":13.391}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":14.757999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":8.4429999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":7.8849999999999998},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":14.007999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444116,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0309999999999997},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":11.27}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":7.7190000000000003},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.1879999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":10.308999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":9.5079999999999991}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":8.3369999999999997},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.585000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":13.702}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.669},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.800000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444116,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":8.1720000000000006},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":4.2949999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.2880000000000003},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":15.016}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":12.768000000000001},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":8.9239999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":11.753},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":12.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.91},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":10.557}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":11.185},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":13.371}]}]},{"osd":6,"up_from":15,"seq":64424509524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":10.481999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.4589999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.2770000000000001},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":12.387}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":11.199999999999999},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.6340000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":14.675000000000001},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":10.146000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":13.196},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.689}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.907999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.652000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:58.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~180s 2024-01-20T11:21:58.348 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.452+0000 7f2747e89640 10 monclient: get_monmap_and_config 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.452+0000 7f2747e89640 10 monclient: build_initial_monmap 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.452+0000 7f2747e89640 10 monclient: monmap: 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:58.452736+0000 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:58.452736+0000 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:58.453 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:58.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.452+0000 7f2747e89640 1 Processor -- start 2024-01-20T11:21:58.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.452+0000 7f2747e89640 1 -- start start 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.452+0000 7f2747e89640 10 monclient: init 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27400a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f27400a9ad0 0x7f27401423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 10 monclient(hunting): picked mon.noname-c con 0x7f27400a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2740142920 0x7f274014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 10 monclient(hunting): picked mon.noname-a con 0x7f2740142920 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27400b25c0 con 0x7f27400a96f0 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27400aaea0 con 0x7f2740142920 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274013fbe0 con 0x7f27400a9ad0 2024-01-20T11:21:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.453+0000 7f2747e89640 10 monclient(hunting): authenticate will time out at 1761.803701s 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2740142920 0x7f274014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2745424640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f27400a9ad0 0x7f27401423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2740142920 0x7f274014c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40878/0 (socket says 172.21.15.110:40878) 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 1 -- 172.21.15.110:0/2704121802 learned_addr learned my addr 172.21.15.110:0/2704121802 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 10 monclient(hunting): get_auth_request con 0x7f2740142920 auth_method 0 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.454+0000 7f2746426640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.455+0000 7f2745424640 10 monclient(hunting): get_auth_request con 0x7f27400a9ad0 auth_method 0 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.455+0000 7f2745424640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:58.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.455+0000 7f2745424640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:58.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.455+0000 7f2745424640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2745424640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2745424640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2745424640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2746426640 10 monclient(hunting): handle_auth_done global_id 5676 payload 274 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2746426640 1 -- 172.21.15.110:0/2704121802 >> v2:172.21.15.134:3300/0 conn(0x7f27400a9ad0 msgr2=0x7f27401423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2746426640 1 --2- 172.21.15.110:0/2704121802 >> v2:172.21.15.134:3300/0 conn(0x7f27400a9ad0 0x7f27401423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2746426640 1 -- 172.21.15.110:0/2704121802 >> v1:172.21.15.134:6789/0 conn(0x7f27400a96f0 legacy=0x7f274013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2746426640 10 monclient: _finish_hunting 0 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2746426640 1 monclient: found mon.noname-a 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2746426640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2746426640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.456+0000 7f2746426640 1 -- 172.21.15.110:0/2704121802 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27400ab0a0 con 0x7f2740142920 2024-01-20T11:21:58.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.458+0000 7f2746426640 1 --2- 172.21.15.110:0/2704121802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 0x7f274014c850 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f274006d0f0 tx=0x7f273c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d960d183614671e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 1 -- 172.21.15.110:0/2704121802 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c031870 con 0x7f2740142920 2024-01-20T11:21:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 10 monclient: dump: 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 10 monclient: _finish_auth 0 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 10 monclient: _check_auth_tickets 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 1 -- 172.21.15.110:0/2704121802 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c0319d0 con 0x7f2740142920 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2747e89640 5 monclient: authenticate success, global_id 5676 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2747e89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2747e89640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 1 -- 172.21.15.110:0/2704121802 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c031d60 con 0x7f2740142920 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2747e89640 4 set_mon_vals no callback set 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.461+0000 7f2744c23640 10 monclient: dump: 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:58.464 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:58.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 10 monclient: shutdown 2024-01-20T11:21:58.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:58.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 1 -- 172.21.15.110:0/2704121802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 msgr2=0x7f274014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:58.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 1 --2- 172.21.15.110:0/2704121802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 0x7f274014c850 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f274006d0f0 tx=0x7f273c02b910 comp rx=0 tx=0).stop 2024-01-20T11:21:58.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 1 -- 172.21.15.110:0/2704121802 shutdown_connections 2024-01-20T11:21:58.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 1 --2- 172.21.15.110:0/2704121802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 0x7f274014c850 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 1 --2- 172.21.15.110:0/2704121802 >> v2:172.21.15.134:3300/0 conn(0x7f27400a9ad0 0x7f27401423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 1 -- 172.21.15.110:0/2704121802 >> 172.21.15.110:0/2704121802 conn(0x7f27400b7bc0 msgr2=0x7f27400b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 1 -- 172.21.15.110:0/2704121802 shutdown_connections 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.462+0000 7f2747e89640 1 -- 172.21.15.110:0/2704121802 wait complete. 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.463+0000 7f2747e89640 10 monclient: build_initial_monmap 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.463+0000 7f2747e89640 10 monclient: monmap: 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:58.468 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.463+0000 7f2747e89640 1 librados: starting msgr at 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.463+0000 7f2747e89640 1 librados: starting objecter 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.463+0000 7f2747e89640 1 Processor -- start 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.463+0000 7f2747e89640 1 -- start start 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.463+0000 7f2747e89640 1 librados: setting wanted keys 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.463+0000 7f2747e89640 1 librados: calling monclient init 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.463+0000 7f2747e89640 10 monclient: init 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27400a96f0 0x7f27401390b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient(hunting): picked mon.noname-b con 0x7f27400a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27400a9ad0 0x7f27401dad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient(hunting): picked mon.noname-c con 0x7f27400a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 0x7f27401db2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient(hunting): picked mon.noname-a con 0x7f2740142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:58.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400af900 con 0x7f2740142920 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400afa70 con 0x7f27400a96f0 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27400afbe0 con 0x7f27400a9ad0 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2747e89640 10 monclient(hunting): authenticate will time out at 1761.814470s 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2746426640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 0x7f27401db2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2745424640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27400a9ad0 0x7f27401dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.464+0000 7f2745c25640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27400a96f0 0x7f27401390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2746426640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 0x7f27401db2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40886/0 (socket says 172.21.15.110:40886) 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2746426640 1 -- 172.21.15.110:0/2503078153 learned_addr learned my addr 172.21.15.110:0/2503078153 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2746426640 10 monclient(hunting): get_auth_request con 0x7f2740142920 auth_method 0 2024-01-20T11:21:58.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2746426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2746426640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2746426640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745c25640 10 monclient(hunting): get_auth_request con 0x7f27400a96f0 auth_method 0 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745c25640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745c25640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745424640 10 monclient(hunting): get_auth_request con 0x7f27400a9ad0 auth_method 0 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745424640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745424640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745424640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2746426640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2746426640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2746426640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745c25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745c25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745c25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745424640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745424640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.465+0000 7f2745424640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 10 monclient(hunting): handle_auth_done global_id 5679 payload 824 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 1 -- 172.21.15.110:0/2503078153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27400a9ad0 msgr2=0x7f27401dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27400a9ad0 0x7f27401dad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 1 -- 172.21.15.110:0/2503078153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27400a96f0 msgr2=0x7f27401390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27400a96f0 0x7f27401390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 10 monclient: _finish_hunting 0 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 1 monclient: found mon.noname-a 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.468+0000 7f2746426640 1 -- 172.21.15.110:0/2503078153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c0094d0 con 0x7f2740142920 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.470+0000 7f2746426640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 0x7f27401db2a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f273c0090b0 tx=0x7f273c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.470+0000 7f27367fc640 10 client.?.objecter ms_handle_connect 0x7f2740142920 2024-01-20T11:21:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.470+0000 7f27367fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 1 -- 172.21.15.110:0/2503078153 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c03ab60 con 0x7f2740142920 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 10 monclient: dump: 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 10 monclient: _finish_auth 0 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 10 monclient: _check_auth_tickets 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 1 -- 172.21.15.110:0/2503078153 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c03acc0 con 0x7f2740142920 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2747e89640 5 monclient: authenticate success, global_id 5679 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2747e89640 10 monclient: _renew_subs 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 1 -- 172.21.15.110:0/2503078153 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c044630 con 0x7f2740142920 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2747e89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27400afda0 con 0x7f2740142920 2024-01-20T11:21:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f27367fc640 10 monclient: dump: 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2737fff640 4 set_mon_vals no callback set 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2747e89640 10 client.5679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2747e89640 10 monclient: _renew_subs 2024-01-20T11:21:58.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2747e89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:58.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27400b01f0 con 0x7f2740142920 2024-01-20T11:21:58.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.474+0000 7f2747e89640 1 librados: init done 2024-01-20T11:21:58.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.475+0000 7f2747e89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:58.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.475+0000 7f2747e89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:58.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.475+0000 7f2747e89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:58.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.475+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274011fa80 con 0x7f2740142920 2024-01-20T11:21:58.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.717+0000 7f27367fc640 1 -- 172.21.15.110:0/2503078153 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f273c04d450 con 0x7f2740142920 2024-01-20T11:21:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.717+0000 7f27367fc640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f272406c8e0 0x7f272406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.717+0000 7f2745c25640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f272406c8e0 0x7f272406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.717+0000 7f2745c25640 10 monclient: get_auth_request con 0x7f272406c8e0 auth_method 0 2024-01-20T11:21:58.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.718+0000 7f27367fc640 10 client.5679.objecter ms_handle_connect 0x7f272406c8e0 2024-01-20T11:21:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.718+0000 7f2745c25640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f272406c8e0 0x7f272406eda0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2738001f10 tx=0x7f2738009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:58.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.729+0000 7f27367fc640 1 -- 172.21.15.110:0/2503078153 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f273c02d040 con 0x7f2740142920 2024-01-20T11:21:58.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.729+0000 7f27367fc640 10 client.5679.objecter ms_dispatch 0x7f2740000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:58.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.729+0000 7f27367fc640 3 client.5679.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:58.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.729+0000 7f27367fc640 3 client.5679.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:58.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.729+0000 7f27367fc640 20 client.5679.objecter dump_active .. 0 homeless 2024-01-20T11:21:58.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.729+0000 7f27367fc640 1 -- 172.21.15.110:0/2503078153 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f273c076b40 con 0x7f2740142920 2024-01-20T11:21:58.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.729+0000 7f27367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:58.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.729+0000 7f27367fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:58.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.907+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27400b4400 con 0x7f272406c8e0 2024-01-20T11:21:58.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.910+0000 7f27367fc640 1 -- 172.21.15.110:0/2503078153 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63895 (secure 0 0 0) 0x7f2738002060 con 0x7f272406c8e0 2024-01-20T11:21:58.910 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:58.911 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:58.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 10 librados: watch_flush enter 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 10 librados: watch_flush exit 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 20 client.5679.objecter shutdown clearing up homeless session... 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 10 client.5679.objecter successfully canceled tick 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f272406c8e0 msgr2=0x7f272406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f272406c8e0 0x7f272406eda0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2738001f10 tx=0x7f2738009040 comp rx=0 tx=0).stop 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 10 monclient: shutdown 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 msgr2=0x7f27401db2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 0x7f27401db2a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f273c0090b0 tx=0x7f273c0046c0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 shutdown_connections 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f272406c8e0 0x7f272406eda0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2740142920 0x7f27401db2a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27400a9ad0 0x7f27401dad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 --2- 172.21.15.110:0/2503078153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27400a96f0 0x7f27401390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:58.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 >> 172.21.15.110:0/2503078153 conn(0x7f27400b7bc0 msgr2=0x7f274013c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:58.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 shutdown_connections 2024-01-20T11:21:58.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 -- 172.21.15.110:0/2503078153 wait complete. 2024-01-20T11:21:58.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:58.912+0000 7f2747e89640 1 librados: shutdown 2024-01-20T11:21:58.924 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-01-20T11:21:58.790979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":6.8300000000000001},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":8.9570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":8.2840000000000007},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":7.4720000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.121},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":11.374000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.317},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":6.1070000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":10.736000000000001},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":13.49}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":9.984},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.387}]}]},{"osd":7,"up_from":21,"seq":90194313301,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":13.166},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":6.3300000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.5899999999999999},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":16.009}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":15.18},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.609}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":9.3699999999999992},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":16.853999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":10.811999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":11.505000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":10.103},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":12.349}]}]},{"osd":3,"up_from":21,"seq":90194313302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.3550000000000004},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":4.7729999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.504},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":7.0410000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":8.8149999999999995},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":12.369}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":11.824999999999999},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":9.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":9.9610000000000003},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":13.289}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":14.976000000000001},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":14.198}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444116,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.9299999999999997},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.806}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":12.186999999999999},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":17.27}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.798999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":10.308}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":7.8200000000000003},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":9.6880000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":14.470000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":7.5469999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.585000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.286999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444117,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0309999999999997},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":11.27}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":7.7190000000000003},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.1879999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":10.308999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":9.5079999999999991}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":8.3369999999999997},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.585000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":13.702}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.669},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.800000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444117,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":8.5180000000000007},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":9.8859999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":9.0899999999999999},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":6.2649999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":10.952},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.162000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":7.2709999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":15.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":14.42},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.542}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":13.702},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":12.919}]}]},{"osd":6,"up_from":15,"seq":64424509525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":7.7140000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":13.568}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":10.785},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":10.185}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":15.058},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":5.9960000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.541},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":12.686999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":14.319000000000001},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":6.7080000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":8.4039999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":9.3420000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:58.926 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:21:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.025+0000 7f4bf91f5640 10 monclient: get_monmap_and_config 2024-01-20T11:21:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.025+0000 7f4bf91f5640 10 monclient: build_initial_monmap 2024-01-20T11:21:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.025+0000 7f4bf91f5640 10 monclient: monmap: 2024-01-20T11:21:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:59.025855+0000 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:59.025855+0000 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.025+0000 7f4bf91f5640 1 Processor -- start 2024-01-20T11:21:59.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.025+0000 7f4bf91f5640 1 -- start start 2024-01-20T11:21:59.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.025+0000 7f4bf91f5640 10 monclient: init 2024-01-20T11:21:59.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:59.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-01-20T11:21:59.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bf40a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:59.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bf40a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf4142af0 0x7f4bf414d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 10 monclient(hunting): picked mon.noname-b con 0x7f4bf4142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf40b25c0 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf4142f50 con 0x7f4bf40a96f0 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf40aaea0 con 0x7f4bf4142af0 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.026+0000 7f4bf91f5640 10 monclient(hunting): authenticate will time out at 1762.376653s 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf359e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf4142af0 0x7f4bf414d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf259c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4bf40a9ad0 0x7f4bf41423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60524/0 (socket says 172.21.15.110:60524) 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf259c640 1 -- 172.21.15.110:0/3760906005 learned_addr learned my addr 172.21.15.110:0/3760906005 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf359e640 10 monclient(hunting): get_auth_request con 0x7f4bf4142af0 auth_method 0 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf359e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf359e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.027+0000 7f4bf359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf1d9b640 1 -- 172.21.15.110:0/3760906005 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028014141 0 0) 0x7f4be8001110 con 0x7f4bf40a96f0 2024-01-20T11:21:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf1d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:21:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf1d9b640 10 monclient(hunting): my global_id is 5767 2024-01-20T11:21:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf1d9b640 1 -- 172.21.15.110:0/3760906005 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc003520 con 0x7f4bf40a96f0 2024-01-20T11:21:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf359e640 10 monclient(hunting): handle_auth_done global_id 5375 payload 274 2024-01-20T11:21:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf1d9b640 1 -- 172.21.15.110:0/3760906005 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265123463 0 0) 0x7f4bf40b25c0 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf359e640 1 -- 172.21.15.110:0/3760906005 >> v1:172.21.15.134:6789/0 conn(0x7f4bf40a96f0 legacy=0x7f4bf413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf359e640 1 -- 172.21.15.110:0/3760906005 >> v1:172.21.15.110:6789/0 conn(0x7f4bf40a9ad0 legacy=0x7f4bf41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf359e640 10 monclient: _finish_hunting 0 2024-01-20T11:21:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf359e640 1 monclient: found mon.noname-b 2024-01-20T11:21:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:21:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf359e640 1 -- 172.21.15.110:0/3760906005 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf413fbe0 con 0x7f4bf4142af0 2024-01-20T11:21:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.029+0000 7f4bf1d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:21:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.031+0000 7f4bf359e640 1 --2- 172.21.15.110:0/3760906005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf4142af0 0x7f4bf414d860 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009dc0 tx=0x7f4bec02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6839b2e7a73be092 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 1 -- 172.21.15.110:0/3760906005 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec02cc40 con 0x7f4bf4142af0 2024-01-20T11:21:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 10 monclient: dump: 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 10 monclient: _finish_auth 0 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 1 -- 172.21.15.110:0/3760906005 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bec02cda0 con 0x7f4bf4142af0 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf91f5640 5 monclient: authenticate success, global_id 5375 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf91f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf91f5640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf91f5640 4 set_mon_vals no callback set 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 1 -- 172.21.15.110:0/3760906005 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec034660 con 0x7f4bf4142af0 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.034+0000 7f4bf1d9b640 10 monclient: dump: 2024-01-20T11:21:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.035+0000 7f4bf91f5640 10 monclient: shutdown 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.035+0000 7f4bf91f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.035+0000 7f4bf91f5640 1 -- 172.21.15.110:0/3760906005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf4142af0 msgr2=0x7f4bf414d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.035+0000 7f4bf91f5640 1 --2- 172.21.15.110:0/3760906005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf4142af0 0x7f4bf414d860 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009dc0 tx=0x7f4bec02be90 comp rx=0 tx=0).stop 2024-01-20T11:21:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.035+0000 7f4bf91f5640 1 -- 172.21.15.110:0/3760906005 shutdown_connections 2024-01-20T11:21:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.035+0000 7f4bf91f5640 1 --2- 172.21.15.110:0/3760906005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf4142af0 0x7f4bf414d860 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.035+0000 7f4bf91f5640 1 -- 172.21.15.110:0/3760906005 >> 172.21.15.110:0/3760906005 conn(0x7f4bf40b7bc0 msgr2=0x7f4bf40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.035+0000 7f4bf91f5640 1 -- 172.21.15.110:0/3760906005 shutdown_connections 2024-01-20T11:21:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.035+0000 7f4bf91f5640 1 -- 172.21.15.110:0/3760906005 wait complete. 2024-01-20T11:21:59.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.036+0000 7f4bf91f5640 10 monclient: build_initial_monmap 2024-01-20T11:21:59.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.036+0000 7f4bf91f5640 10 monclient: monmap: 2024-01-20T11:21:59.042 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:59.042 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:59.042 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:59.042 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 librados: starting msgr at 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 librados: starting objecter 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 Processor -- start 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 -- start start 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 librados: setting wanted keys 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 librados: calling monclient init 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient: init 2024-01-20T11:21:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bf40a96f0 0x7f4bf4138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient(hunting): picked mon.noname-a con 0x7f4bf40a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf40a9ad0 0x7f4bf4139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient(hunting): picked mon.noname-c con 0x7f4bf40a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bf4142af0 0x7f4bf41d7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient(hunting): picked mon.noname-b con 0x7f4bf4142af0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf41399d0 con 0x7f4bf40a96f0 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf41d7100 con 0x7f4bf4142af0 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf413bad0 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.037+0000 7f4bf91f5640 10 monclient(hunting): authenticate will time out at 1762.388128s 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf40a9ad0 0x7f4bf4139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf2d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bf40a96f0 0x7f4bf4138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf359e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bf4142af0 0x7f4bf41d7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf40a9ad0 0x7f4bf4139df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:39462/0 (socket says 172.21.15.110:39462) 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf259c640 1 -- 172.21.15.110:0/1267548523 learned_addr learned my addr 172.21.15.110:0/1267548523 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf259c640 10 monclient(hunting): get_auth_request con 0x7f4bf40a9ad0 auth_method 0 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf359e640 10 monclient(hunting): get_auth_request con 0x7f4bf4142af0 auth_method 0 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf359e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf359e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf2d9d640 10 monclient(hunting): get_auth_request con 0x7f4bf40a96f0 auth_method 0 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.038+0000 7f4bf2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.039+0000 7f4bf259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.039+0000 7f4bf259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.039+0000 7f4bf259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.039+0000 7f4bf359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.039+0000 7f4bf359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.039+0000 7f4bf359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.039+0000 7f4bf2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.039+0000 7f4bf2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.039+0000 7f4bf2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 10 monclient(hunting): handle_auth_done global_id 5378 payload 824 2024-01-20T11:21:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 1 -- 172.21.15.110:0/1267548523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bf4142af0 msgr2=0x7f4bf41d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bf4142af0 0x7f4bf41d7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 1 -- 172.21.15.110:0/1267548523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bf40a96f0 msgr2=0x7f4bf4138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bf40a96f0 0x7f4bf4138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 10 monclient: _finish_hunting 0 2024-01-20T11:21:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 1 monclient: found mon.noname-c 2024-01-20T11:21:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:21:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.042+0000 7f4bf259c640 1 -- 172.21.15.110:0/1267548523 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec009a70 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.044+0000 7f4bf259c640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf40a9ad0 0x7f4bf4139df0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4be400c460 tx=0x7f4be400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:59.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.044+0000 7f4bdb7fe640 10 client.?.objecter ms_handle_connect 0x7f4bf40a9ad0 2024-01-20T11:21:59.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.044+0000 7f4bdb7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 1 -- 172.21.15.110:0/1267548523 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be40074e0 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:21:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 10 monclient: dump: 2024-01-20T11:21:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 10 monclient: _finish_auth 0 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 1 -- 172.21.15.110:0/1267548523 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be4007680 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 1 -- 172.21.15.110:0/1267548523 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be4004a00 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf91f5640 5 monclient: authenticate success, global_id 5378 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf91f5640 10 monclient: _renew_subs 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf91f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf413bdb0 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bdb7fe640 10 monclient: dump: 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:59.051 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf159a640 4 set_mon_vals no callback set 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf91f5640 10 client.5378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf91f5640 10 monclient: _renew_subs 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf91f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf413a5c0 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.048+0000 7f4bf91f5640 1 librados: init done 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.049+0000 7f4bf91f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.049+0000 7f4bf91f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.049+0000 7f4bf91f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:21:59.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.049+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf4075490 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.291+0000 7f4bdb7fe640 1 -- 172.21.15.110:0/1267548523 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4be401c3b0 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.291+0000 7f4bdb7fe640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be006c8b0 0x7f4be006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.291+0000 7f4bf2d9d640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be006c8b0 0x7f4be006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.291+0000 7f4bf2d9d640 10 monclient: get_auth_request con 0x7f4be006c8b0 auth_method 0 2024-01-20T11:21:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.292+0000 7f4bf2d9d640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be006c8b0 0x7f4be006ed70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4be8007fd0 tx=0x7f4be8009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.292+0000 7f4bdb7fe640 10 client.5378.objecter ms_handle_connect 0x7f4be006c8b0 2024-01-20T11:21:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.304+0000 7f4bdb7fe640 1 -- 172.21.15.110:0/1267548523 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4be4057c20 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.304+0000 7f4bdb7fe640 10 client.5378.objecter ms_dispatch 0x7f4bf4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.304+0000 7f4bdb7fe640 3 client.5378.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.304+0000 7f4bdb7fe640 3 client.5378.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.304+0000 7f4bdb7fe640 20 client.5378.objecter dump_active .. 0 homeless 2024-01-20T11:21:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.304+0000 7f4bdb7fe640 1 -- 172.21.15.110:0/1267548523 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4be4053f60 con 0x7f4bf40a9ad0 2024-01-20T11:21:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.304+0000 7f4bdb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:59.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.304+0000 7f4bdb7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:21:59.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.482+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4bf40b4400 con 0x7f4be006c8b0 2024-01-20T11:21:59.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.485+0000 7f4bdb7fe640 1 -- 172.21.15.110:0/1267548523 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63895 (secure 0 0 0) 0x7f4be8007df0 con 0x7f4be006c8b0 2024-01-20T11:21:59.486 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:21:59.486 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:21:59.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.487+0000 7f4bf91f5640 10 librados: watch_flush enter 2024-01-20T11:21:59.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.487+0000 7f4bf91f5640 10 librados: watch_flush exit 2024-01-20T11:21:59.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.487+0000 7f4bf91f5640 20 client.5378.objecter shutdown clearing up homeless session... 2024-01-20T11:21:59.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.487+0000 7f4bf91f5640 10 client.5378.objecter successfully canceled tick 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be006c8b0 msgr2=0x7f4be006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be006c8b0 0x7f4be006ed70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4be8007fd0 tx=0x7f4be8009040 comp rx=0 tx=0).stop 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 10 monclient: shutdown 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf40a9ad0 msgr2=0x7f4bf4139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf40a9ad0 0x7f4bf4139df0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4be400c460 tx=0x7f4be400c920 comp rx=0 tx=0).stop 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 shutdown_connections 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4be006c8b0 0x7f4be006ed70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bf4142af0 0x7f4bf41d7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bf40a9ad0 0x7f4bf4139df0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 --2- 172.21.15.110:0/1267548523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bf40a96f0 0x7f4bf4138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 >> 172.21.15.110:0/1267548523 conn(0x7f4bf40b7bc0 msgr2=0x7f4bf413f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 shutdown_connections 2024-01-20T11:21:59.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 -- 172.21.15.110:0/1267548523 wait complete. 2024-01-20T11:21:59.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.488+0000 7f4bf91f5640 1 librados: shutdown 2024-01-20T11:21:59.498 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-01-20T11:21:58.790979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":6.8300000000000001},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":8.9570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":8.2840000000000007},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":7.4720000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.121},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":11.374000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.317},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":6.1070000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":10.736000000000001},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":13.49}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":9.984},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.387}]}]},{"osd":7,"up_from":21,"seq":90194313301,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":13.166},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":6.3300000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.5899999999999999},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":16.009}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":15.18},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.609}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":9.3699999999999992},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":16.853999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":10.811999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":11.505000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":10.103},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":12.349}]}]},{"osd":3,"up_from":21,"seq":90194313302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.3550000000000004},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":4.7729999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.504},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":7.0410000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":8.8149999999999995},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":12.369}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":11.824999999999999},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":9.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":9.9610000000000003},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":13.289}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":14.976000000000001},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":14.198}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444116,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.9299999999999997},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.806}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":12.186999999999999},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":17.27}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.798999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":10.308}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":7.8200000000000003},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":9.6880000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":14.470000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":7.5469999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.585000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.286999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444117,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0309999999999997},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":11.27}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":7.7190000000000003},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.1879999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":10.308999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":9.5079999999999991}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":8.3369999999999997},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.585000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":13.702}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.669},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.800000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444117,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":8.5180000000000007},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":9.8859999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":9.0899999999999999},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":6.2649999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":10.952},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.162000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":7.2709999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":15.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":14.42},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.542}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":13.702},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":12.919}]}]},{"osd":6,"up_from":15,"seq":64424509525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":7.7140000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":13.568}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":10.785},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":10.185}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":15.058},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":5.9960000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.541},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":12.686999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":14.319000000000001},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":6.7080000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":8.4039999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":9.3420000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:21:59.500 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:21:59.500 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:21:59.500 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:21:59.500 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:21:59.500 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:21:59.500 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:21:59.500 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:21:59.500 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:21:59.501 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:21:59.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.599+0000 7fd2202ec640 10 monclient: get_monmap_and_config 2024-01-20T11:21:59.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.599+0000 7fd2202ec640 10 monclient: build_initial_monmap 2024-01-20T11:21:59.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.599+0000 7fd2202ec640 10 monclient: monmap: 2024-01-20T11:21:59.599 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:21:59.600394+0000 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:21:59.600394+0000 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:21:59.600 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.600+0000 7fd2202ec640 1 Processor -- start 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.600+0000 7fd2202ec640 1 -- start start 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.600+0000 7fd2202ec640 10 monclient: init 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd21806a860 addr v1:172.21.15.110:6789/0 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2180af730 addr v1:172.21.15.134:6789/0 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd218145cc0 0x7fd21814d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 10 monclient(hunting): picked mon.noname-a con 0x7fd218145cc0 addr v2:172.21.15.110:3300/0 2024-01-20T11:21:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2180afc50 con 0x7fd21806a860 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2180b0170 con 0x7fd2180af730 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd21806e450 con 0x7fd218145cc0 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd2202ec640 10 monclient(hunting): authenticate will time out at 1762.951414s 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.601+0000 7fd21e889640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd218145cc0 0x7fd21814d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.602+0000 7fd21d887640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd2180af730 0x7fd2181455b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:41862/0 (socket says 172.21.15.110:41862) 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.602+0000 7fd21d887640 1 -- 172.21.15.110:0/3092928154 learned_addr learned my addr 172.21.15.110:0/3092928154 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.602+0000 7fd21e889640 10 monclient(hunting): get_auth_request con 0x7fd218145cc0 auth_method 0 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.602+0000 7fd21e889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.602+0000 7fd21e889640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.602+0000 7fd21e889640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.603+0000 7fd21e889640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.603+0000 7fd21e889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.603+0000 7fd21e889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.605+0000 7fd21d086640 1 -- 172.21.15.110:0/3092928154 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649371714 0 0) 0x7fd2180b0170 con 0x7fd2180af730 2024-01-20T11:21:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.605+0000 7fd21d086640 10 monclient(hunting): _negotiate 2024-01-20T11:21:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.605+0000 7fd21d086640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.605+0000 7fd21d086640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.605+0000 7fd21d086640 10 monclient(hunting): my global_id is 5773 2024-01-20T11:21:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.605+0000 7fd21d086640 1 -- 172.21.15.110:0/3092928154 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd204003520 con 0x7fd2180af730 2024-01-20T11:21:59.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.606+0000 7fd21e889640 10 monclient(hunting): handle_auth_done global_id 5688 payload 274 2024-01-20T11:21:59.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.606+0000 7fd21e889640 1 -- 172.21.15.110:0/3092928154 >> v1:172.21.15.134:6789/0 conn(0x7fd2180af730 legacy=0x7fd2181455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.606+0000 7fd21e889640 1 -- 172.21.15.110:0/3092928154 >> v1:172.21.15.110:6789/0 conn(0x7fd21806a860 legacy=0x7fd21813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.606+0000 7fd21e889640 10 monclient: _finish_hunting 0 2024-01-20T11:21:59.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.606+0000 7fd21e889640 1 monclient: found mon.noname-a 2024-01-20T11:21:59.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.606+0000 7fd21e889640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:59.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.606+0000 7fd21e889640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:59.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.606+0000 7fd21e889640 1 -- 172.21.15.110:0/3092928154 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21813fb60 con 0x7fd218145cc0 2024-01-20T11:21:59.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.609+0000 7fd21e889640 1 --2- 172.21.15.110:0/3092928154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 0x7fd21814d950 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd210009d90 tx=0x7fd21002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6fe9fd3b1878ce server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 1 -- 172.21.15.110:0/3092928154 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21002cbd0 con 0x7fd218145cc0 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 10 monclient: dump: 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 10 monclient: _finish_auth 0 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 10 monclient: _check_auth_tickets 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 1 -- 172.21.15.110:0/3092928154 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21002cd30 con 0x7fd218145cc0 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 1 -- 172.21.15.110:0/3092928154 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21002cea0 con 0x7fd218145cc0 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd2202ec640 5 monclient: authenticate success, global_id 5688 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd2202ec640 10 monclient: get_monmap_and_config success 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd2202ec640 4 set_mon_vals no callback set 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:59.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.613+0000 7fd21d086640 10 monclient: dump: 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:59.615 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 10 monclient: shutdown 2024-01-20T11:21:59.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 1 -- 172.21.15.110:0/3092928154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 msgr2=0x7fd21814d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 1 --2- 172.21.15.110:0/3092928154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 0x7fd21814d950 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd210009d90 tx=0x7fd21002be20 comp rx=0 tx=0).stop 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 1 -- 172.21.15.110:0/3092928154 shutdown_connections 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 1 --2- 172.21.15.110:0/3092928154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 0x7fd21814d950 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 1 -- 172.21.15.110:0/3092928154 >> 172.21.15.110:0/3092928154 conn(0x7fd2180b47d0 msgr2=0x7fd2180b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 1 -- 172.21.15.110:0/3092928154 shutdown_connections 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 1 -- 172.21.15.110:0/3092928154 wait complete. 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 10 monclient: build_initial_monmap 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.614+0000 7fd2202ec640 10 monclient: monmap: 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:21:59.619 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 librados: starting msgr at 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 librados: starting objecter 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 Processor -- start 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 -- start start 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 librados: setting wanted keys 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 librados: calling monclient init 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient: init 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient: _reopen_session rank -1 2024-01-20T11:21:59.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd21806a860 0x7fd218071920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient(hunting): picked mon.noname-c con 0x7fd21806a860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2180af730 0x7fd21813a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient(hunting): picked mon.noname-b con 0x7fd2180af730 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 0x7fd21813a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient(hunting): picked mon.noname-a con 0x7fd218145cc0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21806f690 con 0x7fd218145cc0 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd218070eb0 con 0x7fd2180af730 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd218071190 con 0x7fd21806a860 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient(hunting): _renew_subs 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.615+0000 7fd2202ec640 10 monclient(hunting): authenticate will time out at 1762.966101s 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e889640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 0x7fd21813a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e088640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd21806a860 0x7fd218071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e889640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 0x7fd21813a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:59364/0 (socket says 172.21.15.110:59364) 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e889640 1 -- 172.21.15.110:0/118897185 learned_addr learned my addr 172.21.15.110:0/118897185 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e889640 10 monclient(hunting): get_auth_request con 0x7fd218145cc0 auth_method 0 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e889640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e889640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21d887640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2180af730 0x7fd21813a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e088640 10 monclient(hunting): get_auth_request con 0x7fd21806a860 auth_method 0 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e088640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e088640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.616+0000 7fd21e088640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21d887640 10 monclient(hunting): get_auth_request con 0x7fd2180af730 auth_method 0 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21d887640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21d887640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:21:59.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21d887640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:21:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21e889640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21e889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21e889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21e088640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21e088640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21e088640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21d887640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:21:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21d887640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:21:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.617+0000 7fd21d887640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:21:59.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 10 monclient(hunting): handle_auth_done global_id 5694 payload 824 2024-01-20T11:21:59.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 1 -- 172.21.15.110:0/118897185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd21806a860 msgr2=0x7fd218071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd21806a860 0x7fd218071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 1 -- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2180af730 msgr2=0x7fd21813a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:21:59.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2180af730 0x7fd21813a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:21:59.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 10 monclient: _finish_hunting 0 2024-01-20T11:21:59.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 1 monclient: found mon.noname-a 2024-01-20T11:21:59.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:21:59.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:21:59.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.620+0000 7fd21e889640 1 -- 172.21.15.110:0/118897185 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd210009a70 con 0x7fd218145cc0 2024-01-20T11:21:59.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.622+0000 7fd21e889640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 0x7fd21813a600 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd210009eb0 tx=0x7fd210004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:59.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.622+0000 7fd20effd640 10 client.?.objecter ms_handle_connect 0x7fd218145cc0 2024-01-20T11:21:59.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.622+0000 7fd20effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:21:59.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 1 -- 172.21.15.110:0/118897185 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21002cea0 con 0x7fd218145cc0 2024-01-20T11:21:59.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 10 monclient: dump: 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 10 monclient: _finish_auth 0 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 10 monclient: _check_auth_tickets 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:21:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 1 -- 172.21.15.110:0/118897185 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21003e610 con 0x7fd218145cc0 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 1 -- 172.21.15.110:0/118897185 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21003e9c0 con 0x7fd218145cc0 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd2202ec640 5 monclient: authenticate success, global_id 5694 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd2202ec640 10 monclient: _renew_subs 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd2202ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd21c885640 4 set_mon_vals no callback set 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21806fe70 con 0x7fd218145cc0 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd20effd640 10 monclient: dump: 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:21:59.628 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd2202ec640 10 client.5694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd2202ec640 10 monclient: _renew_subs 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd2202ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd21806ebe0 con 0x7fd218145cc0 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.626+0000 7fd2202ec640 1 librados: init done 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.627+0000 7fd2202ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.627+0000 7fd2202ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:59.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.627+0000 7fd2202ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:21:59.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.627+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21806f800 con 0x7fd218145cc0 2024-01-20T11:21:59.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.868+0000 7fd20effd640 1 -- 172.21.15.110:0/118897185 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd21003b050 con 0x7fd218145cc0 2024-01-20T11:21:59.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.868+0000 7fd20effd640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1fc06c860 0x7fd1fc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:21:59.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.869+0000 7fd21e088640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1fc06c860 0x7fd1fc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:21:59.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.869+0000 7fd21e088640 10 monclient: get_auth_request con 0x7fd1fc06c860 auth_method 0 2024-01-20T11:21:59.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.869+0000 7fd20effd640 10 client.5694.objecter ms_handle_connect 0x7fd1fc06c860 2024-01-20T11:21:59.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.869+0000 7fd21e088640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1fc06c860 0x7fd1fc06ed20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd214001780 tx=0x7fd214009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:21:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.883+0000 7fd20effd640 1 -- 172.21.15.110:0/118897185 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd21007ae80 con 0x7fd218145cc0 2024-01-20T11:21:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.883+0000 7fd20effd640 10 client.5694.objecter ms_dispatch 0x7fd218000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:21:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.883+0000 7fd20effd640 3 client.5694.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:21:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.883+0000 7fd20effd640 3 client.5694.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:21:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.883+0000 7fd20effd640 20 client.5694.objecter dump_active .. 0 homeless 2024-01-20T11:21:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.883+0000 7fd20effd640 1 -- 172.21.15.110:0/118897185 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd2100771c0 con 0x7fd218145cc0 2024-01-20T11:21:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.883+0000 7fd20effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:21:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:21:59.883+0000 7fd20effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.130+0000 7fd2202ec640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.130+0000 7fd2202ec640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.130+0000 7fd2202ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.130+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd218070000 con 0x7fd218145cc0 2024-01-20T11:22:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.136+0000 7fd20effd640 1 -- 172.21.15.110:0/118897185 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fd210039520 con 0x7fd218145cc0 2024-01-20T11:22:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.136+0000 7fd20effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:00.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.136+0000 7fd20effd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:00.136 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:00.136 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":454,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:00.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 10 librados: watch_flush enter 2024-01-20T11:22:00.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 10 librados: watch_flush exit 2024-01-20T11:22:00.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 20 client.5694.objecter shutdown clearing up homeless session... 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 10 client.5694.objecter successfully canceled tick 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1fc06c860 msgr2=0x7fd1fc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1fc06c860 0x7fd1fc06ed20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd214001780 tx=0x7fd214009040 comp rx=0 tx=0).stop 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 10 monclient: shutdown 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 msgr2=0x7fd21813a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 0x7fd21813a600 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd210009eb0 tx=0x7fd210004a40 comp rx=0 tx=0).stop 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 shutdown_connections 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1fc06c860 0x7fd1fc06ed20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd218145cc0 0x7fd21813a600 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2180af730 0x7fd21813a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 --2- 172.21.15.110:0/118897185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd21806a860 0x7fd218071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 >> 172.21.15.110:0/118897185 conn(0x7fd2180b47d0 msgr2=0x7fd2180aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 shutdown_connections 2024-01-20T11:22:00.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 -- 172.21.15.110:0/118897185 wait complete. 2024-01-20T11:22:00.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.138+0000 7fd2202ec640 1 librados: shutdown 2024-01-20T11:22:00.148 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:00.148 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.247+0000 7f6fbd39e640 10 monclient: get_monmap_and_config 2024-01-20T11:22:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.247+0000 7f6fbd39e640 10 monclient: build_initial_monmap 2024-01-20T11:22:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.247+0000 7f6fbd39e640 10 monclient: monmap: 2024-01-20T11:22:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:00.248201+0000 2024-01-20T11:22:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:00.248201+0000 2024-01-20T11:22:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:00.247 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:00.248 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:00.248 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:00.248 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:00.248 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:00.248 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:00.248 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.248+0000 7f6fbd39e640 1 Processor -- start 2024-01-20T11:22:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.248+0000 7f6fbd39e640 1 -- start start 2024-01-20T11:22:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.248+0000 7f6fbd39e640 10 monclient: init 2024-01-20T11:22:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.248+0000 7f6fbd39e640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.248+0000 7f6fbd39e640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-01-20T11:22:00.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.248+0000 7f6fbd39e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6fb806c2d0 0x7f6fb80b1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.248+0000 7f6fbd39e640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb806c2d0 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fbd39e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fb806c6b0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fbd39e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fb80a9b50 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fbd39e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80b3d20 con 0x7f6fb806c6b0 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fbd39e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80aab50 con 0x7f6fb80a9b50 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fbd39e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fbd39e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb80b39f0 con 0x7f6fb806c2d0 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fbd39e640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fbd39e640 10 monclient(hunting): authenticate will time out at 1763.599245s 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fb7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6fb806c2d0 0x7f6fb80b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fbcb9d640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6fb80a9b50 0x7f6fb81425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:41874/0 (socket says 172.21.15.110:41874) 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.249+0000 7f6fb7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6fb806c2d0 0x7f6fb80b1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:59374/0 (socket says 172.21.15.110:59374) 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.250+0000 7f6fbcb9d640 1 -- 172.21.15.110:0/822602943 learned_addr learned my addr 172.21.15.110:0/822602943 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.250+0000 7f6fb7fff640 10 monclient(hunting): get_auth_request con 0x7f6fb806c2d0 auth_method 0 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.250+0000 7f6fb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:00.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.250+0000 7f6fb7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:00.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.250+0000 7f6fb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:00.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.251+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:00.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.251+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:00.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.251+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.252+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_done global_id 5697 payload 274 2024-01-20T11:22:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.252+0000 7f6fb7fff640 1 -- 172.21.15.110:0/822602943 >> v1:172.21.15.134:6789/0 conn(0x7f6fb80a9b50 legacy=0x7f6fb81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.252+0000 7f6fb7fff640 1 -- 172.21.15.110:0/822602943 >> v1:172.21.15.110:6789/0 conn(0x7f6fb806c6b0 legacy=0x7f6fb8141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.252+0000 7f6fb7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:00.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.252+0000 7f6fb7fff640 1 monclient: found mon.noname-a 2024-01-20T11:22:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.252+0000 7f6fb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.252+0000 7f6fb7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:00.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.252+0000 7f6fb7fff640 1 -- 172.21.15.110:0/822602943 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80aadc0 con 0x7f6fb806c2d0 2024-01-20T11:22:00.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.255+0000 7f6fb7fff640 1 --2- 172.21.15.110:0/822602943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb806c2d0 0x7f6fb80b1990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6fac0097a0 tx=0x7f6fac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c8d9d72b89ba1b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:00.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.259+0000 7f6fb6ffd640 1 -- 172.21.15.110:0/822602943 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac031830 con 0x7f6fb806c2d0 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.259+0000 7f6fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.259+0000 7f6fb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.259+0000 7f6fb6ffd640 10 monclient: dump: 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:00.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fb6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fb6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fb6ffd640 1 -- 172.21.15.110:0/822602943 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac031990 con 0x7f6fb806c2d0 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 5 monclient: authenticate success, global_id 5697 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fb6ffd640 1 -- 172.21.15.110:0/822602943 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac031d20 con 0x7f6fb806c2d0 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 4 set_mon_vals no callback set 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fb6ffd640 10 monclient: dump: 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:00.261 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:00.262 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:00.262 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:00.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 10 monclient: shutdown 2024-01-20T11:22:00.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:00.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 1 -- 172.21.15.110:0/822602943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb806c2d0 msgr2=0x7f6fb80b1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:00.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 1 --2- 172.21.15.110:0/822602943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb806c2d0 0x7f6fb80b1990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6fac0097a0 tx=0x7f6fac02b8d0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 1 -- 172.21.15.110:0/822602943 shutdown_connections 2024-01-20T11:22:00.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 1 --2- 172.21.15.110:0/822602943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb806c2d0 0x7f6fb80b1990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 1 -- 172.21.15.110:0/822602943 >> 172.21.15.110:0/822602943 conn(0x7f6fb80b7cf0 msgr2=0x7f6fb80b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:00.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 1 -- 172.21.15.110:0/822602943 shutdown_connections 2024-01-20T11:22:00.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.260+0000 7f6fbd39e640 1 -- 172.21.15.110:0/822602943 wait complete. 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.261+0000 7f6fbd39e640 10 monclient: build_initial_monmap 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.261+0000 7f6fbd39e640 10 monclient: monmap: 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.261+0000 7f6fbd39e640 1 librados: starting msgr at 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.261+0000 7f6fbd39e640 1 librados: starting objecter 2024-01-20T11:22:00.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.261+0000 7f6fbd39e640 1 Processor -- start 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 1 -- start start 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 1 librados: setting wanted keys 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 1 librados: calling monclient init 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient: init 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fb806c2d0 0x7f6fb813b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient(hunting): picked mon.noname-b con 0x7f6fb806c2d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb80a9b50 0x7f6fb813db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb80a9b50 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fb813c2a0 0x7f6fb813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient(hunting): picked mon.noname-c con 0x7f6fb813c2a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb813e040 con 0x7f6fb80a9b50 2024-01-20T11:22:00.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb813e2b0 con 0x7f6fb806c2d0 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb8143a30 con 0x7f6fb813c2a0 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.262+0000 7f6fbd39e640 10 monclient(hunting): authenticate will time out at 1763.612950s 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fb77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb80a9b50 0x7f6fb813db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fbcb9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fb813c2a0 0x7f6fb813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fb77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb80a9b50 0x7f6fb813db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:59380/0 (socket says 172.21.15.110:59380) 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fb77fe640 1 -- 172.21.15.110:0/3509686154 learned_addr learned my addr 172.21.15.110:0/3509686154 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fb77fe640 10 monclient(hunting): get_auth_request con 0x7f6fb80a9b50 auth_method 0 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fb77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:00.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fb7fff640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fb806c2d0 0x7f6fb813b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fbcb9d640 10 monclient(hunting): get_auth_request con 0x7f6fb813c2a0 auth_method 0 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fbcb9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fbcb9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.263+0000 7f6fbcb9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb7fff640 10 monclient(hunting): get_auth_request con 0x7f6fb806c2d0 auth_method 0 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fbcb9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fbcb9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fbcb9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:00.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.264+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 10 monclient(hunting): handle_auth_done global_id 5384 payload 824 2024-01-20T11:22:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 1 -- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fb806c2d0 msgr2=0x7f6fb813b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fb806c2d0 0x7f6fb813b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 1 -- 172.21.15.110:0/3509686154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb80a9b50 msgr2=0x7f6fb813db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb80a9b50 0x7f6fb813db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 10 monclient: _finish_hunting 0 2024-01-20T11:22:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 1 monclient: found mon.noname-c 2024-01-20T11:22:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.267+0000 7f6fbcb9d640 1 -- 172.21.15.110:0/3509686154 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac009400 con 0x7f6fb813c2a0 2024-01-20T11:22:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.269+0000 7f6fbcb9d640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fb813c2a0 0x7f6fb813e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000cad0 tx=0x7f6fb000cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.269+0000 7f6fb4ff9640 10 client.?.objecter ms_handle_connect 0x7f6fb813c2a0 2024-01-20T11:22:00.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.269+0000 7f6fb4ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 1 -- 172.21.15.110:0/3509686154 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb000ab70 con 0x7f6fb813c2a0 2024-01-20T11:22:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 10 monclient: dump: 2024-01-20T11:22:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:00.273 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 10 monclient: _finish_auth 0 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fbd39e640 5 monclient: authenticate success, global_id 5384 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 1 -- 172.21.15.110:0/3509686154 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb000acd0 con 0x7f6fb813c2a0 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 1 -- 172.21.15.110:0/3509686154 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0012600 con 0x7f6fb813c2a0 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb4ff9640 10 monclient: dump: 2024-01-20T11:22:00.274 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fb67fc640 4 set_mon_vals no callback set 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fbd39e640 10 monclient: _renew_subs 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fbd39e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb8143d10 con 0x7f6fb813c2a0 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fbd39e640 10 client.5384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fbd39e640 10 monclient: _renew_subs 2024-01-20T11:22:00.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fbd39e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb813ee80 con 0x7f6fb813c2a0 2024-01-20T11:22:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.273+0000 7f6fbd39e640 1 librados: init done 2024-01-20T11:22:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.274+0000 7f6fbd39e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.274+0000 7f6fbd39e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.274+0000 7f6fbd39e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:00.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.274+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb8075490 con 0x7f6fb813c2a0 2024-01-20T11:22:00.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.293+0000 7f6fb4ff9640 1 -- 172.21.15.110:0/3509686154 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6fb0012760 con 0x7f6fb813c2a0 2024-01-20T11:22:00.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.293+0000 7f6fb4ff9640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fa406c890 0x7f6fa406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:00.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.294+0000 7f6fb7fff640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fa406c890 0x7f6fa406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:00.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.294+0000 7f6fb7fff640 10 monclient: get_auth_request con 0x7f6fa406c890 auth_method 0 2024-01-20T11:22:00.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.294+0000 7f6fb7fff640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fa406c890 0x7f6fa406ed50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6fac036000 tx=0x7f6fac03c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:00.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.294+0000 7f6fb4ff9640 10 client.5384.objecter ms_handle_connect 0x7f6fa406c890 2024-01-20T11:22:00.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.533+0000 7f6fb4ff9640 1 -- 172.21.15.110:0/3509686154 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6fb000ae40 con 0x7f6fb813c2a0 2024-01-20T11:22:00.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.533+0000 7f6fb4ff9640 10 client.5384.objecter ms_dispatch 0x7f6fb8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:00.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.533+0000 7f6fb4ff9640 3 client.5384.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:00.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.533+0000 7f6fb4ff9640 3 client.5384.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:00.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.533+0000 7f6fb4ff9640 20 client.5384.objecter dump_active .. 0 homeless 2024-01-20T11:22:00.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.533+0000 7f6fb4ff9640 1 -- 172.21.15.110:0/3509686154 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6fb0054b00 con 0x7f6fb813c2a0 2024-01-20T11:22:00.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.533+0000 7f6fb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:00.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.533+0000 7f6fb4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:00.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.707+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fb8143b70 con 0x7f6fa406c890 2024-01-20T11:22:00.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.709+0000 7f6fb4ff9640 1 -- 172.21.15.110:0/3509686154 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63895 (secure 0 0 0) 0x7f6fb8143b70 con 0x7f6fa406c890 2024-01-20T11:22:00.710 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:00.710 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.711+0000 7f6fbd39e640 10 librados: watch_flush enter 2024-01-20T11:22:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.711+0000 7f6fbd39e640 10 librados: watch_flush exit 2024-01-20T11:22:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.711+0000 7f6fbd39e640 20 client.5384.objecter shutdown clearing up homeless session... 2024-01-20T11:22:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.711+0000 7f6fbd39e640 10 client.5384.objecter successfully canceled tick 2024-01-20T11:22:00.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.711+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fa406c890 msgr2=0x7f6fa406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.711+0000 7f6fbd39e640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fa406c890 0x7f6fa406ed50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6fac036000 tx=0x7f6fac03c040 comp rx=0 tx=0).stop 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 10 monclient: shutdown 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fb813c2a0 msgr2=0x7f6fb813e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fb813c2a0 0x7f6fb813e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000cad0 tx=0x7f6fb000cf90 comp rx=0 tx=0).stop 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 shutdown_connections 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6fa406c890 0x7f6fa406ed50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6fb813c2a0 0x7f6fb813e760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fb80a9b50 0x7f6fb813db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 --2- 172.21.15.110:0/3509686154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6fb806c2d0 0x7f6fb813b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 >> 172.21.15.110:0/3509686154 conn(0x7f6fb80b7cf0 msgr2=0x7f6fb8133080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 shutdown_connections 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 -- 172.21.15.110:0/3509686154 wait complete. 2024-01-20T11:22:00.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:00.712+0000 7f6fbd39e640 1 librados: shutdown 2024-01-20T11:22:00.722 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-01-20T11:21:58.790979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:17:41.672067+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:17:41.672067+0000","last_peered":"2024-01-20T11:17:41.672067+0000","last_clean":"2024-01-20T11:17:41.672067+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:17:41.672067+0000","last_undegraded":"2024-01-20T11:17:41.672067+0000","last_fullsized":"2024-01-20T11:17:41.672067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":6.8300000000000001},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":8.9570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":8.2840000000000007},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":7.4720000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.121},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":11.374000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.317},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":6.1070000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":10.736000000000001},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":13.49}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":9.984},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.387}]}]},{"osd":7,"up_from":21,"seq":90194313301,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":13.166},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":6.3300000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.5899999999999999},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":16.009}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":15.18},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.609}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":9.3699999999999992},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":16.853999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":10.811999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":11.505000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":10.103},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":12.349}]}]},{"osd":3,"up_from":21,"seq":90194313302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.3550000000000004},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":4.7729999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.504},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":7.0410000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":8.8149999999999995},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":12.369}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":11.824999999999999},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":9.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":9.9610000000000003},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":13.289}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":14.976000000000001},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":14.198}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444116,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.9299999999999997},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.806}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":12.186999999999999},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":17.27}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.798999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":10.308}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":7.8200000000000003},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":9.6880000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":14.470000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":7.5469999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.585000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.286999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444117,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0309999999999997},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":11.27}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":7.7190000000000003},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.1879999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":10.308999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":9.5079999999999991}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":8.3369999999999997},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.585000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":13.702}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.669},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.800000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444117,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":8.5180000000000007},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":9.8859999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":9.0899999999999999},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":6.2649999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":10.952},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.162000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":7.2709999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":15.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":14.42},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.542}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":13.702},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":12.919}]}]},{"osd":6,"up_from":15,"seq":64424509525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":7.7140000000000004},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":13.568}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":10.785},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":10.185}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":15.058},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":5.9960000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.541},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":12.686999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":14.319000000000001},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":6.7080000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":8.4039999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":9.3420000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:03.724 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.837+0000 7f540ed2c640 10 monclient: get_monmap_and_config 2024-01-20T11:22:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.837+0000 7f540ed2c640 10 monclient: build_initial_monmap 2024-01-20T11:22:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.837+0000 7f540ed2c640 10 monclient: monmap: 2024-01-20T11:22:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:03.838101+0000 2024-01-20T11:22:03.838 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:03.838101+0000 2024-01-20T11:22:03.838 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:03.838 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:03.838 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:03.838 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:03.838 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:03.838 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:03.838 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:03.838 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 1 Processor -- start 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 1 -- start start 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient: init 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f54080b0c00 0x7f540813f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient(hunting): picked mon.noname-a con 0x7f54080b0c00 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f54080adbc0 0x7f540813f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient(hunting): picked mon.noname-c con 0x7f54080adbc0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:03.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54081420b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54080aaa40 con 0x7f54081420b0 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5408072680 con 0x7f54080b0c00 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54080aac90 con 0x7f54080adbc0 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.838+0000 7f540ed2c640 10 monclient(hunting): authenticate will time out at 1767.189060s 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.839+0000 7f540cac8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f54080b0c00 0x7f540813f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.839+0000 7f5407fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f54080adbc0 0x7f540813f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.839+0000 7f540cac8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f54080b0c00 0x7f540813f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:59382/0 (socket says 172.21.15.110:59382) 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.839+0000 7f540cac8640 1 -- 172.21.15.110:0/2480160786 learned_addr learned my addr 172.21.15.110:0/2480160786 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.839+0000 7f540cac8640 10 monclient(hunting): get_auth_request con 0x7f54080b0c00 auth_method 0 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.839+0000 7f540cac8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.839+0000 7f540cac8640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:03.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.839+0000 7f540cac8640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:03.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.840+0000 7f5407fff640 10 monclient(hunting): get_auth_request con 0x7f54080adbc0 auth_method 0 2024-01-20T11:22:03.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.840+0000 7f5407fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:03.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.840+0000 7f5407fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:03.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.840+0000 7f5407fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:03.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.840+0000 7f540cac8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:03.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.840+0000 7f540cac8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:03.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.840+0000 7f540cac8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.841+0000 7f5407fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.841+0000 7f5407fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.841+0000 7f5407fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.842+0000 7f540cac8640 10 monclient(hunting): handle_auth_done global_id 5703 payload 274 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.842+0000 7f540cac8640 1 -- 172.21.15.110:0/2480160786 >> v2:172.21.15.134:3300/0 conn(0x7f54080adbc0 msgr2=0x7f540813f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.842+0000 7f540cac8640 1 --2- 172.21.15.110:0/2480160786 >> v2:172.21.15.134:3300/0 conn(0x7f54080adbc0 0x7f540813f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.842+0000 7f540cac8640 1 -- 172.21.15.110:0/2480160786 >> v1:172.21.15.134:6789/0 conn(0x7f54081420b0 legacy=0x7f54081424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.842+0000 7f540cac8640 10 monclient: _finish_hunting 0 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.842+0000 7f540cac8640 1 monclient: found mon.noname-a 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.842+0000 7f540cac8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.842+0000 7f540cac8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:03.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.842+0000 7f540cac8640 1 -- 172.21.15.110:0/2480160786 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54080aaf70 con 0x7f54080b0c00 2024-01-20T11:22:03.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.844+0000 7f540cac8640 1 --2- 172.21.15.110:0/2480160786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54080b0c00 0x7f540813f190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f54080b12a0 tx=0x7f540002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c134a0ddc7aed46 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:03.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 1 -- 172.21.15.110:0/2480160786 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540002ca80 con 0x7f54080b0c00 2024-01-20T11:22:03.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:03.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:03.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 10 monclient: dump: 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:03.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 1 -- 172.21.15.110:0/2480160786 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540002cbe0 con 0x7f54080b0c00 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f540ed2c640 5 monclient: authenticate success, global_id 5703 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f540ed2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f540ed2c640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 1 -- 172.21.15.110:0/2480160786 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54000343f0 con 0x7f54080b0c00 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f540ed2c640 4 set_mon_vals no callback set 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.847+0000 7f54077fe640 10 monclient: dump: 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:03.849 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:03.850 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:03.850 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:03.850 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:03.850 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:03.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 10 monclient: shutdown 2024-01-20T11:22:03.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:03.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 1 -- 172.21.15.110:0/2480160786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54080b0c00 msgr2=0x7f540813f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:03.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 1 --2- 172.21.15.110:0/2480160786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54080b0c00 0x7f540813f190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f54080b12a0 tx=0x7f540002bee0 comp rx=0 tx=0).stop 2024-01-20T11:22:03.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 1 -- 172.21.15.110:0/2480160786 shutdown_connections 2024-01-20T11:22:03.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 1 --2- 172.21.15.110:0/2480160786 >> v2:172.21.15.134:3300/0 conn(0x7f54080adbc0 0x7f540813f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:03.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 1 --2- 172.21.15.110:0/2480160786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54080b0c00 0x7f540813f190 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 1 -- 172.21.15.110:0/2480160786 >> 172.21.15.110:0/2480160786 conn(0x7f54080b3e60 msgr2=0x7f54080b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 1 -- 172.21.15.110:0/2480160786 shutdown_connections 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.848+0000 7f540ed2c640 1 -- 172.21.15.110:0/2480160786 wait complete. 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.849+0000 7f540ed2c640 10 monclient: build_initial_monmap 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.849+0000 7f540ed2c640 10 monclient: monmap: 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:03.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.849+0000 7f540ed2c640 1 librados: starting msgr at 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.849+0000 7f540ed2c640 1 librados: starting objecter 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 Processor -- start 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 -- start start 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 librados: setting wanted keys 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 librados: calling monclient init 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient: init 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54080adbc0 0x7f54081dadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient(hunting): picked mon.noname-b con 0x7f54080adbc0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54080b0c00 0x7f54080b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient(hunting): picked mon.noname-c con 0x7f54080b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54081420b0 0x7f54080b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient(hunting): picked mon.noname-a con 0x7f54081420b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:03.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54080b2ee0 con 0x7f54081420b0 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5408134c70 con 0x7f54080adbc0 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54081db7b0 con 0x7f54080b0c00 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.850+0000 7f540ed2c640 10 monclient(hunting): authenticate will time out at 1767.200939s 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540d2c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54081420b0 0x7f54080b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f5407fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54080b0c00 0x7f54080b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540cac8640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54080adbc0 0x7f54081dadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540d2c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54081420b0 0x7f54080b18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:59384/0 (socket says 172.21.15.110:59384) 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540d2c9640 1 -- 172.21.15.110:0/3150223267 learned_addr learned my addr 172.21.15.110:0/3150223267 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540d2c9640 10 monclient(hunting): get_auth_request con 0x7f54081420b0 auth_method 0 2024-01-20T11:22:03.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540d2c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540d2c9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540d2c9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f5407fff640 10 monclient(hunting): get_auth_request con 0x7f54080b0c00 auth_method 0 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f5407fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f5407fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f5407fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540cac8640 10 monclient(hunting): get_auth_request con 0x7f54080adbc0 auth_method 0 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540cac8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540cac8640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.851+0000 7f540cac8640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.852+0000 7f540d2c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.852+0000 7f540d2c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.852+0000 7f540d2c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.852+0000 7f5407fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.852+0000 7f5407fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.852+0000 7f5407fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:03.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.852+0000 7f540cac8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:03.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.852+0000 7f540cac8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:03.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.852+0000 7f540cac8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:03.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.854+0000 7f540d2c9640 10 monclient(hunting): handle_auth_done global_id 5706 payload 824 2024-01-20T11:22:03.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.854+0000 7f540d2c9640 1 -- 172.21.15.110:0/3150223267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54080b0c00 msgr2=0x7f54080b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.854+0000 7f540d2c9640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54080b0c00 0x7f54080b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.855+0000 7f540d2c9640 1 -- 172.21.15.110:0/3150223267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54080adbc0 msgr2=0x7f54081dadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.855+0000 7f540d2c9640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54080adbc0 0x7f54081dadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.855+0000 7f540d2c9640 10 monclient: _finish_hunting 0 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.855+0000 7f540d2c9640 1 monclient: found mon.noname-a 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.855+0000 7f540d2c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.855+0000 7f540d2c9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.855+0000 7f540d2c9640 1 -- 172.21.15.110:0/3150223267 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5400009b30 con 0x7f54081420b0 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.855+0000 7f5407fff640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54080b0c00 0x7f54080b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.857+0000 7f540d2c9640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54081420b0 0x7f54080b18e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f53f000c390 tx=0x7f53f000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.857+0000 7f54057fa640 10 client.?.objecter ms_handle_connect 0x7f54081420b0 2024-01-20T11:22:03.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.857+0000 7f54057fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f54057fa640 1 -- 172.21.15.110:0/3150223267 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f00052d0 con 0x7f54081420b0 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f54057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f54057fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f54057fa640 10 monclient: dump: 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f54057fa640 10 monclient: _finish_auth 0 2024-01-20T11:22:03.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f54057fa640 10 monclient: _check_auth_tickets 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f54057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f54057fa640 1 -- 172.21.15.110:0/3150223267 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f0005430 con 0x7f54081420b0 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f540ed2c640 5 monclient: authenticate success, global_id 5706 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.860+0000 7f540ed2c640 10 monclient: _renew_subs 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54081db300 con 0x7f54081420b0 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f54057fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f54057fa640 1 -- 172.21.15.110:0/3150223267 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0003810 con 0x7f54081420b0 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f54057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f54057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f54057fa640 10 monclient: dump: 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:03.862 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f5406ffd640 4 set_mon_vals no callback set 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 10 client.5706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 10 monclient: _renew_subs 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54080b2000 con 0x7f54081420b0 2024-01-20T11:22:03.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 1 librados: init done 2024-01-20T11:22:03.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:03.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:03.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:03.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.861+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5408075490 con 0x7f54081420b0 2024-01-20T11:22:03.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.880+0000 7f54057fa640 1 -- 172.21.15.110:0/3150223267 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f53f0003970 con 0x7f54081420b0 2024-01-20T11:22:03.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.880+0000 7f54057fa640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f53e806c8b0 0x7f53e806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:03.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.881+0000 7f540cac8640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f53e806c8b0 0x7f53e806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:03.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.881+0000 7f540cac8640 10 monclient: get_auth_request con 0x7f53e806c8b0 auth_method 0 2024-01-20T11:22:03.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.882+0000 7f54057fa640 10 client.5706.objecter ms_handle_connect 0x7f53e806c8b0 2024-01-20T11:22:03.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.882+0000 7f540cac8640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f53e806c8b0 0x7f53e806ed70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5400035000 tx=0x7f5400030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:03.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.885+0000 7f54057fa640 1 -- 172.21.15.110:0/3150223267 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f53f0057b80 con 0x7f54081420b0 2024-01-20T11:22:03.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.885+0000 7f54057fa640 10 client.5706.objecter ms_dispatch 0x7f5408000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:03.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.885+0000 7f54057fa640 3 client.5706.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:03.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.885+0000 7f54057fa640 3 client.5706.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:03.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:03.885+0000 7f54057fa640 20 client.5706.objecter dump_active .. 0 homeless 2024-01-20T11:22:04.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.120+0000 7f54057fa640 1 -- 172.21.15.110:0/3150223267 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f53f0053ec0 con 0x7f54081420b0 2024-01-20T11:22:04.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.120+0000 7f54057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:04.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.120+0000 7f54057fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:04.270 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:55.294 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:04.274 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:55.299 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:04.289 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:55.313 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:04.318 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:07:55.324 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:04.321 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:07:55.327 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:04.321 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:55.345 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:04.322 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:55.346 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:04.334 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:07:55.341 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:04.336 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:55.360 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:04.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~186s 2024-01-20T11:22:04.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.359+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f540813cd20 con 0x7f53e806c8b0 2024-01-20T11:22:04.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.363+0000 7f54057fa640 1 -- 172.21.15.110:0/3150223267 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63870 (secure 0 0 0) 0x7f540813cd20 con 0x7f53e806c8b0 2024-01-20T11:22:04.364 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:04.364 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 10 librados: watch_flush enter 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 10 librados: watch_flush exit 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 20 client.5706.objecter shutdown clearing up homeless session... 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 10 client.5706.objecter successfully canceled tick 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f53e806c8b0 msgr2=0x7f53e806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f53e806c8b0 0x7f53e806ed70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5400035000 tx=0x7f5400030000 comp rx=0 tx=0).stop 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 10 monclient: shutdown 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54081420b0 msgr2=0x7f54080b18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54081420b0 0x7f54080b18e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f53f000c390 tx=0x7f53f000c850 comp rx=0 tx=0).stop 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 shutdown_connections 2024-01-20T11:22:04.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f53e806c8b0 0x7f53e806ed70 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54081420b0 0x7f54080b18e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54080b0c00 0x7f54080b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 --2- 172.21.15.110:0/3150223267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54080adbc0 0x7f54081dadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 >> 172.21.15.110:0/3150223267 conn(0x7f54080b3e60 msgr2=0x7f5408135100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 shutdown_connections 2024-01-20T11:22:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 -- 172.21.15.110:0/3150223267 wait complete. 2024-01-20T11:22:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.365+0000 7f540ed2c640 1 librados: shutdown 2024-01-20T11:22:04.376 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-01-20T11:22:02.791951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002734"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":6.8300000000000001},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":8.9570000000000007}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":8.2840000000000007},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":7.4720000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.121},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":11.374000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.317},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":6.1070000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":10.736000000000001},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":13.49}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":9.984},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":14.387}]}]},{"osd":7,"up_from":21,"seq":90194313302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.7969999999999997},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":9.3819999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.6029999999999998},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":8.282}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":7.6619999999999999},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":11.936}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":12.738},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":16.634}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":13.467000000000001},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":10.177}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14.164},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":14.919}]}]},{"osd":3,"up_from":21,"seq":90194313303,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":8.3789999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":6.3630000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.196},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.763}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":9.327},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":11.423999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":12.728},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.255000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":7.5289999999999999},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":12.145}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":14.242000000000001},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":16.693999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444116,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.9299999999999997},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.806}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":12.186999999999999},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":17.27}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.798999999999999},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":10.308}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":7.8200000000000003},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":9.6880000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":14.470000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":7.5469999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.585000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.286999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444117,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0309999999999997},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":11.27}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":7.7190000000000003},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.1879999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":10.308999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":9.5079999999999991}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":8.3369999999999997},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":11.904}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.585000000000001},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":13.702}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.669},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.800000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444117,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":8.5180000000000007},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":9.8859999999999992}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":9.0899999999999999},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":6.2649999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":10.952},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.162000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":7.2709999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":15.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":14.42},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.542}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":13.702},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":12.919}]}]},{"osd":6,"up_from":15,"seq":64424509526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.9039999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.2469999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":11.789999999999999},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":5.6619999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":14.785},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":10.743}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":6.5549999999999997},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":15.75}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":16.539000000000001},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":12.616}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.948},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":7.718}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:04.379 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.481+0000 7f5edcde2640 10 monclient: get_monmap_and_config 2024-01-20T11:22:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.481+0000 7f5edcde2640 10 monclient: build_initial_monmap 2024-01-20T11:22:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.481+0000 7f5edcde2640 10 monclient: monmap: 2024-01-20T11:22:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:04.482049+0000 2024-01-20T11:22:04.481 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:04.482049+0000 2024-01-20T11:22:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:04.482 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:04.482 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 1 Processor -- start 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 1 -- start start 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient: init 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5ed80b4d10 0x7f5ed80b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed80b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ed80b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5ed8145980 0x7f5ed8145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed8145980 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:04.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed8146370 con 0x7f5ed80b17c0 2024-01-20T11:22:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed80b50f0 con 0x7f5ed8145980 2024-01-20T11:22:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed80aade0 con 0x7f5ed80b4d10 2024-01-20T11:22:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.482+0000 7f5edcde2640 10 monclient(hunting): authenticate will time out at 1767.832861s 2024-01-20T11:22:04.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.484+0000 7f5ed6d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5ed8145980 0x7f5ed8145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:04.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.486+0000 7f5ed6d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5ed8145980 0x7f5ed8145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:59392/0 (socket says 172.21.15.110:59392) 2024-01-20T11:22:04.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.486+0000 7f5ed6d9d640 1 -- 172.21.15.110:0/1313917737 learned_addr learned my addr 172.21.15.110:0/1313917737 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:04.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.487+0000 7f5ed6d9d640 10 monclient(hunting): get_auth_request con 0x7f5ed8145980 auth_method 0 2024-01-20T11:22:04.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.487+0000 7f5ed6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:04.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.487+0000 7f5ed6d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:04.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.487+0000 7f5ed6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:04.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.489+0000 7f5ed6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:04.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.489+0000 7f5ed6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:04.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.489+0000 7f5ed6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:04.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed659c640 1 --2- 172.21.15.110:0/1313917737 >> v2:172.21.15.134:3300/0 conn(0x7f5ed80b4d10 0x7f5ed80b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:04.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed6d9d640 10 monclient(hunting): handle_auth_done global_id 5709 payload 274 2024-01-20T11:22:04.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed6d9d640 1 -- 172.21.15.110:0/1313917737 >> v2:172.21.15.134:3300/0 conn(0x7f5ed80b4d10 msgr2=0x7f5ed80b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:04.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed6d9d640 1 --2- 172.21.15.110:0/1313917737 >> v2:172.21.15.134:3300/0 conn(0x7f5ed80b4d10 0x7f5ed80b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed6d9d640 1 -- 172.21.15.110:0/1313917737 >> v1:172.21.15.134:6789/0 conn(0x7f5ed80b17c0 legacy=0x7f5ed80b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:04.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed6d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:22:04.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed6d9d640 1 monclient: found mon.noname-a 2024-01-20T11:22:04.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:04.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:04.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.491+0000 7f5ed6d9d640 1 -- 172.21.15.110:0/1313917737 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed8145460 con 0x7f5ed8145980 2024-01-20T11:22:04.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.501+0000 7f5ed6d9d640 1 --2- 172.21.15.110:0/1313917737 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed8145980 0x7f5ed8145e30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80b8cd0 tx=0x7f5ed002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7e813264b1a0b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:04.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 1 -- 172.21.15.110:0/1313917737 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed002c960 con 0x7f5ed8145980 2024-01-20T11:22:04.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:04.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:04.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 10 monclient: dump: 2024-01-20T11:22:04.507 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:04.507 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:04.507 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 10 monclient: _finish_auth 0 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 1 -- 172.21.15.110:0/1313917737 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed002cac0 con 0x7f5ed8145980 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5edcde2640 5 monclient: authenticate success, global_id 5709 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5edcde2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:04.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5edcde2640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5edcde2640 4 set_mon_vals no callback set 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 1 -- 172.21.15.110:0/1313917737 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed002ce50 con 0x7f5ed8145980 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.507+0000 7f5ed5d9b640 10 monclient: dump: 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:04.509 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:04.510 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:04.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 10 monclient: shutdown 2024-01-20T11:22:04.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 1 -- 172.21.15.110:0/1313917737 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed8145980 msgr2=0x7f5ed8145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 1 --2- 172.21.15.110:0/1313917737 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed8145980 0x7f5ed8145e30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80b8cd0 tx=0x7f5ed002bdc0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 1 -- 172.21.15.110:0/1313917737 shutdown_connections 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 1 --2- 172.21.15.110:0/1313917737 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed8145980 0x7f5ed8145e30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 1 --2- 172.21.15.110:0/1313917737 >> v2:172.21.15.134:3300/0 conn(0x7f5ed80b4d10 0x7f5ed80b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 1 -- 172.21.15.110:0/1313917737 >> 172.21.15.110:0/1313917737 conn(0x7f5ed8066590 msgr2=0x7f5ed80669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 1 -- 172.21.15.110:0/1313917737 shutdown_connections 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.508+0000 7f5edcde2640 1 -- 172.21.15.110:0/1313917737 wait complete. 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.509+0000 7f5edcde2640 10 monclient: build_initial_monmap 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.509+0000 7f5edcde2640 10 monclient: monmap: 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:04.512 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.509+0000 7f5edcde2640 1 librados: starting msgr at 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.509+0000 7f5edcde2640 1 librados: starting objecter 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 Processor -- start 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 -- start start 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 librados: setting wanted keys 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 librados: calling monclient init 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient: init 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:04.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ed80b17c0 0x7f5ed8142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed80b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed80b4d10 0x7f5ed8146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed80b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed8145980 0x7f5ed81d7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed8145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed8147120 con 0x7f5ed8145980 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed8147290 con 0x7f5ed80b17c0 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed8147450 con 0x7f5ed80b4d10 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.510+0000 7f5edcde2640 10 monclient(hunting): authenticate will time out at 1767.861041s 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.512+0000 7f5ed659c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ed80b17c0 0x7f5ed8142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:04.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.512+0000 7f5ecdd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed80b4d10 0x7f5ed8146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:04.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.514+0000 7f5ecdd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed80b4d10 0x7f5ed8146800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40720/0 (socket says 172.21.15.110:40720) 2024-01-20T11:22:04.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.514+0000 7f5ecdd9b640 1 -- 172.21.15.110:0/2632014641 learned_addr learned my addr 172.21.15.110:0/2632014641 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:04.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.514+0000 7f5ecdd9b640 10 monclient(hunting): get_auth_request con 0x7f5ed80b4d10 auth_method 0 2024-01-20T11:22:04.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.514+0000 7f5ecdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:04.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.514+0000 7f5ecdd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:04.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.514+0000 7f5ecdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:04.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.517+0000 7f5ecdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:04.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.517+0000 7f5ecdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:04.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.517+0000 7f5ecdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:04.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.517+0000 7f5ed6d9d640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed8145980 0x7f5ed81d7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:04.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.518+0000 7f5ed6d9d640 10 monclient(hunting): get_auth_request con 0x7f5ed8145980 auth_method 0 2024-01-20T11:22:04.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.518+0000 7f5ed6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:04.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.518+0000 7f5ed6d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:04.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.518+0000 7f5ed6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:04.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.520+0000 7f5ed6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:04.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.520+0000 7f5ed6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:04.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.520+0000 7f5ed6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 10 monclient(hunting): handle_auth_done global_id 5390 payload 824 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 1 -- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ed80b17c0 msgr2=0x7f5ed8142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ed80b17c0 0x7f5ed8142ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 1 -- 172.21.15.110:0/2632014641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed8145980 msgr2=0x7f5ed81d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed8145980 0x7f5ed81d7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 10 monclient: _finish_hunting 0 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 1 monclient: found mon.noname-c 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:04.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.522+0000 7f5ecdd9b640 1 -- 172.21.15.110:0/2632014641 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0009b30 con 0x7f5ed80b4d10 2024-01-20T11:22:04.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.527+0000 7f5ecdd9b640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed80b4d10 0x7f5ed8146800 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000b3c0 tx=0x7f5ec000b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:04.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.527+0000 7f5ecf7fe640 10 client.?.objecter ms_handle_connect 0x7f5ed80b4d10 2024-01-20T11:22:04.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.527+0000 7f5ecf7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:04.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5ecf7fe640 1 -- 172.21.15.110:0/2632014641 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0011830 con 0x7f5ed80b4d10 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5ecf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5ecf7fe640 10 monclient: dump: 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5ecf7fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5ecf7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5ecf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5edcde2640 5 monclient: authenticate success, global_id 5390 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5edcde2640 10 monclient: _renew_subs 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5edcde2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.538+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed81d7e90 con 0x7f5ed80b4d10 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.539+0000 7f5edcde2640 10 client.5390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.539+0000 7f5edcde2640 10 monclient: _renew_subs 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.539+0000 7f5edcde2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.539+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed814ba20 con 0x7f5ed80b4d10 2024-01-20T11:22:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.539+0000 7f5edcde2640 1 librados: init done 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.539+0000 7f5edcde2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.539+0000 7f5edcde2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.539+0000 7f5edcde2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.539+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed813bd20 con 0x7f5ed80b4d10 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.540+0000 7f5ecf7fe640 1 -- 172.21.15.110:0/2632014641 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec00091f0 con 0x7f5ed80b4d10 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.540+0000 7f5ecf7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.540+0000 7f5ecf7fe640 1 -- 172.21.15.110:0/2632014641 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0011bd0 con 0x7f5ed80b4d10 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.540+0000 7f5ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.540+0000 7f5ecf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.540+0000 7f5ecf7fe640 10 monclient: dump: 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.540+0000 7f5ed559a640 4 set_mon_vals no callback set 2024-01-20T11:22:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.796+0000 7f5ecf7fe640 1 -- 172.21.15.110:0/2632014641 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5ec001b460 con 0x7f5ed80b4d10 2024-01-20T11:22:04.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.796+0000 7f5ecf7fe640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:04.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.797+0000 7f5ed659c640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:04.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.797+0000 7f5ed659c640 10 monclient: get_auth_request con 0x7f5eb806c860 auth_method 0 2024-01-20T11:22:04.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.797+0000 7f5ecf7fe640 10 client.5390.objecter ms_handle_connect 0x7f5eb806c860 2024-01-20T11:22:04.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.797+0000 7f5ed659c640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb806ed20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8006fd0 tx=0x7f5ec8008040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.813+0000 7f5ecf7fe640 1 -- 172.21.15.110:0/2632014641 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5ec0059130 con 0x7f5ed80b4d10 2024-01-20T11:22:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.813+0000 7f5ecf7fe640 10 client.5390.objecter ms_dispatch 0x7f5ed8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.813+0000 7f5ecf7fe640 3 client.5390.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.813+0000 7f5ecf7fe640 3 client.5390.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.813+0000 7f5ecf7fe640 20 client.5390.objecter dump_active .. 0 homeless 2024-01-20T11:22:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.813+0000 7f5ecf7fe640 1 -- 172.21.15.110:0/2632014641 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5ec0058800 con 0x7f5ed80b4d10 2024-01-20T11:22:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.813+0000 7f5ecf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:04.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.813+0000 7f5ecf7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:04.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.983+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ed8075490 con 0x7f5eb806c860 2024-01-20T11:22:04.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.983+0000 7f5ed659c640 1 -- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 msgr2=0x7f5eb806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-01-20T11:22:04.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.983+0000 7f5ed659c640 1 -- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 msgr2=0x7f5eb806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-20T11:22:04.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.983+0000 7f5ed659c640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb806ed20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8006fd0 tx=0x7f5ec8008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-20T11:22:04.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.983+0000 7f5ed659c640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb806ed20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8006fd0 tx=0x7f5ec8008040 comp rx=0 tx=0).stop 2024-01-20T11:22:04.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.983+0000 7f5ecf7fe640 1 -- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 msgr2=0x7f5eb806ed20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-01-20T11:22:04.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:04.983+0000 7f5ecf7fe640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb806ed20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.797+0000 7f5ecd59a640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb0005560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.797+0000 7f5ecd59a640 1 -- 172.21.15.110:0/2632014641 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5eb0005ac0 con 0x7f5eb806c860 2024-01-20T11:22:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.797+0000 7f5ed659c640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb0005560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.797+0000 7f5ed659c640 10 monclient: get_auth_request con 0x7f5eb806c860 auth_method 0 2024-01-20T11:22:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.797+0000 7f5ed659c640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb0005560 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8008080 tx=0x7f5ec8012000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.798+0000 7f5ecf7fe640 10 client.5390.objecter ms_handle_connect 0x7f5eb806c860 2024-01-20T11:22:05.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.800+0000 7f5ecf7fe640 1 -- 172.21.15.110:0/2632014641 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63849 (secure 0 0 0) 0x7f5eb0005ac0 con 0x7f5eb806c860 2024-01-20T11:22:05.801 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:05.801 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 10 librados: watch_flush enter 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 10 librados: watch_flush exit 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 20 client.5390.objecter shutdown clearing up homeless session... 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 10 client.5390.objecter successfully canceled tick 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 msgr2=0x7f5eb0005560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb0005560 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8008080 tx=0x7f5ec8012000 comp rx=0 tx=0).stop 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 10 monclient: shutdown 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed80b4d10 msgr2=0x7f5ed8146800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed80b4d10 0x7f5ed8146800 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000b3c0 tx=0x7f5ec000b880 comp rx=0 tx=0).stop 2024-01-20T11:22:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 shutdown_connections 2024-01-20T11:22:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5eb806c860 0x7f5eb0005560 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed8145980 0x7f5ed81d7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed80b4d10 0x7f5ed8146800 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.802+0000 7f5edcde2640 1 --2- 172.21.15.110:0/2632014641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ed80b17c0 0x7f5ed8142ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.803+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 >> 172.21.15.110:0/2632014641 conn(0x7f5ed8066590 msgr2=0x7f5ed80b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.803+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 shutdown_connections 2024-01-20T11:22:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.803+0000 7f5edcde2640 1 -- 172.21.15.110:0/2632014641 wait complete. 2024-01-20T11:22:05.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.803+0000 7f5edcde2640 1 librados: shutdown 2024-01-20T11:22:05.813 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-01-20T11:22:04.792505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002775"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":9.7379999999999995},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":7.4690000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":5.1189999999999998},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":10.782}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":5.8789999999999996},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":11.680999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.441000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":13.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":6.4779999999999998},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":16.539000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":14.827},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":12.365}]}]},{"osd":7,"up_from":21,"seq":90194313302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.7969999999999997},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":9.3819999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.6029999999999998},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":8.282}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":7.6619999999999999},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":11.936}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":12.738},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":16.634}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":13.467000000000001},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":10.177}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14.164},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":14.919}]}]},{"osd":3,"up_from":21,"seq":90194313303,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":8.3789999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":6.3630000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.196},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.763}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":9.327},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":11.423999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":12.728},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.255000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":7.5289999999999999},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":12.145}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":14.242000000000001},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":16.693999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444117,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":15.388},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":14.302}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.2359999999999998},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":5.508}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":8.3580000000000005},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":11.475}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":10.837},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":8.8810000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.968},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":8.6080000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.151999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":12.087}]}]},{"osd":1,"up_from":17,"seq":73014444118,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.5069999999999997},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":9.6289999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":10.276},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.8550000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":11.414999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":8.0619999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":12.768000000000001},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":10.856999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":16.792999999999999},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":9.3520000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.602},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":12.093}]}]},{"osd":0,"up_from":17,"seq":73014444118,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":12.885},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.109}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":16.253},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":7.383}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":13.834},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":10.954000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":10.281000000000001},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":8.4719999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":14.446},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":15.138}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":19.408000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":11.611000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":4.9039999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.2469999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":11.789999999999999},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":5.6619999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":14.785},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":10.743}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":6.5549999999999997},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":15.75}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":16.539000000000001},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":12.616}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.948},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":7.718}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:05.815 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:05.815 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:05.815 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:05.815 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:05.815 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:05.815 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:05.815 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:05.815 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:05.816 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.919+0000 7f05a96de640 10 monclient: get_monmap_and_config 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.919+0000 7f05a96de640 10 monclient: build_initial_monmap 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.919+0000 7f05a96de640 10 monclient: monmap: 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:05.919820+0000 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:05.919820+0000 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:05.919 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:05.920 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:05.920 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:05.920 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:05.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.919+0000 7f05a96de640 1 Processor -- start 2024-01-20T11:22:05.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.919+0000 7f05a96de640 1 -- start start 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.919+0000 7f05a96de640 10 monclient: init 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05a40a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a9ad0 0x7f05a41423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 10 monclient(hunting): picked mon.noname-b con 0x7f05a40a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05a4142920 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a40b25c0 con 0x7f05a40a96f0 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a413fbe0 con 0x7f05a4142920 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40aaea0 con 0x7f05a40a9ad0 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.920+0000 7f05a96de640 10 monclient(hunting): authenticate will time out at 1769.270607s 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.921+0000 7f05a27fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a9ad0 0x7f05a41423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:05.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.921+0000 7f05a2ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f05a40a96f0 0x7f05a413f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48734/0 (socket says 172.21.15.110:48734) 2024-01-20T11:22:05.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.921+0000 7f05a2ffd640 1 -- 172.21.15.110:0/400892965 learned_addr learned my addr 172.21.15.110:0/400892965 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:05.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.921+0000 7f05a27fc640 10 monclient(hunting): get_auth_request con 0x7f05a40a9ad0 auth_method 0 2024-01-20T11:22:05.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.921+0000 7f05a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:05.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.921+0000 7f05a27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:05.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.921+0000 7f05a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:05.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.922+0000 7f05a27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:05.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.922+0000 7f05a27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:05.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.922+0000 7f05a27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.923+0000 7f05a1ffb640 1 -- 172.21.15.110:0/400892965 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403487063 0 0) 0x7f05a413fbe0 con 0x7f05a4142920 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.923+0000 7f05a1ffb640 10 monclient(hunting): _negotiate 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.923+0000 7f05a1ffb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.923+0000 7f05a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.923+0000 7f05a1ffb640 10 monclient(hunting): my global_id is 5794 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.923+0000 7f05a1ffb640 1 -- 172.21.15.110:0/400892965 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058c003520 con 0x7f05a4142920 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.924+0000 7f05a27fc640 10 monclient(hunting): handle_auth_done global_id 5393 payload 274 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.924+0000 7f05a27fc640 1 -- 172.21.15.110:0/400892965 >> v1:172.21.15.134:6789/0 conn(0x7f05a4142920 legacy=0x7f05a414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.924+0000 7f05a27fc640 1 -- 172.21.15.110:0/400892965 >> v1:172.21.15.110:6789/0 conn(0x7f05a40a96f0 legacy=0x7f05a413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:05.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.924+0000 7f05a27fc640 10 monclient: _finish_hunting 0 2024-01-20T11:22:05.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.924+0000 7f05a27fc640 1 monclient: found mon.noname-b 2024-01-20T11:22:05.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.924+0000 7f05a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:05.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.924+0000 7f05a27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:05.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.924+0000 7f05a27fc640 1 -- 172.21.15.110:0/400892965 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a40ab0a0 con 0x7f05a40a9ad0 2024-01-20T11:22:05.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.926+0000 7f05a27fc640 1 --2- 172.21.15.110:0/400892965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a9ad0 0x7f05a41423e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f05980097d0 tx=0x7f059802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14bcad720b4bcfe0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:05.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 1 -- 172.21.15.110:0/400892965 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059802d040 con 0x7f05a40a9ad0 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 10 monclient: dump: 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 10 monclient: _finish_auth 0 2024-01-20T11:22:05.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:22:05.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:05.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a96de640 5 monclient: authenticate success, global_id 5393 2024-01-20T11:22:05.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a96de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:05.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 1 -- 172.21.15.110:0/400892965 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0598031810 con 0x7f05a40a9ad0 2024-01-20T11:22:05.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:05.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a96de640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 1 -- 172.21.15.110:0/400892965 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0598031b80 con 0x7f05a40a9ad0 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a96de640 4 set_mon_vals no callback set 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.930+0000 7f05a1ffb640 10 monclient: dump: 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.931+0000 7f05a96de640 10 monclient: shutdown 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.931+0000 7f05a96de640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.931+0000 7f05a96de640 1 -- 172.21.15.110:0/400892965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a9ad0 msgr2=0x7f05a41423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.931+0000 7f05a96de640 1 --2- 172.21.15.110:0/400892965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a9ad0 0x7f05a41423e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f05980097d0 tx=0x7f059802b8b0 comp rx=0 tx=0).stop 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.931+0000 7f05a96de640 1 -- 172.21.15.110:0/400892965 shutdown_connections 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.931+0000 7f05a96de640 1 --2- 172.21.15.110:0/400892965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a9ad0 0x7f05a41423e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.931+0000 7f05a96de640 1 -- 172.21.15.110:0/400892965 >> 172.21.15.110:0/400892965 conn(0x7f05a40b7bc0 msgr2=0x7f05a40b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.931+0000 7f05a96de640 1 -- 172.21.15.110:0/400892965 shutdown_connections 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.931+0000 7f05a96de640 1 -- 172.21.15.110:0/400892965 wait complete. 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.932+0000 7f05a96de640 10 monclient: build_initial_monmap 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.932+0000 7f05a96de640 10 monclient: monmap: 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.932+0000 7f05a96de640 1 librados: starting msgr at 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.932+0000 7f05a96de640 1 librados: starting objecter 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.932+0000 7f05a96de640 1 Processor -- start 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.932+0000 7f05a96de640 1 -- start start 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.932+0000 7f05a96de640 1 librados: setting wanted keys 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.932+0000 7f05a96de640 1 librados: calling monclient init 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.932+0000 7f05a96de640 10 monclient: init 2024-01-20T11:22:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a96f0 0x7f05a413e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient(hunting): picked mon.noname-c con 0x7f05a40a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05a40a9ad0 0x7f05a41dac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient(hunting): picked mon.noname-a con 0x7f05a40a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a4142920 0x7f05a41db170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient(hunting): picked mon.noname-b con 0x7f05a4142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a41da7a0 con 0x7f05a40a9ad0 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40ac5d0 con 0x7f05a4142920 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40ac8b0 con 0x7f05a40a96f0 2024-01-20T11:22:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a96de640 10 monclient(hunting): authenticate will time out at 1769.283555s 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a96f0 0x7f05a413e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.933+0000 7f05a27fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05a40a9ad0 0x7f05a41dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a4142920 0x7f05a41db170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a4142920 0x7f05a41db170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:56166/0 (socket says 172.21.15.110:56166) 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 1 -- 172.21.15.110:0/872034425 learned_addr learned my addr 172.21.15.110:0/872034425 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 10 monclient(hunting): get_auth_request con 0x7f05a4142920 auth_method 0 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a2ffd640 10 monclient(hunting): get_auth_request con 0x7f05a40a96f0 auth_method 0 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:05.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a27fc640 10 monclient(hunting): get_auth_request con 0x7f05a40a9ad0 auth_method 0 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.934+0000 7f05a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.935+0000 7f05a27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.935+0000 7f05a27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.935+0000 7f05a27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 10 monclient(hunting): handle_auth_done global_id 5396 payload 824 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 1 -- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a4142920 msgr2=0x7f05a41db170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a4142920 0x7f05a41db170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 1 -- 172.21.15.110:0/872034425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05a40a9ad0 msgr2=0x7f05a41dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05a40a9ad0 0x7f05a41dac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 1 monclient: found mon.noname-c 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:05.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a2ffd640 1 -- 172.21.15.110:0/872034425 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598009400 con 0x7f05a40a96f0 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.937+0000 7f05a37fe640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a4142920 0x7f05a41db170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.939+0000 7f05a2ffd640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a96f0 0x7f05a413e970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f059400cae0 tx=0x7f059400a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.939+0000 7f058b7fe640 10 client.?.objecter ms_handle_connect 0x7f05a40a96f0 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.939+0000 7f058b7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.941+0000 7f058b7fe640 1 -- 172.21.15.110:0/872034425 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059400ac40 con 0x7f05a40a96f0 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.941+0000 7f058b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.941+0000 7f058b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.941+0000 7f058b7fe640 10 monclient: dump: 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.941+0000 7f058b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f058b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f058b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 5 monclient: authenticate success, global_id 5396 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 10 monclient: _renew_subs 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a4137bb0 con 0x7f05a40a96f0 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 10 client.5396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 10 monclient: _renew_subs 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a40ab550 con 0x7f05a40a96f0 2024-01-20T11:22:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 1 librados: init done 2024-01-20T11:22:05.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:05.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:05.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:05.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.942+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a4075490 con 0x7f05a40a96f0 2024-01-20T11:22:05.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.944+0000 7f058b7fe640 1 -- 172.21.15.110:0/872034425 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0594001ba0 con 0x7f05a40a96f0 2024-01-20T11:22:05.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.944+0000 7f058b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:05.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:05.944+0000 7f05a17fa640 4 set_mon_vals no callback set 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.182+0000 7f058b7fe640 1 -- 172.21.15.110:0/872034425 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059401c440 con 0x7f05a40a96f0 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.182+0000 7f058b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.182+0000 7f058b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.182+0000 7f058b7fe640 10 monclient: dump: 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:06.183 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.182+0000 7f058b7fe640 1 -- 172.21.15.110:0/872034425 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f0594012400 con 0x7f05a40a96f0 2024-01-20T11:22:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.182+0000 7f058b7fe640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f059006c860 0x7f059006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.183+0000 7f05a27fc640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f059006c860 0x7f059006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.183+0000 7f05a27fc640 10 monclient: get_auth_request con 0x7f059006c860 auth_method 0 2024-01-20T11:22:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.183+0000 7f05a27fc640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f059006c860 0x7f059006ed20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0598036000 tx=0x7f059803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.183+0000 7f058b7fe640 10 client.5396.objecter ms_handle_connect 0x7f059006c860 2024-01-20T11:22:06.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.194+0000 7f058b7fe640 1 -- 172.21.15.110:0/872034425 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f05940589c0 con 0x7f05a40a96f0 2024-01-20T11:22:06.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.194+0000 7f058b7fe640 10 client.5396.objecter ms_dispatch 0x7f05a4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:06.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.194+0000 7f058b7fe640 3 client.5396.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:06.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.194+0000 7f058b7fe640 3 client.5396.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:06.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.195+0000 7f058b7fe640 20 client.5396.objecter dump_active .. 0 homeless 2024-01-20T11:22:06.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.195+0000 7f058b7fe640 1 -- 172.21.15.110:0/872034425 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0594054d00 con 0x7f05a40a96f0 2024-01-20T11:22:06.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.195+0000 7f058b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:06.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.195+0000 7f058b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:06.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.442+0000 7f05a96de640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:06.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.442+0000 7f05a96de640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:06.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.442+0000 7f05a96de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:06.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.442+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f05a40ac9f0 con 0x7f05a40a96f0 2024-01-20T11:22:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.448+0000 7f058b7fe640 1 -- 172.21.15.110:0/872034425 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f05a40ac9f0 con 0x7f05a40a96f0 2024-01-20T11:22:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.448+0000 7f058b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.448+0000 7f058b7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:06.448 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:06.448 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":460,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 10 librados: watch_flush enter 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 10 librados: watch_flush exit 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 20 client.5396.objecter shutdown clearing up homeless session... 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 10 client.5396.objecter successfully canceled tick 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f059006c860 msgr2=0x7f059006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f059006c860 0x7f059006ed20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0598036000 tx=0x7f059803d040 comp rx=0 tx=0).stop 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 10 monclient: shutdown 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a96f0 msgr2=0x7f05a413e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a96f0 0x7f05a413e970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f059400cae0 tx=0x7f059400a060 comp rx=0 tx=0).stop 2024-01-20T11:22:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 shutdown_connections 2024-01-20T11:22:06.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f059006c860 0x7f059006ed20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:06.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a4142920 0x7f05a41db170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:06.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05a40a9ad0 0x7f05a41dac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:06.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 --2- 172.21.15.110:0/872034425 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05a40a96f0 0x7f05a413e970 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:06.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 >> 172.21.15.110:0/872034425 conn(0x7f05a40b7bc0 msgr2=0x7f05a413bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:06.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 shutdown_connections 2024-01-20T11:22:06.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 -- 172.21.15.110:0/872034425 wait complete. 2024-01-20T11:22:06.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.450+0000 7f05a96de640 1 librados: shutdown 2024-01-20T11:22:06.461 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:06.461 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.566+0000 7fd306ca2640 10 monclient: get_monmap_and_config 2024-01-20T11:22:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.566+0000 7fd306ca2640 10 monclient: build_initial_monmap 2024-01-20T11:22:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.566+0000 7fd306ca2640 10 monclient: monmap: 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:06.567480+0000 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:06.567480+0000 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:06.567 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.567+0000 7fd306ca2640 1 Processor -- start 2024-01-20T11:22:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.567+0000 7fd306ca2640 1 -- start start 2024-01-20T11:22:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.567+0000 7fd306ca2640 10 monclient: init 2024-01-20T11:22:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-01-20T11:22:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd30006bb00 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3000af790 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3000aeb20 0x7fd3000aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 10 monclient(hunting): picked mon.noname-b con 0x7fd3000aeb20 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000ada30 con 0x7fd30006bb00 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000add10 con 0x7fd3000af790 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000adff0 con 0x7fd3000aeb20 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd306ca2640 10 monclient(hunting): authenticate will time out at 1769.918411s 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd30523f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3000aeb20 0x7fd3000aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd304a3e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd30006bb00 0x7fd3000b03c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48744/0 (socket says 172.21.15.110:48744) 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.568+0000 7fd304a3e640 1 -- 172.21.15.110:0/1230687130 learned_addr learned my addr 172.21.15.110:0/1230687130 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.569+0000 7fd30523f640 10 monclient(hunting): get_auth_request con 0x7fd3000aeb20 auth_method 0 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.569+0000 7fd30523f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.569+0000 7fd30523f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.569+0000 7fd30523f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.569+0000 7fd30523f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.569+0000 7fd30523f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.569+0000 7fd30523f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd2f77fe640 1 -- 172.21.15.110:0/1230687130 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549880701 0 0) 0x7fd3000ada30 con 0x7fd30006bb00 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd2f77fe640 10 monclient(hunting): _negotiate 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd2f77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd2f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd2f77fe640 10 monclient(hunting): my global_id is 5721 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd2f77fe640 1 -- 172.21.15.110:0/1230687130 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e8003520 con 0x7fd30006bb00 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd30523f640 10 monclient(hunting): handle_auth_done global_id 5399 payload 274 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd30523f640 1 -- 172.21.15.110:0/1230687130 >> v1:172.21.15.134:6789/0 conn(0x7fd3000af790 legacy=0x7fd3000ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd30523f640 1 -- 172.21.15.110:0/1230687130 >> v1:172.21.15.110:6789/0 conn(0x7fd30006bb00 legacy=0x7fd3000b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd30523f640 10 monclient: _finish_hunting 0 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd30523f640 1 monclient: found mon.noname-b 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd30523f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd30523f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.571+0000 7fd30523f640 1 -- 172.21.15.110:0/1230687130 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30006c910 con 0x7fd3000aeb20 2024-01-20T11:22:06.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.573+0000 7fd30523f640 1 --2- 172.21.15.110:0/1230687130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3000aeb20 0x7fd3000aef70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009dc0 tx=0x7fd2f802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2620011d660516f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.575+0000 7fd2f77fe640 1 -- 172.21.15.110:0/1230687130 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f802cc40 con 0x7fd3000aeb20 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.575+0000 7fd2f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.575+0000 7fd2f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.575+0000 7fd2f77fe640 10 monclient: dump: 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.575+0000 7fd2f77fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.575+0000 7fd2f77fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.575+0000 7fd2f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.575+0000 7fd306ca2640 5 monclient: authenticate success, global_id 5399 2024-01-20T11:22:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.575+0000 7fd306ca2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.576+0000 7fd2f77fe640 1 -- 172.21.15.110:0/1230687130 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f802cda0 con 0x7fd3000aeb20 2024-01-20T11:22:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.576+0000 7fd2f77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.576+0000 7fd2f77fe640 1 -- 172.21.15.110:0/1230687130 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8034640 con 0x7fd3000aeb20 2024-01-20T11:22:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.576+0000 7fd2f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.576+0000 7fd2f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.576+0000 7fd2f77fe640 10 monclient: dump: 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.576+0000 7fd306ca2640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.576+0000 7fd306ca2640 4 set_mon_vals no callback set 2024-01-20T11:22:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.577+0000 7fd306ca2640 10 monclient: shutdown 2024-01-20T11:22:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.577+0000 7fd306ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.577+0000 7fd306ca2640 1 -- 172.21.15.110:0/1230687130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3000aeb20 msgr2=0x7fd3000aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.577+0000 7fd306ca2640 1 --2- 172.21.15.110:0/1230687130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3000aeb20 0x7fd3000aef70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009dc0 tx=0x7fd2f802be90 comp rx=0 tx=0).stop 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.577+0000 7fd306ca2640 1 -- 172.21.15.110:0/1230687130 shutdown_connections 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.577+0000 7fd306ca2640 1 --2- 172.21.15.110:0/1230687130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3000aeb20 0x7fd3000aef70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.577+0000 7fd306ca2640 1 -- 172.21.15.110:0/1230687130 >> 172.21.15.110:0/1230687130 conn(0x7fd300066d60 msgr2=0x7fd3000b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.577+0000 7fd306ca2640 1 -- 172.21.15.110:0/1230687130 shutdown_connections 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.577+0000 7fd306ca2640 1 -- 172.21.15.110:0/1230687130 wait complete. 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.578+0000 7fd306ca2640 10 monclient: build_initial_monmap 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.578+0000 7fd306ca2640 10 monclient: monmap: 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.578+0000 7fd306ca2640 1 librados: starting msgr at 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.578+0000 7fd306ca2640 1 librados: starting objecter 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.578+0000 7fd306ca2640 1 Processor -- start 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.578+0000 7fd306ca2640 1 -- start start 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.578+0000 7fd306ca2640 1 librados: setting wanted keys 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.578+0000 7fd306ca2640 1 librados: calling monclient init 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.578+0000 7fd306ca2640 10 monclient: init 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd30006bb00 0x7fd30011d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient(hunting): picked mon.noname-c con 0x7fd30006bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd3000aeb20 0x7fd3001d7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient(hunting): picked mon.noname-a con 0x7fd3000aeb20 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3000af790 0x7fd3001d7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient(hunting): picked mon.noname-b con 0x7fd3000af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30013f390 con 0x7fd3000aeb20 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30011ea00 con 0x7fd3000af790 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30011ece0 con 0x7fd30006bb00 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd306ca2640 10 monclient(hunting): authenticate will time out at 1769.929696s 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.579+0000 7fd304a3e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd30006bb00 0x7fd30011d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd2f7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd3000aeb20 0x7fd3001d7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:06.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd30523f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3000af790 0x7fd3001d7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:06.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd304a3e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd30006bb00 0x7fd30011d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40758/0 (socket says 172.21.15.110:40758) 2024-01-20T11:22:06.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd304a3e640 1 -- 172.21.15.110:0/4023783277 learned_addr learned my addr 172.21.15.110:0/4023783277 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:06.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd304a3e640 10 monclient(hunting): get_auth_request con 0x7fd30006bb00 auth_method 0 2024-01-20T11:22:06.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd304a3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:06.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd304a3e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:06.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd304a3e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd2f7fff640 10 monclient(hunting): get_auth_request con 0x7fd3000aeb20 auth_method 0 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd2f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd2f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd30523f640 10 monclient(hunting): get_auth_request con 0x7fd3000af790 auth_method 0 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd30523f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd30523f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd30523f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd304a3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd304a3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.580+0000 7fd304a3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.581+0000 7fd2f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.581+0000 7fd2f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.581+0000 7fd2f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.581+0000 7fd30523f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.581+0000 7fd30523f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.581+0000 7fd30523f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 10 monclient(hunting): handle_auth_done global_id 5402 payload 824 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 1 -- 172.21.15.110:0/4023783277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3000af790 msgr2=0x7fd3001d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3000af790 0x7fd3001d7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 1 -- 172.21.15.110:0/4023783277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd3000aeb20 msgr2=0x7fd3001d7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd3000aeb20 0x7fd3001d7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 10 monclient: _finish_hunting 0 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 1 monclient: found mon.noname-c 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.583+0000 7fd304a3e640 1 -- 172.21.15.110:0/4023783277 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f8009a70 con 0x7fd30006bb00 2024-01-20T11:22:06.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.585+0000 7fd304a3e640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd30006bb00 0x7fd30011d810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00c500 tx=0x7fd2fc00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:06.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.585+0000 7fd2f57fa640 10 client.?.objecter ms_handle_connect 0x7fd30006bb00 2024-01-20T11:22:06.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.585+0000 7fd2f57fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:06.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 1 -- 172.21.15.110:0/4023783277 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc005ce0 con 0x7fd30006bb00 2024-01-20T11:22:06.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:06.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:06.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 10 monclient: dump: 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 10 monclient: _finish_auth 0 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 10 monclient: _check_auth_tickets 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 1 -- 172.21.15.110:0/4023783277 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2fc005e40 con 0x7fd30006bb00 2024-01-20T11:22:06.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 1 -- 172.21.15.110:0/4023783277 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc007330 con 0x7fd30006bb00 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd306ca2640 5 monclient: authenticate success, global_id 5402 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd306ca2640 10 monclient: _renew_subs 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd306ca2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3001d7ac0 con 0x7fd30006bb00 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f6ffd640 4 set_mon_vals no callback set 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd2f57fa640 10 monclient: dump: 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:06.591 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:06.592 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:06.592 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:06.592 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:06.592 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:06.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd306ca2640 10 client.5402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:06.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd306ca2640 10 monclient: _renew_subs 2024-01-20T11:22:06.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd306ca2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:06.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30013fb70 con 0x7fd30006bb00 2024-01-20T11:22:06.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.589+0000 7fd306ca2640 1 librados: init done 2024-01-20T11:22:06.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.590+0000 7fd306ca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:06.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.590+0000 7fd306ca2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:06.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.590+0000 7fd306ca2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:06.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.590+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd300075490 con 0x7fd30006bb00 2024-01-20T11:22:06.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.610+0000 7fd2f57fa640 1 -- 172.21.15.110:0/4023783277 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd2fc007550 con 0x7fd30006bb00 2024-01-20T11:22:06.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.610+0000 7fd2f57fa640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2ec06c860 0x7fd2ec06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:06.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.611+0000 7fd2f7fff640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2ec06c860 0x7fd2ec06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:06.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.611+0000 7fd2f7fff640 10 monclient: get_auth_request con 0x7fd2ec06c860 auth_method 0 2024-01-20T11:22:06.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.611+0000 7fd2f57fa640 10 client.5402.objecter ms_handle_connect 0x7fd2ec06c860 2024-01-20T11:22:06.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.611+0000 7fd2f7fff640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2ec06c860 0x7fd2ec06ed20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd30006f690 tx=0x7fd2f0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.838+0000 7fd2f57fa640 1 -- 172.21.15.110:0/4023783277 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd2fc057dc0 con 0x7fd30006bb00 2024-01-20T11:22:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.838+0000 7fd2f57fa640 10 client.5402.objecter ms_dispatch 0x7fd300000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.838+0000 7fd2f57fa640 3 client.5402.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.838+0000 7fd2f57fa640 3 client.5402.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.839+0000 7fd2f57fa640 20 client.5402.objecter dump_active .. 0 homeless 2024-01-20T11:22:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.839+0000 7fd2f57fa640 1 -- 172.21.15.110:0/4023783277 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd2fc054100 con 0x7fd30006bb00 2024-01-20T11:22:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.839+0000 7fd2f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:06.839+0000 7fd2f57fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:07.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.008+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd300143930 con 0x7fd2ec06c860 2024-01-20T11:22:07.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.011+0000 7fd2f57fa640 1 -- 172.21.15.110:0/4023783277 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63947 (secure 0 0 0) 0x7fd300143930 con 0x7fd2ec06c860 2024-01-20T11:22:07.011 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:07.012 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 10 librados: watch_flush enter 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 10 librados: watch_flush exit 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 20 client.5402.objecter shutdown clearing up homeless session... 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 10 client.5402.objecter successfully canceled tick 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2ec06c860 msgr2=0x7fd2ec06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2ec06c860 0x7fd2ec06ed20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd30006f690 tx=0x7fd2f0009040 comp rx=0 tx=0).stop 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 10 monclient: shutdown 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd30006bb00 msgr2=0x7fd30011d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd30006bb00 0x7fd30011d810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00c500 tx=0x7fd2fc00c9c0 comp rx=0 tx=0).stop 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 shutdown_connections 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd2ec06c860 0x7fd2ec06ed20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd3000af790 0x7fd3001d7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd3000aeb20 0x7fd3001d7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 --2- 172.21.15.110:0/4023783277 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd30006bb00 0x7fd30011d810 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 >> 172.21.15.110:0/4023783277 conn(0x7fd300066d60 msgr2=0x7fd3000b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 shutdown_connections 2024-01-20T11:22:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 -- 172.21.15.110:0/4023783277 wait complete. 2024-01-20T11:22:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:07.013+0000 7fd306ca2640 1 librados: shutdown 2024-01-20T11:22:07.024 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-01-20T11:22:06.792981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002777"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":9.7379999999999995},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":7.4690000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":5.1189999999999998},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":10.782}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":5.8789999999999996},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":11.680999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.441000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":13.928000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":6.4779999999999998},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":16.539000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":14.827},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":12.365}]}]},{"osd":7,"up_from":21,"seq":90194313303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.7009999999999996},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":8.3049999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":6.9619999999999997},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":10.212999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":11.582000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":14.561999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":10.51},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":12.249000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":13.003},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":10.986000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":16.085000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":15.382999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313303,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":8.3789999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":6.3630000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":10.196},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.763}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":9.327},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":11.423999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":12.728},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.255000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":7.5289999999999999},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":12.145}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":14.242000000000001},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":16.693999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444117,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":15.388},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":14.302}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.2359999999999998},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":5.508}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":8.3580000000000005},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":11.475}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":10.837},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":8.8810000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.968},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":8.6080000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.151999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":12.087}]}]},{"osd":1,"up_from":17,"seq":73014444118,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.5069999999999997},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":9.6289999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":10.276},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":6.8550000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":11.414999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":8.0619999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":12.768000000000001},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":10.856999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":16.792999999999999},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":9.3520000000000003}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.602},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":12.093}]}]},{"osd":0,"up_from":17,"seq":73014444118,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":12.885},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":8.109}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":16.253},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":7.383}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":13.834},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":10.954000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":10.281000000000001},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":8.4719999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":14.446},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":15.138}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":19.408000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":11.611000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":9.8810000000000002},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":4.4580000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.3659999999999997},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":11.718}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":13.066000000000001},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":8.2469999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":12.397},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":7.6600000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":13.677},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":8.9049999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":9.4939999999999998},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.396000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:10.027 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:10.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.126+0000 7efd7891d640 10 monclient: get_monmap_and_config 2024-01-20T11:22:10.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.126+0000 7efd7891d640 10 monclient: build_initial_monmap 2024-01-20T11:22:10.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.126+0000 7efd7891d640 10 monclient: monmap: 2024-01-20T11:22:10.126 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:10.126 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.126 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:10.126807+0000 2024-01-20T11:22:10.126 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:10.126807+0000 2024-01-20T11:22:10.127 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:10.127 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.127 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:10.127 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:10.127 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:10.127 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:10.127 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:10.127 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.126+0000 7efd7891d640 1 Processor -- start 2024-01-20T11:22:10.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 1 -- start start 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient: init 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd740b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efd740b1ed0 0x7efd740b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient(hunting): picked mon.noname-a con 0x7efd740b1ed0 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efd74145980 0x7efd74145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient(hunting): picked mon.noname-c con 0x7efd74145980 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd74146370 con 0x7efd740b4d10 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd740b50f0 con 0x7efd740b1ed0 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd740aade0 con 0x7efd74145980 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.127+0000 7efd7891d640 10 monclient(hunting): authenticate will time out at 1773.477788s 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd6bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efd740b1ed0 0x7efd740b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd72d9d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efd74145980 0x7efd74145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd7259c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7efd740b4d10 0x7efd740b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58728/0 (socket says 172.21.15.110:58728) 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd7259c640 1 -- 172.21.15.110:0/4138650221 learned_addr learned my addr 172.21.15.110:0/4138650221 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd6bfff640 10 monclient(hunting): get_auth_request con 0x7efd740b1ed0 auth_method 0 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd6bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd72d9d640 10 monclient(hunting): get_auth_request con 0x7efd74145980 auth_method 0 2024-01-20T11:22:10.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd72d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd72d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.128+0000 7efd72d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.129+0000 7efd72d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.129+0000 7efd72d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.129+0000 7efd72d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.129+0000 7efd6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.129+0000 7efd6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.129+0000 7efd6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.131+0000 7efd72d9d640 10 monclient(hunting): handle_auth_done global_id 5806 payload 274 2024-01-20T11:22:10.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.131+0000 7efd72d9d640 1 -- 172.21.15.110:0/4138650221 >> v2:172.21.15.110:3300/0 conn(0x7efd740b1ed0 msgr2=0x7efd740b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:10.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.131+0000 7efd72d9d640 1 --2- 172.21.15.110:0/4138650221 >> v2:172.21.15.110:3300/0 conn(0x7efd740b1ed0 0x7efd740b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.131+0000 7efd72d9d640 1 -- 172.21.15.110:0/4138650221 >> v1:172.21.15.110:6789/0 conn(0x7efd740b4d10 legacy=0x7efd740b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.131+0000 7efd72d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:22:10.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.131+0000 7efd72d9d640 1 monclient: found mon.noname-c 2024-01-20T11:22:10.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.131+0000 7efd72d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:10.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.131+0000 7efd72d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:22:10.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.131+0000 7efd72d9d640 1 -- 172.21.15.110:0/4138650221 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd74145460 con 0x7efd74145980 2024-01-20T11:22:10.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.133+0000 7efd72d9d640 1 --2- 172.21.15.110:0/4138650221 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd74145980 0x7efd74145e30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7efd740aa1c0 tx=0x7efd6c02bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51a1f6e2fa28e0f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:10.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 1 -- 172.21.15.110:0/4138650221 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c02cb60 con 0x7efd74145980 2024-01-20T11:22:10.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:10.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:10.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 10 monclient: dump: 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 10 monclient: _finish_auth 0 2024-01-20T11:22:10.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 1 -- 172.21.15.110:0/4138650221 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c02ccc0 con 0x7efd74145980 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 1 -- 172.21.15.110:0/4138650221 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c034660 con 0x7efd74145980 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd71d9b640 10 monclient: dump: 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:10.139 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.140 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:10.140 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:10.140 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:10.140 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd7891d640 5 monclient: authenticate success, global_id 5806 2024-01-20T11:22:10.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd7891d640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:10.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.137+0000 7efd7891d640 4 set_mon_vals no callback set 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 10 monclient: shutdown 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 1 -- 172.21.15.110:0/4138650221 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd74145980 msgr2=0x7efd74145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 1 --2- 172.21.15.110:0/4138650221 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd74145980 0x7efd74145e30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7efd740aa1c0 tx=0x7efd6c02bdb0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 1 -- 172.21.15.110:0/4138650221 shutdown_connections 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 1 --2- 172.21.15.110:0/4138650221 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd74145980 0x7efd74145e30 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 1 --2- 172.21.15.110:0/4138650221 >> v2:172.21.15.110:3300/0 conn(0x7efd740b1ed0 0x7efd740b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 1 -- 172.21.15.110:0/4138650221 >> 172.21.15.110:0/4138650221 conn(0x7efd74066590 msgr2=0x7efd740669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 1 -- 172.21.15.110:0/4138650221 shutdown_connections 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.138+0000 7efd7891d640 1 -- 172.21.15.110:0/4138650221 wait complete. 2024-01-20T11:22:10.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.139+0000 7efd7891d640 10 monclient: build_initial_monmap 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.139+0000 7efd7891d640 10 monclient: monmap: 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.139+0000 7efd7891d640 1 librados: starting msgr at 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.139+0000 7efd7891d640 1 librados: starting objecter 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.139+0000 7efd7891d640 1 Processor -- start 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.139+0000 7efd7891d640 1 -- start start 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.139+0000 7efd7891d640 1 librados: setting wanted keys 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.139+0000 7efd7891d640 1 librados: calling monclient init 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.139+0000 7efd7891d640 10 monclient: init 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd740b1ed0 0x7efd7414b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient(hunting): picked mon.noname-b con 0x7efd740b1ed0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd740b4d10 0x7efd74139370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient(hunting): picked mon.noname-c con 0x7efd740b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:10.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd74145980 0x7efd741daf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient(hunting): picked mon.noname-a con 0x7efd74145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74139a90 con 0x7efd74145980 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd741398b0 con 0x7efd740b1ed0 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7413fd00 con 0x7efd740b4d10 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7891d640 10 monclient(hunting): authenticate will time out at 1773.490385s 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd72d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd74145980 0x7efd741daf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd7259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd740b1ed0 0x7efd7414b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.140+0000 7efd6bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd740b4d10 0x7efd74139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd72d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd74145980 0x7efd741daf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44606/0 (socket says 172.21.15.110:44606) 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd72d9d640 1 -- 172.21.15.110:0/815167460 learned_addr learned my addr 172.21.15.110:0/815167460 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd72d9d640 10 monclient(hunting): get_auth_request con 0x7efd74145980 auth_method 0 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd72d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd72d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd72d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd7259c640 10 monclient(hunting): get_auth_request con 0x7efd740b1ed0 auth_method 0 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd7259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd7259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd6bfff640 10 monclient(hunting): get_auth_request con 0x7efd740b4d10 auth_method 0 2024-01-20T11:22:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd6bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd72d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd72d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd72d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.141+0000 7efd6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 10 monclient(hunting): handle_auth_done global_id 5733 payload 824 2024-01-20T11:22:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 1 -- 172.21.15.110:0/815167460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd740b4d10 msgr2=0x7efd74139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd740b4d10 0x7efd74139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 1 -- 172.21.15.110:0/815167460 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd740b1ed0 msgr2=0x7efd7414b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd740b1ed0 0x7efd7414b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:22:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 1 monclient: found mon.noname-a 2024-01-20T11:22:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:10.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.144+0000 7efd72d9d640 1 -- 172.21.15.110:0/815167460 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c009a70 con 0x7efd74145980 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.146+0000 7efd72d9d640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd74145980 0x7efd741daf00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efd6c02c3c0 tx=0x7efd6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.146+0000 7efd6affd640 10 client.?.objecter ms_handle_connect 0x7efd74145980 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.146+0000 7efd6affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 1 -- 172.21.15.110:0/815167460 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c03e610 con 0x7efd74145980 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 10 monclient: dump: 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 10 monclient: _finish_auth 0 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 1 -- 172.21.15.110:0/815167460 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c03e770 con 0x7efd74145980 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7891d640 5 monclient: authenticate success, global_id 5733 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7891d640 10 monclient: _renew_subs 2024-01-20T11:22:10.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 1 -- 172.21.15.110:0/815167460 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c03eb20 con 0x7efd74145980 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7891d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd741418c0 con 0x7efd74145980 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7159a640 4 set_mon_vals no callback set 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd6affd640 10 monclient: dump: 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7891d640 10 client.5733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7891d640 10 monclient: _renew_subs 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7891d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd74141c50 con 0x7efd74145980 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.149+0000 7efd7891d640 1 librados: init done 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.150+0000 7efd7891d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.150+0000 7efd7891d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.150+0000 7efd7891d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.150+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd74075490 con 0x7efd74145980 2024-01-20T11:22:10.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.169+0000 7efd6affd640 1 -- 172.21.15.110:0/815167460 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7efd6c03b050 con 0x7efd74145980 2024-01-20T11:22:10.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.169+0000 7efd6affd640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.170+0000 7efd7259c640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.170+0000 7efd7259c640 10 monclient: get_auth_request con 0x7efd5406c8b0 auth_method 0 2024-01-20T11:22:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.170+0000 7efd7259c640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efd64007fd0 tx=0x7efd64007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:10.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.170+0000 7efd6affd640 10 client.5733.objecter ms_handle_connect 0x7efd5406c8b0 2024-01-20T11:22:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.171+0000 7efd6affd640 1 -- 172.21.15.110:0/815167460 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efd6c07ae10 con 0x7efd74145980 2024-01-20T11:22:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.171+0000 7efd6affd640 10 client.5733.objecter ms_dispatch 0x7efd74000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:10.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.171+0000 7efd6affd640 3 client.5733.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.171+0000 7efd6affd640 3 client.5733.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:10.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.171+0000 7efd6affd640 20 client.5733.objecter dump_active .. 0 homeless 2024-01-20T11:22:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.401+0000 7efd6affd640 1 -- 172.21.15.110:0/815167460 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7efd6c077150 con 0x7efd74145980 2024-01-20T11:22:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.401+0000 7efd6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.401+0000 7efd6affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:10.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~193s 2024-01-20T11:22:10.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.618+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd740b55c0 con 0x7efd5406c8b0 2024-01-20T11:22:10.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.621+0000 7efd6affd640 1 -- 172.21.15.110:0/815167460 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64039 (secure 0 0 0) 0x7efd740b55c0 con 0x7efd5406c8b0 2024-01-20T11:22:10.622 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:10.623 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 10 librados: watch_flush enter 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 10 librados: watch_flush exit 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 20 client.5733.objecter shutdown clearing up homeless session... 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 10 client.5733.objecter successfully canceled tick 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 msgr2=0x7efd5406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efd64007fd0 tx=0x7efd64007e80 comp rx=0 tx=0).stop 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 10 monclient: shutdown 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd74145980 msgr2=0x7efd741daf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd74145980 0x7efd741daf00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efd6c02c3c0 tx=0x7efd6c004a40 comp rx=0 tx=0).stop 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 shutdown_connections 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efd5406c8b0 0x7efd5406ed70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd74145980 0x7efd741daf00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd740b4d10 0x7efd74139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 --2- 172.21.15.110:0/815167460 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd740b1ed0 0x7efd7414b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 >> 172.21.15.110:0/815167460 conn(0x7efd74066590 msgr2=0x7efd740b83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 shutdown_connections 2024-01-20T11:22:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 -- 172.21.15.110:0/815167460 wait complete. 2024-01-20T11:22:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.623+0000 7efd7891d640 1 librados: shutdown 2024-01-20T11:22:10.635 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-01-20T11:22:08.793389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002771"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":15.798999999999999},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":10.657999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":12.972},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":5.9189999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":14.473000000000001},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":7.827}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":9.1890000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":9.9570000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":6.6559999999999997},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":13.659000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":8.4640000000000004},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":11.371}]}]},{"osd":7,"up_from":21,"seq":90194313303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.7009999999999996},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":8.3049999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":6.9619999999999997},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":10.212999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":11.582000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":14.561999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":10.51},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":12.249000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":13.003},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":10.986000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":16.085000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":15.382999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313304,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0410000000000004},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":14.662000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":13.605},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":11}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":11.731},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":15.254}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":7.9530000000000003},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.872999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":8.5220000000000002},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":5.8879999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":9.9540000000000006},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.946999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444118,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":10.851000000000001},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.7409999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":14.66},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.1799999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.004},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":8.4700000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":10.16},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":13.271000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.882999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":7.8979999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":15.534000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":9.5760000000000005}]}]},{"osd":1,"up_from":17,"seq":73014444119,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0590000000000002},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":4.9470000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":8.6400000000000006},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":10.534000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":13.951000000000001},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":11.557}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":13.555},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.872}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.9370000000000003},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":12.773999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":10.901},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":12.164}]}]},{"osd":0,"up_from":17,"seq":73014444119,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":8.4450000000000003},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":6.8159999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.383},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":14.018000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":14.930999999999999},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":11.196999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":12.097},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":9.4629999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":19.693999999999999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":13.271000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":16.280000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":8.0429999999999993}]}]},{"osd":6,"up_from":15,"seq":64424509527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":9.8810000000000002},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":4.4580000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.3659999999999997},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":11.718}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":13.066000000000001},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":8.2469999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":12.397},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":7.6600000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":13.677},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":8.9049999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":9.4939999999999998},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.396000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:10.636 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:10.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.734+0000 7fcf228a2640 10 monclient: get_monmap_and_config 2024-01-20T11:22:10.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.734+0000 7fcf228a2640 10 monclient: build_initial_monmap 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.734+0000 7fcf228a2640 10 monclient: monmap: 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:10.736400+0000 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:10.736400+0000 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:10.736 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.735+0000 7fcf228a2640 1 Processor -- start 2024-01-20T11:22:10.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.735+0000 7fcf228a2640 1 -- start start 2024-01-20T11:22:10.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.735+0000 7fcf228a2640 10 monclient: init 2024-01-20T11:22:10.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:10.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c0accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1c0b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcf1c13d0e0 0x7fcf1c13d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c13d0e0 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf1c143200 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c06dcc0 con 0x7fcf1c143200 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c0ad450 con 0x7fcf1c13d0e0 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0ad8e0 con 0x7fcf1c0b0c00 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf228a2640 10 monclient(hunting): authenticate will time out at 1774.087368s 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf1b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcf1c13d0e0 0x7fcf1c13d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.736+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c0accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf20e3f640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fcf1c143200 0x7fcf1c143690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:40656/0 (socket says 172.21.15.110:40656) 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf20e3f640 1 -- 172.21.15.110:0/1325779757 learned_addr learned my addr 172.21.15.110:0/1325779757 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1bfff640 10 monclient(hunting): get_auth_request con 0x7fcf1c0b0c00 auth_method 0 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1b7fe640 10 monclient(hunting): get_auth_request con 0x7fcf1c13d0e0 auth_method 0 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.737+0000 7fcf1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1bfff640 10 monclient(hunting): handle_auth_done global_id 5408 payload 274 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1bfff640 1 -- 172.21.15.110:0/1325779757 >> v2:172.21.15.110:3300/0 conn(0x7fcf1c13d0e0 msgr2=0x7fcf1c13d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1bfff640 1 --2- 172.21.15.110:0/1325779757 >> v2:172.21.15.110:3300/0 conn(0x7fcf1c13d0e0 0x7fcf1c13d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1bfff640 1 -- 172.21.15.110:0/1325779757 >> v1:172.21.15.134:6789/0 conn(0x7fcf1c143200 legacy=0x7fcf1c143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1bfff640 1 monclient: found mon.noname-b 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1bfff640 1 -- 172.21.15.110:0/1325779757 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c143da0 con 0x7fcf1c0b0c00 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.739+0000 7fcf1b7fe640 1 --2- 172.21.15.110:0/1325779757 >> v2:172.21.15.110:3300/0 conn(0x7fcf1c13d0e0 0x7fcf1c13d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:22:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.741+0000 7fcf1bfff640 1 --2- 172.21.15.110:0/1325779757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c0accf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0097a0 tx=0x7fcf0c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1095a2055d51f9f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:10.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 1 -- 172.21.15.110:0/1325779757 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c02e070 con 0x7fcf1c0b0c00 2024-01-20T11:22:10.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:10.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:10.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 10 monclient: dump: 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 10 monclient: _finish_auth 0 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 1 -- 172.21.15.110:0/1325779757 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0315a0 con 0x7fcf1c0b0c00 2024-01-20T11:22:10.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 1 -- 172.21.15.110:0/1325779757 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c031930 con 0x7fcf1c0b0c00 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf228a2640 5 monclient: authenticate success, global_id 5408 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf228a2640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf228a2640 4 set_mon_vals no callback set 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.744+0000 7fcf1affd640 10 monclient: dump: 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:10.747 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:10.748 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:10.748 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 10 monclient: shutdown 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 1 -- 172.21.15.110:0/1325779757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 msgr2=0x7fcf1c0accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 1 --2- 172.21.15.110:0/1325779757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c0accf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0097a0 tx=0x7fcf0c02b840 comp rx=0 tx=0).stop 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 1 -- 172.21.15.110:0/1325779757 shutdown_connections 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 1 --2- 172.21.15.110:0/1325779757 >> v2:172.21.15.110:3300/0 conn(0x7fcf1c13d0e0 0x7fcf1c13d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 1 --2- 172.21.15.110:0/1325779757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c0accf0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 1 -- 172.21.15.110:0/1325779757 >> 172.21.15.110:0/1325779757 conn(0x7fcf1c0b3e60 msgr2=0x7fcf1c0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 1 -- 172.21.15.110:0/1325779757 shutdown_connections 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.745+0000 7fcf228a2640 1 -- 172.21.15.110:0/1325779757 wait complete. 2024-01-20T11:22:10.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.746+0000 7fcf228a2640 10 monclient: build_initial_monmap 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.746+0000 7fcf228a2640 10 monclient: monmap: 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.746+0000 7fcf228a2640 1 librados: starting msgr at 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.746+0000 7fcf228a2640 1 librados: starting objecter 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.746+0000 7fcf228a2640 1 Processor -- start 2024-01-20T11:22:10.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.746+0000 7fcf228a2640 1 -- start start 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.746+0000 7fcf228a2640 1 librados: setting wanted keys 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.746+0000 7fcf228a2640 1 librados: calling monclient init 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.746+0000 7fcf228a2640 10 monclient: init 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c13fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1c0b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c13d0e0 0x7fcf1c1486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c13d0e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf1c143200 0x7fcf1c148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1c143200 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0ac2b0 con 0x7fcf1c13d0e0 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1328d0 con 0x7fcf1c143200 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0aa970 con 0x7fcf1c0b0c00 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf228a2640 10 monclient(hunting): authenticate will time out at 1774.098480s 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf1b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c13d0e0 0x7fcf1c1486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c13fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.747+0000 7fcf20e3f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf1c143200 0x7fcf1c148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c13d0e0 0x7fcf1c1486e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44620/0 (socket says 172.21.15.110:44620) 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1b7fe640 1 -- 172.21.15.110:0/485749312 learned_addr learned my addr 172.21.15.110:0/485749312 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1bfff640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c13fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56468/0 (socket says 172.21.15.110:56468) 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1b7fe640 10 monclient(hunting): get_auth_request con 0x7fcf1c13d0e0 auth_method 0 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1bfff640 10 monclient(hunting): get_auth_request con 0x7fcf1c0b0c00 auth_method 0 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf20e3f640 10 monclient(hunting): get_auth_request con 0x7fcf1c143200 auth_method 0 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf20e3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf20e3f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf20e3f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf20e3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:10.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf20e3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:10.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.748+0000 7fcf20e3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:10.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 10 monclient(hunting): handle_auth_done global_id 5739 payload 824 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 1 -- 172.21.15.110:0/485749312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 msgr2=0x7fcf1c13fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c13fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 1 -- 172.21.15.110:0/485749312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf1c143200 msgr2=0x7fcf1c148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf1c143200 0x7fcf1c148c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 1 monclient: found mon.noname-a 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.751+0000 7fcf1b7fe640 1 -- 172.21.15.110:0/485749312 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf04009400 con 0x7fcf1c13d0e0 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.753+0000 7fcf1b7fe640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c13d0e0 0x7fcf1c1486e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400ebe0 tx=0x7fcf0400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.753+0000 7fcf18ff9640 10 client.?.objecter ms_handle_connect 0x7fcf1c13d0e0 2024-01-20T11:22:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.753+0000 7fcf18ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 1 -- 172.21.15.110:0/485749312 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04009760 con 0x7fcf1c13d0e0 2024-01-20T11:22:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 10 monclient: dump: 2024-01-20T11:22:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 10 monclient: _finish_auth 0 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 1 -- 172.21.15.110:0/485749312 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0400cc30 con 0x7fcf1c13d0e0 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf228a2640 5 monclient: authenticate success, global_id 5739 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf228a2640 10 monclient: _renew_subs 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf228a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c009400 con 0x7fcf1c13d0e0 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 1 -- 172.21.15.110:0/485749312 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0400eee0 con 0x7fcf1c13d0e0 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf18ff9640 10 monclient: dump: 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf1a7fc640 4 set_mon_vals no callback set 2024-01-20T11:22:10.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf228a2640 10 client.5739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf228a2640 10 monclient: _renew_subs 2024-01-20T11:22:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf228a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c0aa590 con 0x7fcf1c13d0e0 2024-01-20T11:22:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.757+0000 7fcf228a2640 1 librados: init done 2024-01-20T11:22:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.758+0000 7fcf228a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.758+0000 7fcf228a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.758+0000 7fcf228a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:10.758+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c0aab30 con 0x7fcf1c13d0e0 2024-01-20T11:22:11.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.000+0000 7fcf18ff9640 1 -- 172.21.15.110:0/485749312 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fcf04013470 con 0x7fcf1c13d0e0 2024-01-20T11:22:11.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.000+0000 7fcf18ff9640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcefc06c8b0 0x7fcefc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.001+0000 7fcf1bfff640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcefc06c8b0 0x7fcefc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.001+0000 7fcf1bfff640 10 monclient: get_auth_request con 0x7fcefc06c8b0 auth_method 0 2024-01-20T11:22:11.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.001+0000 7fcf1bfff640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcefc06c8b0 0x7fcefc06ed70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c036000 tx=0x7fcf0c03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:11.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.001+0000 7fcf18ff9640 10 client.5739.objecter ms_handle_connect 0x7fcefc06c8b0 2024-01-20T11:22:11.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.014+0000 7fcf18ff9640 1 -- 172.21.15.110:0/485749312 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcf04058cd0 con 0x7fcf1c13d0e0 2024-01-20T11:22:11.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.014+0000 7fcf18ff9640 10 client.5739.objecter ms_dispatch 0x7fcf1c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:11.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.014+0000 7fcf18ff9640 3 client.5739.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:11.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.014+0000 7fcf18ff9640 3 client.5739.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:11.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.014+0000 7fcf18ff9640 20 client.5739.objecter dump_active .. 0 homeless 2024-01-20T11:22:11.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.014+0000 7fcf18ff9640 1 -- 172.21.15.110:0/485749312 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fcf04055010 con 0x7fcf1c13d0e0 2024-01-20T11:22:11.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.014+0000 7fcf18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:11.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.014+0000 7fcf18ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:11.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.189+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcf1c0adcf0 con 0x7fcefc06c8b0 2024-01-20T11:22:11.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.193+0000 7fcf18ff9640 1 -- 172.21.15.110:0/485749312 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64039 (secure 0 0 0) 0x7fcf0c031990 con 0x7fcefc06c8b0 2024-01-20T11:22:11.194 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:11.194 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.194+0000 7fcf228a2640 10 librados: watch_flush enter 2024-01-20T11:22:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 10 librados: watch_flush exit 2024-01-20T11:22:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 20 client.5739.objecter shutdown clearing up homeless session... 2024-01-20T11:22:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 10 client.5739.objecter successfully canceled tick 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcefc06c8b0 msgr2=0x7fcefc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcefc06c8b0 0x7fcefc06ed70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c036000 tx=0x7fcf0c03d040 comp rx=0 tx=0).stop 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 10 monclient: shutdown 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c13d0e0 msgr2=0x7fcf1c1486e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c13d0e0 0x7fcf1c1486e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400ebe0 tx=0x7fcf0400c520 comp rx=0 tx=0).stop 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 shutdown_connections 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fcefc06c8b0 0x7fcefc06ed70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf1c143200 0x7fcf1c148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c13d0e0 0x7fcf1c1486e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 --2- 172.21.15.110:0/485749312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1c0b0c00 0x7fcf1c13fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 >> 172.21.15.110:0/485749312 conn(0x7fcf1c0b3e60 msgr2=0x7fcf1c0b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 shutdown_connections 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 -- 172.21.15.110:0/485749312 wait complete. 2024-01-20T11:22:11.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.195+0000 7fcf228a2640 1 librados: shutdown 2024-01-20T11:22:11.207 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-01-20T11:22:10.793966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":15.798999999999999},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":10.657999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":12.972},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":5.9189999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":14.473000000000001},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":7.827}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":9.1890000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":9.9570000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":6.6559999999999997},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":13.659000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":8.4640000000000004},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":11.371}]}]},{"osd":7,"up_from":21,"seq":90194313303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.7009999999999996},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":8.3049999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":6.9619999999999997},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":10.212999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":11.582000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":14.561999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":10.51},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":12.249000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":13.003},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":10.986000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":16.085000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":15.382999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313304,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0410000000000004},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":14.662000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":13.605},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":11}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":11.731},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":15.254}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":7.9530000000000003},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.872999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":8.5220000000000002},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":5.8879999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":9.9540000000000006},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.946999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444118,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":10.851000000000001},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.7409999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":14.66},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.1799999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.004},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":8.4700000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":10.16},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":13.271000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.882999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":7.8979999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":15.534000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":9.5760000000000005}]}]},{"osd":1,"up_from":17,"seq":73014444119,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0590000000000002},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":4.9470000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":8.6400000000000006},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":10.534000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":13.951000000000001},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":11.557}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":13.555},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.872}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.9370000000000003},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":12.773999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":10.901},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":12.164}]}]},{"osd":0,"up_from":17,"seq":73014444119,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":8.4450000000000003},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":6.8159999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.383},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":14.018000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":14.930999999999999},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":11.196999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":12.097},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":9.4629999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":19.693999999999999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":13.271000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":16.280000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":8.0429999999999993}]}]},{"osd":6,"up_from":15,"seq":64424509527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":9.8810000000000002},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":4.4580000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.3659999999999997},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":11.718}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":13.066000000000001},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":8.2469999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":12.397},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":7.6600000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":13.677},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":8.9049999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":9.4939999999999998},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.396000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:11.208 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:11.208 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:11.208 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:11.208 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:11.209 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:11.209 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:11.209 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:11.209 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:11.209 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:11.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.306+0000 7f881ac46640 10 monclient: get_monmap_and_config 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.306+0000 7f881ac46640 10 monclient: build_initial_monmap 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.306+0000 7f881ac46640 10 monclient: monmap: 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:11.308493+0000 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:11.308493+0000 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:11.308 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.307+0000 7f881ac46640 1 Processor -- start 2024-01-20T11:22:11.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.307+0000 7f881ac46640 1 -- start start 2024-01-20T11:22:11.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.307+0000 7f881ac46640 10 monclient: init 2024-01-20T11:22:11.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:11.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-01-20T11:22:11.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88140a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f88141423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 10 monclient(hunting): picked mon.noname-b con 0x7f88140a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8814142920 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140b25c0 con 0x7f88140a96f0 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881413fbe0 con 0x7f8814142920 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88140aaea0 con 0x7f88140a9ad0 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f881ac46640 10 monclient(hunting): authenticate will time out at 1774.659541s 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f8813fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f88141423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f88189e2640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f88140a96f0 0x7f881413f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58738/0 (socket says 172.21.15.110:58738) 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.308+0000 7f88189e2640 1 -- 172.21.15.110:0/400629425 learned_addr learned my addr 172.21.15.110:0/400629425 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.309+0000 7f8813fff640 10 monclient(hunting): get_auth_request con 0x7f88140a9ad0 auth_method 0 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.309+0000 7f8813fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.309+0000 7f8813fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:11.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.309+0000 7f8813fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:11.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.309+0000 7f8813fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:11.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.309+0000 7f8813fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:11.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.309+0000 7f8813fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:11.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f8813fff640 10 monclient(hunting): handle_auth_done global_id 5414 payload 274 2024-01-20T11:22:11.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f8813fff640 1 -- 172.21.15.110:0/400629425 >> v1:172.21.15.134:6789/0 conn(0x7f8814142920 legacy=0x7f881414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f8813fff640 1 -- 172.21.15.110:0/400629425 >> v1:172.21.15.110:6789/0 conn(0x7f88140a96f0 legacy=0x7f881413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f8813fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:11.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f8813fff640 1 monclient: found mon.noname-b 2024-01-20T11:22:11.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f8813fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:11.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f8813fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:11.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f8813fff640 1 -- 172.21.15.110:0/400629425 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88140ab0a0 con 0x7f88140a9ad0 2024-01-20T11:22:11.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f88137fe640 1 -- 172.21.15.110:0/400629425 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014261643 0 0) 0x7f88140b25c0 con 0x7f88140a96f0 2024-01-20T11:22:11.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.311+0000 7f88137fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:22:11.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.313+0000 7f8813fff640 1 --2- 172.21.15.110:0/400629425 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f88141423e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f88000097d0 tx=0x7f880002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b503d0c346f129e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 1 -- 172.21.15.110:0/400629425 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800031800 con 0x7f88140a9ad0 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 10 monclient: dump: 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 1 -- 172.21.15.110:0/400629425 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8800031960 con 0x7f88140a9ad0 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 5 monclient: authenticate success, global_id 5414 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 1 -- 172.21.15.110:0/400629425 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800031cf0 con 0x7f88140a9ad0 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 4 set_mon_vals no callback set 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f88137fe640 10 monclient: dump: 2024-01-20T11:22:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:11.320 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 10 monclient: shutdown 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 1 -- 172.21.15.110:0/400629425 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 msgr2=0x7f88141423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 1 --2- 172.21.15.110:0/400629425 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f88141423e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f88000097d0 tx=0x7f880002b8a0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 1 -- 172.21.15.110:0/400629425 shutdown_connections 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 1 --2- 172.21.15.110:0/400629425 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f88141423e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 1 -- 172.21.15.110:0/400629425 >> 172.21.15.110:0/400629425 conn(0x7f88140b7bc0 msgr2=0x7f88140b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 1 -- 172.21.15.110:0/400629425 shutdown_connections 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.317+0000 7f881ac46640 1 -- 172.21.15.110:0/400629425 wait complete. 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.318+0000 7f881ac46640 10 monclient: build_initial_monmap 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.318+0000 7f881ac46640 10 monclient: monmap: 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.318+0000 7f881ac46640 1 librados: starting msgr at 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.318+0000 7f881ac46640 1 librados: starting objecter 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.318+0000 7f881ac46640 1 Processor -- start 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 1 -- start start 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 1 librados: setting wanted keys 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 1 librados: calling monclient init 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient: init 2024-01-20T11:22:11.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88140a96f0 0x7f8814140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient(hunting): picked mon.noname-b con 0x7f88140a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f8814145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient(hunting): picked mon.noname-c con 0x7f88140a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8814142920 0x7f8814144270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient(hunting): picked mon.noname-a con 0x7f8814142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8814136aa0 con 0x7f8814142920 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8814145540 con 0x7f88140a96f0 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88141d7440 con 0x7f88140a9ad0 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.319+0000 7f881ac46640 10 monclient(hunting): authenticate will time out at 1774.670951s 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f8813fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f8814145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88191e3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8814142920 0x7f8814144270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88189e2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88140a96f0 0x7f8814140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f8813fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f8814145000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56488/0 (socket says 172.21.15.110:56488) 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f8813fff640 1 -- 172.21.15.110:0/2552006673 learned_addr learned my addr 172.21.15.110:0/2552006673 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f8813fff640 10 monclient(hunting): get_auth_request con 0x7f88140a9ad0 auth_method 0 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f8813fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f8813fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f8813fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88191e3640 10 monclient(hunting): get_auth_request con 0x7f8814142920 auth_method 0 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88191e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88191e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88191e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88189e2640 10 monclient(hunting): get_auth_request con 0x7f88140a96f0 auth_method 0 2024-01-20T11:22:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88189e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88189e2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.320+0000 7f88189e2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.321+0000 7f8813fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.321+0000 7f8813fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.321+0000 7f8813fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.321+0000 7f88191e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.321+0000 7f88191e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.321+0000 7f88191e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.321+0000 7f88189e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.321+0000 7f88189e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:11.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.321+0000 7f88189e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:11.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.323+0000 7f8813fff640 10 monclient(hunting): handle_auth_done global_id 5417 payload 824 2024-01-20T11:22:11.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.324+0000 7f8813fff640 1 -- 172.21.15.110:0/2552006673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88140a96f0 msgr2=0x7f8814140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.324+0000 7f8813fff640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88140a96f0 0x7f8814140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.324+0000 7f8813fff640 1 -- 172.21.15.110:0/2552006673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8814142920 msgr2=0x7f8814144270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.324+0000 7f8813fff640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8814142920 0x7f8814144270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.324+0000 7f8813fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:11.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.324+0000 7f8813fff640 1 monclient: found mon.noname-c 2024-01-20T11:22:11.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.324+0000 7f8813fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:11.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.324+0000 7f8813fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:11.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.324+0000 7f8813fff640 1 -- 172.21.15.110:0/2552006673 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8800009400 con 0x7f88140a9ad0 2024-01-20T11:22:11.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.326+0000 7f8813fff640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f8814145000 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f88000098c0 tx=0x7f88000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:11.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.326+0000 7f88117fa640 10 client.?.objecter ms_handle_connect 0x7f88140a9ad0 2024-01-20T11:22:11.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.326+0000 7f88117fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:11.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.329+0000 7f88117fa640 1 -- 172.21.15.110:0/2552006673 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880003bdb0 con 0x7f88140a9ad0 2024-01-20T11:22:11.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 10 monclient: dump: 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 10 monclient: _finish_auth 0 2024-01-20T11:22:11.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 10 monclient: _check_auth_tickets 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 1 -- 172.21.15.110:0/2552006673 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88000453f0 con 0x7f88140a9ad0 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 5 monclient: authenticate success, global_id 5417 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 1 -- 172.21.15.110:0/2552006673 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88000457a0 con 0x7f88140a9ad0 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 10 monclient: _renew_subs 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88141447b0 con 0x7f88140a9ad0 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f8812ffd640 4 set_mon_vals no callback set 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f88117fa640 10 monclient: dump: 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:11.333 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 10 client.5417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 10 monclient: _renew_subs 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88141d7960 con 0x7f88140a9ad0 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 1 librados: init done 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.330+0000 7f881ac46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.331+0000 7f881ac46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.331+0000 7f881ac46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:11.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.331+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88141d72b0 con 0x7f88140a9ad0 2024-01-20T11:22:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.349+0000 7f88117fa640 1 -- 172.21.15.110:0/2552006673 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f880004e450 con 0x7f88140a9ad0 2024-01-20T11:22:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.349+0000 7f88117fa640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f87f806c8b0 0x7f87f806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.350+0000 7f88189e2640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f87f806c8b0 0x7f87f806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.350+0000 7f88189e2640 10 monclient: get_auth_request con 0x7f87f806c8b0 auth_method 0 2024-01-20T11:22:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.350+0000 7f88189e2640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f87f806c8b0 0x7f87f806ed70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8808007f70 tx=0x7f8808007f00 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.350+0000 7f88117fa640 10 client.5417.objecter ms_handle_connect 0x7f87f806c8b0 2024-01-20T11:22:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.353+0000 7f88117fa640 1 -- 172.21.15.110:0/2552006673 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f880002e070 con 0x7f88140a9ad0 2024-01-20T11:22:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.353+0000 7f88117fa640 10 client.5417.objecter ms_dispatch 0x7f8814000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.353+0000 7f88117fa640 3 client.5417.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.353+0000 7f88117fa640 3 client.5417.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.353+0000 7f88117fa640 20 client.5417.objecter dump_active .. 0 homeless 2024-01-20T11:22:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.588+0000 7f88117fa640 1 -- 172.21.15.110:0/2552006673 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8800076ef0 con 0x7f88140a9ad0 2024-01-20T11:22:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.588+0000 7f88117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.588+0000 7f88117fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:11.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.835+0000 7f881ac46640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:11.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.835+0000 7f881ac46640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:11.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.835+0000 7f881ac46640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:11.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.835+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f8814075490 con 0x7f88140a9ad0 2024-01-20T11:22:11.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.861+0000 7f88117fa640 1 -- 172.21.15.110:0/2552006673 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f8800035080 con 0x7f88140a9ad0 2024-01-20T11:22:11.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.861+0000 7f88117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:11.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.861+0000 7f88117fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:11.862 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:11.862 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":466,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 10 librados: watch_flush enter 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 10 librados: watch_flush exit 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 20 client.5417.objecter shutdown clearing up homeless session... 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 10 client.5417.objecter successfully canceled tick 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f87f806c8b0 msgr2=0x7f87f806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f87f806c8b0 0x7f87f806ed70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8808007f70 tx=0x7f8808007f00 comp rx=0 tx=0).stop 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 10 monclient: shutdown 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 msgr2=0x7f8814145000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f8814145000 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f88000098c0 tx=0x7f88000046c0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 shutdown_connections 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f87f806c8b0 0x7f87f806ed70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8814142920 0x7f8814144270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88140a9ad0 0x7f8814145000 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 --2- 172.21.15.110:0/2552006673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88140a96f0 0x7f8814140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 >> 172.21.15.110:0/2552006673 conn(0x7f88140b7bc0 msgr2=0x7f881413be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:11.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 shutdown_connections 2024-01-20T11:22:11.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 -- 172.21.15.110:0/2552006673 wait complete. 2024-01-20T11:22:11.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.863+0000 7f881ac46640 1 librados: shutdown 2024-01-20T11:22:11.879 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:11.879 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:11.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.977+0000 7fedb5541640 10 monclient: get_monmap_and_config 2024-01-20T11:22:11.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.977+0000 7fedb5541640 10 monclient: build_initial_monmap 2024-01-20T11:22:11.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.977+0000 7fedb5541640 10 monclient: monmap: 2024-01-20T11:22:11.978 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:11.978 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:11.978 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:11.978702+0000 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:11.978702+0000 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.977+0000 7fedb5541640 1 Processor -- start 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.977+0000 7fedb5541640 1 -- start start 2024-01-20T11:22:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.977+0000 7fedb5541640 10 monclient: init 2024-01-20T11:22:11.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:11.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-01-20T11:22:11.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb00a96f0 0x7fedb013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 10 monclient(hunting): picked mon.noname-b con 0x7fedb00a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:11.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fedb013fa10 0x7fedb01423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 10 monclient(hunting): picked mon.noname-a con 0x7fedb013fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:11.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fedb00a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:11.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb00b25c0 con 0x7fedb00a9ad0 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb00aaea0 con 0x7fedb013fa10 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb00ab0a0 con 0x7fedb00a96f0 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedb5541640 10 monclient(hunting): authenticate will time out at 1775.329539s 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.978+0000 7fedaeffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb00a96f0 0x7fedb013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedae7fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fedb013fa10 0x7fedb01423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedaeffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb00a96f0 0x7fedb013f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56492/0 (socket says 172.21.15.110:56492) 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedaeffd640 1 -- 172.21.15.110:0/3204594943 learned_addr learned my addr 172.21.15.110:0/3204594943 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedaeffd640 10 monclient(hunting): get_auth_request con 0x7fedb00a96f0 auth_method 0 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedaeffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedae7fc640 10 monclient(hunting): get_auth_request con 0x7fedb013fa10 auth_method 0 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedae7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.979+0000 7fedaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.980+0000 7fedae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.980+0000 7fedae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:11.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.980+0000 7fedae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.981+0000 7fedaeffd640 10 monclient(hunting): handle_auth_done global_id 5420 payload 274 2024-01-20T11:22:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.981+0000 7fedaeffd640 1 -- 172.21.15.110:0/3204594943 >> v2:172.21.15.110:3300/0 conn(0x7fedb013fa10 msgr2=0x7fedb01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.981+0000 7fedaeffd640 1 --2- 172.21.15.110:0/3204594943 >> v2:172.21.15.110:3300/0 conn(0x7fedb013fa10 0x7fedb01423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.981+0000 7fedaeffd640 1 -- 172.21.15.110:0/3204594943 >> v1:172.21.15.110:6789/0 conn(0x7fedb00a9ad0 legacy=0x7fedb0142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.981+0000 7fedaeffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.981+0000 7fedaeffd640 1 monclient: found mon.noname-b 2024-01-20T11:22:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.981+0000 7fedaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.981+0000 7fedaeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:11.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.981+0000 7fedaeffd640 1 -- 172.21.15.110:0/3204594943 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb00b3fb0 con 0x7fedb00a96f0 2024-01-20T11:22:11.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.983+0000 7fedaeffd640 1 --2- 172.21.15.110:0/3204594943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb00a96f0 0x7fedb013f4d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feda0009870 tx=0x7feda002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a89e50afc10fdb61 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 1 -- 172.21.15.110:0/3204594943 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda0031640 con 0x7fedb00a96f0 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 10 monclient: dump: 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 10 monclient: _finish_auth 0 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 10 monclient: _check_auth_tickets 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 1 -- 172.21.15.110:0/3204594943 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda00317a0 con 0x7fedb00a96f0 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 1 -- 172.21.15.110:0/3204594943 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda0031b30 con 0x7fedb00a96f0 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedb5541640 5 monclient: authenticate success, global_id 5420 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedb5541640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedb5541640 4 set_mon_vals no callback set 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedadffb640 10 monclient: dump: 2024-01-20T11:22:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:11.990 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedb5541640 10 monclient: shutdown 2024-01-20T11:22:11.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedb5541640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedb5541640 1 -- 172.21.15.110:0/3204594943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb00a96f0 msgr2=0x7fedb013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.987+0000 7fedb5541640 1 --2- 172.21.15.110:0/3204594943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb00a96f0 0x7fedb013f4d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feda0009870 tx=0x7feda002b8f0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.988+0000 7fedb5541640 1 -- 172.21.15.110:0/3204594943 shutdown_connections 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.988+0000 7fedb5541640 1 --2- 172.21.15.110:0/3204594943 >> v2:172.21.15.110:3300/0 conn(0x7fedb013fa10 0x7fedb01423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.988+0000 7fedb5541640 1 --2- 172.21.15.110:0/3204594943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb00a96f0 0x7fedb013f4d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.988+0000 7fedb5541640 1 -- 172.21.15.110:0/3204594943 >> 172.21.15.110:0/3204594943 conn(0x7fedb00b7bc0 msgr2=0x7fedb00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.988+0000 7fedb5541640 1 -- 172.21.15.110:0/3204594943 shutdown_connections 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.988+0000 7fedb5541640 1 -- 172.21.15.110:0/3204594943 wait complete. 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.989+0000 7fedb5541640 10 monclient: build_initial_monmap 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.989+0000 7fedb5541640 10 monclient: monmap: 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.989+0000 7fedb5541640 1 librados: starting msgr at 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.989+0000 7fedb5541640 1 librados: starting objecter 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.989+0000 7fedb5541640 1 Processor -- start 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.989+0000 7fedb5541640 1 -- start start 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.989+0000 7fedb5541640 1 librados: setting wanted keys 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.989+0000 7fedb5541640 1 librados: calling monclient init 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.989+0000 7fedb5541640 10 monclient: init 2024-01-20T11:22:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedb00a96f0 0x7fedb0143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient(hunting): picked mon.noname-a con 0x7fedb00a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedb00a9ad0 0x7fedb01da750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient(hunting): picked mon.noname-b con 0x7fedb00a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb0137f10 0x7fedb01dac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient(hunting): picked mon.noname-c con 0x7fedb0137f10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb0145e90 con 0x7fedb00a96f0 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb0146130 con 0x7fedb00a9ad0 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:11.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb0139370 con 0x7fedb0137f10 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedb5541640 10 monclient(hunting): authenticate will time out at 1775.341373s 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedaf7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb0137f10 0x7fedb01dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.990+0000 7fedaeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedb00a96f0 0x7fedb0143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedae7fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedb00a9ad0 0x7fedb01da750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedb00a96f0 0x7fedb0143300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44644/0 (socket says 172.21.15.110:44644) 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaeffd640 1 -- 172.21.15.110:0/688062327 learned_addr learned my addr 172.21.15.110:0/688062327 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaeffd640 10 monclient(hunting): get_auth_request con 0x7fedb00a96f0 auth_method 0 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaeffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaf7fe640 10 monclient(hunting): get_auth_request con 0x7fedb0137f10 auth_method 0 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:11.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedae7fc640 10 monclient(hunting): get_auth_request con 0x7fedb00a9ad0 auth_method 0 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedae7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.991+0000 7fedaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.992+0000 7fedae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.992+0000 7fedae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.992+0000 7fedae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:11.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 10 monclient(hunting): handle_auth_done global_id 5754 payload 824 2024-01-20T11:22:11.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 1 -- 172.21.15.110:0/688062327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb0137f10 msgr2=0x7fedb01dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:11.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb0137f10 0x7fedb01dac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 1 -- 172.21.15.110:0/688062327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedb00a9ad0 msgr2=0x7fedb01da750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedb00a9ad0 0x7fedb01da750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 1 monclient: found mon.noname-a 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.994+0000 7fedaeffd640 1 -- 172.21.15.110:0/688062327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda00094d0 con 0x7fedb00a96f0 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.996+0000 7fedaeffd640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedb00a96f0 0x7fedb0143300 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7feda0009870 tx=0x7feda00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.996+0000 7fed977fe640 10 client.?.objecter ms_handle_connect 0x7fedb00a96f0 2024-01-20T11:22:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:11.996+0000 7fed977fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 1 -- 172.21.15.110:0/688062327 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda003a930 con 0x7fedb00a96f0 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 10 monclient: dump: 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:12.002 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 1 -- 172.21.15.110:0/688062327 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda003aa90 con 0x7fedb00a96f0 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 1 -- 172.21.15.110:0/688062327 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda003ae40 con 0x7fedb00a96f0 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fedb5541640 5 monclient: authenticate success, global_id 5754 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fedb5541640 10 monclient: _renew_subs 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fedb5541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb0145620 con 0x7fedb00a96f0 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fed977fe640 10 monclient: dump: 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:12.003 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.000+0000 7fedad7fa640 4 set_mon_vals no callback set 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.001+0000 7fedb5541640 10 client.5754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.001+0000 7fedb5541640 10 monclient: _renew_subs 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.001+0000 7fedb5541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.001+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb01db3b0 con 0x7fedb00a96f0 2024-01-20T11:22:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.001+0000 7fedb5541640 1 librados: init done 2024-01-20T11:22:12.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.001+0000 7fedb5541640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:12.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.001+0000 7fedb5541640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:12.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.001+0000 7fedb5541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:12.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.001+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb0075490 con 0x7fedb00a96f0 2024-01-20T11:22:12.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.241+0000 7fed977fe640 1 -- 172.21.15.110:0/688062327 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7feda002f050 con 0x7fedb00a96f0 2024-01-20T11:22:12.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.241+0000 7fed977fe640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed9c06c8b0 0x7fed9c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:12.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.242+0000 7fedae7fc640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed9c06c8b0 0x7fed9c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:12.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.242+0000 7fedae7fc640 10 monclient: get_auth_request con 0x7fed9c06c8b0 auth_method 0 2024-01-20T11:22:12.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.242+0000 7fedae7fc640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed9c06c8b0 0x7fed9c06ed70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fedb013bf20 tx=0x7feda4009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:12.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.242+0000 7fed977fe640 10 client.5754.objecter ms_handle_connect 0x7fed9c06c8b0 2024-01-20T11:22:12.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.254+0000 7fed977fe640 1 -- 172.21.15.110:0/688062327 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feda007a550 con 0x7fedb00a96f0 2024-01-20T11:22:12.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.254+0000 7fed977fe640 10 client.5754.objecter ms_dispatch 0x7fedb0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:12.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.254+0000 7fed977fe640 3 client.5754.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:12.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.254+0000 7fed977fe640 3 client.5754.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:12.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.254+0000 7fed977fe640 20 client.5754.objecter dump_active .. 0 homeless 2024-01-20T11:22:12.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.254+0000 7fed977fe640 1 -- 172.21.15.110:0/688062327 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7feda0076890 con 0x7fedb00a96f0 2024-01-20T11:22:12.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.254+0000 7fed977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:12.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.254+0000 7fed977fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:12.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.428+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fedb0069850 con 0x7fed9c06c8b0 2024-01-20T11:22:12.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.431+0000 7fed977fe640 1 -- 172.21.15.110:0/688062327 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64039 (secure 0 0 0) 0x7fedb0069850 con 0x7fed9c06c8b0 2024-01-20T11:22:12.432 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:12.432 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:12.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.432+0000 7fedb5541640 10 librados: watch_flush enter 2024-01-20T11:22:12.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.432+0000 7fedb5541640 10 librados: watch_flush exit 2024-01-20T11:22:12.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.432+0000 7fedb5541640 20 client.5754.objecter shutdown clearing up homeless session... 2024-01-20T11:22:12.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.432+0000 7fedb5541640 10 client.5754.objecter successfully canceled tick 2024-01-20T11:22:12.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed9c06c8b0 msgr2=0x7fed9c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:12.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed9c06c8b0 0x7fed9c06ed70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fedb013bf20 tx=0x7feda4009190 comp rx=0 tx=0).stop 2024-01-20T11:22:12.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 10 monclient: shutdown 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedb00a96f0 msgr2=0x7fedb0143300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedb00a96f0 0x7fedb0143300 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7feda0009870 tx=0x7feda00046c0 comp rx=0 tx=0).stop 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 shutdown_connections 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fed9c06c8b0 0x7fed9c06ed70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fedb0137f10 0x7fedb01dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedb00a9ad0 0x7fedb01da750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 --2- 172.21.15.110:0/688062327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedb00a96f0 0x7fedb0143300 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 >> 172.21.15.110:0/688062327 conn(0x7fedb00b7bc0 msgr2=0x7fedb00b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 shutdown_connections 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 -- 172.21.15.110:0/688062327 wait complete. 2024-01-20T11:22:12.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:12.433+0000 7fedb5541640 1 librados: shutdown 2024-01-20T11:22:12.444 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-01-20T11:22:10.793966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":15.798999999999999},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":10.657999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":12.972},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":5.9189999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":14.473000000000001},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":7.827}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":9.1890000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":9.9570000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":6.6559999999999997},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":13.659000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":8.4640000000000004},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":11.371}]}]},{"osd":7,"up_from":21,"seq":90194313303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.7009999999999996},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":8.3049999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":6.9619999999999997},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":10.212999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":11.582000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":14.561999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":10.51},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":12.249000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":13.003},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":10.986000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":16.085000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":15.382999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313304,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0410000000000004},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":14.662000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":13.605},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":11}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":11.731},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":15.254}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":7.9530000000000003},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.872999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":8.5220000000000002},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":5.8879999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":9.9540000000000006},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":15.946999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444118,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":10.851000000000001},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":5.7409999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":14.66},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.1799999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.004},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":8.4700000000000006}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":10.16},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":13.271000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.882999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":7.8979999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":15.534000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":9.5760000000000005}]}]},{"osd":1,"up_from":17,"seq":73014444119,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.0590000000000002},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":4.9470000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":8.6400000000000006},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":10.534000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":13.951000000000001},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":11.557}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":13.555},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.872}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.9370000000000003},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":12.773999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":10.901},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":12.164}]}]},{"osd":0,"up_from":17,"seq":73014444119,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":8.4450000000000003},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":6.8159999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":7.383},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":14.018000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":14.930999999999999},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":11.196999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":12.097},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":9.4629999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":19.693999999999999},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":13.271000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":16.280000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":8.0429999999999993}]}]},{"osd":6,"up_from":15,"seq":64424509527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":9.8810000000000002},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":4.4580000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.3659999999999997},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":11.718}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":13.066000000000001},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":8.2469999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":12.397},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":7.6600000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":13.677},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":8.9049999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":9.4939999999999998},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.396000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:15.446 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:15.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.545+0000 7f2c1b615640 10 monclient: get_monmap_and_config 2024-01-20T11:22:15.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.545+0000 7f2c1b615640 10 monclient: build_initial_monmap 2024-01-20T11:22:15.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.545+0000 7f2c1b615640 10 monclient: monmap: 2024-01-20T11:22:15.546 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:15.546 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:15.546 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:15.546840+0000 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:15.546840+0000 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.545+0000 7f2c1b615640 1 Processor -- start 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 1 -- start start 2024-01-20T11:22:15.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient: init 2024-01-20T11:22:15.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:15.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-01-20T11:22:15.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2c140b4d10 0x7f2c140b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:15.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient(hunting): picked mon.noname-a con 0x7f2c140b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:15.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c140b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c14145980 0x7f2c14145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient(hunting): picked mon.noname-b con 0x7f2c14145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c14146370 con 0x7f2c140b17c0 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c140b50f0 con 0x7f2c140b4d10 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c140aade0 con 0x7f2c14145980 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.546+0000 7f2c1b615640 10 monclient(hunting): authenticate will time out at 1778.897785s 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c193b1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2c140b4d10 0x7f2c140b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c19bb2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c14145980 0x7f2c14145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c19bb2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c14145980 0x7f2c14145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56512/0 (socket says 172.21.15.110:56512) 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c19bb2640 1 -- 172.21.15.110:0/1906243456 learned_addr learned my addr 172.21.15.110:0/1906243456 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c19bb2640 10 monclient(hunting): get_auth_request con 0x7f2c14145980 auth_method 0 2024-01-20T11:22:15.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c19bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c19bb2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c19bb2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c193b1640 10 monclient(hunting): get_auth_request con 0x7f2c140b4d10 auth_method 0 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c193b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c193b1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.547+0000 7f2c193b1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.548+0000 7f2c19bb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.548+0000 7f2c19bb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.548+0000 7f2c19bb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.548+0000 7f2c193b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.548+0000 7f2c193b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:15.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.548+0000 7f2c193b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:15.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.549+0000 7f2c19bb2640 10 monclient(hunting): handle_auth_done global_id 5426 payload 274 2024-01-20T11:22:15.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.549+0000 7f2c19bb2640 1 -- 172.21.15.110:0/1906243456 >> v2:172.21.15.110:3300/0 conn(0x7f2c140b4d10 msgr2=0x7f2c140b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:15.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.549+0000 7f2c19bb2640 1 --2- 172.21.15.110:0/1906243456 >> v2:172.21.15.110:3300/0 conn(0x7f2c140b4d10 0x7f2c140b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:15.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.549+0000 7f2c19bb2640 1 -- 172.21.15.110:0/1906243456 >> v1:172.21.15.134:6789/0 conn(0x7f2c140b17c0 legacy=0x7f2c140b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:15.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.549+0000 7f2c19bb2640 10 monclient: _finish_hunting 0 2024-01-20T11:22:15.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.549+0000 7f2c19bb2640 1 monclient: found mon.noname-b 2024-01-20T11:22:15.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.549+0000 7f2c19bb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:15.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.549+0000 7f2c19bb2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:15.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.549+0000 7f2c19bb2640 1 -- 172.21.15.110:0/1906243456 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c14145460 con 0x7f2c14145980 2024-01-20T11:22:15.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.551+0000 7f2c19bb2640 1 --2- 172.21.15.110:0/1906243456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c14145980 0x7f2c14145e30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009870 tx=0x7f2c1002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=195c59c896a3e91f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:15.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 1 -- 172.21.15.110:0/1906243456 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10031640 con 0x7f2c14145980 2024-01-20T11:22:15.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:15.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 10 monclient: dump: 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 10 monclient: _finish_auth 0 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:15.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 1 -- 172.21.15.110:0/1906243456 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c100317a0 con 0x7f2c14145980 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 5 monclient: authenticate success, global_id 5426 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 1 -- 172.21.15.110:0/1906243456 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10031b30 con 0x7f2c14145980 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 4 set_mon_vals no callback set 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c0bfff640 10 monclient: dump: 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 10 monclient: shutdown 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 1 -- 172.21.15.110:0/1906243456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c14145980 msgr2=0x7f2c14145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 1 --2- 172.21.15.110:0/1906243456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c14145980 0x7f2c14145e30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009870 tx=0x7f2c1002b8f0 comp rx=0 tx=0).stop 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 1 -- 172.21.15.110:0/1906243456 shutdown_connections 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 1 --2- 172.21.15.110:0/1906243456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c14145980 0x7f2c14145e30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 1 --2- 172.21.15.110:0/1906243456 >> v2:172.21.15.110:3300/0 conn(0x7f2c140b4d10 0x7f2c140b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 1 -- 172.21.15.110:0/1906243456 >> 172.21.15.110:0/1906243456 conn(0x7f2c14066590 msgr2=0x7f2c140669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:15.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 1 -- 172.21.15.110:0/1906243456 shutdown_connections 2024-01-20T11:22:15.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.555+0000 7f2c1b615640 1 -- 172.21.15.110:0/1906243456 wait complete. 2024-01-20T11:22:15.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.556+0000 7f2c1b615640 10 monclient: build_initial_monmap 2024-01-20T11:22:15.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.556+0000 7f2c1b615640 10 monclient: monmap: 2024-01-20T11:22:15.563 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:15.563 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:15.563 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:15.563 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:15.563 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:15.563 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:15.563 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.556+0000 7f2c1b615640 1 librados: starting msgr at 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.556+0000 7f2c1b615640 1 librados: starting objecter 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.556+0000 7f2c1b615640 1 Processor -- start 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.556+0000 7f2c1b615640 1 -- start start 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.556+0000 7f2c1b615640 1 librados: setting wanted keys 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.556+0000 7f2c1b615640 1 librados: calling monclient init 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.556+0000 7f2c1b615640 10 monclient: init 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c140b17c0 0x7f2c14140740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:15.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient(hunting): picked mon.noname-a con 0x7f2c140b17c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c140b4d10 0x7f2c14137d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient(hunting): picked mon.noname-c con 0x7f2c140b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c14145980 0x7f2c14138250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient(hunting): picked mon.noname-b con 0x7f2c14145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c14143380 con 0x7f2c140b17c0 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c140accf0 con 0x7f2c14145980 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c140acf40 con 0x7f2c140b4d10 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.557+0000 7f2c1b615640 10 monclient(hunting): authenticate will time out at 1778.908717s 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c18bb0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c140b4d10 0x7f2c14137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c193b1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c140b17c0 0x7f2c14140740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:15.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c19bb2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c14145980 0x7f2c14138250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c18bb0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c140b4d10 0x7f2c14137d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56518/0 (socket says 172.21.15.110:56518) 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c18bb0640 1 -- 172.21.15.110:0/471204549 learned_addr learned my addr 172.21.15.110:0/471204549 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c18bb0640 10 monclient(hunting): get_auth_request con 0x7f2c140b4d10 auth_method 0 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c18bb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c18bb0640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c18bb0640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c193b1640 10 monclient(hunting): get_auth_request con 0x7f2c140b17c0 auth_method 0 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c193b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c193b1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c193b1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c19bb2640 10 monclient(hunting): get_auth_request con 0x7f2c14145980 auth_method 0 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c19bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c19bb2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.558+0000 7f2c19bb2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.559+0000 7f2c18bb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:15.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.559+0000 7f2c18bb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.559+0000 7f2c18bb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.559+0000 7f2c193b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.559+0000 7f2c193b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.559+0000 7f2c193b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.559+0000 7f2c19bb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.559+0000 7f2c19bb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.559+0000 7f2c19bb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 10 monclient(hunting): handle_auth_done global_id 5830 payload 824 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 1 -- 172.21.15.110:0/471204549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c140b4d10 msgr2=0x7f2c14137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c140b4d10 0x7f2c14137d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 1 -- 172.21.15.110:0/471204549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c140b17c0 msgr2=0x7f2c14140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c140b17c0 0x7f2c14140740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 10 monclient: _finish_hunting 0 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 1 monclient: found mon.noname-b 2024-01-20T11:22:15.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:15.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:15.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.562+0000 7f2c19bb2640 1 -- 172.21.15.110:0/471204549 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c100094d0 con 0x7f2c14145980 2024-01-20T11:22:15.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.565+0000 7f2c19bb2640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c14145980 0x7f2c14138250 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2c10033000 tx=0x7f2c100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:15.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.565+0000 7f2c09ffb640 10 client.?.objecter ms_handle_connect 0x7f2c14145980 2024-01-20T11:22:15.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.565+0000 7f2c09ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 1 -- 172.21.15.110:0/471204549 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1003a970 con 0x7f2c14145980 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 10 monclient: dump: 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:15.570 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 10 monclient: _finish_auth 0 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 1 -- 172.21.15.110:0/471204549 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1003aaf0 con 0x7f2c14145980 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 1 -- 172.21.15.110:0/471204549 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1003aea0 con 0x7f2c14145980 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c09ffb640 10 monclient: dump: 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:15.571 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.568+0000 7f2c0b7fe640 4 set_mon_vals no callback set 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.569+0000 7f2c1b615640 5 monclient: authenticate success, global_id 5830 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.569+0000 7f2c1b615640 10 monclient: _renew_subs 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.569+0000 7f2c1b615640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.569+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c140ad220 con 0x7f2c14145980 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.569+0000 7f2c1b615640 10 client.5830.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.569+0000 7f2c1b615640 10 monclient: _renew_subs 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.569+0000 7f2c1b615640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:15.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.569+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c140ab470 con 0x7f2c14145980 2024-01-20T11:22:15.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.569+0000 7f2c1b615640 1 librados: init done 2024-01-20T11:22:15.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.570+0000 7f2c1b615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:15.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.570+0000 7f2c1b615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:15.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.570+0000 7f2c1b615640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:15.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.570+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c141430e0 con 0x7f2c14145980 2024-01-20T11:22:15.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.812+0000 7f2c09ffb640 1 -- 172.21.15.110:0/471204549 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2c1002f050 con 0x7f2c14145980 2024-01-20T11:22:15.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.812+0000 7f2c09ffb640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8e0 0x7f2bf806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:15.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.813+0000 7f2c193b1640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8e0 0x7f2bf806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:15.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.813+0000 7f2c193b1640 10 monclient: get_auth_request con 0x7f2bf806c8e0 auth_method 0 2024-01-20T11:22:15.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.813+0000 7f2c193b1640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8e0 0x7f2bf806eda0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0113e0 tx=0x7f2c0c00b040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:15.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.813+0000 7f2c09ffb640 10 client.5830.objecter ms_handle_connect 0x7f2bf806c8e0 2024-01-20T11:22:15.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.817+0000 7f2c09ffb640 1 -- 172.21.15.110:0/471204549 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2c1007a500 con 0x7f2c14145980 2024-01-20T11:22:15.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.817+0000 7f2c09ffb640 10 client.5830.objecter ms_dispatch 0x7f2c14000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:15.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.817+0000 7f2c09ffb640 3 client.5830.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:15.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.817+0000 7f2c09ffb640 3 client.5830.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:15.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.817+0000 7f2c09ffb640 20 client.5830.objecter dump_active .. 0 homeless 2024-01-20T11:22:15.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.827+0000 7f2c09ffb640 1 -- 172.21.15.110:0/471204549 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2c10076840 con 0x7f2c14145980 2024-01-20T11:22:15.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.827+0000 7f2c09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:15.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.827+0000 7f2c09ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:15.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.994+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c1414c7f0 con 0x7f2bf806c8e0 2024-01-20T11:22:15.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.997+0000 7f2c09ffb640 1 -- 172.21.15.110:0/471204549 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63981 (secure 0 0 0) 0x7f2c0c00f040 con 0x7f2bf806c8e0 2024-01-20T11:22:16.049 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:16.049 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:16.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 10 librados: watch_flush enter 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 10 librados: watch_flush exit 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 20 client.5830.objecter shutdown clearing up homeless session... 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 10 client.5830.objecter successfully canceled tick 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8e0 msgr2=0x7f2bf806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8e0 0x7f2bf806eda0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0113e0 tx=0x7f2c0c00b040 comp rx=0 tx=0).stop 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 10 monclient: shutdown 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c14145980 msgr2=0x7f2c14138250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c14145980 0x7f2c14138250 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2c10033000 tx=0x7f2c100046c0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 shutdown_connections 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2bf806c8e0 0x7f2bf806eda0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c14145980 0x7f2c14138250 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2c140b4d10 0x7f2c14137d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 --2- 172.21.15.110:0/471204549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c140b17c0 0x7f2c14140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 >> 172.21.15.110:0/471204549 conn(0x7f2c14066590 msgr2=0x7f2c1413ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 shutdown_connections 2024-01-20T11:22:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 -- 172.21.15.110:0/471204549 wait complete. 2024-01-20T11:22:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:15.999+0000 7f2c1b615640 1 librados: shutdown 2024-01-20T11:22:16.053 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-01-20T11:22:14.794921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002970"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280602,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":7.6470000000000002},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.931}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":13.731},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":8.8170000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.327999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":14.414}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":11.644},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":11.000999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":15.122999999999999},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":10.304}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":12.967000000000001},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":9.5800000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313304,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.7969999999999997},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":8.2200000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":10.855},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":14.816000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":17.186},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":9.7059999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":11.744999999999999},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":18.042999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":12.446},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":13.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14.090999999999999},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":16.053999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313305,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0679999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":8.6210000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":15.157999999999999},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":9.5830000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":13.747},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":7.4279999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":10.816000000000001},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":11.456}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":14.393000000000001},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":12.212}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.457000000000001},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":10.185}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444119,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.2669999999999995},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":7.0910000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":7.9889999999999999},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.390000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.315},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.3219999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":17.117000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.398}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.449999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":12.859}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":16.056999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":12.154}]}]},{"osd":1,"up_from":17,"seq":73014444120,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.1739999999999995},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.1459999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":14.872},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":9.7710000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":13.946},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":6.9029999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.1760000000000002},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.504000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.57},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":10.42}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":13.201000000000001},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":10.983000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444120,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":7.21},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":6.6970000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":8.7349999999999994},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":13.452999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.853999999999999},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":7.9500000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":9.3059999999999992},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":14.339}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.127000000000001},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":9.8640000000000008}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":10.441000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":12.66}]}]},{"osd":6,"up_from":15,"seq":64424509528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":10.039},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":10.760999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.6429999999999998},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":8.4900000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":8.8399999999999999},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":14.379}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":6.9009999999999998},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":11.965999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":11.407999999999999},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":15.222}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":15.983000000000001},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":13.43}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:16.054 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.152+0000 7f0137a46640 10 monclient: get_monmap_and_config 2024-01-20T11:22:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.152+0000 7f0137a46640 10 monclient: build_initial_monmap 2024-01-20T11:22:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.152+0000 7f0137a46640 10 monclient: monmap: 2024-01-20T11:22:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:16.154295+0000 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:16.154295+0000 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:16.154 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.153+0000 7f0137a46640 1 Processor -- start 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.153+0000 7f0137a46640 1 -- start start 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.153+0000 7f0137a46640 10 monclient: init 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.153+0000 7f0137a46640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.153+0000 7f0137a46640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.153+0000 7f0137a46640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0130066620 0x7f013013f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 10 monclient(hunting): picked mon.noname-a con 0x7f0130066620 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f013006e2a0 0x7f01301456a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 10 monclient(hunting): picked mon.noname-c con 0x7f013006e2a0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f013013f780 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01301462f0 con 0x7f013013f780 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0130066a00 con 0x7f0130066620 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f013006dfe0 con 0x7f013006e2a0 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0137a46640 10 monclient(hunting): authenticate will time out at 1779.505233s 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f01357e2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0130066620 0x7f013013f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f0134fe1640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f013006e2a0 0x7f01301456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.154+0000 7f01357e2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0130066620 0x7f013013f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44678/0 (socket says 172.21.15.110:44678) 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f01357e2640 1 -- 172.21.15.110:0/1319347883 learned_addr learned my addr 172.21.15.110:0/1319347883 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f01357e2640 10 monclient(hunting): get_auth_request con 0x7f0130066620 auth_method 0 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f01357e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f01357e2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f01357e2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f01357e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f01357e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f01357e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f0134fe1640 10 monclient(hunting): get_auth_request con 0x7f013006e2a0 auth_method 0 2024-01-20T11:22:16.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f0134fe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f0134fe1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.155+0000 7f0134fe1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.156+0000 7f0134fe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.156+0000 7f0134fe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.156+0000 7f0134fe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.157+0000 7f01357e2640 10 monclient(hunting): handle_auth_done global_id 5763 payload 274 2024-01-20T11:22:16.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.157+0000 7f01357e2640 1 -- 172.21.15.110:0/1319347883 >> v2:172.21.15.134:3300/0 conn(0x7f013006e2a0 msgr2=0x7f01301456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:16.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.157+0000 7f01357e2640 1 --2- 172.21.15.110:0/1319347883 >> v2:172.21.15.134:3300/0 conn(0x7f013006e2a0 0x7f01301456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.157+0000 7f01357e2640 1 -- 172.21.15.110:0/1319347883 >> v1:172.21.15.134:6789/0 conn(0x7f013013f780 legacy=0x7f0130145be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.157+0000 7f01357e2640 10 monclient: _finish_hunting 0 2024-01-20T11:22:16.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.157+0000 7f01357e2640 1 monclient: found mon.noname-a 2024-01-20T11:22:16.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.157+0000 7f01357e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:16.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.157+0000 7f01357e2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:16.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.157+0000 7f01357e2640 1 -- 172.21.15.110:0/1319347883 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300ae130 con 0x7f0130066620 2024-01-20T11:22:16.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.158+0000 7f01357e2640 1 --2- 172.21.15.110:0/1319347883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0130066620 0x7f013013f240 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f01300b3eb0 tx=0x7f012c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f94158447e8ef2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 1 -- 172.21.15.110:0/1319347883 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c031870 con 0x7f0130066620 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 10 monclient: dump: 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 10 monclient: _finish_auth 0 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 10 monclient: _check_auth_tickets 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 1 -- 172.21.15.110:0/1319347883 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c0319d0 con 0x7f0130066620 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0137a46640 5 monclient: authenticate success, global_id 5763 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 1 -- 172.21.15.110:0/1319347883 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c031d60 con 0x7f0130066620 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0137a46640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0137a46640 4 set_mon_vals no callback set 2024-01-20T11:22:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.162+0000 7f0127fff640 10 monclient: dump: 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:16.165 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 10 monclient: shutdown 2024-01-20T11:22:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 1 -- 172.21.15.110:0/1319347883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0130066620 msgr2=0x7f013013f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 1 --2- 172.21.15.110:0/1319347883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0130066620 0x7f013013f240 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f01300b3eb0 tx=0x7f012c02b910 comp rx=0 tx=0).stop 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 1 -- 172.21.15.110:0/1319347883 shutdown_connections 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 1 --2- 172.21.15.110:0/1319347883 >> v2:172.21.15.134:3300/0 conn(0x7f013006e2a0 0x7f01301456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 1 --2- 172.21.15.110:0/1319347883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0130066620 0x7f013013f240 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 1 -- 172.21.15.110:0/1319347883 >> 172.21.15.110:0/1319347883 conn(0x7f01300b09e0 msgr2=0x7f01300b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 1 -- 172.21.15.110:0/1319347883 shutdown_connections 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.163+0000 7f0137a46640 1 -- 172.21.15.110:0/1319347883 wait complete. 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.164+0000 7f0137a46640 10 monclient: build_initial_monmap 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.164+0000 7f0137a46640 10 monclient: monmap: 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.164+0000 7f0137a46640 1 librados: starting msgr at 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.164+0000 7f0137a46640 1 librados: starting objecter 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.164+0000 7f0137a46640 1 Processor -- start 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.164+0000 7f0137a46640 1 -- start start 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.164+0000 7f0137a46640 1 librados: setting wanted keys 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.164+0000 7f0137a46640 1 librados: calling monclient init 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.164+0000 7f0137a46640 10 monclient: init 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0130066620 0x7f0130140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient(hunting): picked mon.noname-b con 0x7f0130066620 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:16.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f013013f780 0x7f01301415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient(hunting): picked mon.noname-a con 0x7f013013f780 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0130070210 0x7f01300706c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient(hunting): picked mon.noname-c con 0x7f0130070210 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0130142a60 con 0x7f013013f780 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0130142180 con 0x7f0130066620 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0130139280 con 0x7f0130070210 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0137a46640 10 monclient(hunting): authenticate will time out at 1779.516357s 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0134fe1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f013013f780 0x7f01301415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.165+0000 7f0135fe3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0130070210 0x7f01300706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0134fe1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f013013f780 0x7f01301415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44686/0 (socket says 172.21.15.110:44686) 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0134fe1640 1 -- 172.21.15.110:0/1429143460 learned_addr learned my addr 172.21.15.110:0/1429143460 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0134fe1640 10 monclient(hunting): get_auth_request con 0x7f013013f780 auth_method 0 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0134fe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0134fe1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0134fe1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f01357e2640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0130066620 0x7f0130140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0135fe3640 10 monclient(hunting): get_auth_request con 0x7f0130070210 auth_method 0 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0135fe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0135fe3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0135fe3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f01357e2640 10 monclient(hunting): get_auth_request con 0x7f0130066620 auth_method 0 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f01357e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f01357e2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f01357e2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0134fe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0134fe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0134fe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f01357e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f01357e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f01357e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0135fe3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0135fe3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.166+0000 7f0135fe3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 10 monclient(hunting): handle_auth_done global_id 5836 payload 824 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 1 -- 172.21.15.110:0/1429143460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0130070210 msgr2=0x7f01300706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0130070210 0x7f01300706c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 1 -- 172.21.15.110:0/1429143460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f013013f780 msgr2=0x7f01301415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f013013f780 0x7f01301415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 10 monclient: _finish_hunting 0 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 1 monclient: found mon.noname-b 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.169+0000 7f01357e2640 1 -- 172.21.15.110:0/1429143460 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c0094d0 con 0x7f0130066620 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.171+0000 7f01357e2640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0130066620 0x7f0130140e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f012c033000 tx=0x7f012c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.171+0000 7f0125ffb640 10 client.?.objecter ms_handle_connect 0x7f0130066620 2024-01-20T11:22:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.171+0000 7f0125ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:16.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 1 -- 172.21.15.110:0/1429143460 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c03ac60 con 0x7f0130066620 2024-01-20T11:22:16.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:16.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:16.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 10 monclient: dump: 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 10 monclient: _finish_auth 0 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 1 -- 172.21.15.110:0/1429143460 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c03ade0 con 0x7f0130066620 2024-01-20T11:22:16.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 1 -- 172.21.15.110:0/1429143460 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c044630 con 0x7f0130066620 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0125ffb640 10 monclient: dump: 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0137a46640 5 monclient: authenticate success, global_id 5836 2024-01-20T11:22:16.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0137a46640 10 monclient: _renew_subs 2024-01-20T11:22:16.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0137a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:16.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f01277fe640 4 set_mon_vals no callback set 2024-01-20T11:22:16.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0130139560 con 0x7f0130066620 2024-01-20T11:22:16.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0137a46640 10 client.5836.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:16.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0137a46640 10 monclient: _renew_subs 2024-01-20T11:22:16.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0137a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:16.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0130070c00 con 0x7f0130066620 2024-01-20T11:22:16.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.175+0000 7f0137a46640 1 librados: init done 2024-01-20T11:22:16.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.176+0000 7f0137a46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:16.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.176+0000 7f0137a46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:16.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.176+0000 7f0137a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:16.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.176+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0130141ea0 con 0x7f0130066620 2024-01-20T11:22:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.418+0000 7f0125ffb640 1 -- 172.21.15.110:0/1429143460 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f012c04d450 con 0x7f0130066620 2024-01-20T11:22:16.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.418+0000 7f0125ffb640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f011406c8b0 0x7f011406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.419+0000 7f0134fe1640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f011406c8b0 0x7f011406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.419+0000 7f0134fe1640 10 monclient: get_auth_request con 0x7f011406c8b0 auth_method 0 2024-01-20T11:22:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.419+0000 7f0125ffb640 10 client.5836.objecter ms_handle_connect 0x7f011406c8b0 2024-01-20T11:22:16.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.419+0000 7f0134fe1640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f011406c8b0 0x7f011406ed70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f013013fec0 tx=0x7f0120009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:16.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.422+0000 7f0125ffb640 1 -- 172.21.15.110:0/1429143460 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f012c02d040 con 0x7f0130066620 2024-01-20T11:22:16.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.422+0000 7f0125ffb640 10 client.5836.objecter ms_dispatch 0x7f0130000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:16.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.422+0000 7f0125ffb640 3 client.5836.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:16.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.422+0000 7f0125ffb640 3 client.5836.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:16.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.422+0000 7f0125ffb640 20 client.5836.objecter dump_active .. 0 homeless 2024-01-20T11:22:16.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.433+0000 7f0125ffb640 1 -- 172.21.15.110:0/1429143460 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f012c076af0 con 0x7f0130066620 2024-01-20T11:22:16.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.433+0000 7f0125ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:16.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.433+0000 7f0125ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:16.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.609+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f013014c7c0 con 0x7f011406c8b0 2024-01-20T11:22:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.612+0000 7f0125ffb640 1 -- 172.21.15.110:0/1429143460 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63981 (secure 0 0 0) 0x7f0120011af0 con 0x7f011406c8b0 2024-01-20T11:22:16.613 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 10 librados: watch_flush enter 2024-01-20T11:22:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 10 librados: watch_flush exit 2024-01-20T11:22:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 20 client.5836.objecter shutdown clearing up homeless session... 2024-01-20T11:22:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 10 client.5836.objecter successfully canceled tick 2024-01-20T11:22:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f011406c8b0 msgr2=0x7f011406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f011406c8b0 0x7f011406ed70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f013013fec0 tx=0x7f0120009190 comp rx=0 tx=0).stop 2024-01-20T11:22:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 10 monclient: shutdown 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0130066620 msgr2=0x7f0130140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0130066620 0x7f0130140e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f012c033000 tx=0x7f012c0046c0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 shutdown_connections 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f011406c8b0 0x7f011406ed70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0130070210 0x7f01300706c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f013013f780 0x7f01301415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 --2- 172.21.15.110:0/1429143460 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0130066620 0x7f0130140e30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 >> 172.21.15.110:0/1429143460 conn(0x7f01300b09e0 msgr2=0x7f0130073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 shutdown_connections 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 -- 172.21.15.110:0/1429143460 wait complete. 2024-01-20T11:22:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.614+0000 7f0137a46640 1 librados: shutdown 2024-01-20T11:22:16.625 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-01-20T11:22:14.794921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002970"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280602,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":7.6470000000000002},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.931}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":13.731},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":8.8170000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.327999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":14.414}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":11.644},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":11.000999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":15.122999999999999},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":10.304}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":12.967000000000001},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":9.5800000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313304,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.7969999999999997},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":8.2200000000000006}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":10.855},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":14.816000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":17.186},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":9.7059999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":11.744999999999999},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":18.042999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":12.446},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":13.382999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14.090999999999999},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":16.053999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313305,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0679999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":8.6210000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":15.157999999999999},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":9.5830000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":13.747},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":7.4279999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":10.816000000000001},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":11.456}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":14.393000000000001},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":12.212}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.457000000000001},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":10.185}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444119,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.2669999999999995},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":7.0910000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":7.9889999999999999},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.390000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.315},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.3219999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":17.117000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.398}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.449999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":12.859}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":16.056999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":12.154}]}]},{"osd":1,"up_from":17,"seq":73014444120,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.1739999999999995},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.1459999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":14.872},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":9.7710000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":13.946},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":6.9029999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.1760000000000002},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.504000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.57},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":10.42}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":13.201000000000001},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":10.983000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444120,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":7.21},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":6.6970000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":8.7349999999999994},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":13.452999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.853999999999999},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":7.9500000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":9.3059999999999992},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":14.339}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.127000000000001},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":9.8640000000000008}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":10.441000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":12.66}]}]},{"osd":6,"up_from":15,"seq":64424509528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":10.039},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":10.760999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.6429999999999998},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":8.4900000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":8.8399999999999999},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":14.379}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":6.9009999999999998},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":11.965999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":11.407999999999999},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":15.222}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":15.983000000000001},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":13.43}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:16.627 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:16.627 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:16.627 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:16.627 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:16.627 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:16.627 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:16.627 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:16.627 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:16.628 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:16.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.723+0000 7fa01ad05640 10 monclient: get_monmap_and_config 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.723+0000 7fa01ad05640 10 monclient: build_initial_monmap 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.723+0000 7fa01ad05640 10 monclient: monmap: 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:16.725473+0000 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:16.725473+0000 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:16.725 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.724+0000 7fa01ad05640 1 Processor -- start 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.724+0000 7fa01ad05640 1 -- start start 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.724+0000 7fa01ad05640 10 monclient: init 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0140a96f0 0x7fa01413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 10 monclient(hunting): picked mon.noname-b con 0x7fa0140a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa01413fa10 0x7fa0141423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 10 monclient(hunting): picked mon.noname-c con 0x7fa01413fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:16.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0140a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0140b25c0 con 0x7fa0140a9ad0 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0140aaea0 con 0x7fa01413fa10 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140ab0a0 con 0x7fa0140a96f0 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa01ad05640 10 monclient(hunting): authenticate will time out at 1780.076505s 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.725+0000 7fa018aa1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0140a96f0 0x7fa01413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa013fff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa01413fa10 0x7fa0141423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa018aa1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0140a96f0 0x7fa01413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56524/0 (socket says 172.21.15.110:56524) 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa018aa1640 1 -- 172.21.15.110:0/3095110163 learned_addr learned my addr 172.21.15.110:0/3095110163 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa018aa1640 10 monclient(hunting): get_auth_request con 0x7fa0140a96f0 auth_method 0 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa018aa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa018aa1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa018aa1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa013fff640 10 monclient(hunting): get_auth_request con 0x7fa01413fa10 auth_method 0 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa013fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa013fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa013fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa018aa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa018aa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.726+0000 7fa018aa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.727+0000 7fa013fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.727+0000 7fa013fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.727+0000 7fa013fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.728+0000 7fa018aa1640 10 monclient(hunting): handle_auth_done global_id 5435 payload 274 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.728+0000 7fa018aa1640 1 -- 172.21.15.110:0/3095110163 >> v2:172.21.15.134:3300/0 conn(0x7fa01413fa10 msgr2=0x7fa0141423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.728+0000 7fa018aa1640 1 --2- 172.21.15.110:0/3095110163 >> v2:172.21.15.134:3300/0 conn(0x7fa01413fa10 0x7fa0141423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.728+0000 7fa018aa1640 1 -- 172.21.15.110:0/3095110163 >> v1:172.21.15.134:6789/0 conn(0x7fa0140a9ad0 legacy=0x7fa014142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.728+0000 7fa018aa1640 10 monclient: _finish_hunting 0 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.728+0000 7fa018aa1640 1 monclient: found mon.noname-b 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.728+0000 7fa018aa1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.728+0000 7fa018aa1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.728+0000 7fa018aa1640 1 -- 172.21.15.110:0/3095110163 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0140b3fb0 con 0x7fa0140a96f0 2024-01-20T11:22:16.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.730+0000 7fa018aa1640 1 --2- 172.21.15.110:0/3095110163 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0140a96f0 0x7fa01413f4d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009e50 tx=0x7fa00c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c0b5f1accf6685a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:16.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 1 -- 172.21.15.110:0/3095110163 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c02ca70 con 0x7fa0140a96f0 2024-01-20T11:22:16.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:16.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:16.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 10 monclient: dump: 2024-01-20T11:22:16.735 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:16.735 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa01ad05640 5 monclient: authenticate success, global_id 5435 2024-01-20T11:22:16.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa01ad05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 1 -- 172.21.15.110:0/3095110163 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c02cbd0 con 0x7fa0140a96f0 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa01ad05640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa01ad05640 4 set_mon_vals no callback set 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 1 -- 172.21.15.110:0/3095110163 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c0343f0 con 0x7fa0140a96f0 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.733+0000 7fa0137fe640 10 monclient: dump: 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:16.737 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 10 monclient: shutdown 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 1 -- 172.21.15.110:0/3095110163 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0140a96f0 msgr2=0x7fa01413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 1 --2- 172.21.15.110:0/3095110163 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0140a96f0 0x7fa01413f4d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009e50 tx=0x7fa00c02bed0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 1 -- 172.21.15.110:0/3095110163 shutdown_connections 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 1 --2- 172.21.15.110:0/3095110163 >> v2:172.21.15.134:3300/0 conn(0x7fa01413fa10 0x7fa0141423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 1 --2- 172.21.15.110:0/3095110163 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0140a96f0 0x7fa01413f4d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 1 -- 172.21.15.110:0/3095110163 >> 172.21.15.110:0/3095110163 conn(0x7fa0140b7bc0 msgr2=0x7fa0140b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 1 -- 172.21.15.110:0/3095110163 shutdown_connections 2024-01-20T11:22:16.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.734+0000 7fa01ad05640 1 -- 172.21.15.110:0/3095110163 wait complete. 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.735+0000 7fa01ad05640 10 monclient: build_initial_monmap 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.735+0000 7fa01ad05640 10 monclient: monmap: 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:16.742 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.735+0000 7fa01ad05640 1 librados: starting msgr at 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.735+0000 7fa01ad05640 1 librados: starting objecter 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.735+0000 7fa01ad05640 1 Processor -- start 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.735+0000 7fa01ad05640 1 -- start start 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.735+0000 7fa01ad05640 1 librados: setting wanted keys 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.735+0000 7fa01ad05640 1 librados: calling monclient init 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.735+0000 7fa01ad05640 10 monclient: init 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0140a96f0 0x7fa014132420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient(hunting): picked mon.noname-b con 0x7fa0140a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0140a9ad0 0x7fa014132b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient(hunting): picked mon.noname-a con 0x7fa0140a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01413cf60 0x7fa01413d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient(hunting): picked mon.noname-c con 0x7fa01413cf60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa014133090 con 0x7fa0140a9ad0 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01413ea60 con 0x7fa0140a96f0 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa014134250 con 0x7fa01413cf60 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa01ad05640 10 monclient(hunting): authenticate will time out at 1780.087638s 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.736+0000 7fa0192a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01413cf60 0x7fa01413d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa013fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0140a9ad0 0x7fa014132b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa018aa1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0140a96f0 0x7fa014132420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa0192a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01413cf60 0x7fa01413d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56532/0 (socket says 172.21.15.110:56532) 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa0192a2640 1 -- 172.21.15.110:0/4127485899 learned_addr learned my addr 172.21.15.110:0/4127485899 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa0192a2640 10 monclient(hunting): get_auth_request con 0x7fa01413cf60 auth_method 0 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa0192a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa0192a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa0192a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa013fff640 10 monclient(hunting): get_auth_request con 0x7fa0140a9ad0 auth_method 0 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa013fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa013fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa013fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa018aa1640 10 monclient(hunting): get_auth_request con 0x7fa0140a96f0 auth_method 0 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa018aa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa018aa1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa018aa1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:16.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa0192a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa0192a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa0192a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa013fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa013fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa013fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.737+0000 7fa018aa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.738+0000 7fa018aa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.738+0000 7fa018aa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 10 monclient(hunting): handle_auth_done global_id 5438 payload 824 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 1 -- 172.21.15.110:0/4127485899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0140a96f0 msgr2=0x7fa014132420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0140a96f0 0x7fa014132420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 1 -- 172.21.15.110:0/4127485899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0140a9ad0 msgr2=0x7fa014132b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:16.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0140a9ad0 0x7fa014132b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:16.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 10 monclient: _finish_hunting 0 2024-01-20T11:22:16.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 1 monclient: found mon.noname-c 2024-01-20T11:22:16.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:16.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:16.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.740+0000 7fa0192a2640 1 -- 172.21.15.110:0/4127485899 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c009b30 con 0x7fa01413cf60 2024-01-20T11:22:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.742+0000 7fa0192a2640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01413cf60 0x7fa01413d3f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa004001eb0 tx=0x7fa00400c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.742+0000 7fa0117fa640 10 client.?.objecter ms_handle_connect 0x7fa01413cf60 2024-01-20T11:22:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.742+0000 7fa0117fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa0117fa640 1 -- 172.21.15.110:0/4127485899 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0040052d0 con 0x7fa01413cf60 2024-01-20T11:22:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa0117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa0117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa0117fa640 10 monclient: dump: 2024-01-20T11:22:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa0117fa640 10 monclient: _finish_auth 0 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa0117fa640 10 monclient: _check_auth_tickets 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa0117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa01ad05640 5 monclient: authenticate success, global_id 5438 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa01ad05640 10 monclient: _renew_subs 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa01ad05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa014134530 con 0x7fa01413cf60 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa01ad05640 10 client.5438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa01ad05640 10 monclient: _renew_subs 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa01ad05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01413d930 con 0x7fa01413cf60 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.744+0000 7fa01ad05640 1 librados: init done 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.745+0000 7fa01ad05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.745+0000 7fa01ad05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.745+0000 7fa01ad05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.745+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0141334b0 con 0x7fa01413cf60 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.746+0000 7fa0117fa640 1 -- 172.21.15.110:0/4127485899 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa004005430 con 0x7fa01413cf60 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.746+0000 7fa0117fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.746+0000 7fa0117fa640 1 -- 172.21.15.110:0/4127485899 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004007220 con 0x7fa01413cf60 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.746+0000 7fa0117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.746+0000 7fa0117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.746+0000 7fa0117fa640 10 monclient: dump: 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.746+0000 7fa012ffd640 4 set_mon_vals no callback set 2024-01-20T11:22:16.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~199s 2024-01-20T11:22:16.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.992+0000 7fa0117fa640 1 -- 172.21.15.110:0/4127485899 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa004007380 con 0x7fa01413cf60 2024-01-20T11:22:16.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.992+0000 7fa0117fa640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00006c860 0x7fa00006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:16.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.993+0000 7fa018aa1640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00006c860 0x7fa00006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:16.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.993+0000 7fa018aa1640 10 monclient: get_auth_request con 0x7fa00006c860 auth_method 0 2024-01-20T11:22:16.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.993+0000 7fa0117fa640 10 client.5438.objecter ms_handle_connect 0x7fa00006c860 2024-01-20T11:22:16.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:16.993+0000 7fa018aa1640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00006c860 0x7fa00006ed20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa00c035000 tx=0x7fa00c030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:17.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.005+0000 7fa0117fa640 1 -- 172.21.15.110:0/4127485899 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa004057a90 con 0x7fa01413cf60 2024-01-20T11:22:17.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.005+0000 7fa0117fa640 10 client.5438.objecter ms_dispatch 0x7fa014000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:17.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.005+0000 7fa0117fa640 3 client.5438.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:17.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.005+0000 7fa0117fa640 3 client.5438.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:17.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.005+0000 7fa0117fa640 20 client.5438.objecter dump_active .. 0 homeless 2024-01-20T11:22:17.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.005+0000 7fa0117fa640 1 -- 172.21.15.110:0/4127485899 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa004053dd0 con 0x7fa01413cf60 2024-01-20T11:22:17.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.005+0000 7fa0117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:17.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.005+0000 7fa0117fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:17.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.255+0000 7fa01ad05640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:17.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.255+0000 7fa01ad05640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:17.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.255+0000 7fa01ad05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:17.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.255+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa0141334b0 con 0x7fa01413cf60 2024-01-20T11:22:17.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.260+0000 7fa0117fa640 1 -- 172.21.15.110:0/4127485899 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fa004017520 con 0x7fa01413cf60 2024-01-20T11:22:17.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.260+0000 7fa0117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:17.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.260+0000 7fa0117fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:17.261 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:17.262 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":471,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:17.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 10 librados: watch_flush enter 2024-01-20T11:22:17.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 10 librados: watch_flush exit 2024-01-20T11:22:17.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 20 client.5438.objecter shutdown clearing up homeless session... 2024-01-20T11:22:17.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 10 client.5438.objecter successfully canceled tick 2024-01-20T11:22:17.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00006c860 msgr2=0x7fa00006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:17.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00006c860 0x7fa00006ed20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa00c035000 tx=0x7fa00c030000 comp rx=0 tx=0).stop 2024-01-20T11:22:17.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 10 monclient: shutdown 2024-01-20T11:22:17.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:17.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01413cf60 msgr2=0x7fa01413d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01413cf60 0x7fa01413d3f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa004001eb0 tx=0x7fa00400c890 comp rx=0 tx=0).stop 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 shutdown_connections 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa00006c860 0x7fa00006ed20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa01413cf60 0x7fa01413d3f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0140a9ad0 0x7fa014132b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 --2- 172.21.15.110:0/4127485899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0140a96f0 0x7fa014132420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.262+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 >> 172.21.15.110:0/4127485899 conn(0x7fa0140b7bc0 msgr2=0x7fa014136930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.263+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 shutdown_connections 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.263+0000 7fa01ad05640 1 -- 172.21.15.110:0/4127485899 wait complete. 2024-01-20T11:22:17.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.263+0000 7fa01ad05640 1 librados: shutdown 2024-01-20T11:22:17.275 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:17.275 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.373+0000 7f376a826640 10 monclient: get_monmap_and_config 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.373+0000 7f376a826640 10 monclient: build_initial_monmap 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.373+0000 7f376a826640 10 monclient: monmap: 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:17.374757+0000 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:17.374757+0000 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:17.375 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:17.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.373+0000 7f376a826640 1 Processor -- start 2024-01-20T11:22:17.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.373+0000 7f376a826640 1 -- start start 2024-01-20T11:22:17.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.373+0000 7f376a826640 10 monclient: init 2024-01-20T11:22:17.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:17.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-01-20T11:22:17.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37640a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:17.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37640a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3764142af0 0x7f376414d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 10 monclient(hunting): picked mon.noname-a con 0x7f3764142af0 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37640b25c0 con 0x7f37640a9ad0 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3764142f50 con 0x7f37640a96f0 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640aaea0 con 0x7f3764142af0 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.374+0000 7f376a826640 10 monclient(hunting): authenticate will time out at 1780.725791s 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.375+0000 7f3768dc3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3764142af0 0x7f376414d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.375+0000 7f3763fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f37640a96f0 0x7f376413f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:40688/0 (socket says 172.21.15.110:40688) 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.375+0000 7f3763fff640 1 -- 172.21.15.110:0/1028367872 learned_addr learned my addr 172.21.15.110:0/1028367872 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.375+0000 7f3768dc3640 10 monclient(hunting): get_auth_request con 0x7f3764142af0 auth_method 0 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.375+0000 7f3768dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.375+0000 7f3768dc3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:17.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.375+0000 7f3768dc3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.376+0000 7f3768dc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.376+0000 7f3768dc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.376+0000 7f3768dc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.377+0000 7f3762ffd640 1 -- 172.21.15.110:0/1028367872 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189993151 0 0) 0x7f3754001110 con 0x7f37640a96f0 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.377+0000 7f3762ffd640 10 monclient(hunting): _negotiate 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.377+0000 7f3762ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.377+0000 7f3762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.377+0000 7f3762ffd640 10 monclient(hunting): my global_id is 5845 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.377+0000 7f3762ffd640 1 -- 172.21.15.110:0/1028367872 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f374c003520 con 0x7f37640a96f0 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.378+0000 7f3768dc3640 10 monclient(hunting): handle_auth_done global_id 5772 payload 274 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.378+0000 7f3768dc3640 1 -- 172.21.15.110:0/1028367872 >> v1:172.21.15.134:6789/0 conn(0x7f37640a96f0 legacy=0x7f376413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:17.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.378+0000 7f3768dc3640 1 -- 172.21.15.110:0/1028367872 >> v1:172.21.15.110:6789/0 conn(0x7f37640a9ad0 legacy=0x7f37641423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:17.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.378+0000 7f3768dc3640 10 monclient: _finish_hunting 0 2024-01-20T11:22:17.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.378+0000 7f3768dc3640 1 monclient: found mon.noname-a 2024-01-20T11:22:17.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.378+0000 7f3768dc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:17.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.378+0000 7f3768dc3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:17.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.378+0000 7f3768dc3640 1 -- 172.21.15.110:0/1028367872 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376413fbe0 con 0x7f3764142af0 2024-01-20T11:22:17.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.383+0000 7f3768dc3640 1 --2- 172.21.15.110:0/1028367872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 0x7f376414d860 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f375c009d90 tx=0x7f375c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a7c0d2882bc3dc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:17.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.385+0000 7f3762ffd640 1 -- 172.21.15.110:0/1028367872 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c02cc60 con 0x7f3764142af0 2024-01-20T11:22:17.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.385+0000 7f3762ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.385+0000 7f3762ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.385+0000 7f3762ffd640 10 monclient: dump: 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.385+0000 7f3762ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.385+0000 7f3762ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.385+0000 7f3762ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.385+0000 7f376a826640 5 monclient: authenticate success, global_id 5772 2024-01-20T11:22:17.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.385+0000 7f376a826640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:17.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.386+0000 7f3762ffd640 1 -- 172.21.15.110:0/1028367872 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c02cdc0 con 0x7f3764142af0 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.386+0000 7f3762ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.386+0000 7f3762ffd640 1 -- 172.21.15.110:0/1028367872 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c034640 con 0x7f3764142af0 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.386+0000 7f3762ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.386+0000 7f3762ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.386+0000 7f3762ffd640 10 monclient: dump: 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.386+0000 7f376a826640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.386+0000 7f376a826640 4 set_mon_vals no callback set 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 10 monclient: shutdown 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 1 -- 172.21.15.110:0/1028367872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 msgr2=0x7f376414d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 1 --2- 172.21.15.110:0/1028367872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 0x7f376414d860 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f375c009d90 tx=0x7f375c02beb0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 1 -- 172.21.15.110:0/1028367872 shutdown_connections 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 1 --2- 172.21.15.110:0/1028367872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 0x7f376414d860 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 1 -- 172.21.15.110:0/1028367872 >> 172.21.15.110:0/1028367872 conn(0x7f37640b7bc0 msgr2=0x7f37640b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 1 -- 172.21.15.110:0/1028367872 shutdown_connections 2024-01-20T11:22:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 1 -- 172.21.15.110:0/1028367872 wait complete. 2024-01-20T11:22:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 10 monclient: build_initial_monmap 2024-01-20T11:22:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.387+0000 7f376a826640 10 monclient: monmap: 2024-01-20T11:22:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 librados: starting msgr at 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 librados: starting objecter 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 Processor -- start 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 -- start start 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 librados: setting wanted keys 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 librados: calling monclient init 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient: init 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37640a96f0 0x7f376413e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient(hunting): picked mon.noname-c con 0x7f37640a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37640a9ad0 0x7f3764135700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient(hunting): picked mon.noname-b con 0x7f37640a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 0x7f37641da5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient(hunting): picked mon.noname-a con 0x7f3764142af0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37641374c0 con 0x7f3764142af0 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3764135d20 con 0x7f37640a9ad0 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3764135200 con 0x7f37640a96f0 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.388+0000 7f376a826640 10 monclient(hunting): authenticate will time out at 1780.740110s 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3768dc3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 0x7f37641da5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:17.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3763fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37640a96f0 0x7f376413e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f37637fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37640a9ad0 0x7f3764135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3768dc3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 0x7f37641da5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44716/0 (socket says 172.21.15.110:44716) 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3768dc3640 1 -- 172.21.15.110:0/1602817479 learned_addr learned my addr 172.21.15.110:0/1602817479 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3768dc3640 10 monclient(hunting): get_auth_request con 0x7f3764142af0 auth_method 0 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3768dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3768dc3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3768dc3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3763fff640 10 monclient(hunting): get_auth_request con 0x7f37640a96f0 auth_method 0 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3763fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:17.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f3763fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f37637fe640 10 monclient(hunting): get_auth_request con 0x7f37640a9ad0 auth_method 0 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f37637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f37637fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.389+0000 7f37637fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.390+0000 7f3768dc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.390+0000 7f3768dc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.390+0000 7f3768dc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.390+0000 7f3763fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.390+0000 7f3763fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.390+0000 7f3763fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.390+0000 7f37637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.390+0000 7f37637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:17.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.390+0000 7f37637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:17.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.392+0000 7f3768dc3640 10 monclient(hunting): handle_auth_done global_id 5775 payload 824 2024-01-20T11:22:17.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.393+0000 7f3768dc3640 1 -- 172.21.15.110:0/1602817479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37640a96f0 msgr2=0x7f376413e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:17.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.393+0000 7f3768dc3640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37640a96f0 0x7f376413e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.393+0000 7f3768dc3640 1 -- 172.21.15.110:0/1602817479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37640a9ad0 msgr2=0x7f3764135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.393+0000 7f3768dc3640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37640a9ad0 0x7f3764135700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.393+0000 7f3768dc3640 10 monclient: _finish_hunting 0 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.393+0000 7f3768dc3640 1 monclient: found mon.noname-a 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.393+0000 7f3768dc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.393+0000 7f3768dc3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.393+0000 7f3768dc3640 1 -- 172.21.15.110:0/1602817479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c009a70 con 0x7f3764142af0 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.395+0000 7f3768dc3640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 0x7f37641da5d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f375c037040 tx=0x7f375c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.395+0000 7f3760ff9640 10 client.?.objecter ms_handle_connect 0x7f3764142af0 2024-01-20T11:22:17.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.395+0000 7f3760ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:17.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 1 -- 172.21.15.110:0/1602817479 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c004800 con 0x7f3764142af0 2024-01-20T11:22:17.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 10 monclient: dump: 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 10 monclient: _finish_auth 0 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:22:17.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 1 -- 172.21.15.110:0/1602817479 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c009450 con 0x7f3764142af0 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 1 -- 172.21.15.110:0/1602817479 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c034680 con 0x7f3764142af0 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f3760ff9640 10 monclient: dump: 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:17.401 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f376a826640 5 monclient: authenticate success, global_id 5775 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f376a826640 10 monclient: _renew_subs 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f376a826640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3764135420 con 0x7f3764142af0 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f37627fc640 4 set_mon_vals no callback set 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f376a826640 10 client.5775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f376a826640 10 monclient: _renew_subs 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f376a826640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37641dacf0 con 0x7f3764142af0 2024-01-20T11:22:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.398+0000 7f376a826640 1 librados: init done 2024-01-20T11:22:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.399+0000 7f376a826640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.399+0000 7f376a826640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.399+0000 7f376a826640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.399+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3764135ee0 con 0x7f3764142af0 2024-01-20T11:22:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.636+0000 7f3760ff9640 1 -- 172.21.15.110:0/1602817479 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f375c0347e0 con 0x7f3764142af0 2024-01-20T11:22:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.636+0000 7f3760ff9640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f375006c8b0 0x7f375006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.636+0000 7f3763fff640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f375006c8b0 0x7f375006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.636+0000 7f3763fff640 10 monclient: get_auth_request con 0x7f375006c8b0 auth_method 0 2024-01-20T11:22:17.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.637+0000 7f3763fff640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f375006c8b0 0x7f375006ed70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3754001270 tx=0x7f3754007bb0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:17.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.637+0000 7f3760ff9640 10 client.5775.objecter ms_handle_connect 0x7f375006c8b0 2024-01-20T11:22:17.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.650+0000 7f3760ff9640 1 -- 172.21.15.110:0/1602817479 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f375c07b200 con 0x7f3764142af0 2024-01-20T11:22:17.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.650+0000 7f3760ff9640 10 client.5775.objecter ms_dispatch 0x7f3764000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.650+0000 7f3760ff9640 3 client.5775.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.650+0000 7f3760ff9640 3 client.5775.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.650+0000 7f3760ff9640 20 client.5775.objecter dump_active .. 0 homeless 2024-01-20T11:22:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.650+0000 7f3760ff9640 1 -- 172.21.15.110:0/1602817479 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f375c077540 con 0x7f3764142af0 2024-01-20T11:22:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.650+0000 7f3760ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.650+0000 7f3760ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:17.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.827+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f37640b4400 con 0x7f375006c8b0 2024-01-20T11:22:17.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.830+0000 7f3760ff9640 1 -- 172.21.15.110:0/1602817479 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63994 (secure 0 0 0) 0x7f3754005850 con 0x7f375006c8b0 2024-01-20T11:22:17.831 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:17.832 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:17.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 10 librados: watch_flush enter 2024-01-20T11:22:17.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 10 librados: watch_flush exit 2024-01-20T11:22:17.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 20 client.5775.objecter shutdown clearing up homeless session... 2024-01-20T11:22:17.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 10 client.5775.objecter successfully canceled tick 2024-01-20T11:22:17.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f375006c8b0 msgr2=0x7f375006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:17.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f375006c8b0 0x7f375006ed70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3754001270 tx=0x7f3754007bb0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 10 monclient: shutdown 2024-01-20T11:22:17.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:17.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 msgr2=0x7f37641da5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:17.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 0x7f37641da5d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f375c037040 tx=0x7f375c004a40 comp rx=0 tx=0).stop 2024-01-20T11:22:17.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 shutdown_connections 2024-01-20T11:22:17.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f375006c8b0 0x7f375006ed70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3764142af0 0x7f37641da5d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37640a9ad0 0x7f3764135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 --2- 172.21.15.110:0/1602817479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37640a96f0 0x7f376413e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:17.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 >> 172.21.15.110:0/1602817479 conn(0x7f37640b7bc0 msgr2=0x7f37640b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:17.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 shutdown_connections 2024-01-20T11:22:17.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 -- 172.21.15.110:0/1602817479 wait complete. 2024-01-20T11:22:17.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:17.832+0000 7f376a826640 1 librados: shutdown 2024-01-20T11:22:17.844 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-01-20T11:22:16.795404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002899"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280602,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":7.6470000000000002},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.931}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":13.731},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":8.8170000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.327999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":14.414}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":11.644},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":11.000999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":15.122999999999999},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":10.304}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":12.967000000000001},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":9.5800000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313305,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":7.0720000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":4.4240000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.5039999999999996},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":11.449}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":12.989000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.8659999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":11.087999999999999},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":13.630000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":14.829000000000001},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.363}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":15.558},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":16.283999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313305,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0679999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":8.6210000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":15.157999999999999},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":9.5830000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":13.747},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":7.4279999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":10.816000000000001},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":11.456}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":14.393000000000001},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":12.212}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":16.457000000000001},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":10.185}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444119,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":8.2669999999999995},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":7.0910000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":7.9889999999999999},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.390000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.315},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.3219999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":17.117000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.398}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.449999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":12.859}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":16.056999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":12.154}]}]},{"osd":1,"up_from":17,"seq":73014444120,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.1739999999999995},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.1459999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":14.872},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":9.7710000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":13.946},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":6.9029999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.1760000000000002},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.504000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":12.57},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":10.42}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":13.201000000000001},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":10.983000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444120,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":7.21},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":6.6970000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":8.7349999999999994},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":13.452999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.853999999999999},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":7.9500000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":9.3059999999999992},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":14.339}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.127000000000001},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":9.8640000000000008}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":10.441000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":12.66}]}]},{"osd":6,"up_from":15,"seq":64424509529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.5599999999999996},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.3949999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":15.577999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":9.0679999999999996},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":9.7520000000000007}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":16.440000000000001},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":14.420999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":12.340999999999999},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":13.019}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":11.336},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":17.273}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:20.846 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:20.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.945+0000 7f09e857c640 10 monclient: get_monmap_and_config 2024-01-20T11:22:20.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.945+0000 7f09e857c640 10 monclient: build_initial_monmap 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.945+0000 7f09e857c640 10 monclient: monmap: 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:20.947458+0000 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:20.947458+0000 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:20.947 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.946+0000 7f09e857c640 1 Processor -- start 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.946+0000 7f09e857c640 1 -- start start 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.946+0000 7f09e857c640 10 monclient: init 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f09e00af3a0 0x7f09e013b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 10 monclient(hunting): picked mon.noname-c con 0x7f09e00af3a0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09e00ab6d0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00adb30 0x7f09e013bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 10 monclient(hunting): picked mon.noname-b con 0x7f09e00adb30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e0141930 con 0x7f09e00ab6d0 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:20.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e0142920 con 0x7f09e00af3a0 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e00af780 con 0x7f09e00adb30 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e857c640 10 monclient(hunting): authenticate will time out at 1784.298542s 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e6b19640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00adb30 0x7f09e013bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.947+0000 7f09e5b17640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f09e00ab6d0 0x7f09e00ad420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53148/0 (socket says 172.21.15.110:53148) 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e5b17640 1 -- 172.21.15.110:0/4159998867 learned_addr learned my addr 172.21.15.110:0/4159998867 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6b19640 10 monclient(hunting): get_auth_request con 0x7f09e00adb30 auth_method 0 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6b19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6b19640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6b19640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6318640 1 --2- 172.21.15.110:0/4159998867 >> v2:172.21.15.134:3300/0 conn(0x7f09e00af3a0 0x7f09e013b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6318640 10 monclient(hunting): get_auth_request con 0x7f09e00af3a0 auth_method 0 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6318640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:20.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6318640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:20.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6318640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:20.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6b19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:20.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6b19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:20.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.948+0000 7f09e6b19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:20.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.949+0000 7f09e6318640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:20.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.949+0000 7f09e6318640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:20.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.949+0000 7f09e6318640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:20.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e5316640 1 -- 172.21.15.110:0/4159998867 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483614578 0 0) 0x7f09e0141930 con 0x7f09e00ab6d0 2024-01-20T11:22:20.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e5316640 10 monclient(hunting): _negotiate 2024-01-20T11:22:20.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e5316640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:20.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e5316640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:20.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e5316640 10 monclient(hunting): my global_id is 5778 2024-01-20T11:22:20.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e5316640 1 -- 172.21.15.110:0/4159998867 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc005730 con 0x7f09e00ab6d0 2024-01-20T11:22:20.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e6b19640 10 monclient(hunting): handle_auth_done global_id 5444 payload 274 2024-01-20T11:22:20.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e6b19640 1 -- 172.21.15.110:0/4159998867 >> v2:172.21.15.134:3300/0 conn(0x7f09e00af3a0 msgr2=0x7f09e013b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:20.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e6b19640 1 --2- 172.21.15.110:0/4159998867 >> v2:172.21.15.134:3300/0 conn(0x7f09e00af3a0 0x7f09e013b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:20.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e6b19640 1 -- 172.21.15.110:0/4159998867 >> v1:172.21.15.110:6789/0 conn(0x7f09e00ab6d0 legacy=0x7f09e00ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:20.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e6b19640 10 monclient: _finish_hunting 0 2024-01-20T11:22:20.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e6b19640 1 monclient: found mon.noname-b 2024-01-20T11:22:20.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e6b19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:20.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e6b19640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:20.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.950+0000 7f09e6b19640 1 -- 172.21.15.110:0/4159998867 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00ab2b0 con 0x7f09e00adb30 2024-01-20T11:22:20.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.952+0000 7f09e6b19640 1 --2- 172.21.15.110:0/4159998867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00adb30 0x7f09e013bc70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0097a0 tx=0x7f09dc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20225c2bdf198b55 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:20.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 1 -- 172.21.15.110:0/4159998867 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc031830 con 0x7f09e00adb30 2024-01-20T11:22:20.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:20.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:20.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 10 monclient: dump: 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 10 monclient: _finish_auth 0 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 10 monclient: _check_auth_tickets 2024-01-20T11:22:20.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 1 -- 172.21.15.110:0/4159998867 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc031990 con 0x7f09e00adb30 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 1 -- 172.21.15.110:0/4159998867 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc031d20 con 0x7f09e00adb30 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e857c640 5 monclient: authenticate success, global_id 5444 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e857c640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e857c640 4 set_mon_vals no callback set 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.956+0000 7f09e5316640 10 monclient: dump: 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:20.959 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:20.960 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:20.960 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:20.960 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:20.960 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:20.960 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:20.960 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:20.960 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:20.960 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 10 monclient: shutdown 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 1 -- 172.21.15.110:0/4159998867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00adb30 msgr2=0x7f09e013bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 1 --2- 172.21.15.110:0/4159998867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00adb30 0x7f09e013bc70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0097a0 tx=0x7f09dc02b8d0 comp rx=0 tx=0).stop 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 1 -- 172.21.15.110:0/4159998867 shutdown_connections 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 1 --2- 172.21.15.110:0/4159998867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00adb30 0x7f09e013bc70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 1 --2- 172.21.15.110:0/4159998867 >> v2:172.21.15.134:3300/0 conn(0x7f09e00af3a0 0x7f09e013b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 1 -- 172.21.15.110:0/4159998867 >> 172.21.15.110:0/4159998867 conn(0x7f09e00b3e60 msgr2=0x7f09e00b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 1 -- 172.21.15.110:0/4159998867 shutdown_connections 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.957+0000 7f09e857c640 1 -- 172.21.15.110:0/4159998867 wait complete. 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.958+0000 7f09e857c640 10 monclient: build_initial_monmap 2024-01-20T11:22:20.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.958+0000 7f09e857c640 10 monclient: monmap: 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.958+0000 7f09e857c640 1 librados: starting msgr at 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.958+0000 7f09e857c640 1 librados: starting objecter 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.958+0000 7f09e857c640 1 Processor -- start 2024-01-20T11:22:20.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.958+0000 7f09e857c640 1 -- start start 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.958+0000 7f09e857c640 1 librados: setting wanted keys 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.958+0000 7f09e857c640 1 librados: calling monclient init 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.958+0000 7f09e857c640 10 monclient: init 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00ab6d0 0x7f09e0071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient(hunting): picked mon.noname-c con 0x7f09e00ab6d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e00adb30 0x7f09e013c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient(hunting): picked mon.noname-a con 0x7f09e00adb30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e00af3a0 0x7f09e01daca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient(hunting): picked mon.noname-b con 0x7f09e00af3a0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01da780 con 0x7f09e00adb30 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:20.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e0139280 con 0x7f09e00af3a0 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01db1e0 con 0x7f09e00ab6d0 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e857c640 10 monclient(hunting): authenticate will time out at 1784.310670s 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e5b17640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e00adb30 0x7f09e013c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.959+0000 7f09e6318640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00ab6d0 0x7f09e0071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6b19640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e00af3a0 0x7f09e01daca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e5b17640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e00adb30 0x7f09e013c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52792/0 (socket says 172.21.15.110:52792) 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e5b17640 1 -- 172.21.15.110:0/2498048147 learned_addr learned my addr 172.21.15.110:0/2498048147 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e5b17640 10 monclient(hunting): get_auth_request con 0x7f09e00adb30 auth_method 0 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e5b17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e5b17640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e5b17640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:20.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6318640 10 monclient(hunting): get_auth_request con 0x7f09e00ab6d0 auth_method 0 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6318640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6318640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6318640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6b19640 10 monclient(hunting): get_auth_request con 0x7f09e00af3a0 auth_method 0 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6b19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6b19640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6b19640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6318640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6318640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e6318640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e5b17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e5b17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.960+0000 7f09e5b17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.961+0000 7f09e6b19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:20.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.961+0000 7f09e6b19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.961+0000 7f09e6b19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 10 monclient(hunting): handle_auth_done global_id 5447 payload 824 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 1 -- 172.21.15.110:0/2498048147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e00af3a0 msgr2=0x7f09e01daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e00af3a0 0x7f09e01daca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 1 -- 172.21.15.110:0/2498048147 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e00adb30 msgr2=0x7f09e013c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e00adb30 0x7f09e013c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 10 monclient: _finish_hunting 0 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 1 monclient: found mon.noname-c 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:20.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.963+0000 7f09e6318640 1 -- 172.21.15.110:0/2498048147 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc009400 con 0x7f09e00ab6d0 2024-01-20T11:22:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.966+0000 7f09e6318640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00ab6d0 0x7f09e0071850 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09d800ebc0 tx=0x7f09d800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.966+0000 7f09d6ffd640 10 client.?.objecter ms_handle_connect 0x7f09e00ab6d0 2024-01-20T11:22:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.966+0000 7f09d6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 1 -- 172.21.15.110:0/2498048147 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d800cda0 con 0x7f09e00ab6d0 2024-01-20T11:22:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 10 monclient: dump: 2024-01-20T11:22:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 1 -- 172.21.15.110:0/2498048147 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d800f040 con 0x7f09e00ab6d0 2024-01-20T11:22:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09e857c640 5 monclient: authenticate success, global_id 5447 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09e857c640 10 monclient: _renew_subs 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09e857c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e01da3a0 con 0x7f09e00ab6d0 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 1 -- 172.21.15.110:0/2498048147 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d80146c0 con 0x7f09e00ab6d0 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09d6ffd640 10 monclient: dump: 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09e4b15640 4 set_mon_vals no callback set 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09e857c640 10 client.5447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.969+0000 7f09e857c640 10 monclient: _renew_subs 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.970+0000 7f09e857c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.970+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e01389c0 con 0x7f09e00ab6d0 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.970+0000 7f09e857c640 1 librados: init done 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.971+0000 7f09e857c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.971+0000 7f09e857c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:20.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.971+0000 7f09e857c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:20.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.971+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e01da8f0 con 0x7f09e00ab6d0 2024-01-20T11:22:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.989+0000 7f09d6ffd640 1 -- 172.21.15.110:0/2498048147 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f09d8014820 con 0x7f09e00ab6d0 2024-01-20T11:22:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.989+0000 7f09d6ffd640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f09c406c8e0 0x7f09c406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.989+0000 7f09e5b17640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f09c406c8e0 0x7f09c406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.989+0000 7f09e5b17640 10 monclient: get_auth_request con 0x7f09c406c8e0 auth_method 0 2024-01-20T11:22:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.990+0000 7f09d6ffd640 10 client.5447.objecter ms_handle_connect 0x7f09c406c8e0 2024-01-20T11:22:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.990+0000 7f09e5b17640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f09c406c8e0 0x7f09c406eda0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f09e011ba40 tx=0x7f09d0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:20.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.993+0000 7f09d6ffd640 1 -- 172.21.15.110:0/2498048147 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f09d8059410 con 0x7f09e00ab6d0 2024-01-20T11:22:20.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.993+0000 7f09d6ffd640 10 client.5447.objecter ms_dispatch 0x7f09e0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:20.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.993+0000 7f09d6ffd640 3 client.5447.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:20.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.993+0000 7f09d6ffd640 3 client.5447.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:20.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:20.993+0000 7f09d6ffd640 20 client.5447.objecter dump_active .. 0 homeless 2024-01-20T11:22:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.225+0000 7f09d6ffd640 1 -- 172.21.15.110:0/2498048147 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f09d8055750 con 0x7f09e00ab6d0 2024-01-20T11:22:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.225+0000 7f09d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:21.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.225+0000 7f09d6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:21.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.401+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f09e00acd20 con 0x7f09c406c8e0 2024-01-20T11:22:21.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.404+0000 7f09d6ffd640 1 -- 172.21.15.110:0/2498048147 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63997 (secure 0 0 0) 0x7f09d000f820 con 0x7f09c406c8e0 2024-01-20T11:22:21.406 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:21.406 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:21.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 10 librados: watch_flush enter 2024-01-20T11:22:21.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 10 librados: watch_flush exit 2024-01-20T11:22:21.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 20 client.5447.objecter shutdown clearing up homeless session... 2024-01-20T11:22:21.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 10 client.5447.objecter successfully canceled tick 2024-01-20T11:22:21.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f09c406c8e0 msgr2=0x7f09c406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:21.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f09c406c8e0 0x7f09c406eda0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f09e011ba40 tx=0x7f09d0009040 comp rx=0 tx=0).stop 2024-01-20T11:22:21.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 10 monclient: shutdown 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00ab6d0 msgr2=0x7f09e0071850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00ab6d0 0x7f09e0071850 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09d800ebc0 tx=0x7f09d800c490 comp rx=0 tx=0).stop 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 shutdown_connections 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f09c406c8e0 0x7f09c406eda0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e00af3a0 0x7f09e01daca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e00adb30 0x7f09e013c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 --2- 172.21.15.110:0/2498048147 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e00ab6d0 0x7f09e0071850 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.406+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 >> 172.21.15.110:0/2498048147 conn(0x7f09e00b3e60 msgr2=0x7f09e00b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.407+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 shutdown_connections 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.407+0000 7f09e857c640 1 -- 172.21.15.110:0/2498048147 wait complete. 2024-01-20T11:22:21.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.407+0000 7f09e857c640 1 librados: shutdown 2024-01-20T11:22:21.418 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-01-20T11:22:20.796316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":24.600000000000001},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.0830000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":22.195},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":22.798999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":19.283999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":20.315000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":27.010999999999999},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":18.577000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":23.946999999999999},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":25.286000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":20.991},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":27.356000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313305,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":7.0720000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":4.4240000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.5039999999999996},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":11.449}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":12.989000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.8659999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":11.087999999999999},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":13.630000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":14.829000000000001},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.363}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":15.558},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":16.283999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313306,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.7110000000000003},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":9.1850000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":5.9820000000000002},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":7.4080000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":9.7940000000000005},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":12.494}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":13.291},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":11.898999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":8.5199999999999996},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":14.414999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":15.163},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":10.500999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444120,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":7.2220000000000004},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":11.699}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.577},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":15.31}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.116},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.0739999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.348000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.381}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.739000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.084}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.5129999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.219999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444121,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.4320000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.9809999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":9.8670000000000009},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":7.9109999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":13.08},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":12.395}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":10.769},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.257999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.449},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":15.853}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":12.034000000000001},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":16.934000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444121,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":7.7549999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":4.6159999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":8.5600000000000005},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.8849999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.721},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.888}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.737},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":6.7619999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.339},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.321999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":9.2100000000000009}]}]},{"osd":6,"up_from":15,"seq":64424509529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.5599999999999996},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.3949999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":15.577999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":9.0679999999999996},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":9.7520000000000007}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":16.440000000000001},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":14.420999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":12.340999999999999},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":13.019}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":11.336},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":17.273}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:21.419 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:21.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.520+0000 7f6854541640 10 monclient: get_monmap_and_config 2024-01-20T11:22:21.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.520+0000 7f6854541640 10 monclient: build_initial_monmap 2024-01-20T11:22:21.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.520+0000 7f6854541640 10 monclient: monmap: 2024-01-20T11:22:21.521 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:21.521 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:21.521 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:21.522139+0000 2024-01-20T11:22:21.521 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:21.522139+0000 2024-01-20T11:22:21.521 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:21.522 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:21.522 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:21.522 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:21.522 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:21.522 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:21.522 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:21.522 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:21.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 1 Processor -- start 2024-01-20T11:22:21.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 1 -- start start 2024-01-20T11:22:21.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient: init 2024-01-20T11:22:21.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f684c06f920 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c143950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient(hunting): picked mon.noname-b con 0x7f684c06fd00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f684c143e90 0x7f684c14a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient(hunting): picked mon.noname-c con 0x7f684c143e90 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684c0715b0 con 0x7f684c06f920 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684c1442e0 con 0x7f684c143e90 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c071890 con 0x7f684c06fd00 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:21.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.521+0000 7f6854541640 10 monclient(hunting): authenticate will time out at 1784.873171s 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.522+0000 7f68522dd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f684c06f920 0x7f684c13d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53162/0 (socket says 172.21.15.110:53162) 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.522+0000 7f68522dd640 1 -- 172.21.15.110:0/1458160326 learned_addr learned my addr 172.21.15.110:0/1458160326 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.522+0000 7f6851adc640 1 --2- 172.21.15.110:0/1458160326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.522+0000 7f6852ade640 1 --2- 172.21.15.110:0/1458160326 >> v2:172.21.15.134:3300/0 conn(0x7f684c143e90 0x7f684c14a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6851adc640 10 monclient(hunting): get_auth_request con 0x7f684c06fd00 auth_method 0 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6851adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6851adc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6851adc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6852ade640 10 monclient(hunting): get_auth_request con 0x7f684c143e90 auth_method 0 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6852ade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6852ade640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6852ade640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6851adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6851adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:21.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6851adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:21.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6852ade640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:21.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6852ade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:21.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.523+0000 7f6852ade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f6851adc640 10 monclient(hunting): handle_auth_done global_id 5450 payload 274 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f68512db640 1 -- 172.21.15.110:0/1458160326 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680805812 0 0) 0x7f684c0715b0 con 0x7f684c06f920 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f6851adc640 1 -- 172.21.15.110:0/1458160326 >> v2:172.21.15.134:3300/0 conn(0x7f684c143e90 msgr2=0x7f684c14a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f6851adc640 1 --2- 172.21.15.110:0/1458160326 >> v2:172.21.15.134:3300/0 conn(0x7f684c143e90 0x7f684c14a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f6851adc640 1 -- 172.21.15.110:0/1458160326 >> v1:172.21.15.110:6789/0 conn(0x7f684c06f920 legacy=0x7f684c13d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f6851adc640 10 monclient: _finish_hunting 0 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f6851adc640 1 monclient: found mon.noname-b 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f6851adc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f6851adc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f6851adc640 1 -- 172.21.15.110:0/1458160326 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c071b70 con 0x7f684c06fd00 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.525+0000 7f68512db640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:22:21.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.527+0000 7f6851adc640 1 --2- 172.21.15.110:0/1458160326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c143950 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f683c009790 tx=0x7f683c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b932c35882ae85f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:21.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 1 -- 172.21.15.110:0/1458160326 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c031770 con 0x7f684c06fd00 2024-01-20T11:22:21.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 10 monclient: dump: 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:21.532 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 10 monclient: _finish_auth 0 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 10 monclient: _check_auth_tickets 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 1 -- 172.21.15.110:0/1458160326 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c0318d0 con 0x7f684c06fd00 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f6854541640 5 monclient: authenticate success, global_id 5450 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f6854541640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f6854541640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f6854541640 4 set_mon_vals no callback set 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 1 -- 172.21.15.110:0/1458160326 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c031c60 con 0x7f684c06fd00 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:21.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.530+0000 7f68512db640 10 monclient: dump: 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:21.534 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:21.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 10 monclient: shutdown 2024-01-20T11:22:21.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:21.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 1 -- 172.21.15.110:0/1458160326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 msgr2=0x7f684c143950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:21.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 1 --2- 172.21.15.110:0/1458160326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c143950 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f683c009790 tx=0x7f683c02b810 comp rx=0 tx=0).stop 2024-01-20T11:22:21.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 1 -- 172.21.15.110:0/1458160326 shutdown_connections 2024-01-20T11:22:21.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 1 --2- 172.21.15.110:0/1458160326 >> v2:172.21.15.134:3300/0 conn(0x7f684c143e90 0x7f684c14a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 1 --2- 172.21.15.110:0/1458160326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c143950 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 1 -- 172.21.15.110:0/1458160326 >> 172.21.15.110:0/1458160326 conn(0x7f684c066d60 msgr2=0x7f684c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 1 -- 172.21.15.110:0/1458160326 shutdown_connections 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.531+0000 7f6854541640 1 -- 172.21.15.110:0/1458160326 wait complete. 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.532+0000 7f6854541640 10 monclient: build_initial_monmap 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.532+0000 7f6854541640 10 monclient: monmap: 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:21.539 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.532+0000 7f6854541640 1 librados: starting msgr at 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.532+0000 7f6854541640 1 librados: starting objecter 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.532+0000 7f6854541640 1 Processor -- start 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.532+0000 7f6854541640 1 -- start start 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.532+0000 7f6854541640 1 librados: setting wanted keys 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.532+0000 7f6854541640 1 librados: calling monclient init 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.532+0000 7f6854541640 10 monclient: init 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f684c06f920 0x7f684c1d88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient(hunting): picked mon.noname-b con 0x7f684c06f920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c0a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:21.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient(hunting): picked mon.noname-c con 0x7f684c06fd00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f684c143e90 0x7f684c0ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient(hunting): picked mon.noname-a con 0x7f684c143e90 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0ac1d0 con 0x7f684c143e90 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c1411c0 con 0x7f684c06f920 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c1d9330 con 0x7f684c06fd00 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6854541640 10 monclient(hunting): authenticate will time out at 1784.884422s 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6851adc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c0a9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f68522dd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f684c06f920 0x7f684c1d88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.533+0000 7f6852ade640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f684c143e90 0x7f684c0ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:21.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6851adc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c0a9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47878/0 (socket says 172.21.15.110:47878) 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6851adc640 1 -- 172.21.15.110:0/1444029684 learned_addr learned my addr 172.21.15.110:0/1444029684 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6851adc640 10 monclient(hunting): get_auth_request con 0x7f684c06fd00 auth_method 0 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6851adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6851adc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6851adc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f68522dd640 10 monclient(hunting): get_auth_request con 0x7f684c06f920 auth_method 0 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f68522dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f68522dd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f68522dd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6852ade640 10 monclient(hunting): get_auth_request con 0x7f684c143e90 auth_method 0 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6852ade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6852ade640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6852ade640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6851adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:21.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6851adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6851adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f68522dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f68522dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f68522dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6852ade640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6852ade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.534+0000 7f6852ade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 10 monclient(hunting): handle_auth_done global_id 5453 payload 824 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 1 -- 172.21.15.110:0/1444029684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f684c06f920 msgr2=0x7f684c1d88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f684c06f920 0x7f684c1d88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 1 -- 172.21.15.110:0/1444029684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f684c143e90 msgr2=0x7f684c0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f684c143e90 0x7f684c0ae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 10 monclient: _finish_hunting 0 2024-01-20T11:22:21.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 1 monclient: found mon.noname-c 2024-01-20T11:22:21.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:21.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:21.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.537+0000 7f6851adc640 1 -- 172.21.15.110:0/1444029684 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c0093a0 con 0x7f684c06fd00 2024-01-20T11:22:21.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.539+0000 7f6851adc640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c0a9a90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f683c034040 tx=0x7f683c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:21.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.539+0000 7f6842ffd640 10 client.?.objecter ms_handle_connect 0x7f684c06fd00 2024-01-20T11:22:21.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.539+0000 7f6842ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:21.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 1 -- 172.21.15.110:0/1444029684 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c005da0 con 0x7f684c06fd00 2024-01-20T11:22:21.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:21.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 10 monclient: dump: 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:21.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 1 -- 172.21.15.110:0/1444029684 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c006030 con 0x7f684c06fd00 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 1 -- 172.21.15.110:0/1444029684 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0063e0 con 0x7f684c06fd00 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6842ffd640 10 monclient: dump: 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:21.547 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.543+0000 7f6854541640 5 monclient: authenticate success, global_id 5453 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.543+0000 7f6854541640 10 monclient: _renew_subs 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.543+0000 7f6854541640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.543+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684c1d94f0 con 0x7f684c06fd00 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.542+0000 7f6850ada640 4 set_mon_vals no callback set 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.543+0000 7f6854541640 10 client.5453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.543+0000 7f6854541640 10 monclient: _renew_subs 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.543+0000 7f6854541640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.543+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684c0af080 con 0x7f684c06fd00 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.543+0000 7f6854541640 1 librados: init done 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.544+0000 7f6854541640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.544+0000 7f6854541640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.544+0000 7f6854541640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:21.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.544+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684c075490 con 0x7f684c06fd00 2024-01-20T11:22:21.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.562+0000 7f6842ffd640 1 -- 172.21.15.110:0/1444029684 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f683c006540 con 0x7f684c06fd00 2024-01-20T11:22:21.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.563+0000 7f6842ffd640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683006c8b0 0x7f683006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:21.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.563+0000 7f68522dd640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683006c8b0 0x7f683006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:21.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.563+0000 7f68522dd640 10 monclient: get_auth_request con 0x7f683006c8b0 auth_method 0 2024-01-20T11:22:21.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.564+0000 7f68522dd640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683006c8b0 0x7f683006ed70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6848000f60 tx=0x7f6848007e40 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:21.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.564+0000 7f6842ffd640 10 client.5453.objecter ms_handle_connect 0x7f683006c8b0 2024-01-20T11:22:21.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.798+0000 7f6842ffd640 1 -- 172.21.15.110:0/1444029684 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f683c07a8f0 con 0x7f684c06fd00 2024-01-20T11:22:21.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.798+0000 7f6842ffd640 10 client.5453.objecter ms_dispatch 0x7f684c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:21.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.798+0000 7f6842ffd640 3 client.5453.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:21.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.798+0000 7f6842ffd640 3 client.5453.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:21.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.798+0000 7f6842ffd640 20 client.5453.objecter dump_active .. 0 homeless 2024-01-20T11:22:21.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.798+0000 7f6842ffd640 1 -- 172.21.15.110:0/1444029684 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f683c076c30 con 0x7f684c06fd00 2024-01-20T11:22:21.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.798+0000 7f6842ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:21.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.798+0000 7f6842ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:21.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.967+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f684c0b03c0 con 0x7f683006c8b0 2024-01-20T11:22:21.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.971+0000 7f6842ffd640 1 -- 172.21.15.110:0/1444029684 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63997 (secure 0 0 0) 0x7f6848004c70 con 0x7f683006c8b0 2024-01-20T11:22:21.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:21.972 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:21.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 10 librados: watch_flush enter 2024-01-20T11:22:21.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 10 librados: watch_flush exit 2024-01-20T11:22:21.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 20 client.5453.objecter shutdown clearing up homeless session... 2024-01-20T11:22:21.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 10 client.5453.objecter successfully canceled tick 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683006c8b0 msgr2=0x7f683006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683006c8b0 0x7f683006ed70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6848000f60 tx=0x7f6848007e40 comp rx=0 tx=0).stop 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 10 monclient: shutdown 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 msgr2=0x7f684c0a9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c0a9a90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f683c034040 tx=0x7f683c0046c0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 shutdown_connections 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f683006c8b0 0x7f683006ed70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f684c143e90 0x7f684c0ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f684c06fd00 0x7f684c0a9a90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 --2- 172.21.15.110:0/1444029684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f684c06f920 0x7f684c1d88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 >> 172.21.15.110:0/1444029684 conn(0x7f684c066d60 msgr2=0x7f684c0b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 shutdown_connections 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.972+0000 7f6854541640 1 -- 172.21.15.110:0/1444029684 wait complete. 2024-01-20T11:22:21.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:21.973+0000 7f6854541640 1 librados: shutdown 2024-01-20T11:22:21.984 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-01-20T11:22:20.796316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":24.600000000000001},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.0830000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":22.195},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":22.798999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":19.283999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":20.315000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":27.010999999999999},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":18.577000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":23.946999999999999},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":25.286000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":20.991},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":27.356000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313305,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":7.0720000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":4.4240000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.5039999999999996},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":11.449}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":12.989000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":7.8659999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":11.087999999999999},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":13.630000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":14.829000000000001},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.363}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":15.558},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":16.283999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313306,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.7110000000000003},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":9.1850000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":5.9820000000000002},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":7.4080000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":9.7940000000000005},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":12.494}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":13.291},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":11.898999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":8.5199999999999996},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":14.414999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":15.163},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":10.500999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444120,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":7.2220000000000004},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":11.699}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.577},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":15.31}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.116},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.0739999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.348000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.381}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.739000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.084}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.5129999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.219999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444121,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.4320000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.9809999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":9.8670000000000009},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":7.9109999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":13.08},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":12.395}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":10.769},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.257999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.449},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":15.853}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":12.034000000000001},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":16.934000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444121,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":7.7549999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":4.6159999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":8.5600000000000005},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.8849999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.721},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.888}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.737},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":6.7619999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.339},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.321999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":9.2100000000000009}]}]},{"osd":6,"up_from":15,"seq":64424509529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.5599999999999996},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.3949999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.2279999999999998},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":15.577999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":9.0679999999999996},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":9.7520000000000007}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":16.440000000000001},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":14.420999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":12.340999999999999},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":13.019}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":11.336},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":17.273}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:21.985 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:21.985 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:21.986 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:21.986 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:21.986 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:21.986 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:21.986 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:21.986 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:21.986 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:22.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.083+0000 7f6bb89f5640 10 monclient: get_monmap_and_config 2024-01-20T11:22:22.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.083+0000 7f6bb89f5640 10 monclient: build_initial_monmap 2024-01-20T11:22:22.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.083+0000 7f6bb89f5640 10 monclient: monmap: 2024-01-20T11:22:22.085 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:22.085 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.085 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:22.085161+0000 2024-01-20T11:22:22.085 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:22.085161+0000 2024-01-20T11:22:22.086 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:22.086 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.086 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:22.086 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:22.086 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:22.086 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:22.086 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:22.087 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 1 Processor -- start 2024-01-20T11:22:22.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 1 -- start start 2024-01-20T11:22:22.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient: init 2024-01-20T11:22:22.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:22.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-01-20T11:22:22.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bb40af3a0 0x7f6bb413b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient(hunting): picked mon.noname-a con 0x7f6bb40af3a0 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6bb40ab6d0 0x7f6bb413bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient(hunting): picked mon.noname-c con 0x7f6bb40ab6d0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6bb40ad420 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb4141930 con 0x7f6bb40ad420 2024-01-20T11:22:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb4142920 con 0x7f6bb40af3a0 2024-01-20T11:22:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb40af780 con 0x7f6bb40ab6d0 2024-01-20T11:22:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.084+0000 7f6bb89f5640 10 monclient(hunting): authenticate will time out at 1785.436080s 2024-01-20T11:22:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.085+0000 7f6bb259c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bb40af3a0 0x7f6bb413b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.085+0000 7f6babfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f6bb40ab6d0 0x7f6bb413bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6bb259c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bb40af3a0 0x7f6bb413b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52814/0 (socket says 172.21.15.110:52814) 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6bb259c640 1 -- 172.21.15.110:0/3807810764 learned_addr learned my addr 172.21.15.110:0/3807810764 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6bb259c640 10 monclient(hunting): get_auth_request con 0x7f6bb40af3a0 auth_method 0 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6bb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6bb259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6bb259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6bb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6bb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6bb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6babfff640 10 monclient(hunting): get_auth_request con 0x7f6bb40ab6d0 auth_method 0 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6babfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6babfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.086+0000 7f6babfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.087+0000 7f6babfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.087+0000 7f6babfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.087+0000 7f6babfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.088+0000 7f6bb259c640 10 monclient(hunting): handle_auth_done global_id 5790 payload 274 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.088+0000 7f6bb259c640 1 -- 172.21.15.110:0/3807810764 >> v2:172.21.15.134:3300/0 conn(0x7f6bb40ab6d0 msgr2=0x7f6bb413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.088+0000 7f6bb259c640 1 --2- 172.21.15.110:0/3807810764 >> v2:172.21.15.134:3300/0 conn(0x7f6bb40ab6d0 0x7f6bb413bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.088+0000 7f6bb259c640 1 -- 172.21.15.110:0/3807810764 >> v1:172.21.15.134:6789/0 conn(0x7f6bb40ad420 legacy=0x7f6bb40ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.088+0000 7f6bb259c640 10 monclient: _finish_hunting 0 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.088+0000 7f6bb259c640 1 monclient: found mon.noname-a 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.088+0000 7f6bb259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.088+0000 7f6bb259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.088+0000 7f6bb259c640 1 -- 172.21.15.110:0/3807810764 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb40ab2b0 con 0x7f6bb40af3a0 2024-01-20T11:22:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.090+0000 7f6bb259c640 1 --2- 172.21.15.110:0/3807810764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 0x7f6bb413b730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40b0120 tx=0x7f6ba402df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ba2a9beba18067 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:22.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 1 -- 172.21.15.110:0/3807810764 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba402ece0 con 0x7f6bb40af3a0 2024-01-20T11:22:22.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 10 monclient: dump: 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 10 monclient: _finish_auth 0 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 1 -- 172.21.15.110:0/3807810764 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba4032070 con 0x7f6bb40af3a0 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 1 -- 172.21.15.110:0/3807810764 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4038620 con 0x7f6bb40af3a0 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb1d9b640 10 monclient: dump: 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb89f5640 5 monclient: authenticate success, global_id 5790 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb89f5640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.093+0000 7f6bb89f5640 4 set_mon_vals no callback set 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 10 monclient: shutdown 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 1 -- 172.21.15.110:0/3807810764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 msgr2=0x7f6bb413b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 1 --2- 172.21.15.110:0/3807810764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 0x7f6bb413b730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40b0120 tx=0x7f6ba402df30 comp rx=0 tx=0).stop 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 1 -- 172.21.15.110:0/3807810764 shutdown_connections 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 1 --2- 172.21.15.110:0/3807810764 >> v2:172.21.15.134:3300/0 conn(0x7f6bb40ab6d0 0x7f6bb413bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 1 --2- 172.21.15.110:0/3807810764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 0x7f6bb413b730 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 1 -- 172.21.15.110:0/3807810764 >> 172.21.15.110:0/3807810764 conn(0x7f6bb40b3e60 msgr2=0x7f6bb40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 1 -- 172.21.15.110:0/3807810764 shutdown_connections 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.094+0000 7f6bb89f5640 1 -- 172.21.15.110:0/3807810764 wait complete. 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.095+0000 7f6bb89f5640 10 monclient: build_initial_monmap 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.095+0000 7f6bb89f5640 10 monclient: monmap: 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.095+0000 7f6bb89f5640 1 librados: starting msgr at 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.095+0000 7f6bb89f5640 1 librados: starting objecter 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.095+0000 7f6bb89f5640 1 Processor -- start 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.095+0000 7f6bb89f5640 1 -- start start 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.095+0000 7f6bb89f5640 1 librados: setting wanted keys 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.095+0000 7f6bb89f5640 1 librados: calling monclient init 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.095+0000 7f6bb89f5640 10 monclient: init 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bb40ab6d0 0x7f6bb4071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient(hunting): picked mon.noname-b con 0x7f6bb40ab6d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bb40ad420 0x7f6bb413e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient(hunting): picked mon.noname-c con 0x7f6bb40ad420 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 0x7f6bb41dab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient(hunting): picked mon.noname-a con 0x7f6bb40af3a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb41da670 con 0x7f6bb40af3a0 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb413fc20 con 0x7f6bb40ab6d0 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb41414e0 con 0x7f6bb40ad420 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb89f5640 10 monclient(hunting): authenticate will time out at 1785.447574s 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bb40ab6d0 0x7f6bb4071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6babfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bb40ad420 0x7f6bb413e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.096+0000 7f6bb2d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 0x7f6bb41dab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb2d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 0x7f6bb41dab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52824/0 (socket says 172.21.15.110:52824) 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb2d9d640 1 -- 172.21.15.110:0/359734319 learned_addr learned my addr 172.21.15.110:0/359734319 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb2d9d640 10 monclient(hunting): get_auth_request con 0x7f6bb40af3a0 auth_method 0 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb259c640 10 monclient(hunting): get_auth_request con 0x7f6bb40ab6d0 auth_method 0 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb259c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6babfff640 10 monclient(hunting): get_auth_request con 0x7f6bb40ad420 auth_method 0 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6babfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6babfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6babfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6bb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6babfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6babfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.097+0000 7f6babfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 10 monclient(hunting): handle_auth_done global_id 5793 payload 824 2024-01-20T11:22:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 1 -- 172.21.15.110:0/359734319 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bb40ad420 msgr2=0x7f6bb413e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bb40ad420 0x7f6bb413e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 1 -- 172.21.15.110:0/359734319 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bb40ab6d0 msgr2=0x7f6bb4071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bb40ab6d0 0x7f6bb4071850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 10 monclient: _finish_hunting 0 2024-01-20T11:22:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 1 monclient: found mon.noname-a 2024-01-20T11:22:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:22.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.100+0000 7f6bb2d9d640 1 -- 172.21.15.110:0/359734319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba400b760 con 0x7f6bb40af3a0 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.103+0000 7f6bb2d9d640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 0x7f6bb41dab00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6bac00c7a0 tx=0x7f6bac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.103+0000 7f6baaffd640 10 client.?.objecter ms_handle_connect 0x7f6bb40af3a0 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.103+0000 7f6baaffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6baaffd640 1 -- 172.21.15.110:0/359734319 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac003b70 con 0x7f6bb40af3a0 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6baaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6baaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6baaffd640 10 monclient: dump: 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.110 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6baaffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6baaffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6baaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6baaffd640 1 -- 172.21.15.110:0/359734319 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac003cf0 con 0x7f6bb40af3a0 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6bb89f5640 5 monclient: authenticate success, global_id 5793 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6bb89f5640 10 monclient: _renew_subs 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6bb89f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb413ef10 con 0x7f6bb40af3a0 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.106+0000 7f6baaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6baaffd640 1 -- 172.21.15.110:0/359734319 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac003ea0 con 0x7f6bb40af3a0 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6baaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:22.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6baaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6baaffd640 10 monclient: dump: 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb159a640 4 set_mon_vals no callback set 2024-01-20T11:22:22.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb89f5640 10 client.5793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb89f5640 10 monclient: _renew_subs 2024-01-20T11:22:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb89f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb41db220 con 0x7f6bb40af3a0 2024-01-20T11:22:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb89f5640 1 librados: init done 2024-01-20T11:22:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb89f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb89f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb89f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.107+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb413f290 con 0x7f6bb40af3a0 2024-01-20T11:22:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.126+0000 7f6baaffd640 1 -- 172.21.15.110:0/359734319 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6bac00aa90 con 0x7f6bb40af3a0 2024-01-20T11:22:22.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.127+0000 7f6baaffd640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b9406c860 0x7f6b9406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.127+0000 7f6bb259c640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b9406c860 0x7f6b9406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.127+0000 7f6bb259c640 10 monclient: get_auth_request con 0x7f6b9406c860 auth_method 0 2024-01-20T11:22:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.128+0000 7f6baaffd640 10 client.5793.objecter ms_handle_connect 0x7f6b9406c860 2024-01-20T11:22:22.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.128+0000 7f6bb259c640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b9406c860 0x7f6b9406ed20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4039000 tx=0x7f6ba4034000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:22.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.365+0000 7f6baaffd640 1 -- 172.21.15.110:0/359734319 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6bac058a40 con 0x7f6bb40af3a0 2024-01-20T11:22:22.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.365+0000 7f6baaffd640 10 client.5793.objecter ms_dispatch 0x7f6bb4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:22.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.365+0000 7f6baaffd640 3 client.5793.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:22.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.365+0000 7f6baaffd640 3 client.5793.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:22.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.365+0000 7f6baaffd640 20 client.5793.objecter dump_active .. 0 homeless 2024-01-20T11:22:22.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.365+0000 7f6baaffd640 1 -- 172.21.15.110:0/359734319 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6bac054d80 con 0x7f6bb40af3a0 2024-01-20T11:22:22.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.365+0000 7f6baaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:22.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.365+0000 7f6baaffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:22.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.614+0000 7f6bb89f5640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:22.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.614+0000 7f6bb89f5640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:22.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.614+0000 7f6bb89f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.614+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f6bb4075490 con 0x7f6bb40af3a0 2024-01-20T11:22:22.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.619+0000 7f6baaffd640 1 -- 172.21.15.110:0/359734319 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f6bac0583a0 con 0x7f6bb40af3a0 2024-01-20T11:22:22.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.619+0000 7f6baaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:22.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.619+0000 7f6baaffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:22.620 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:22.620 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":477,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 10 librados: watch_flush enter 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 10 librados: watch_flush exit 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 20 client.5793.objecter shutdown clearing up homeless session... 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 10 client.5793.objecter successfully canceled tick 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b9406c860 msgr2=0x7f6b9406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b9406c860 0x7f6b9406ed20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4039000 tx=0x7f6ba4034000 comp rx=0 tx=0).stop 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 10 monclient: shutdown 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 msgr2=0x7f6bb41dab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.621+0000 7f6bb89f5640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 0x7f6bb41dab00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6bac00c7a0 tx=0x7f6bac00cc60 comp rx=0 tx=0).stop 2024-01-20T11:22:22.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.622+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 shutdown_connections 2024-01-20T11:22:22.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.622+0000 7f6bb89f5640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6b9406c860 0x7f6b9406ed20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.622+0000 7f6bb89f5640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bb40af3a0 0x7f6bb41dab00 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.622+0000 7f6bb89f5640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bb40ad420 0x7f6bb413e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.622+0000 7f6bb89f5640 1 --2- 172.21.15.110:0/359734319 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bb40ab6d0 0x7f6bb4071850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.622+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 >> 172.21.15.110:0/359734319 conn(0x7f6bb40b3e60 msgr2=0x7f6bb40aa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:22.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.622+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 shutdown_connections 2024-01-20T11:22:22.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.622+0000 7f6bb89f5640 1 -- 172.21.15.110:0/359734319 wait complete. 2024-01-20T11:22:22.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.622+0000 7f6bb89f5640 1 librados: shutdown 2024-01-20T11:22:22.633 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:22.633 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:22.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.732+0000 7f5a3c62b640 10 monclient: get_monmap_and_config 2024-01-20T11:22:22.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.732+0000 7f5a3c62b640 10 monclient: build_initial_monmap 2024-01-20T11:22:22.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.732+0000 7f5a3c62b640 10 monclient: monmap: 2024-01-20T11:22:22.733 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:22.733 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.733 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:22.733627+0000 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:22.733627+0000 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.732+0000 7f5a3c62b640 1 Processor -- start 2024-01-20T11:22:22.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.732+0000 7f5a3c62b640 1 -- start start 2024-01-20T11:22:22.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.732+0000 7f5a3c62b640 10 monclient: init 2024-01-20T11:22:22.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:22.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-01-20T11:22:22.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a340b4d10 0x7f5a340b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a340b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:22.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5a340b17c0 0x7f5a340b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a340b17c0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:22.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a34145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:22.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a340b2120 con 0x7f5a34145980 2024-01-20T11:22:22.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a340b50f0 con 0x7f5a340b17c0 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a340aade0 con 0x7f5a340b4d10 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.733+0000 7f5a3c62b640 10 monclient(hunting): authenticate will time out at 1786.084607s 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a3a3c7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a340b4d10 0x7f5a340b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a3abc8640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5a34145980 0x7f5a34145e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53174/0 (socket says 172.21.15.110:53174) 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a3abc8640 1 -- 172.21.15.110:0/3704367193 learned_addr learned my addr 172.21.15.110:0/3704367193 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a39bc6640 1 --2- 172.21.15.110:0/3704367193 >> v2:172.21.15.134:3300/0 conn(0x7f5a340b17c0 0x7f5a340b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a3a3c7640 10 monclient(hunting): get_auth_request con 0x7f5a340b4d10 auth_method 0 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a3a3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a3a3c7640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a3a3c7640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a39bc6640 10 monclient(hunting): get_auth_request con 0x7f5a340b17c0 auth_method 0 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a39bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a39bc6640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.734+0000 7f5a39bc6640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.735+0000 7f5a3a3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.735+0000 7f5a3a3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.735+0000 7f5a3a3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.735+0000 7f5a39bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.735+0000 7f5a39bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.735+0000 7f5a39bc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a3a3c7640 10 monclient(hunting): handle_auth_done global_id 5459 payload 274 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a3a3c7640 1 -- 172.21.15.110:0/3704367193 >> v2:172.21.15.134:3300/0 conn(0x7f5a340b17c0 msgr2=0x7f5a340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a3a3c7640 1 --2- 172.21.15.110:0/3704367193 >> v2:172.21.15.134:3300/0 conn(0x7f5a340b17c0 0x7f5a340b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a3a3c7640 1 -- 172.21.15.110:0/3704367193 >> v1:172.21.15.110:6789/0 conn(0x7f5a34145980 legacy=0x7f5a34145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a3a3c7640 10 monclient: _finish_hunting 0 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a3a3c7640 1 monclient: found mon.noname-b 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a3a3c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a3a3c7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a3a3c7640 1 -- 172.21.15.110:0/3704367193 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a34145460 con 0x7f5a340b4d10 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a393c5640 1 -- 172.21.15.110:0/3704367193 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410381571 0 0) 0x7f5a340b2120 con 0x7f5a34145980 2024-01-20T11:22:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.736+0000 7f5a393c5640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:22:22.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.738+0000 7f5a3a3c7640 1 --2- 172.21.15.110:0/3704367193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a340b4d10 0x7f5a340b3da0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009d90 tx=0x7f5a3002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a843701125e289c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:22.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.741+0000 7f5a393c5640 1 -- 172.21.15.110:0/3704367193 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3002cbc0 con 0x7f5a340b4d10 2024-01-20T11:22:22.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.741+0000 7f5a393c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.741+0000 7f5a393c5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.741+0000 7f5a393c5640 10 monclient: dump: 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.741+0000 7f5a393c5640 10 monclient: _finish_auth 0 2024-01-20T11:22:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.741+0000 7f5a393c5640 10 monclient: _check_auth_tickets 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.741+0000 7f5a393c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 5 monclient: authenticate success, global_id 5459 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a393c5640 1 -- 172.21.15.110:0/3704367193 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3002cd20 con 0x7f5a340b4d10 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a393c5640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a393c5640 1 -- 172.21.15.110:0/3704367193 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30034660 con 0x7f5a340b4d10 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 4 set_mon_vals no callback set 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a393c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a393c5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a393c5640 10 monclient: dump: 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:22.745 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 10 monclient: shutdown 2024-01-20T11:22:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3704367193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a340b4d10 msgr2=0x7f5a340b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 1 --2- 172.21.15.110:0/3704367193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a340b4d10 0x7f5a340b3da0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009d90 tx=0x7f5a3002be10 comp rx=0 tx=0).stop 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3704367193 shutdown_connections 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 1 --2- 172.21.15.110:0/3704367193 >> v2:172.21.15.134:3300/0 conn(0x7f5a340b17c0 0x7f5a340b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 1 --2- 172.21.15.110:0/3704367193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a340b4d10 0x7f5a340b3da0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3704367193 >> 172.21.15.110:0/3704367193 conn(0x7f5a34066590 msgr2=0x7f5a340669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3704367193 shutdown_connections 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.742+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3704367193 wait complete. 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.743+0000 7f5a3c62b640 10 monclient: build_initial_monmap 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.743+0000 7f5a3c62b640 10 monclient: monmap: 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:22.749 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.743+0000 7f5a3c62b640 1 librados: starting msgr at 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.743+0000 7f5a3c62b640 1 librados: starting objecter 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.743+0000 7f5a3c62b640 1 Processor -- start 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.743+0000 7f5a3c62b640 1 -- start start 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.743+0000 7f5a3c62b640 1 librados: setting wanted keys 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.743+0000 7f5a3c62b640 1 librados: calling monclient init 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.743+0000 7f5a3c62b640 10 monclient: init 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5a340b17c0 0x7f5a34142e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a340b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a340b4d10 0x7f5a34146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient(hunting): picked mon.noname-a con 0x7f5a340b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a34145980 0x7f5a34146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a34145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a34147ea0 con 0x7f5a340b4d10 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a341415c0 con 0x7f5a340b17c0 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a341480b0 con 0x7f5a34145980 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3c62b640 10 monclient(hunting): authenticate will time out at 1786.095549s 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a39bc6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a340b4d10 0x7f5a34146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.744+0000 7f5a3abc8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a34145980 0x7f5a34146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3a3c7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5a340b17c0 0x7f5a34142e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3abc8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a34145980 0x7f5a34146a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47904/0 (socket says 172.21.15.110:47904) 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3abc8640 1 -- 172.21.15.110:0/3950129987 learned_addr learned my addr 172.21.15.110:0/3950129987 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a39bc6640 10 monclient(hunting): get_auth_request con 0x7f5a340b4d10 auth_method 0 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a39bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a39bc6640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a39bc6640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3abc8640 10 monclient(hunting): get_auth_request con 0x7f5a34145980 auth_method 0 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3abc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3abc8640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3abc8640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3a3c7640 10 monclient(hunting): get_auth_request con 0x7f5a340b17c0 auth_method 0 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3a3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3a3c7640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3a3c7640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a39bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a39bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a39bc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3abc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3abc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.745+0000 7f5a3abc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.746+0000 7f5a3a3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.746+0000 7f5a3a3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:22.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.746+0000 7f5a3a3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 10 monclient(hunting): handle_auth_done global_id 5799 payload 824 2024-01-20T11:22:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 1 -- 172.21.15.110:0/3950129987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a34145980 msgr2=0x7f5a34146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a34145980 0x7f5a34146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 1 -- 172.21.15.110:0/3950129987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5a340b17c0 msgr2=0x7f5a34142e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5a340b17c0 0x7f5a34142e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 10 monclient: _finish_hunting 0 2024-01-20T11:22:22.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 1 monclient: found mon.noname-a 2024-01-20T11:22:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.748+0000 7f5a39bc6640 1 -- 172.21.15.110:0/3950129987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a30009a70 con 0x7f5a340b4d10 2024-01-20T11:22:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.750+0000 7f5a39bc6640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a340b4d10 0x7f5a34146550 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400e850 tx=0x7f5a2400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.750+0000 7f5a2affd640 10 client.?.objecter ms_handle_connect 0x7f5a340b4d10 2024-01-20T11:22:22.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.750+0000 7f5a2affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 1 -- 172.21.15.110:0/3950129987 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a240095b0 con 0x7f5a340b4d10 2024-01-20T11:22:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 10 monclient: dump: 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 10 monclient: _finish_auth 0 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 1 -- 172.21.15.110:0/3950129987 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2400c940 con 0x7f5a340b4d10 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 1 -- 172.21.15.110:0/3950129987 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2400ccd0 con 0x7f5a340b4d10 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a2affd640 10 monclient: dump: 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a38bc4640 4 set_mon_vals no callback set 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a3c62b640 5 monclient: authenticate success, global_id 5799 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a3c62b640 10 monclient: _renew_subs 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a3c62b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a34148780 con 0x7f5a340b4d10 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a3c62b640 10 client.5799.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a3c62b640 10 monclient: _renew_subs 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a3c62b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a341471d0 con 0x7f5a340b4d10 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.754+0000 7f5a3c62b640 1 librados: init done 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.755+0000 7f5a3c62b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.755+0000 7f5a3c62b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.755+0000 7f5a3c62b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:22.755+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a34148450 con 0x7f5a340b4d10 2024-01-20T11:22:22.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~205s 2024-01-20T11:22:23.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.000+0000 7f5a2affd640 1 -- 172.21.15.110:0/3950129987 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5a24013470 con 0x7f5a340b4d10 2024-01-20T11:22:23.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.000+0000 7f5a2affd640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5a1806c890 0x7f5a1806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:23.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.000+0000 7f5a3a3c7640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5a1806c890 0x7f5a1806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:23.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.000+0000 7f5a3a3c7640 10 monclient: get_auth_request con 0x7f5a1806c890 auth_method 0 2024-01-20T11:22:23.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.001+0000 7f5a2affd640 10 client.5799.objecter ms_handle_connect 0x7f5a1806c890 2024-01-20T11:22:23.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.001+0000 7f5a3a3c7640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5a1806c890 0x7f5a1806ed50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5a30036000 tx=0x7f5a30030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:23.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.013+0000 7f5a2affd640 1 -- 172.21.15.110:0/3950129987 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5a2400ce30 con 0x7f5a340b4d10 2024-01-20T11:22:23.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.013+0000 7f5a2affd640 10 client.5799.objecter ms_dispatch 0x7f5a34000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:23.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.013+0000 7f5a2affd640 3 client.5799.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:23.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.013+0000 7f5a2affd640 3 client.5799.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:23.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.013+0000 7f5a2affd640 20 client.5799.objecter dump_active .. 0 homeless 2024-01-20T11:22:23.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.013+0000 7f5a2affd640 1 -- 172.21.15.110:0/3950129987 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5a24054e50 con 0x7f5a340b4d10 2024-01-20T11:22:23.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.013+0000 7f5a2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:23.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.013+0000 7f5a2affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:23.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.182+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a34075490 con 0x7f5a1806c890 2024-01-20T11:22:23.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.186+0000 7f5a2affd640 1 -- 172.21.15.110:0/3950129987 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63984 (secure 0 0 0) 0x7f5a34075490 con 0x7f5a1806c890 2024-01-20T11:22:23.187 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:23.187 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:23.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 10 librados: watch_flush enter 2024-01-20T11:22:23.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 10 librados: watch_flush exit 2024-01-20T11:22:23.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 20 client.5799.objecter shutdown clearing up homeless session... 2024-01-20T11:22:23.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 10 client.5799.objecter successfully canceled tick 2024-01-20T11:22:23.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5a1806c890 msgr2=0x7f5a1806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5a1806c890 0x7f5a1806ed50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5a30036000 tx=0x7f5a30030000 comp rx=0 tx=0).stop 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 10 monclient: shutdown 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a340b4d10 msgr2=0x7f5a34146550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a340b4d10 0x7f5a34146550 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400e850 tx=0x7f5a2400ed10 comp rx=0 tx=0).stop 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 shutdown_connections 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5a1806c890 0x7f5a1806ed50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a34145980 0x7f5a34146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a340b4d10 0x7f5a34146550 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 --2- 172.21.15.110:0/3950129987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5a340b17c0 0x7f5a34142e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 >> 172.21.15.110:0/3950129987 conn(0x7f5a34066590 msgr2=0x7f5a3413b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 shutdown_connections 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 -- 172.21.15.110:0/3950129987 wait complete. 2024-01-20T11:22:23.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:23.187+0000 7f5a3c62b640 1 librados: shutdown 2024-01-20T11:22:23.199 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-01-20T11:22:22.796725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002759"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":24.600000000000001},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.0830000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":22.195},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":22.798999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":19.283999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":20.315000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":27.010999999999999},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":18.577000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":23.946999999999999},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":25.286000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":20.991},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":27.356000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313306,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.0199999999999996},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":5.7069999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.8469999999999995},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.5090000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":12.314},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":10.275}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":14.202999999999999},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":11.531000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":15.744},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":13.558999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14.983000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.111000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":15.608000000000001},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":14.705}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1489999999999991},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.307}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":6.2149999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":9.6980000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":12.025},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.0759999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":7.8140000000000001},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":8.4600000000000009}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":12.65},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":13.725}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444120,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":7.2220000000000004},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":11.699}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":6.577},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":15.31}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.116},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.0739999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.348000000000001},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.381}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":13.739000000000001},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":10.084}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.5129999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.219999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444121,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":7.4320000000000004},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.9809999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":9.8670000000000009},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":7.9109999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":13.08},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":12.395}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":10.769},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":15.257999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":11.449},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":15.853}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":12.034000000000001},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":16.934000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444121,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":7.7549999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":4.6159999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":8.5600000000000005},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.8849999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.721},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.888}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.737},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":6.7619999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.339},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.321999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":9.2100000000000009}]}]},{"osd":6,"up_from":15,"seq":64424509530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":9.8109999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":5.5090000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.5810000000000004},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":13.565}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":12.215999999999999},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":11.638999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.356999999999999},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":6.2290000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":10.493},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.063000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":12.925000000000001},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":8.3439999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:24.242 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:15.266 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:24.247 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:15.271 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:24.262 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:15.286 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:24.292 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:15.316 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:24.294 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:15.318 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:24.297 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:15.303 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:24.299 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:15.305 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:24.310 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:15.334 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:24.313 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:15.319 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:26.202 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.302+0000 7f9f34be2640 10 monclient: get_monmap_and_config 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.302+0000 7f9f34be2640 10 monclient: build_initial_monmap 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.303+0000 7f9f34be2640 10 monclient: monmap: 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:26.304588+0000 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:26.304588+0000 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:26.305 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:26.305 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:26.305 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:26.305 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:26.305 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.303+0000 7f9f34be2640 1 Processor -- start 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.303+0000 7f9f34be2640 1 -- start start 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.303+0000 7f9f34be2640 10 monclient: init 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f3006bb00 0x7f9f300b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3006bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f300af790 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f300b0900 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3006bee0 con 0x7f9f300b0900 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f300ada30 con 0x7f9f300af790 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:26.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f301466b0 con 0x7f9f3006bb00 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.304+0000 7f9f34be2640 10 monclient(hunting): authenticate will time out at 1789.655574s 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.305+0000 7f9f2e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f3006bb00 0x7f9f300b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.305+0000 7f9f27fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9f300af790 0x7f9f30145800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:53226/0 (socket says 172.21.15.110:53226) 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.305+0000 7f9f27fff640 1 -- 172.21.15.110:0/2769309152 learned_addr learned my addr 172.21.15.110:0/2769309152 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.305+0000 7f9f2e59c640 10 monclient(hunting): get_auth_request con 0x7f9f3006bb00 auth_method 0 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.305+0000 7f9f2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.305+0000 7f9f2e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.305+0000 7f9f2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.306+0000 7f9f2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.306+0000 7f9f2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:26.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.306+0000 7f9f2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:26.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.307+0000 7f9f2dd9b640 1 -- 172.21.15.110:0/2769309152 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888329068 0 0) 0x7f9f300ada30 con 0x7f9f300af790 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.307+0000 7f9f2dd9b640 10 monclient(hunting): _negotiate 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.307+0000 7f9f2dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.307+0000 7f9f2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.307+0000 7f9f2dd9b640 10 monclient(hunting): my global_id is 5878 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.307+0000 7f9f2dd9b640 1 -- 172.21.15.110:0/2769309152 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f1c003520 con 0x7f9f300af790 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.308+0000 7f9f2e59c640 10 monclient(hunting): handle_auth_done global_id 5465 payload 274 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.308+0000 7f9f2e59c640 1 -- 172.21.15.110:0/2769309152 >> v1:172.21.15.134:6789/0 conn(0x7f9f300af790 legacy=0x7f9f30145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.308+0000 7f9f2e59c640 1 -- 172.21.15.110:0/2769309152 >> v1:172.21.15.110:6789/0 conn(0x7f9f300b0900 legacy=0x7f9f30145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.308+0000 7f9f2e59c640 10 monclient: _finish_hunting 0 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.308+0000 7f9f2e59c640 1 monclient: found mon.noname-b 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.308+0000 7f9f2e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:26.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.308+0000 7f9f2e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:26.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.308+0000 7f9f2e59c640 1 -- 172.21.15.110:0/2769309152 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300add10 con 0x7f9f3006bb00 2024-01-20T11:22:26.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.310+0000 7f9f2e59c640 1 --2- 172.21.15.110:0/2769309152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f3006bb00 0x7f9f300b03c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9f200097d0 tx=0x7f9f2002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dff6ff90224820d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:26.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 1 -- 172.21.15.110:0/2769309152 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f20031800 con 0x7f9f3006bb00 2024-01-20T11:22:26.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:26.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:26.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 10 monclient: dump: 2024-01-20T11:22:26.315 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:26.315 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 10 monclient: _finish_auth 0 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 10 monclient: _check_auth_tickets 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 1 -- 172.21.15.110:0/2769309152 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f20031960 con 0x7f9f3006bb00 2024-01-20T11:22:26.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f34be2640 5 monclient: authenticate success, global_id 5465 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f34be2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f34be2640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 1 -- 172.21.15.110:0/2769309152 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f20031cf0 con 0x7f9f3006bb00 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f34be2640 4 set_mon_vals no callback set 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.314+0000 7f9f2dd9b640 10 monclient: dump: 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:26.317 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.318 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:26.318 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:26.318 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:26.318 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.315+0000 7f9f34be2640 10 monclient: shutdown 2024-01-20T11:22:26.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.315+0000 7f9f34be2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:26.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.315+0000 7f9f34be2640 1 -- 172.21.15.110:0/2769309152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f3006bb00 msgr2=0x7f9f300b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.315+0000 7f9f34be2640 1 --2- 172.21.15.110:0/2769309152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f3006bb00 0x7f9f300b03c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9f200097d0 tx=0x7f9f2002b8a0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.315+0000 7f9f34be2640 1 -- 172.21.15.110:0/2769309152 shutdown_connections 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.315+0000 7f9f34be2640 1 --2- 172.21.15.110:0/2769309152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f3006bb00 0x7f9f300b03c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.315+0000 7f9f34be2640 1 -- 172.21.15.110:0/2769309152 >> 172.21.15.110:0/2769309152 conn(0x7f9f30066d60 msgr2=0x7f9f300b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.315+0000 7f9f34be2640 1 -- 172.21.15.110:0/2769309152 shutdown_connections 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.315+0000 7f9f34be2640 1 -- 172.21.15.110:0/2769309152 wait complete. 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.316+0000 7f9f34be2640 10 monclient: build_initial_monmap 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.316+0000 7f9f34be2640 10 monclient: monmap: 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:26.322 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.316+0000 7f9f34be2640 1 librados: starting msgr at 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.316+0000 7f9f34be2640 1 librados: starting objecter 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.316+0000 7f9f34be2640 1 Processor -- start 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.316+0000 7f9f34be2640 1 -- start start 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.316+0000 7f9f34be2640 1 librados: setting wanted keys 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.316+0000 7f9f34be2640 1 librados: calling monclient init 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.316+0000 7f9f34be2640 10 monclient: init 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f3006bb00 0x7f9f3011dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3006bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:26.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f300af790 0x7f9f3013f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient(hunting): picked mon.noname-c con 0x7f9f300af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:26.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f300b0900 0x7f9f301d79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient(hunting): picked mon.noname-b con 0x7f9f300b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:26.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:26.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3011f0e0 con 0x7f9f3006bb00 2024-01-20T11:22:26.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:26.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f301d7490 con 0x7f9f300b0900 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3013c850 con 0x7f9f300af790 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f34be2640 10 monclient(hunting): authenticate will time out at 1789.668376s 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f2e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f3006bb00 0x7f9f3011dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f27fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f300af790 0x7f9f3013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f2ed9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f300b0900 0x7f9f301d79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f27fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f300af790 0x7f9f3013f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47918/0 (socket says 172.21.15.110:47918) 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f27fff640 1 -- 172.21.15.110:0/2072169456 learned_addr learned my addr 172.21.15.110:0/2072169456 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f27fff640 10 monclient(hunting): get_auth_request con 0x7f9f300af790 auth_method 0 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f27fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f27fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:26.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f2e59c640 10 monclient(hunting): get_auth_request con 0x7f9f3006bb00 auth_method 0 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f2e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.317+0000 7f9f2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2ed9d640 10 monclient(hunting): get_auth_request con 0x7f9f300b0900 auth_method 0 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:26.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.318+0000 7f9f2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:26.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 10 monclient(hunting): handle_auth_done global_id 5805 payload 824 2024-01-20T11:22:26.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 1 -- 172.21.15.110:0/2072169456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f300af790 msgr2=0x7f9f3013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f300af790 0x7f9f3013f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 1 -- 172.21.15.110:0/2072169456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f300b0900 msgr2=0x7f9f301d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f300b0900 0x7f9f301d79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 10 monclient: _finish_hunting 0 2024-01-20T11:22:26.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 1 monclient: found mon.noname-a 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.321+0000 7f9f2e59c640 1 -- 172.21.15.110:0/2072169456 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f20009400 con 0x7f9f3006bb00 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.323+0000 7f9f2e59c640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f3006bb00 0x7f9f3011dab0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9f20033000 tx=0x7f9f200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.323+0000 7f9f26ffd640 10 client.?.objecter ms_handle_connect 0x7f9f3006bb00 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.323+0000 7f9f26ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 1 -- 172.21.15.110:0/2072169456 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2003bc70 con 0x7f9f3006bb00 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 10 monclient: dump: 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:26.330 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 1 -- 172.21.15.110:0/2072169456 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2003be10 con 0x7f9f3006bb00 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f34be2640 5 monclient: authenticate success, global_id 5805 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f34be2640 10 monclient: _renew_subs 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f34be2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:26.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3013f970 con 0x7f9f3006bb00 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 1 -- 172.21.15.110:0/2072169456 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f20045630 con 0x7f9f3006bb00 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f26ffd640 10 monclient: dump: 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:26.332 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f2d59a640 4 set_mon_vals no callback set 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f34be2640 10 client.5805.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f34be2640 10 monclient: _renew_subs 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f34be2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f301d80d0 con 0x7f9f3006bb00 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.327+0000 7f9f34be2640 1 librados: init done 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.328+0000 7f9f34be2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.328+0000 7f9f34be2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.328+0000 7f9f34be2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:26.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.328+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3011f220 con 0x7f9f3006bb00 2024-01-20T11:22:26.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.564+0000 7f9f26ffd640 1 -- 172.21.15.110:0/2072169456 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9f2004e450 con 0x7f9f3006bb00 2024-01-20T11:22:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.565+0000 7f9f26ffd640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9f1006c8b0 0x7f9f1006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.565+0000 7f9f27fff640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9f1006c8b0 0x7f9f1006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.565+0000 7f9f27fff640 10 monclient: get_auth_request con 0x7f9f1006c8b0 auth_method 0 2024-01-20T11:22:26.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.566+0000 7f9f26ffd640 10 client.5805.objecter ms_handle_connect 0x7f9f1006c8b0 2024-01-20T11:22:26.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.566+0000 7f9f27fff640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9f1006c8b0 0x7f9f1006ed70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9f301426b0 tx=0x7f9f18009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.578+0000 7f9f26ffd640 1 -- 172.21.15.110:0/2072169456 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9f2002e070 con 0x7f9f3006bb00 2024-01-20T11:22:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.578+0000 7f9f26ffd640 10 client.5805.objecter ms_dispatch 0x7f9f30000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.578+0000 7f9f26ffd640 3 client.5805.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.578+0000 7f9f26ffd640 3 client.5805.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.578+0000 7f9f26ffd640 20 client.5805.objecter dump_active .. 0 homeless 2024-01-20T11:22:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.578+0000 7f9f26ffd640 1 -- 172.21.15.110:0/2072169456 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9f20076b60 con 0x7f9f3006bb00 2024-01-20T11:22:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.578+0000 7f9f26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.578+0000 7f9f26ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:26.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.746+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9f300b3390 con 0x7f9f1006c8b0 2024-01-20T11:22:26.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.749+0000 7f9f26ffd640 1 -- 172.21.15.110:0/2072169456 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64032 (secure 0 0 0) 0x7f9f300b3390 con 0x7f9f1006c8b0 2024-01-20T11:22:26.750 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:26.751 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:26.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 10 librados: watch_flush enter 2024-01-20T11:22:26.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 10 librados: watch_flush exit 2024-01-20T11:22:26.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 20 client.5805.objecter shutdown clearing up homeless session... 2024-01-20T11:22:26.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 10 client.5805.objecter successfully canceled tick 2024-01-20T11:22:26.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9f1006c8b0 msgr2=0x7f9f1006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9f1006c8b0 0x7f9f1006ed70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9f301426b0 tx=0x7f9f18009040 comp rx=0 tx=0).stop 2024-01-20T11:22:26.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 10 monclient: shutdown 2024-01-20T11:22:26.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:26.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f3006bb00 msgr2=0x7f9f3011dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f3006bb00 0x7f9f3011dab0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9f20033000 tx=0x7f9f200046c0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 shutdown_connections 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9f1006c8b0 0x7f9f1006ed70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f300b0900 0x7f9f301d79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f300af790 0x7f9f3013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 --2- 172.21.15.110:0/2072169456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f3006bb00 0x7f9f3011dab0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 >> 172.21.15.110:0/2072169456 conn(0x7f9f30066d60 msgr2=0x7f9f30145ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 shutdown_connections 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.751+0000 7f9f34be2640 1 -- 172.21.15.110:0/2072169456 wait complete. 2024-01-20T11:22:26.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.752+0000 7f9f34be2640 1 librados: shutdown 2024-01-20T11:22:26.763 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-01-20T11:22:24.797211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":14.331},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.9690000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":6.0750000000000002},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":9.0999999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.44},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":9.7639999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.178000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":7.7480000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":11.835000000000001},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":16.007000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":10.927},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":13.052}]}]},{"osd":7,"up_from":21,"seq":90194313306,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.0199999999999996},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":5.7069999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":8.8469999999999995},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.5090000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":12.314},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":10.275}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":14.202999999999999},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":11.531000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":15.744},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":13.558999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":14.983000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.111000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":15.608000000000001},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":14.705}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1489999999999991},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.307}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":6.2149999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":9.6980000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":12.025},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.0759999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":7.8140000000000001},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":8.4600000000000009}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":12.65},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":13.725}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":20.539999999999999},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":15.042999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":16.329999999999998},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.334}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":19.061},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.8829999999999991}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":18.103999999999999},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.303000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":19.635999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":9.5269999999999992}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.5129999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.219999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444122,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.6769999999999996},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":4.8419999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":11.897},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":8.2599999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":14.247999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":9.032}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":12.960000000000001},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.327999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.606999999999999},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":9.7159999999999993}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.41},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":10.316000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444122,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":7.7549999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":4.6159999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":8.5600000000000005},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.8849999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.721},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.888}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.737},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":6.7619999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.339},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.321999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":9.2100000000000009}]}]},{"osd":6,"up_from":15,"seq":64424509530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":9.8109999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":5.5090000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.5810000000000004},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":13.565}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":12.215999999999999},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":11.638999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.356999999999999},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":6.2290000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":10.493},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.063000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":12.925000000000001},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":8.3439999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:26.765 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:26.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.865+0000 7f959f2b4640 10 monclient: get_monmap_and_config 2024-01-20T11:22:26.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.865+0000 7f959f2b4640 10 monclient: build_initial_monmap 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.865+0000 7f959f2b4640 10 monclient: monmap: 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:26.867462+0000 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:26.867462+0000 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:26.867 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.866+0000 7f959f2b4640 1 Processor -- start 2024-01-20T11:22:26.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.866+0000 7f959f2b4640 1 -- start start 2024-01-20T11:22:26.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.866+0000 7f959f2b4640 10 monclient: init 2024-01-20T11:22:26.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:26.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-01-20T11:22:26.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95980a96f0 0x7f959813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 10 monclient(hunting): picked mon.noname-b con 0x7f95980a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f959813fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f95980a9ad0 0x7f9598142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 10 monclient(hunting): picked mon.noname-a con 0x7f95980a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95980b25c0 con 0x7f959813fa10 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95980aaea0 con 0x7f95980a9ad0 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95980ab0a0 con 0x7f95980a96f0 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959f2b4640 10 monclient(hunting): authenticate will time out at 1790.218487s 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.867+0000 7f959d050640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95980a96f0 0x7f959813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.868+0000 7f959d050640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95980a96f0 0x7f959813f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47932/0 (socket says 172.21.15.110:47932) 2024-01-20T11:22:26.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.868+0000 7f959d050640 1 -- 172.21.15.110:0/3064094754 learned_addr learned my addr 172.21.15.110:0/3064094754 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:26.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.868+0000 7f959d050640 10 monclient(hunting): get_auth_request con 0x7f95980a96f0 auth_method 0 2024-01-20T11:22:26.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.868+0000 7f959d050640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:26.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.868+0000 7f959d050640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:26.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.868+0000 7f959d050640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:26.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.868+0000 7f959d050640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:26.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.868+0000 7f959d050640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:26.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.868+0000 7f959d050640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d851640 1 --2- 172.21.15.110:0/3064094754 >> v2:172.21.15.110:3300/0 conn(0x7f95980a9ad0 0x7f9598142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d050640 10 monclient(hunting): handle_auth_done global_id 5471 payload 274 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d050640 1 -- 172.21.15.110:0/3064094754 >> v2:172.21.15.110:3300/0 conn(0x7f95980a9ad0 msgr2=0x7f9598142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d050640 1 --2- 172.21.15.110:0/3064094754 >> v2:172.21.15.110:3300/0 conn(0x7f95980a9ad0 0x7f9598142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d050640 1 -- 172.21.15.110:0/3064094754 >> v1:172.21.15.110:6789/0 conn(0x7f959813fa10 legacy=0x7f95981423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d050640 10 monclient: _finish_hunting 0 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d050640 1 monclient: found mon.noname-b 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d050640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d050640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:26.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.870+0000 7f959d050640 1 -- 172.21.15.110:0/3064094754 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95980b3fb0 con 0x7f95980a96f0 2024-01-20T11:22:26.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.873+0000 7f959d050640 1 --2- 172.21.15.110:0/3064094754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95980a96f0 0x7f959813f4d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9588009870 tx=0x7f958802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bddde1a4ec9d78a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:26.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 1 -- 172.21.15.110:0/3064094754 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95880317c0 con 0x7f95980a96f0 2024-01-20T11:22:26.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:26.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 10 monclient: dump: 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 10 monclient: _finish_auth 0 2024-01-20T11:22:26.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 1 -- 172.21.15.110:0/3064094754 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9588031920 con 0x7f95980a96f0 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 1 -- 172.21.15.110:0/3064094754 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9588031cb0 con 0x7f95980a96f0 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f959f2b4640 5 monclient: authenticate success, global_id 5471 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f959f2b4640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f959f2b4640 4 set_mon_vals no callback set 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.878+0000 7f958ffff640 10 monclient: dump: 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:26.881 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.882 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.882 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.882 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:26.882 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.882 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:26.882 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:26.882 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:26.882 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 10 monclient: shutdown 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 1 -- 172.21.15.110:0/3064094754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95980a96f0 msgr2=0x7f959813f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 1 --2- 172.21.15.110:0/3064094754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95980a96f0 0x7f959813f4d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9588009870 tx=0x7f958802b860 comp rx=0 tx=0).stop 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 1 -- 172.21.15.110:0/3064094754 shutdown_connections 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 1 --2- 172.21.15.110:0/3064094754 >> v2:172.21.15.110:3300/0 conn(0x7f95980a9ad0 0x7f9598142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 1 --2- 172.21.15.110:0/3064094754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95980a96f0 0x7f959813f4d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 1 -- 172.21.15.110:0/3064094754 >> 172.21.15.110:0/3064094754 conn(0x7f95980b7bc0 msgr2=0x7f95980b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 1 -- 172.21.15.110:0/3064094754 shutdown_connections 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.879+0000 7f959f2b4640 1 -- 172.21.15.110:0/3064094754 wait complete. 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.880+0000 7f959f2b4640 10 monclient: build_initial_monmap 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.880+0000 7f959f2b4640 10 monclient: monmap: 2024-01-20T11:22:26.886 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.880+0000 7f959f2b4640 1 librados: starting msgr at 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.880+0000 7f959f2b4640 1 librados: starting objecter 2024-01-20T11:22:26.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.880+0000 7f959f2b4640 1 Processor -- start 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 1 -- start start 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 1 librados: setting wanted keys 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 1 librados: calling monclient init 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient: init 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95980a96f0 0x7f959813b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient(hunting): picked mon.noname-a con 0x7f95980a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95980a9ad0 0x7f95981dad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient(hunting): picked mon.noname-b con 0x7f95980a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f959813b660 0x7f95981db2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient(hunting): picked mon.noname-c con 0x7f959813b660 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:26.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95980abd20 con 0x7f95980a96f0 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95981da840 con 0x7f95980a9ad0 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9598137ff0 con 0x7f959813b660 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.881+0000 7f959f2b4640 10 monclient(hunting): authenticate will time out at 1790.232903s 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959d851640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f959813b660 0x7f95981db2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959c84f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95980a9ad0 0x7f95981dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959d851640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f959813b660 0x7f95981db2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47940/0 (socket says 172.21.15.110:47940) 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959d851640 1 -- 172.21.15.110:0/3265512036 learned_addr learned my addr 172.21.15.110:0/3265512036 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959d851640 10 monclient(hunting): get_auth_request con 0x7f959813b660 auth_method 0 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959d851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:26.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959d851640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959d851640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959c84f640 10 monclient(hunting): get_auth_request con 0x7f95980a9ad0 auth_method 0 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959c84f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959c84f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.882+0000 7f959c84f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.883+0000 7f959d050640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95980a96f0 0x7f959813b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.883+0000 7f959d851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.883+0000 7f959d851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.883+0000 7f959d851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.883+0000 7f959c84f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.883+0000 7f959c84f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:26.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.883+0000 7f959c84f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:26.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 10 monclient(hunting): handle_auth_done global_id 5474 payload 824 2024-01-20T11:22:26.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 1 -- 172.21.15.110:0/3265512036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95980a9ad0 msgr2=0x7f95981dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95980a9ad0 0x7f95981dad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 1 -- 172.21.15.110:0/3265512036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95980a96f0 msgr2=0x7f959813b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:26.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95980a96f0 0x7f959813b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:26.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 10 monclient: _finish_hunting 0 2024-01-20T11:22:26.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 1 monclient: found mon.noname-c 2024-01-20T11:22:26.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:26.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:26.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.886+0000 7f959d851640 1 -- 172.21.15.110:0/3265512036 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95880094d0 con 0x7f959813b660 2024-01-20T11:22:26.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.889+0000 7f959d851640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f959813b660 0x7f95981db2a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f959000cd70 tx=0x7f959000a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:26.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.889+0000 7f958dffb640 10 client.?.objecter ms_handle_connect 0x7f959813b660 2024-01-20T11:22:26.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.889+0000 7f958dffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 1 -- 172.21.15.110:0/3265512036 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590003b90 con 0x7f959813b660 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 10 monclient: dump: 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.894 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 10 monclient: _finish_auth 0 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 10 monclient: _check_auth_tickets 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 1 -- 172.21.15.110:0/3265512036 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9590003cf0 con 0x7f959813b660 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f959f2b4640 5 monclient: authenticate success, global_id 5474 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f959f2b4640 10 monclient: _renew_subs 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f959f2b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f959813a4a0 con 0x7f959813b660 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 1 -- 172.21.15.110:0/3265512036 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590004240 con 0x7f959813b660 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:26.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958dffb640 10 monclient: dump: 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.892+0000 7f958effd640 4 set_mon_vals no callback set 2024-01-20T11:22:26.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.893+0000 7f959f2b4640 10 client.5474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:26.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.893+0000 7f959f2b4640 10 monclient: _renew_subs 2024-01-20T11:22:26.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.893+0000 7f959f2b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:26.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.893+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9598133df0 con 0x7f959813b660 2024-01-20T11:22:26.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.893+0000 7f959f2b4640 1 librados: init done 2024-01-20T11:22:26.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.894+0000 7f959f2b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:26.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.894+0000 7f959f2b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:26.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.894+0000 7f959f2b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:26.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.894+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95980aa800 con 0x7f959813b660 2024-01-20T11:22:26.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.913+0000 7f958dffb640 1 -- 172.21.15.110:0/3265512036 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f95900043a0 con 0x7f959813b660 2024-01-20T11:22:26.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.913+0000 7f958dffb640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f957c06c8b0 0x7f957c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:26.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.914+0000 7f959d050640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f957c06c8b0 0x7f957c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:26.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.914+0000 7f959d050640 10 monclient: get_auth_request con 0x7f957c06c8b0 auth_method 0 2024-01-20T11:22:26.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.914+0000 7f959d050640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f957c06c8b0 0x7f957c06ed70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9588036000 tx=0x7f958803d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:26.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:26.914+0000 7f958dffb640 10 client.5474.objecter ms_handle_connect 0x7f957c06c8b0 2024-01-20T11:22:27.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.149+0000 7f958dffb640 1 -- 172.21.15.110:0/3265512036 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9590012070 con 0x7f959813b660 2024-01-20T11:22:27.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.149+0000 7f958dffb640 10 client.5474.objecter ms_dispatch 0x7f9598000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:27.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.149+0000 7f958dffb640 3 client.5474.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:27.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.149+0000 7f958dffb640 3 client.5474.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:27.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.149+0000 7f958dffb640 20 client.5474.objecter dump_active .. 0 homeless 2024-01-20T11:22:27.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.149+0000 7f958dffb640 1 -- 172.21.15.110:0/3265512036 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9590055270 con 0x7f959813b660 2024-01-20T11:22:27.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.149+0000 7f958dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:27.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.149+0000 7f958dffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:27.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.329+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f959813a7d0 con 0x7f957c06c8b0 2024-01-20T11:22:27.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.332+0000 7f958dffb640 1 -- 172.21.15.110:0/3265512036 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64019 (secure 0 0 0) 0x7f959813a7d0 con 0x7f957c06c8b0 2024-01-20T11:22:27.333 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:27.334 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 10 librados: watch_flush enter 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 10 librados: watch_flush exit 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 20 client.5474.objecter shutdown clearing up homeless session... 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 10 client.5474.objecter successfully canceled tick 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f957c06c8b0 msgr2=0x7f957c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f957c06c8b0 0x7f957c06ed70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9588036000 tx=0x7f958803d040 comp rx=0 tx=0).stop 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 10 monclient: shutdown 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f959813b660 msgr2=0x7f95981db2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f959813b660 0x7f95981db2a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f959000cd70 tx=0x7f959000a660 comp rx=0 tx=0).stop 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 shutdown_connections 2024-01-20T11:22:27.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f957c06c8b0 0x7f957c06ed70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f959813b660 0x7f95981db2a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95980a9ad0 0x7f95981dad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 --2- 172.21.15.110:0/3265512036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95980a96f0 0x7f959813b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 >> 172.21.15.110:0/3265512036 conn(0x7f95980b7bc0 msgr2=0x7f95980b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:27.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 shutdown_connections 2024-01-20T11:22:27.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 -- 172.21.15.110:0/3265512036 wait complete. 2024-01-20T11:22:27.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.334+0000 7f959f2b4640 1 librados: shutdown 2024-01-20T11:22:27.345 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-01-20T11:22:26.797643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002722"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":14.331},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.9690000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":6.0750000000000002},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":9.0999999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.44},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":9.7639999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.178000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":7.7480000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":11.835000000000001},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":16.007000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":10.927},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":13.052}]}]},{"osd":7,"up_from":21,"seq":90194313307,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.9870000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":10.77}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":16.486000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":8.7330000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":9.5090000000000003},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":11.441000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":7.1189999999999998},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":14.724}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":10.138999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.077999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":12.781000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":15.523}]}]},{"osd":3,"up_from":21,"seq":90194313307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":15.608000000000001},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":14.705}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1489999999999991},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.307}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":6.2149999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":9.6980000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":12.025},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.0759999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":7.8140000000000001},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":8.4600000000000009}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":12.65},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":13.725}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":20.539999999999999},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":15.042999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":16.329999999999998},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.334}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":19.061},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.8829999999999991}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":18.103999999999999},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.303000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":19.635999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":9.5269999999999992}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.5129999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.219999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444122,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.6769999999999996},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":4.8419999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":11.897},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":8.2599999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":14.247999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":9.032}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":12.960000000000001},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.327999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.606999999999999},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":9.7159999999999993}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.41},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":10.316000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444122,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":7.7549999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":4.6159999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":8.5600000000000005},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.8849999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.721},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.888}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.737},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":6.7619999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.339},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.321999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":9.2100000000000009}]}]},{"osd":6,"up_from":15,"seq":64424509531,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.0289999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":13.965999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.175000000000001},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":10.401999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":15.935},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":8.5730000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.7899999999999991},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":14.943}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":17.617000000000001},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.536}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.257999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":7.9189999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:27.347 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:27.347 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:27.347 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:27.347 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:27.347 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:27.347 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:27.347 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:27.347 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:27.348 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:27.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.445+0000 7f7801446640 10 monclient: get_monmap_and_config 2024-01-20T11:22:27.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.445+0000 7f7801446640 10 monclient: build_initial_monmap 2024-01-20T11:22:27.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.445+0000 7f7801446640 10 monclient: monmap: 2024-01-20T11:22:27.446 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:27.446 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:27.446 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:27.446802+0000 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:27.446802+0000 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.445+0000 7f7801446640 1 Processor -- start 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.445+0000 7f7801446640 1 -- start start 2024-01-20T11:22:27.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.445+0000 7f7801446640 10 monclient: init 2024-01-20T11:22:27.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:27.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-01-20T11:22:27.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a96f0 0x7f77fc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:27.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 10 monclient(hunting): picked mon.noname-b con 0x7f77fc0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77fc13fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f77fc0a9ad0 0x7f77fc142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 10 monclient(hunting): picked mon.noname-a con 0x7f77fc0a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc0b25c0 con 0x7f77fc13fa10 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77fc0aaea0 con 0x7f77fc0a9ad0 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc0ab0a0 con 0x7f77fc0a96f0 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f7801446640 10 monclient(hunting): authenticate will time out at 1790.797572s 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f77fb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f77fc0a9ad0 0x7f77fc142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.446+0000 7f77faffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a96f0 0x7f77fc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77fb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f77fc0a9ad0 0x7f77fc142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52884/0 (socket says 172.21.15.110:52884) 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77fb7fe640 1 -- 172.21.15.110:0/3659289372 learned_addr learned my addr 172.21.15.110:0/3659289372 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:27.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77fb7fe640 10 monclient(hunting): get_auth_request con 0x7f77fc0a9ad0 auth_method 0 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77fb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77faffd640 10 monclient(hunting): get_auth_request con 0x7f77fc0a96f0 auth_method 0 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77faffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77faffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.447+0000 7f77fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.448+0000 7f77faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.448+0000 7f77faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:27.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.448+0000 7f77faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:27.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.450+0000 7f77fb7fe640 10 monclient(hunting): handle_auth_done global_id 5808 payload 274 2024-01-20T11:22:27.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.450+0000 7f77fb7fe640 1 -- 172.21.15.110:0/3659289372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a96f0 msgr2=0x7f77fc13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:27.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.450+0000 7f77fb7fe640 1 --2- 172.21.15.110:0/3659289372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a96f0 0x7f77fc13f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.450+0000 7f77fb7fe640 1 -- 172.21.15.110:0/3659289372 >> v1:172.21.15.134:6789/0 conn(0x7f77fc13fa10 legacy=0x7f77fc1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:27.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.450+0000 7f77fb7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:27.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.450+0000 7f77fb7fe640 1 monclient: found mon.noname-a 2024-01-20T11:22:27.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.450+0000 7f77fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:27.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.450+0000 7f77fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:27.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.450+0000 7f77fb7fe640 1 -- 172.21.15.110:0/3659289372 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc0b3fb0 con 0x7f77fc0a9ad0 2024-01-20T11:22:27.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.452+0000 7f77fb7fe640 1 --2- 172.21.15.110:0/3659289372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a9ad0 0x7f77fc142af0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009e50 tx=0x7f77ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fadd2e95b2ca0cd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:27.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 1 -- 172.21.15.110:0/3659289372 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec02ca70 con 0x7f77fc0a9ad0 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 10 monclient: dump: 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:27.458 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 10 monclient: _finish_auth 0 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 1 -- 172.21.15.110:0/3659289372 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ec02cbd0 con 0x7f77fc0a9ad0 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f77f9ffb640 1 -- 172.21.15.110:0/3659289372 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0343f0 con 0x7f77fc0a9ad0 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f7801446640 5 monclient: authenticate success, global_id 5808 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.456+0000 7f7801446640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 4 set_mon_vals no callback set 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f77f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:27.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f77f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f77f9ffb640 10 monclient: dump: 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:27.460 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:27.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 10 monclient: shutdown 2024-01-20T11:22:27.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 1 -- 172.21.15.110:0/3659289372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a9ad0 msgr2=0x7f77fc142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 1 --2- 172.21.15.110:0/3659289372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a9ad0 0x7f77fc142af0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009e50 tx=0x7f77ec02bed0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 1 -- 172.21.15.110:0/3659289372 shutdown_connections 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 1 --2- 172.21.15.110:0/3659289372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a9ad0 0x7f77fc142af0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 1 --2- 172.21.15.110:0/3659289372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a96f0 0x7f77fc13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 1 -- 172.21.15.110:0/3659289372 >> 172.21.15.110:0/3659289372 conn(0x7f77fc0b7bc0 msgr2=0x7f77fc0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 1 -- 172.21.15.110:0/3659289372 shutdown_connections 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.457+0000 7f7801446640 1 -- 172.21.15.110:0/3659289372 wait complete. 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.458+0000 7f7801446640 10 monclient: build_initial_monmap 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.458+0000 7f7801446640 10 monclient: monmap: 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:27.464 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.458+0000 7f7801446640 1 librados: starting msgr at 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.458+0000 7f7801446640 1 librados: starting objecter 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 Processor -- start 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 -- start start 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 librados: setting wanted keys 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 librados: calling monclient init 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient: init 2024-01-20T11:22:27.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a96f0 0x7f77fc134dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient(hunting): picked mon.noname-a con 0x7f77fc0a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a9ad0 0x7f77fc1374f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient(hunting): picked mon.noname-c con 0x7f77fc0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77fc1dacf0 0x7f77fc1db180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient(hunting): picked mon.noname-b con 0x7f77fc1dacf0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc135400 con 0x7f77fc0a96f0 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1366c0 con 0x7f77fc1dacf0 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1369a0 con 0x7f77fc0a9ad0 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:27.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.459+0000 7f7801446640 10 monclient(hunting): authenticate will time out at 1790.811074s 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77faffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a96f0 0x7f77fc134dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77fa7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a9ad0 0x7f77fc1374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77faffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a96f0 0x7f77fc134dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52900/0 (socket says 172.21.15.110:52900) 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77faffd640 1 -- 172.21.15.110:0/3582227857 learned_addr learned my addr 172.21.15.110:0/3582227857 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77faffd640 10 monclient(hunting): get_auth_request con 0x7f77fc0a96f0 auth_method 0 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77faffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77faffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77fb7fe640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77fc1dacf0 0x7f77fc1db180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77fa7fc640 10 monclient(hunting): get_auth_request con 0x7f77fc0a9ad0 auth_method 0 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77fa7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.460+0000 7f77fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fb7fe640 10 monclient(hunting): get_auth_request con 0x7f77fc1dacf0 auth_method 0 2024-01-20T11:22:27.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:27.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.461+0000 7f77fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:27.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.463+0000 7f77faffd640 10 monclient(hunting): handle_auth_done global_id 5811 payload 824 2024-01-20T11:22:27.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.464+0000 7f77faffd640 1 -- 172.21.15.110:0/3582227857 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a9ad0 msgr2=0x7f77fc1374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:27.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.464+0000 7f77faffd640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a9ad0 0x7f77fc1374f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.464+0000 7f77faffd640 1 -- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77fc1dacf0 msgr2=0x7f77fc1db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:27.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.464+0000 7f77faffd640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77fc1dacf0 0x7f77fc1db180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.464+0000 7f77faffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:27.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.464+0000 7f77faffd640 1 monclient: found mon.noname-a 2024-01-20T11:22:27.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.464+0000 7f77faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:27.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.464+0000 7f77faffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:27.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.464+0000 7f77faffd640 1 -- 172.21.15.110:0/3582227857 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec009b30 con 0x7f77fc0a96f0 2024-01-20T11:22:27.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.466+0000 7f77faffd640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a96f0 0x7f77fc134dd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f77f400e9b0 tx=0x7f77f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:27.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.466+0000 7f77e37fe640 10 client.?.objecter ms_handle_connect 0x7f77fc0a96f0 2024-01-20T11:22:27.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.466+0000 7f77e37fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 1 -- 172.21.15.110:0/3582227857 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4009670 con 0x7f77fc0a96f0 2024-01-20T11:22:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 10 monclient: dump: 2024-01-20T11:22:27.471 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 1 -- 172.21.15.110:0/3582227857 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f400c9a0 con 0x7f77fc0a96f0 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f7801446640 5 monclient: authenticate success, global_id 5811 2024-01-20T11:22:27.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f7801446640 10 monclient: _renew_subs 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f7801446640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77fc136c80 con 0x7f77fc0a96f0 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 1 -- 172.21.15.110:0/3582227857 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f400cd50 con 0x7f77fc0a96f0 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77e37fe640 10 monclient: dump: 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:27.473 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f77f97fa640 4 set_mon_vals no callback set 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f7801446640 10 client.5811.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f7801446640 10 monclient: _renew_subs 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f7801446640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77fc1db6c0 con 0x7f77fc0a96f0 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.469+0000 7f7801446640 1 librados: init done 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.470+0000 7f7801446640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.470+0000 7f7801446640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.470+0000 7f7801446640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:27.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.470+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77fc075490 con 0x7f77fc0a96f0 2024-01-20T11:22:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.489+0000 7f77e37fe640 1 -- 172.21.15.110:0/3582227857 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f77f4013470 con 0x7f77fc0a96f0 2024-01-20T11:22:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.489+0000 7f77e37fe640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f77e806c8b0 0x7f77e806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:27.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.490+0000 7f77fa7fc640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f77e806c8b0 0x7f77e806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.490+0000 7f77fa7fc640 10 monclient: get_auth_request con 0x7f77e806c8b0 auth_method 0 2024-01-20T11:22:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.490+0000 7f77fa7fc640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f77e806c8b0 0x7f77e806ed70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f77fc1337b0 tx=0x7f77f0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:27.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.490+0000 7f77e37fe640 10 client.5811.objecter ms_handle_connect 0x7f77e806c8b0 2024-01-20T11:22:27.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.491+0000 7f77e37fe640 1 -- 172.21.15.110:0/3582227857 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f77f4058bd0 con 0x7f77fc0a96f0 2024-01-20T11:22:27.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.491+0000 7f77e37fe640 10 client.5811.objecter ms_dispatch 0x7f77fc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:27.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.491+0000 7f77e37fe640 3 client.5811.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:27.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.491+0000 7f77e37fe640 3 client.5811.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:27.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.491+0000 7f77e37fe640 20 client.5811.objecter dump_active .. 0 homeless 2024-01-20T11:22:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.723+0000 7f77e37fe640 1 -- 172.21.15.110:0/3582227857 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f77f4054f10 con 0x7f77fc0a96f0 2024-01-20T11:22:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.723+0000 7f77e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.723+0000 7f77e37fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:27.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.971+0000 7f7801446640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:27.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.971+0000 7f7801446640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:27.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.971+0000 7f7801446640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:27.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.971+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f77fc0b5f20 con 0x7f77fc0a96f0 2024-01-20T11:22:27.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.977+0000 7f77e37fe640 1 -- 172.21.15.110:0/3582227857 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f77f4021520 con 0x7f77fc0a96f0 2024-01-20T11:22:27.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.977+0000 7f77e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:27.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.977+0000 7f77e37fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:27.978 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:27.978 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":482,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:27.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 10 librados: watch_flush enter 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 10 librados: watch_flush exit 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 20 client.5811.objecter shutdown clearing up homeless session... 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 10 client.5811.objecter successfully canceled tick 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f77e806c8b0 msgr2=0x7f77e806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f77e806c8b0 0x7f77e806ed70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f77fc1337b0 tx=0x7f77f0009040 comp rx=0 tx=0).stop 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 10 monclient: shutdown 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a96f0 msgr2=0x7f77fc134dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a96f0 0x7f77fc134dd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f77f400e9b0 tx=0x7f77f400ee70 comp rx=0 tx=0).stop 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 shutdown_connections 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f77e806c8b0 0x7f77e806ed70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77fc1dacf0 0x7f77fc1db180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77fc0a9ad0 0x7f77fc1374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 --2- 172.21.15.110:0/3582227857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77fc0a96f0 0x7f77fc134dd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 >> 172.21.15.110:0/3582227857 conn(0x7f77fc0b7bc0 msgr2=0x7f77fc0b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:27.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 shutdown_connections 2024-01-20T11:22:27.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 -- 172.21.15.110:0/3582227857 wait complete. 2024-01-20T11:22:27.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:27.979+0000 7f7801446640 1 librados: shutdown 2024-01-20T11:22:27.991 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:27.991 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:28.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.091+0000 7f59cd7e2640 10 monclient: get_monmap_and_config 2024-01-20T11:22:28.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.091+0000 7f59cd7e2640 10 monclient: build_initial_monmap 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.091+0000 7f59cd7e2640 10 monclient: monmap: 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:28.093351+0000 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:28.093351+0000 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:28.094 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.092+0000 7f59cd7e2640 1 Processor -- start 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.092+0000 7f59cd7e2640 1 -- start start 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.092+0000 7f59cd7e2640 10 monclient: init 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80b4d10 0x7f59c80b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f59c80b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f59c80b17c0 0x7f59c80b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 10 monclient(hunting): picked mon.noname-c con 0x7f59c80b17c0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59c8145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80b2120 con 0x7f59c8145980 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c80b50f0 con 0x7f59c80b17c0 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80aade0 con 0x7f59c80b4d10 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59cd7e2640 10 monclient(hunting): authenticate will time out at 1791.444369s 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59c77fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f59c8145980 0x7f59c8145e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53196/0 (socket says 172.21.15.110:53196) 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59c77fe640 1 -- 172.21.15.110:0/3330473185 learned_addr learned my addr 172.21.15.110:0/3330473185 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59c6ffd640 1 --2- 172.21.15.110:0/3330473185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80b4d10 0x7f59c80b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.093+0000 7f59bffff640 1 --2- 172.21.15.110:0/3330473185 >> v2:172.21.15.134:3300/0 conn(0x7f59c80b17c0 0x7f59c80b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59c6ffd640 10 monclient(hunting): get_auth_request con 0x7f59c80b4d10 auth_method 0 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59c6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59bffff640 10 monclient(hunting): get_auth_request con 0x7f59c80b17c0 auth_method 0 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59bffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59bffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.094+0000 7f59bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c67fc640 1 -- 172.21.15.110:0/3330473185 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964677945 0 0) 0x7f59c80b2120 con 0x7f59c8145980 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c67fc640 10 monclient(hunting): _negotiate 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c67fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c67fc640 10 monclient(hunting): my global_id is 5814 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c67fc640 1 -- 172.21.15.110:0/3330473185 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b4003520 con 0x7f59c8145980 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c6ffd640 10 monclient(hunting): handle_auth_done global_id 5483 payload 274 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c6ffd640 1 -- 172.21.15.110:0/3330473185 >> v2:172.21.15.134:3300/0 conn(0x7f59c80b17c0 msgr2=0x7f59c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c6ffd640 1 --2- 172.21.15.110:0/3330473185 >> v2:172.21.15.134:3300/0 conn(0x7f59c80b17c0 0x7f59c80b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c6ffd640 1 -- 172.21.15.110:0/3330473185 >> v1:172.21.15.110:6789/0 conn(0x7f59c8145980 legacy=0x7f59c8145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c6ffd640 1 monclient: found mon.noname-b 2024-01-20T11:22:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.096+0000 7f59c6ffd640 1 -- 172.21.15.110:0/3330473185 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c8145460 con 0x7f59c80b4d10 2024-01-20T11:22:28.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.098+0000 7f59c6ffd640 1 --2- 172.21.15.110:0/3330473185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80b4d10 0x7f59c80b3da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f59b80097a0 tx=0x7f59b802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0f9de8d852086b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 1 -- 172.21.15.110:0/3330473185 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b802e070 con 0x7f59c80b4d10 2024-01-20T11:22:28.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 10 monclient: dump: 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:28.103 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 10 monclient: _finish_auth 0 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 1 -- 172.21.15.110:0/3330473185 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b80315a0 con 0x7f59c80b4d10 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59cd7e2640 5 monclient: authenticate success, global_id 5483 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59cd7e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59cd7e2640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 1 -- 172.21.15.110:0/3330473185 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8031930 con 0x7f59c80b4d10 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59cd7e2640 4 set_mon_vals no callback set 2024-01-20T11:22:28.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.101+0000 7f59c67fc640 10 monclient: dump: 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:28.105 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 10 monclient: shutdown 2024-01-20T11:22:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:28.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 1 -- 172.21.15.110:0/3330473185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80b4d10 msgr2=0x7f59c80b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 1 --2- 172.21.15.110:0/3330473185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80b4d10 0x7f59c80b3da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f59b80097a0 tx=0x7f59b802b840 comp rx=0 tx=0).stop 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 1 -- 172.21.15.110:0/3330473185 shutdown_connections 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 1 --2- 172.21.15.110:0/3330473185 >> v2:172.21.15.134:3300/0 conn(0x7f59c80b17c0 0x7f59c80b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 1 --2- 172.21.15.110:0/3330473185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80b4d10 0x7f59c80b3da0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 1 -- 172.21.15.110:0/3330473185 >> 172.21.15.110:0/3330473185 conn(0x7f59c8066590 msgr2=0x7f59c80669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 1 -- 172.21.15.110:0/3330473185 shutdown_connections 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.102+0000 7f59cd7e2640 1 -- 172.21.15.110:0/3330473185 wait complete. 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.103+0000 7f59cd7e2640 10 monclient: build_initial_monmap 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.103+0000 7f59cd7e2640 10 monclient: monmap: 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:28.110 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.103+0000 7f59cd7e2640 1 librados: starting msgr at 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.103+0000 7f59cd7e2640 1 librados: starting objecter 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.103+0000 7f59cd7e2640 1 Processor -- start 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.103+0000 7f59cd7e2640 1 -- start start 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.103+0000 7f59cd7e2640 1 librados: setting wanted keys 2024-01-20T11:22:28.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.103+0000 7f59cd7e2640 1 librados: calling monclient init 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.103+0000 7f59cd7e2640 10 monclient: init 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59c80b17c0 0x7f59c813ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f59c80b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80b4d10 0x7f59c813c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient(hunting): picked mon.noname-a con 0x7f59c80b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c8145980 0x7f59c813c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient(hunting): picked mon.noname-c con 0x7f59c8145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c8137b50 con 0x7f59c80b4d10 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:28.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c813cac0 con 0x7f59c80b17c0 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80abc30 con 0x7f59c8145980 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59cd7e2640 10 monclient(hunting): authenticate will time out at 1791.455425s 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59bffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80b4d10 0x7f59c813c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59c77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c8145980 0x7f59c813c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.104+0000 7f59c6ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59c80b17c0 0x7f59c813ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59bffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80b4d10 0x7f59c813c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52906/0 (socket says 172.21.15.110:52906) 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59bffff640 1 -- 172.21.15.110:0/983097149 learned_addr learned my addr 172.21.15.110:0/983097149 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59bffff640 10 monclient(hunting): get_auth_request con 0x7f59c80b4d10 auth_method 0 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59bffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:28.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59bffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c77fe640 10 monclient(hunting): get_auth_request con 0x7f59c8145980 auth_method 0 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c6ffd640 10 monclient(hunting): get_auth_request con 0x7f59c80b17c0 auth_method 0 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:28.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.105+0000 7f59c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 10 monclient(hunting): handle_auth_done global_id 5896 payload 824 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 1 -- 172.21.15.110:0/983097149 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c8145980 msgr2=0x7f59c813c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c8145980 0x7f59c813c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 1 -- 172.21.15.110:0/983097149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80b4d10 msgr2=0x7f59c813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80b4d10 0x7f59c813c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 1 monclient: found mon.noname-b 2024-01-20T11:22:28.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:28.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.108+0000 7f59c6ffd640 1 -- 172.21.15.110:0/983097149 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8009400 con 0x7f59c80b17c0 2024-01-20T11:22:28.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.111+0000 7f59c6ffd640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59c80b17c0 0x7f59c813ebd0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f59b8033000 tx=0x7f59b80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:28.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.111+0000 7f59bf7fe640 10 client.?.objecter ms_handle_connect 0x7f59c80b17c0 2024-01-20T11:22:28.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.111+0000 7f59bf7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:28.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 1 -- 172.21.15.110:0/983097149 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8032840 con 0x7f59c80b17c0 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 10 monclient: dump: 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:28.118 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 1 -- 172.21.15.110:0/983097149 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b80329c0 con 0x7f59c80b17c0 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59cd7e2640 5 monclient: authenticate success, global_id 5896 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59cd7e2640 10 monclient: _renew_subs 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59cd7e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c80ac860 con 0x7f59c80b17c0 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 1 -- 172.21.15.110:0/983097149 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8032d70 con 0x7f59c80b17c0 2024-01-20T11:22:28.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59bf7fe640 10 monclient: dump: 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:28.120 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59c5ffb640 4 set_mon_vals no callback set 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59cd7e2640 10 client.5896.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59cd7e2640 10 monclient: _renew_subs 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59cd7e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c80ab710 con 0x7f59c80b17c0 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.115+0000 7f59cd7e2640 1 librados: init done 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.116+0000 7f59cd7e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.116+0000 7f59cd7e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.116+0000 7f59cd7e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:28.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.116+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c814c7b0 con 0x7f59c80b17c0 2024-01-20T11:22:28.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.135+0000 7f59bf7fe640 1 -- 172.21.15.110:0/983097149 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f59b804e430 con 0x7f59c80b17c0 2024-01-20T11:22:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.136+0000 7f59bf7fe640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59a806c860 0x7f59a806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.136+0000 7f59bffff640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59a806c860 0x7f59a806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.136+0000 7f59bffff640 10 monclient: get_auth_request con 0x7f59a806c860 auth_method 0 2024-01-20T11:22:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.136+0000 7f59bffff640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59a806c860 0x7f59a806ed20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f59c813fdd0 tx=0x7f59b0009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:28.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.136+0000 7f59bf7fe640 10 client.5896.objecter ms_handle_connect 0x7f59a806c860 2024-01-20T11:22:28.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.356+0000 7f59bf7fe640 1 -- 172.21.15.110:0/983097149 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f59b807a760 con 0x7f59c80b17c0 2024-01-20T11:22:28.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.356+0000 7f59bf7fe640 10 client.5896.objecter ms_dispatch 0x7f59c8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:28.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.356+0000 7f59bf7fe640 3 client.5896.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:28.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.356+0000 7f59bf7fe640 3 client.5896.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:28.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.356+0000 7f59bf7fe640 20 client.5896.objecter dump_active .. 0 homeless 2024-01-20T11:22:28.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.365+0000 7f59bf7fe640 1 -- 172.21.15.110:0/983097149 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f59b8076aa0 con 0x7f59c80b17c0 2024-01-20T11:22:28.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.365+0000 7f59bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:28.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.365+0000 7f59bf7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:28.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.535+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59c80acc50 con 0x7f59a806c860 2024-01-20T11:22:28.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.538+0000 7f59bf7fe640 1 -- 172.21.15.110:0/983097149 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64019 (secure 0 0 0) 0x7f59c80acc50 con 0x7f59a806c860 2024-01-20T11:22:28.539 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:28.540 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:28.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 10 librados: watch_flush enter 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 10 librados: watch_flush exit 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 20 client.5896.objecter shutdown clearing up homeless session... 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 10 client.5896.objecter successfully canceled tick 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59a806c860 msgr2=0x7f59a806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59a806c860 0x7f59a806ed20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f59c813fdd0 tx=0x7f59b0009190 comp rx=0 tx=0).stop 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 10 monclient: shutdown 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59c80b17c0 msgr2=0x7f59c813ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59c80b17c0 0x7f59c813ebd0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f59b8033000 tx=0x7f59b80046c0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 shutdown_connections 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59a806c860 0x7f59a806ed20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c8145980 0x7f59c813c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80b4d10 0x7f59c813c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 --2- 172.21.15.110:0/983097149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59c80b17c0 0x7f59c813ebd0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 >> 172.21.15.110:0/983097149 conn(0x7f59c8066590 msgr2=0x7f59c813b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 shutdown_connections 2024-01-20T11:22:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 -- 172.21.15.110:0/983097149 wait complete. 2024-01-20T11:22:28.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:28.540+0000 7f59cd7e2640 1 librados: shutdown 2024-01-20T11:22:28.551 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-01-20T11:22:26.797643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002722"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":14.331},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":6.9690000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":6.0750000000000002},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":9.0999999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":12.44},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":9.7639999999999993}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":15.178000000000001},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":7.7480000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":11.835000000000001},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":16.007000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":10.927},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":13.052}]}]},{"osd":7,"up_from":21,"seq":90194313307,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.9870000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":10.77}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":16.486000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":8.7330000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":9.5090000000000003},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":11.441000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":7.1189999999999998},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":14.724}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":10.138999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.077999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":12.781000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":15.523}]}]},{"osd":3,"up_from":21,"seq":90194313307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":15.608000000000001},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":14.705}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":9.1489999999999991},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.307}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":6.2149999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":9.6980000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":12.025},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.0759999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":7.8140000000000001},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":8.4600000000000009}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":12.65},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":13.725}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.4350000000000005,"5min":8.2490000000000006,"15min":9.6880000000000006},"min":{"1min":4.8529999999999998,"5min":4.0750000000000002,"15min":4.0750000000000002},"max":{"1min":15.361000000000001,"5min":15.361000000000001,"15min":25.603999999999999},"last":20.539999999999999},{"interface":"front","average":{"1min":8.1430000000000007,"5min":8.2910000000000004,"15min":8.5079999999999991},"min":{"1min":4.9210000000000003,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":17.154,"5min":17.154,"15min":17.154},"last":15.042999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":8.423,"5min":8.6799999999999997,"15min":12.42},"min":{"1min":5.2750000000000004,"5min":4.1660000000000004,"15min":4.1660000000000004},"max":{"1min":17.024000000000001,"5min":20.425999999999998,"15min":65.064999999999998},"last":16.329999999999998},{"interface":"front","average":{"1min":11.253,"5min":11.391,"15min":15.015000000000001},"min":{"1min":6.9180000000000001,"5min":6.8869999999999996,"15min":5.0460000000000003},"max":{"1min":15.073,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.334}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":19.061},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.8829999999999991}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":18.103999999999999},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.303000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:21:22 2024","interfaces":[{"interface":"back","average":{"1min":13.759,"5min":13.644,"15min":15.265000000000001},"min":{"1min":8.1470000000000002,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":19.780000000000001,"5min":20.245999999999999,"15min":35.186999999999998},"last":19.635999999999999},{"interface":"front","average":{"1min":10.244,"5min":10.678000000000001,"15min":10.858000000000001},"min":{"1min":6.0579999999999998,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":15.202999999999999,"5min":17.308,"15min":32.984000000000002},"last":9.5269999999999992}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.5129999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.219999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444122,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":6.6769999999999996},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":4.8419999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":11.897},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":8.2599999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":14.247999999999999},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":9.032}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":12.960000000000001},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":16.327999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":13.606999999999999},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":9.7159999999999993}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":15.41},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":10.316000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444122,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":7.7309999999999999,"5min":7.8769999999999998,"15min":12.878},"min":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":13.045999999999999,"5min":17.498999999999999,"15min":114.843},"last":7.7549999999999999},{"interface":"front","average":{"1min":9.0489999999999995,"5min":8.9009999999999998,"15min":13.843999999999999},"min":{"1min":3.6909999999999998,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":18.504999999999999,"5min":18.504999999999999,"15min":121.044},"last":4.6159999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":8.0329999999999995,"5min":7.9329999999999998,"15min":23.983000000000001},"min":{"1min":5.5609999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.177,"5min":18.198,"15min":379.17099999999999},"last":8.5600000000000005},{"interface":"front","average":{"1min":11.212,"5min":10.868,"15min":25.957999999999998},"min":{"1min":4.819,"5min":4.819,"15min":4.819},"max":{"1min":14.321999999999999,"5min":19.094999999999999,"15min":373.88299999999998},"last":9.8849999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":11.721},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.888}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.737},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":6.7619999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:25 2024","interfaces":[{"interface":"back","average":{"1min":12.731,"5min":13.127000000000001,"15min":28.471},"min":{"1min":7.2270000000000003,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":19.376000000000001,"5min":19.376000000000001,"15min":375.733},"last":15.339},{"interface":"front","average":{"1min":13.712999999999999,"5min":13.893000000000001,"15min":26.957000000000001},"min":{"1min":8.2599999999999998,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":17.943000000000001,"5min":19.553000000000001,"15min":374.654},"last":11.441000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.321999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":9.2100000000000009}]}]},{"osd":6,"up_from":15,"seq":64424509531,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.0289999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":13.965999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.175000000000001},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":10.401999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":15.935},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":8.5730000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.7899999999999991},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":14.943}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":17.617000000000001},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.536}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.257999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":7.9189999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:29.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~211s 2024-01-20T11:22:31.554 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.653+0000 7fa871e15640 10 monclient: get_monmap_and_config 2024-01-20T11:22:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.653+0000 7fa871e15640 10 monclient: build_initial_monmap 2024-01-20T11:22:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.653+0000 7fa871e15640 10 monclient: monmap: 2024-01-20T11:22:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:31.655049+0000 2024-01-20T11:22:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:31.655049+0000 2024-01-20T11:22:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:31.655 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.654+0000 7fa871e15640 1 Processor -- start 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.654+0000 7fa871e15640 1 -- start start 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.654+0000 7fa871e15640 10 monclient: init 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa86c0a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 10 monclient(hunting): picked mon.noname-b con 0x7fa86c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa86c142920 0x7fa86c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 10 monclient(hunting): picked mon.noname-c con 0x7fa86c142920 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86c0b25c0 con 0x7fa86c0a96f0 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86c0aaea0 con 0x7fa86c142920 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c13fbe0 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa871e15640 10 monclient(hunting): authenticate will time out at 1795.006418s 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.655+0000 7fa86affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46922/0 (socket says 172.21.15.110:46922) 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86affd640 1 -- 172.21.15.110:0/1628177476 learned_addr learned my addr 172.21.15.110:0/1628177476 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86affd640 10 monclient(hunting): get_auth_request con 0x7fa86c0a9ad0 auth_method 0 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86bfff640 1 --2- 172.21.15.110:0/1628177476 >> v2:172.21.15.134:3300/0 conn(0x7fa86c142920 0x7fa86c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.656+0000 7fa86affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:31.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.657+0000 7fa86bfff640 10 monclient(hunting): get_auth_request con 0x7fa86c142920 auth_method 0 2024-01-20T11:22:31.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.657+0000 7fa86bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:31.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.657+0000 7fa86bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:31.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.657+0000 7fa86bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:31.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.657+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:31.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.657+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:31.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.657+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:31.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.658+0000 7fa86affd640 10 monclient(hunting): handle_auth_done global_id 5489 payload 274 2024-01-20T11:22:31.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.658+0000 7fa86affd640 1 -- 172.21.15.110:0/1628177476 >> v2:172.21.15.134:3300/0 conn(0x7fa86c142920 msgr2=0x7fa86c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:31.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.658+0000 7fa86affd640 1 --2- 172.21.15.110:0/1628177476 >> v2:172.21.15.134:3300/0 conn(0x7fa86c142920 0x7fa86c14c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:31.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.658+0000 7fa86affd640 1 -- 172.21.15.110:0/1628177476 >> v1:172.21.15.134:6789/0 conn(0x7fa86c0a96f0 legacy=0x7fa86c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:31.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.658+0000 7fa86affd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:31.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.658+0000 7fa86affd640 1 monclient: found mon.noname-b 2024-01-20T11:22:31.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.658+0000 7fa86affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:31.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.658+0000 7fa86affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:31.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.658+0000 7fa86affd640 1 -- 172.21.15.110:0/1628177476 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c0ab0a0 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.660+0000 7fa86affd640 1 --2- 172.21.15.110:0/1628177476 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1423e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa860009e50 tx=0x7fa86002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8a5cde5384dc607 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 1 -- 172.21.15.110:0/1628177476 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86002ca70 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 10 monclient: dump: 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 1 -- 172.21.15.110:0/1628177476 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86002cbd0 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa871e15640 5 monclient: authenticate success, global_id 5489 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa871e15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa871e15640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa871e15640 4 set_mon_vals no callback set 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 1 -- 172.21.15.110:0/1628177476 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600343f0 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.663+0000 7fa86a7fc640 10 monclient: dump: 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:31.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 10 monclient: shutdown 2024-01-20T11:22:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 1 -- 172.21.15.110:0/1628177476 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 msgr2=0x7fa86c1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 1 --2- 172.21.15.110:0/1628177476 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1423e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa860009e50 tx=0x7fa86002bed0 comp rx=0 tx=0).stop 2024-01-20T11:22:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 1 -- 172.21.15.110:0/1628177476 shutdown_connections 2024-01-20T11:22:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 1 --2- 172.21.15.110:0/1628177476 >> v2:172.21.15.134:3300/0 conn(0x7fa86c142920 0x7fa86c14c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 1 --2- 172.21.15.110:0/1628177476 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1423e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 1 -- 172.21.15.110:0/1628177476 >> 172.21.15.110:0/1628177476 conn(0x7fa86c0b7bc0 msgr2=0x7fa86c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 1 -- 172.21.15.110:0/1628177476 shutdown_connections 2024-01-20T11:22:31.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.664+0000 7fa871e15640 1 -- 172.21.15.110:0/1628177476 wait complete. 2024-01-20T11:22:31.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.665+0000 7fa871e15640 10 monclient: build_initial_monmap 2024-01-20T11:22:31.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.665+0000 7fa871e15640 10 monclient: monmap: 2024-01-20T11:22:31.672 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:31.672 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:31.672 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:31.672 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:31.672 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:31.672 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.665+0000 7fa871e15640 1 librados: starting msgr at 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.665+0000 7fa871e15640 1 librados: starting objecter 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.665+0000 7fa871e15640 1 Processor -- start 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.665+0000 7fa871e15640 1 -- start start 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.665+0000 7fa871e15640 1 librados: setting wanted keys 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.665+0000 7fa871e15640 1 librados: calling monclient init 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.665+0000 7fa871e15640 10 monclient: init 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:31.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa86c0a96f0 0x7fa86c139070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient(hunting): picked mon.noname-a con 0x7fa86c0a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1dab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient(hunting): picked mon.noname-c con 0x7fa86c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa86c142920 0x7fa86c138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient(hunting): picked mon.noname-b con 0x7fa86c142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c139b10 con 0x7fa86c0a96f0 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1db590 con 0x7fa86c142920 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1db090 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa871e15640 10 monclient(hunting): authenticate will time out at 1795.017603s 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa86affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1dab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:31.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.666+0000 7fa86b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa86c0a96f0 0x7fa86c139070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa86c142920 0x7fa86c138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1dab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46924/0 (socket says 172.21.15.110:46924) 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa86c0a96f0 0x7fa86c139070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47566/0 (socket says 172.21.15.110:47566) 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86affd640 1 -- 172.21.15.110:0/116906276 learned_addr learned my addr 172.21.15.110:0/116906276 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86affd640 10 monclient(hunting): get_auth_request con 0x7fa86c0a9ad0 auth_method 0 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86b7fe640 10 monclient(hunting): get_auth_request con 0x7fa86c0a96f0 auth_method 0 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86bfff640 10 monclient(hunting): get_auth_request con 0x7fa86c142920 auth_method 0 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:31.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.667+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 10 monclient(hunting): handle_auth_done global_id 5492 payload 824 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 1 -- 172.21.15.110:0/116906276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa86c142920 msgr2=0x7fa86c138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa86c142920 0x7fa86c138360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 1 -- 172.21.15.110:0/116906276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa86c0a96f0 msgr2=0x7fa86c139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa86c0a96f0 0x7fa86c139070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:31.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:31.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 1 monclient: found mon.noname-c 2024-01-20T11:22:31.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:31.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:31.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.670+0000 7fa86affd640 1 -- 172.21.15.110:0/116906276 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa860009b30 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.672+0000 7fa86affd640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1dab50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa860000f80 tx=0x7fa860004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:31.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.672+0000 7fa853fff640 10 client.?.objecter ms_handle_connect 0x7fa86c0a9ad0 2024-01-20T11:22:31.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.672+0000 7fa853fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:31.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 1 -- 172.21.15.110:0/116906276 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86003e450 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 10 monclient: dump: 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:31.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 10 monclient: _finish_auth 0 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 10 monclient: _check_auth_tickets 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 1 -- 172.21.15.110:0/116906276 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86003e5f0 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa871e15640 5 monclient: authenticate success, global_id 5492 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa871e15640 10 monclient: _renew_subs 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa871e15640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c1db310 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 1 -- 172.21.15.110:0/116906276 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86003e9a0 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa853fff640 10 monclient: dump: 2024-01-20T11:22:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa869ffb640 4 set_mon_vals no callback set 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa871e15640 10 client.5492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa871e15640 10 monclient: _renew_subs 2024-01-20T11:22:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa871e15640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c138a80 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.676+0000 7fa871e15640 1 librados: init done 2024-01-20T11:22:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.677+0000 7fa871e15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.677+0000 7fa871e15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.677+0000 7fa871e15640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.677+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c1db7b0 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.922+0000 7fa853fff640 1 -- 172.21.15.110:0/116906276 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa86003eb00 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.922+0000 7fa853fff640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa85806c8b0 0x7fa85806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:31.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.922+0000 7fa86b7fe640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa85806c8b0 0x7fa85806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:31.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.922+0000 7fa86b7fe640 10 monclient: get_auth_request con 0x7fa85806c8b0 auth_method 0 2024-01-20T11:22:31.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.923+0000 7fa853fff640 10 client.5492.objecter ms_handle_connect 0x7fa85806c8b0 2024-01-20T11:22:31.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.923+0000 7fa86b7fe640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa85806c8b0 0x7fa85806ed70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa864000f60 tx=0x7fa864009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:31.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.936+0000 7fa853fff640 1 -- 172.21.15.110:0/116906276 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa86007b340 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.936+0000 7fa853fff640 10 client.5492.objecter ms_dispatch 0x7fa86c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:31.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.936+0000 7fa853fff640 3 client.5492.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:31.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.936+0000 7fa853fff640 3 client.5492.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:31.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.936+0000 7fa853fff640 20 client.5492.objecter dump_active .. 0 homeless 2024-01-20T11:22:31.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.936+0000 7fa853fff640 1 -- 172.21.15.110:0/116906276 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa860077680 con 0x7fa86c0a9ad0 2024-01-20T11:22:31.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.936+0000 7fa853fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:31.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:31.936+0000 7fa853fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.107+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa86c0b4400 con 0x7fa85806c8b0 2024-01-20T11:22:32.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.110+0000 7fa853fff640 1 -- 172.21.15.110:0/116906276 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63892 (secure 0 0 0) 0x7fa8640014a0 con 0x7fa85806c8b0 2024-01-20T11:22:32.112 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:32.112 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 10 librados: watch_flush enter 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 10 librados: watch_flush exit 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 20 client.5492.objecter shutdown clearing up homeless session... 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 10 client.5492.objecter successfully canceled tick 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa85806c8b0 msgr2=0x7fa85806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa85806c8b0 0x7fa85806ed70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa864000f60 tx=0x7fa864009040 comp rx=0 tx=0).stop 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 10 monclient: shutdown 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 msgr2=0x7fa86c1dab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1dab50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa860000f80 tx=0x7fa860004a40 comp rx=0 tx=0).stop 2024-01-20T11:22:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 shutdown_connections 2024-01-20T11:22:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa85806c8b0 0x7fa85806ed70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa86c142920 0x7fa86c138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa86c0a9ad0 0x7fa86c1dab50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 --2- 172.21.15.110:0/116906276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa86c0a96f0 0x7fa86c139070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.112+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 >> 172.21.15.110:0/116906276 conn(0x7fa86c0b7bc0 msgr2=0x7fa86c13c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.113+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 shutdown_connections 2024-01-20T11:22:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.113+0000 7fa871e15640 1 -- 172.21.15.110:0/116906276 wait complete. 2024-01-20T11:22:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.113+0000 7fa871e15640 1 librados: shutdown 2024-01-20T11:22:32.125 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-01-20T11:22:30.798466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002570"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":6.6449999999999996},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":5.0670000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":10.723000000000001},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":7.5999999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":11.523999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":12.483000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":13.339},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":9.6259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":13.968},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":14.589}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":8.9209999999999994},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":15.818}]}]},{"osd":7,"up_from":21,"seq":90194313307,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.9870000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":10.77}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":16.486000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":8.7330000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":9.5090000000000003},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":11.441000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":7.1189999999999998},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":14.724}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":10.138999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.077999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":12.781000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":15.523}]}]},{"osd":3,"up_from":21,"seq":90194313308,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0039999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":5.7290000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":12.605},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.426}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":8.6300000000000008},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":7.6219999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":14.590999999999999},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":9.2390000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":9.8100000000000005},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":13.750999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":15.382},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":11.106}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444122,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":10.978999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":9.4740000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":14.817},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.343}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.012},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":7.9550000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":13.605},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.416}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":14.202999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":12.1}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":16.951000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":15.452999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444123,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.6379999999999999},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.5519999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":12.215999999999999},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":13.113}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":11.416},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":5.8220000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.6609999999999996},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.743}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.9809999999999999},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":15.612}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":10.340999999999999},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.494}]}]},{"osd":0,"up_from":17,"seq":73014444123,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.2430000000000003},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":11.881}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":6.3170000000000002},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":8.3059999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":9.3350000000000009},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.521000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.143000000000001},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":16.202000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":10.958},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":13.712}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.398999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":17.552}]}]},{"osd":6,"up_from":15,"seq":64424509531,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.0289999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":13.965999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.175000000000001},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":10.401999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":15.935},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":8.5730000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.7899999999999991},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":14.943}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":17.617000000000001},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.536}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.257999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":7.9189999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:32.126 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.224+0000 7fb904b05640 10 monclient: get_monmap_and_config 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.224+0000 7fb904b05640 10 monclient: build_initial_monmap 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.224+0000 7fb904b05640 10 monclient: monmap: 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:32.225780+0000 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:32.225780+0000 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:32.226 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:32.226 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.224+0000 7fb904b05640 1 Processor -- start 2024-01-20T11:22:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.224+0000 7fb904b05640 1 -- start start 2024-01-20T11:22:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient: init 2024-01-20T11:22:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-01-20T11:22:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9000a96f0 0x7fb90013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient(hunting): picked mon.noname-b con 0x7fb9000a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fb90013fa10 0x7fb9001423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient(hunting): picked mon.noname-c con 0x7fb90013fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9000a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9000b25c0 con 0x7fb9000a9ad0 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9000aaea0 con 0x7fb90013fa10 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9000ab0a0 con 0x7fb9000a96f0 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.225+0000 7fb904b05640 10 monclient(hunting): authenticate will time out at 1795.576843s 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.226+0000 7fb8fe59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9000a96f0 0x7fb90013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.226+0000 7fb8fed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb9000a9ad0 0x7fb900142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33626/0 (socket says 172.21.15.110:33626) 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.226+0000 7fb8fed9d640 1 -- 172.21.15.110:0/8485804 learned_addr learned my addr 172.21.15.110:0/8485804 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.226+0000 7fb8fe59c640 10 monclient(hunting): get_auth_request con 0x7fb9000a96f0 auth_method 0 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.226+0000 7fb8fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.226+0000 7fb8fe59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.226+0000 7fb8fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.226+0000 7fb8f7fff640 1 --2- 172.21.15.110:0/8485804 >> v2:172.21.15.134:3300/0 conn(0x7fb90013fa10 0x7fb9001423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8f7fff640 10 monclient(hunting): get_auth_request con 0x7fb90013fa10 auth_method 0 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.227+0000 7fb8f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fe59c640 10 monclient(hunting): handle_auth_done global_id 5495 payload 274 2024-01-20T11:22:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fe59c640 1 -- 172.21.15.110:0/8485804 >> v2:172.21.15.134:3300/0 conn(0x7fb90013fa10 msgr2=0x7fb9001423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fe59c640 1 --2- 172.21.15.110:0/8485804 >> v2:172.21.15.134:3300/0 conn(0x7fb90013fa10 0x7fb9001423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fe59c640 1 -- 172.21.15.110:0/8485804 >> v1:172.21.15.110:6789/0 conn(0x7fb9000a9ad0 legacy=0x7fb900142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fe59c640 10 monclient: _finish_hunting 0 2024-01-20T11:22:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fe59c640 1 monclient: found mon.noname-b 2024-01-20T11:22:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fdd9b640 1 -- 172.21.15.110:0/8485804 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190042549 0 0) 0x7fb9000b25c0 con 0x7fb9000a9ad0 2024-01-20T11:22:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fe59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fe59c640 1 -- 172.21.15.110:0/8485804 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9000b3fb0 con 0x7fb9000a96f0 2024-01-20T11:22:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.228+0000 7fb8fdd9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:22:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.231+0000 7fb8fe59c640 1 --2- 172.21.15.110:0/8485804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9000a96f0 0x7fb90013f4d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb8f00097a0 tx=0x7fb8f002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51f137d79f4ec571 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:32.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 1 -- 172.21.15.110:0/8485804 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f002e070 con 0x7fb9000a96f0 2024-01-20T11:22:32.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:32.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 10 monclient: dump: 2024-01-20T11:22:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:32.239 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 10 monclient: _finish_auth 0 2024-01-20T11:22:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 10 monclient: _check_auth_tickets 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 1 -- 172.21.15.110:0/8485804 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f00315a0 con 0x7fb9000a96f0 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb904b05640 5 monclient: authenticate success, global_id 5495 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb904b05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb904b05640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb904b05640 4 set_mon_vals no callback set 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 1 -- 172.21.15.110:0/8485804 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0031930 con 0x7fb9000a96f0 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.234+0000 7fb8fdd9b640 10 monclient: dump: 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:32.241 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 10 monclient: shutdown 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 1 -- 172.21.15.110:0/8485804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9000a96f0 msgr2=0x7fb90013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 1 --2- 172.21.15.110:0/8485804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9000a96f0 0x7fb90013f4d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb8f00097a0 tx=0x7fb8f002b840 comp rx=0 tx=0).stop 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 1 -- 172.21.15.110:0/8485804 shutdown_connections 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 1 --2- 172.21.15.110:0/8485804 >> v2:172.21.15.134:3300/0 conn(0x7fb90013fa10 0x7fb9001423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 1 --2- 172.21.15.110:0/8485804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9000a96f0 0x7fb90013f4d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 1 -- 172.21.15.110:0/8485804 >> 172.21.15.110:0/8485804 conn(0x7fb9000b7bc0 msgr2=0x7fb9000b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 1 -- 172.21.15.110:0/8485804 shutdown_connections 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.235+0000 7fb904b05640 1 -- 172.21.15.110:0/8485804 wait complete. 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.236+0000 7fb904b05640 10 monclient: build_initial_monmap 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.236+0000 7fb904b05640 10 monclient: monmap: 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.236+0000 7fb904b05640 1 librados: starting msgr at 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.236+0000 7fb904b05640 1 librados: starting objecter 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 Processor -- start 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 -- start start 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 librados: setting wanted keys 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 librados: calling monclient init 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient: init 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9000a96f0 0x7fb900136d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient(hunting): picked mon.noname-b con 0x7fb9000a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9000a9ad0 0x7fb9001da730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient(hunting): picked mon.noname-a con 0x7fb9000a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9001db420 0x7fb900133ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient(hunting): picked mon.noname-c con 0x7fb9001db420 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9001dac70 con 0x7fb9000a9ad0 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9001daf00 con 0x7fb9000a96f0 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb900137720 con 0x7fb9001db420 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.237+0000 7fb904b05640 10 monclient(hunting): authenticate will time out at 1795.588851s 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9001db420 0x7fb900133ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8f7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9000a9ad0 0x7fb9001da730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8f7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9000a9ad0 0x7fb9001da730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47574/0 (socket says 172.21.15.110:47574) 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8f7fff640 1 -- 172.21.15.110:0/2334377843 learned_addr learned my addr 172.21.15.110:0/2334377843 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8f7fff640 10 monclient(hunting): get_auth_request con 0x7fb9000a9ad0 auth_method 0 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fed9d640 10 monclient(hunting): get_auth_request con 0x7fb9001db420 auth_method 0 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fe59c640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9000a96f0 0x7fb900136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fe59c640 10 monclient(hunting): get_auth_request con 0x7fb9000a96f0 auth_method 0 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fe59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.238+0000 7fb8fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.239+0000 7fb8f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.239+0000 7fb8f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.239+0000 7fb8f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.239+0000 7fb8fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.239+0000 7fb8fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.239+0000 7fb8fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.239+0000 7fb8fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.239+0000 7fb8fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.239+0000 7fb8fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 10 monclient(hunting): handle_auth_done global_id 5826 payload 824 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 1 -- 172.21.15.110:0/2334377843 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9001db420 msgr2=0x7fb900133ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9001db420 0x7fb900133ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 1 -- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9000a96f0 msgr2=0x7fb900136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9000a96f0 0x7fb900136d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 1 monclient: found mon.noname-a 2024-01-20T11:22:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.241+0000 7fb8f7fff640 1 -- 172.21.15.110:0/2334377843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0009400 con 0x7fb9000a9ad0 2024-01-20T11:22:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.244+0000 7fb8f7fff640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9000a9ad0 0x7fb9001da730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800e840 tx=0x7fb8e800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.244+0000 7fb8f6ffd640 10 client.?.objecter ms_handle_connect 0x7fb9000a9ad0 2024-01-20T11:22:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.244+0000 7fb8f6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:32.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 1 -- 172.21.15.110:0/2334377843 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800c730 con 0x7fb9000a9ad0 2024-01-20T11:22:32.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:32.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 10 monclient: dump: 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 1 -- 172.21.15.110:0/2334377843 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e800c890 con 0x7fb9000a9ad0 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 1 -- 172.21.15.110:0/2334377843 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800cc40 con 0x7fb9000a9ad0 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb904b05640 5 monclient: authenticate success, global_id 5826 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb904b05640 10 monclient: _renew_subs 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb904b05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9001378e0 con 0x7fb9000a9ad0 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8fd59a640 4 set_mon_vals no callback set 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb8f6ffd640 10 monclient: dump: 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb904b05640 10 client.5826.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb904b05640 10 monclient: _renew_subs 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb904b05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9001345c0 con 0x7fb9000a9ad0 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.247+0000 7fb904b05640 1 librados: init done 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.248+0000 7fb904b05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.248+0000 7fb904b05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.248+0000 7fb904b05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.248+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9001db070 con 0x7fb9000a9ad0 2024-01-20T11:22:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.267+0000 7fb8f6ffd640 1 -- 172.21.15.110:0/2334377843 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb8e8013470 con 0x7fb9000a9ad0 2024-01-20T11:22:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.267+0000 7fb8f6ffd640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb8e006c8b0 0x7fb8e006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.268+0000 7fb8fe59c640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb8e006c8b0 0x7fb8e006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.268+0000 7fb8fe59c640 10 monclient: get_auth_request con 0x7fb8e006c8b0 auth_method 0 2024-01-20T11:22:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.268+0000 7fb8fe59c640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb8e006c8b0 0x7fb8e006ed70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0036000 tx=0x7fb8f003d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.268+0000 7fb8f6ffd640 10 client.5826.objecter ms_handle_connect 0x7fb8e006c8b0 2024-01-20T11:22:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.497+0000 7fb8f6ffd640 1 -- 172.21.15.110:0/2334377843 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb8e80589d0 con 0x7fb9000a9ad0 2024-01-20T11:22:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.497+0000 7fb8f6ffd640 10 client.5826.objecter ms_dispatch 0x7fb900000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.497+0000 7fb8f6ffd640 3 client.5826.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.497+0000 7fb8f6ffd640 3 client.5826.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.497+0000 7fb8f6ffd640 20 client.5826.objecter dump_active .. 0 homeless 2024-01-20T11:22:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.497+0000 7fb8f6ffd640 1 -- 172.21.15.110:0/2334377843 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb8e8054d10 con 0x7fb9000a9ad0 2024-01-20T11:22:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.497+0000 7fb8f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.497+0000 7fb8f6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:32.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.671+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb900136400 con 0x7fb8e006c8b0 2024-01-20T11:22:32.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.674+0000 7fb8f6ffd640 1 -- 172.21.15.110:0/2334377843 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63892 (secure 0 0 0) 0x7fb900136400 con 0x7fb8e006c8b0 2024-01-20T11:22:32.676 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 10 librados: watch_flush enter 2024-01-20T11:22:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 10 librados: watch_flush exit 2024-01-20T11:22:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 20 client.5826.objecter shutdown clearing up homeless session... 2024-01-20T11:22:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 10 client.5826.objecter successfully canceled tick 2024-01-20T11:22:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb8e006c8b0 msgr2=0x7fb8e006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb8e006c8b0 0x7fb8e006ed70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0036000 tx=0x7fb8f003d040 comp rx=0 tx=0).stop 2024-01-20T11:22:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 10 monclient: shutdown 2024-01-20T11:22:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9000a9ad0 msgr2=0x7fb9001da730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9000a9ad0 0x7fb9001da730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800e840 tx=0x7fb8e800ed00 comp rx=0 tx=0).stop 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 shutdown_connections 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb8e006c8b0 0x7fb8e006ed70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9001db420 0x7fb900133ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9000a9ad0 0x7fb9001da730 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 --2- 172.21.15.110:0/2334377843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9000a96f0 0x7fb900136d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 >> 172.21.15.110:0/2334377843 conn(0x7fb9000b7bc0 msgr2=0x7fb9000b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 shutdown_connections 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 -- 172.21.15.110:0/2334377843 wait complete. 2024-01-20T11:22:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.676+0000 7fb904b05640 1 librados: shutdown 2024-01-20T11:22:32.688 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-01-20T11:22:30.798466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002570"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":6.6449999999999996},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":5.0670000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":10.723000000000001},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":7.5999999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":11.523999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":12.483000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":13.339},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":9.6259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":13.968},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":14.589}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":8.9209999999999994},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":15.818}]}]},{"osd":7,"up_from":21,"seq":90194313307,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.9870000000000001},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":10.77}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":16.486000000000001},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":8.7330000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":9.5090000000000003},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":11.441000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":7.1189999999999998},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":14.724}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":10.138999999999999},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":12.077999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":12.781000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":15.523}]}]},{"osd":3,"up_from":21,"seq":90194313308,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0039999999999996},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":5.7290000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":12.605},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.426}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":8.6300000000000008},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":7.6219999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":14.590999999999999},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":9.2390000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":9.8100000000000005},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":13.750999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":15.382},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":11.106}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444122,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":10.978999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":9.4740000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":14.817},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.343}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.012},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":7.9550000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":13.605},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.416}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":14.202999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":12.1}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":16.951000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":15.452999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444123,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.6379999999999999},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.5519999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":12.215999999999999},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":13.113}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":11.416},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":5.8220000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.6609999999999996},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.743}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.9809999999999999},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":15.612}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":10.340999999999999},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.494}]}]},{"osd":0,"up_from":17,"seq":73014444123,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.2430000000000003},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":11.881}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":6.3170000000000002},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":8.3059999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":9.3350000000000009},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.521000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.143000000000001},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":16.202000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":10.958},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":13.712}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.398999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":17.552}]}]},{"osd":6,"up_from":15,"seq":64424509531,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":7.5069999999999997,"5min":7.5830000000000002,"15min":10.381},"min":{"1min":4.7830000000000004,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":11.6,"5min":16.233000000000001,"15min":106.059},"last":6.0289999999999999},{"interface":"front","average":{"1min":8.0830000000000002,"5min":8.0470000000000006,"15min":12.663},"min":{"1min":4.1840000000000002,"5min":3.96,"15min":3.96},"max":{"1min":13.348000000000001,"5min":14.807,"15min":157.67699999999999},"last":13.965999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8100000000000005,"5min":9.5020000000000007,"15min":23.542000000000002},"min":{"1min":5.5279999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":13.757,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.175000000000001},{"interface":"front","average":{"1min":11.119,"5min":11.215999999999999,"15min":24.893999999999998},"min":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.093,"5min":17.751000000000001,"15min":190.304},"last":10.401999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:23 2024","interfaces":[{"interface":"back","average":{"1min":9.6280000000000001,"5min":10.43,"15min":13.574},"min":{"1min":6.2549999999999999,"5min":5.6369999999999996,"15min":5.3300000000000001},"max":{"1min":17.491,"5min":17.491,"15min":128.67099999999999},"last":15.935},{"interface":"front","average":{"1min":10.757999999999999,"5min":11.084,"15min":14.473000000000001},"min":{"1min":6.3150000000000004,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.484,"5min":17.050000000000001,"15min":101.75700000000001},"last":8.5730000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.7899999999999991},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":14.943}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":17.617000000000001},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":11.536}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.257999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":7.9189999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:32.689 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:32.689 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:32.689 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:32.689 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:32.690 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:32.690 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:32.690 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:32.690 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:32.690 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:32.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.787+0000 7f5a02eb4640 10 monclient: get_monmap_and_config 2024-01-20T11:22:32.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.787+0000 7f5a02eb4640 10 monclient: build_initial_monmap 2024-01-20T11:22:32.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.787+0000 7f5a02eb4640 10 monclient: monmap: 2024-01-20T11:22:32.788 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:32.788 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.788 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:32.789037+0000 2024-01-20T11:22:32.788 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:32.789037+0000 2024-01-20T11:22:32.789 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:32.789 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.789 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:32.789 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:32.789 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:32.789 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:32.789 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:32.789 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 1 Processor -- start 2024-01-20T11:22:32.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 1 -- start start 2024-01-20T11:22:32.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient: init 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f59fc06a860 0x7f59fc13f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient(hunting): picked mon.noname-c con 0x7f59fc06a860 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f59fc0af730 0x7f59fc1455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient(hunting): picked mon.noname-a con 0x7f59fc0af730 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59fc13f990 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc0afc50 con 0x7f59fc13f990 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc06e450 con 0x7f59fc0af730 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc0b0170 con 0x7f59fc06a860 2024-01-20T11:22:32.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.788+0000 7f5a02eb4640 10 monclient(hunting): authenticate will time out at 1796.139974s 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.789+0000 7f59f3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f59fc0af730 0x7f59fc1455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.789+0000 7f59f3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f59fc0af730 0x7f59fc1455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47584/0 (socket says 172.21.15.110:47584) 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.789+0000 7f59f3fff640 1 -- 172.21.15.110:0/773810438 learned_addr learned my addr 172.21.15.110:0/773810438 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.789+0000 7f59f3fff640 10 monclient(hunting): get_auth_request con 0x7f59fc0af730 auth_method 0 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.789+0000 7f59f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.789+0000 7f59f3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.789+0000 7f59f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.789+0000 7f5a00c50640 1 --2- 172.21.15.110:0/773810438 >> v2:172.21.15.134:3300/0 conn(0x7f59fc06a860 0x7f59fc13f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.790+0000 7f59f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.790+0000 7f59f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.790+0000 7f59f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.790+0000 7f5a00c50640 10 monclient(hunting): get_auth_request con 0x7f59fc06a860 auth_method 0 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.790+0000 7f5a00c50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.790+0000 7f5a00c50640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.790+0000 7f5a00c50640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f5a00c50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f5a00c50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f5a00c50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f59f3fff640 10 monclient(hunting): handle_auth_done global_id 5829 payload 274 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f59f3fff640 1 -- 172.21.15.110:0/773810438 >> v2:172.21.15.134:3300/0 conn(0x7f59fc06a860 msgr2=0x7f59fc13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f59f3fff640 1 --2- 172.21.15.110:0/773810438 >> v2:172.21.15.134:3300/0 conn(0x7f59fc06a860 0x7f59fc13f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f59f3fff640 1 -- 172.21.15.110:0/773810438 >> v1:172.21.15.134:6789/0 conn(0x7f59fc13f990 legacy=0x7f59fc145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f59f3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f59f3fff640 1 monclient: found mon.noname-a 2024-01-20T11:22:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f59f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:32.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f59f3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:32.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.791+0000 7f59f3fff640 1 -- 172.21.15.110:0/773810438 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc06ac40 con 0x7f59fc0af730 2024-01-20T11:22:32.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.793+0000 7f59f3fff640 1 --2- 172.21.15.110:0/773810438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc0af730 0x7f59fc1455b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0b5630 tx=0x7f59ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f69b958ee68e312 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 1 -- 172.21.15.110:0/773810438 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0317e0 con 0x7f59fc0af730 2024-01-20T11:22:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 10 monclient: dump: 2024-01-20T11:22:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 1 -- 172.21.15.110:0/773810438 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ec031940 con 0x7f59fc0af730 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 1 -- 172.21.15.110:0/773810438 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec031cd0 con 0x7f59fc0af730 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f5a02eb4640 5 monclient: authenticate success, global_id 5829 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f5a02eb4640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:32.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f5a02eb4640 4 set_mon_vals no callback set 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.797+0000 7f59f37fe640 10 monclient: dump: 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:32.800 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 10 monclient: shutdown 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 1 -- 172.21.15.110:0/773810438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc0af730 msgr2=0x7f59fc1455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 1 --2- 172.21.15.110:0/773810438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc0af730 0x7f59fc1455b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0b5630 tx=0x7f59ec02b880 comp rx=0 tx=0).stop 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 1 -- 172.21.15.110:0/773810438 shutdown_connections 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 1 --2- 172.21.15.110:0/773810438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc0af730 0x7f59fc1455b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 1 --2- 172.21.15.110:0/773810438 >> v2:172.21.15.134:3300/0 conn(0x7f59fc06a860 0x7f59fc13f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 1 -- 172.21.15.110:0/773810438 >> 172.21.15.110:0/773810438 conn(0x7f59fc0b47d0 msgr2=0x7f59fc0b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 1 -- 172.21.15.110:0/773810438 shutdown_connections 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.798+0000 7f5a02eb4640 1 -- 172.21.15.110:0/773810438 wait complete. 2024-01-20T11:22:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.799+0000 7f5a02eb4640 10 monclient: build_initial_monmap 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.799+0000 7f5a02eb4640 10 monclient: monmap: 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.799+0000 7f5a02eb4640 1 librados: starting msgr at 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.799+0000 7f5a02eb4640 1 librados: starting objecter 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.799+0000 7f5a02eb4640 1 Processor -- start 2024-01-20T11:22:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.799+0000 7f5a02eb4640 1 -- start start 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.799+0000 7f5a02eb4640 1 librados: setting wanted keys 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.799+0000 7f5a02eb4640 1 librados: calling monclient init 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.799+0000 7f5a02eb4640 10 monclient: init 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc06a860 0x7f59fc138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient(hunting): picked mon.noname-b con 0x7f59fc06a860 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59fc0af730 0x7f59fc13b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient(hunting): picked mon.noname-c con 0x7f59fc0af730 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc13f990 0x7f59fc1319c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient(hunting): picked mon.noname-a con 0x7f59fc13f990 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc13b210 con 0x7f59fc13f990 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc13c150 con 0x7f59fc06a860 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc13bdb0 con 0x7f59fc0af730 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a02eb4640 10 monclient(hunting): authenticate will time out at 1796.151533s 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a01451640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc13f990 0x7f59fc1319c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.800+0000 7f5a01451640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc13f990 0x7f59fc1319c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47596/0 (socket says 172.21.15.110:47596) 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a01451640 1 -- 172.21.15.110:0/1218191155 learned_addr learned my addr 172.21.15.110:0/1218191155 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a01451640 10 monclient(hunting): get_auth_request con 0x7f59fc13f990 auth_method 0 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a01451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a01451640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a01451640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a00c50640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc06a860 0x7f59fc138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a01451640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a01451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a01451640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a00c50640 10 monclient(hunting): get_auth_request con 0x7f59fc06a860 auth_method 0 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a00c50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a00c50640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f5a00c50640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.801+0000 7f59f3fff640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59fc0af730 0x7f59fc13b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f5a00c50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f5a00c50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f5a00c50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f59f3fff640 10 monclient(hunting): get_auth_request con 0x7f59fc0af730 auth_method 0 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f59f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f59f3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f59f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f59f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f59f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:32.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.802+0000 7f59f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:32.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 10 monclient(hunting): handle_auth_done global_id 5832 payload 824 2024-01-20T11:22:32.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 1 -- 172.21.15.110:0/1218191155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59fc0af730 msgr2=0x7f59fc13b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59fc0af730 0x7f59fc13b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 1 -- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc06a860 msgr2=0x7f59fc138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:32.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc06a860 0x7f59fc138670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:32.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 10 monclient: _finish_hunting 0 2024-01-20T11:22:32.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 1 monclient: found mon.noname-a 2024-01-20T11:22:32.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:32.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:32.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.804+0000 7f5a01451640 1 -- 172.21.15.110:0/1218191155 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec0094d0 con 0x7f59fc13f990 2024-01-20T11:22:32.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.809+0000 7f5a01451640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc13f990 0x7f59fc1319c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f59f400c760 tx=0x7f59f400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:32.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.810+0000 7f59f17fa640 10 client.?.objecter ms_handle_connect 0x7f59fc13f990 2024-01-20T11:22:32.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.810+0000 7f59f17fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:32.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 1 -- 172.21.15.110:0/1218191155 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4003b70 con 0x7f59fc13f990 2024-01-20T11:22:32.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:32.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 10 monclient: dump: 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 10 monclient: _finish_auth 0 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 10 monclient: _check_auth_tickets 2024-01-20T11:22:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 1 -- 172.21.15.110:0/1218191155 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f4003cd0 con 0x7f59fc13f990 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 1 -- 172.21.15.110:0/1218191155 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f400aa90 con 0x7f59fc13f990 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f17fa640 10 monclient: dump: 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f5a02eb4640 5 monclient: authenticate success, global_id 5832 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f5a02eb4640 10 monclient: _renew_subs 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f5a02eb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f59f2ffd640 4 set_mon_vals no callback set 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc13c4a0 con 0x7f59fc13f990 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f5a02eb4640 10 client.5832.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f5a02eb4640 10 monclient: _renew_subs 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f5a02eb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc1320e0 con 0x7f59fc13f990 2024-01-20T11:22:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.826+0000 7f5a02eb4640 1 librados: init done 2024-01-20T11:22:32.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.827+0000 7f5a02eb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:32.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.827+0000 7f5a02eb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:32.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.827+0000 7f5a02eb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:32.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.827+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59fc13c690 con 0x7f59fc13f990 2024-01-20T11:22:32.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.845+0000 7f59f17fa640 1 -- 172.21.15.110:0/1218191155 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f59f400abf0 con 0x7f59fc13f990 2024-01-20T11:22:32.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.846+0000 7f59f17fa640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59e806c8b0 0x7f59e806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:32.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.846+0000 7f5a00c50640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59e806c8b0 0x7f59e806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:32.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.846+0000 7f5a00c50640 10 monclient: get_auth_request con 0x7f59e806c8b0 auth_method 0 2024-01-20T11:22:32.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.847+0000 7f59f17fa640 10 client.5832.objecter ms_handle_connect 0x7f59e806c8b0 2024-01-20T11:22:32.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.847+0000 7f5a00c50640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59e806c8b0 0x7f59e806ed70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009aa0 tx=0x7f59f8009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.848+0000 7f59f17fa640 1 -- 172.21.15.110:0/1218191155 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f59f40589e0 con 0x7f59fc13f990 2024-01-20T11:22:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.848+0000 7f59f17fa640 10 client.5832.objecter ms_dispatch 0x7f59fc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.848+0000 7f59f17fa640 3 client.5832.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.848+0000 7f59f17fa640 3 client.5832.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:32.848+0000 7f59f17fa640 20 client.5832.objecter dump_active .. 0 homeless 2024-01-20T11:22:33.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.082+0000 7f59f17fa640 1 -- 172.21.15.110:0/1218191155 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f59f4054d20 con 0x7f59fc13f990 2024-01-20T11:22:33.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.082+0000 7f59f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:33.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.082+0000 7f59f17fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.329+0000 7f5a02eb4640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.330+0000 7f5a02eb4640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.330+0000 7f5a02eb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.330+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f59fc075490 con 0x7f59fc13f990 2024-01-20T11:22:33.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.335+0000 7f59f17fa640 1 -- 172.21.15.110:0/1218191155 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f59f4058340 con 0x7f59fc13f990 2024-01-20T11:22:33.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.335+0000 7f59f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.335+0000 7f59f17fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:33.336 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:33.336 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":487,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:33.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.336+0000 7f5a02eb4640 10 librados: watch_flush enter 2024-01-20T11:22:33.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.336+0000 7f5a02eb4640 10 librados: watch_flush exit 2024-01-20T11:22:33.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.336+0000 7f5a02eb4640 20 client.5832.objecter shutdown clearing up homeless session... 2024-01-20T11:22:33.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.336+0000 7f5a02eb4640 10 client.5832.objecter successfully canceled tick 2024-01-20T11:22:33.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59e806c8b0 msgr2=0x7f59e806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:33.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59e806c8b0 0x7f59e806ed70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009aa0 tx=0x7f59f8009190 comp rx=0 tx=0).stop 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 10 monclient: shutdown 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc13f990 msgr2=0x7f59fc1319c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc13f990 0x7f59fc1319c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f59f400c760 tx=0x7f59f400cc20 comp rx=0 tx=0).stop 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 shutdown_connections 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f59e806c8b0 0x7f59e806ed70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59fc13f990 0x7f59fc1319c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59fc0af730 0x7f59fc13b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 --2- 172.21.15.110:0/1218191155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc06a860 0x7f59fc138670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 >> 172.21.15.110:0/1218191155 conn(0x7f59fc0b47d0 msgr2=0x7f59fc0adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 shutdown_connections 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 -- 172.21.15.110:0/1218191155 wait complete. 2024-01-20T11:22:33.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.337+0000 7f5a02eb4640 1 librados: shutdown 2024-01-20T11:22:33.348 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:33.348 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:33.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.446+0000 7f5331341640 10 monclient: get_monmap_and_config 2024-01-20T11:22:33.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.446+0000 7f5331341640 10 monclient: build_initial_monmap 2024-01-20T11:22:33.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.446+0000 7f5331341640 10 monclient: monmap: 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:33.448059+0000 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:33.448059+0000 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:33.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 1 Processor -- start 2024-01-20T11:22:33.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 1 -- start start 2024-01-20T11:22:33.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient: init 2024-01-20T11:22:33.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f532c0b4d10 0x7f532c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient(hunting): picked mon.noname-a con 0x7f532c0b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f532c0b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f532c145980 0x7f532c145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient(hunting): picked mon.noname-c con 0x7f532c145980 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c146370 con 0x7f532c0b17c0 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c0b50f0 con 0x7f532c0b4d10 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c0aade0 con 0x7f532c145980 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:33.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.447+0000 7f5331341640 10 monclient(hunting): authenticate will time out at 1796.798885s 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.448+0000 7f532affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f532c0b4d10 0x7f532c0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.448+0000 7f532affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f532c0b4d10 0x7f532c0b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47606/0 (socket says 172.21.15.110:47606) 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.448+0000 7f532affd640 1 -- 172.21.15.110:0/3789357737 learned_addr learned my addr 172.21.15.110:0/3789357737 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.448+0000 7f532affd640 10 monclient(hunting): get_auth_request con 0x7f532c0b4d10 auth_method 0 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.448+0000 7f532affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.448+0000 7f532affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.448+0000 7f532affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.448+0000 7f532b7fe640 1 --2- 172.21.15.110:0/3789357737 >> v2:172.21.15.134:3300/0 conn(0x7f532c145980 0x7f532c145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.449+0000 7f532affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.449+0000 7f532affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.449+0000 7f532affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.449+0000 7f532b7fe640 10 monclient(hunting): get_auth_request con 0x7f532c145980 auth_method 0 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.449+0000 7f532b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:33.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.449+0000 7f532b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:33.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.449+0000 7f532b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:33.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:33.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:33.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:33.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532affd640 10 monclient(hunting): handle_auth_done global_id 5835 payload 274 2024-01-20T11:22:33.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532affd640 1 -- 172.21.15.110:0/3789357737 >> v2:172.21.15.134:3300/0 conn(0x7f532c145980 msgr2=0x7f532c145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:33.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532affd640 1 --2- 172.21.15.110:0/3789357737 >> v2:172.21.15.134:3300/0 conn(0x7f532c145980 0x7f532c145e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532affd640 1 -- 172.21.15.110:0/3789357737 >> v1:172.21.15.134:6789/0 conn(0x7f532c0b17c0 legacy=0x7f532c0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:33.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532affd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:33.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532affd640 1 monclient: found mon.noname-a 2024-01-20T11:22:33.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:33.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:33.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.450+0000 7f532affd640 1 -- 172.21.15.110:0/3789357737 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c145460 con 0x7f532c0b4d10 2024-01-20T11:22:33.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.452+0000 7f532affd640 1 --2- 172.21.15.110:0/3789357737 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 0x7f532c0b3da0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f532c0aa1c0 tx=0x7f532402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5a33d26e9b26ea server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 1 -- 172.21.15.110:0/3789357737 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532402c9f0 con 0x7f532c0b4d10 2024-01-20T11:22:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 10 monclient: dump: 2024-01-20T11:22:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:33.458 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 10 monclient: _finish_auth 0 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 1 -- 172.21.15.110:0/3789357737 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532402cb50 con 0x7f532c0b4d10 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5331341640 5 monclient: authenticate success, global_id 5835 2024-01-20T11:22:33.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5331341640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5331341640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5331341640 4 set_mon_vals no callback set 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 1 -- 172.21.15.110:0/3789357737 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53240343f0 con 0x7f532c0b4d10 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.456+0000 7f5329ffb640 10 monclient: dump: 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:33.460 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 10 monclient: shutdown 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 1 -- 172.21.15.110:0/3789357737 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 msgr2=0x7f532c0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 1 --2- 172.21.15.110:0/3789357737 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 0x7f532c0b3da0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f532c0aa1c0 tx=0x7f532402be50 comp rx=0 tx=0).stop 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 1 -- 172.21.15.110:0/3789357737 shutdown_connections 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 1 --2- 172.21.15.110:0/3789357737 >> v2:172.21.15.134:3300/0 conn(0x7f532c145980 0x7f532c145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 1 --2- 172.21.15.110:0/3789357737 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 0x7f532c0b3da0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 1 -- 172.21.15.110:0/3789357737 >> 172.21.15.110:0/3789357737 conn(0x7f532c066590 msgr2=0x7f532c0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 1 -- 172.21.15.110:0/3789357737 shutdown_connections 2024-01-20T11:22:33.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.457+0000 7f5331341640 1 -- 172.21.15.110:0/3789357737 wait complete. 2024-01-20T11:22:33.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.458+0000 7f5331341640 10 monclient: build_initial_monmap 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.458+0000 7f5331341640 10 monclient: monmap: 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.458+0000 7f5331341640 1 librados: starting msgr at 2024-01-20T11:22:33.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.458+0000 7f5331341640 1 librados: starting objecter 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.458+0000 7f5331341640 1 Processor -- start 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.458+0000 7f5331341640 1 -- start start 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.458+0000 7f5331341640 1 librados: setting wanted keys 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.458+0000 7f5331341640 1 librados: calling monclient init 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.458+0000 7f5331341640 10 monclient: init 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f532c0b17c0 0x7f532c146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient(hunting): picked mon.noname-b con 0x7f532c0b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 0x7f532c140280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient(hunting): picked mon.noname-a con 0x7f532c0b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c145980 0x7f532c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient(hunting): picked mon.noname-c con 0x7f532c145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:33.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c147fe0 con 0x7f532c0b4d10 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1482c0 con 0x7f532c0b17c0 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1470d0 con 0x7f532c145980 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f5331341640 10 monclient(hunting): authenticate will time out at 1796.810648s 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.459+0000 7f532a7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 0x7f532c140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c145980 0x7f532c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532affd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f532c0b17c0 0x7f532c146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532a7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 0x7f532c140280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47620/0 (socket says 172.21.15.110:47620) 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532a7fc640 1 -- 172.21.15.110:0/3770349281 learned_addr learned my addr 172.21.15.110:0/3770349281 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532a7fc640 10 monclient(hunting): get_auth_request con 0x7f532c0b4d10 auth_method 0 2024-01-20T11:22:33.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532b7fe640 10 monclient(hunting): get_auth_request con 0x7f532c145980 auth_method 0 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532affd640 10 monclient(hunting): get_auth_request con 0x7f532c0b17c0 auth_method 0 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.460+0000 7f532a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:33.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.461+0000 7f532b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.461+0000 7f532b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.461+0000 7f532b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.461+0000 7f532affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.461+0000 7f532affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.461+0000 7f532affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 10 monclient(hunting): handle_auth_done global_id 5838 payload 824 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 1 -- 172.21.15.110:0/3770349281 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c145980 msgr2=0x7f532c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c145980 0x7f532c1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 1 -- 172.21.15.110:0/3770349281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f532c0b17c0 msgr2=0x7f532c146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f532c0b17c0 0x7f532c146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 10 monclient: _finish_hunting 0 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 1 monclient: found mon.noname-a 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:33.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:33.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.463+0000 7f532a7fc640 1 -- 172.21.15.110:0/3770349281 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5324009b30 con 0x7f532c0b4d10 2024-01-20T11:22:33.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.466+0000 7f532a7fc640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 0x7f532c140280 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f532000c440 tx=0x7f532000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.466+0000 7f53137fe640 10 client.?.objecter ms_handle_connect 0x7f532c0b4d10 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.466+0000 7f53137fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 1 -- 172.21.15.110:0/3770349281 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53200052d0 con 0x7f532c0b4d10 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 10 monclient: dump: 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:33.471 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 1 -- 172.21.15.110:0/3770349281 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5320005430 con 0x7f532c0b4d10 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 1 -- 172.21.15.110:0/3770349281 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320007260 con 0x7f532c0b4d10 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53137fe640 10 monclient: dump: 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:33.472 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f5331341640 5 monclient: authenticate success, global_id 5838 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f5331341640 10 monclient: _renew_subs 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f5331341640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c146d40 con 0x7f532c0b4d10 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f53297fa640 4 set_mon_vals no callback set 2024-01-20T11:22:33.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f5331341640 10 client.5838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:33.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f5331341640 10 monclient: _renew_subs 2024-01-20T11:22:33.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f5331341640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:33.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c142fe0 con 0x7f532c0b4d10 2024-01-20T11:22:33.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.469+0000 7f5331341640 1 librados: init done 2024-01-20T11:22:33.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.470+0000 7f5331341640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:33.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.470+0000 7f5331341640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:33.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.470+0000 7f5331341640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:33.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.470+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532c142b20 con 0x7f532c0b4d10 2024-01-20T11:22:33.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.489+0000 7f53137fe640 1 -- 172.21.15.110:0/3770349281 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f53200073c0 con 0x7f532c0b4d10 2024-01-20T11:22:33.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.489+0000 7f53137fe640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f531806c8e0 0x7f531806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:33.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.490+0000 7f532affd640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f531806c8e0 0x7f531806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.490+0000 7f532affd640 10 monclient: get_auth_request con 0x7f531806c8e0 auth_method 0 2024-01-20T11:22:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.490+0000 7f532affd640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f531806c8e0 0x7f531806eda0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5324035000 tx=0x7f5324030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:33.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.490+0000 7f53137fe640 10 client.5838.objecter ms_handle_connect 0x7f531806c8e0 2024-01-20T11:22:33.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.723+0000 7f53137fe640 1 -- 172.21.15.110:0/3770349281 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5320007670 con 0x7f532c0b4d10 2024-01-20T11:22:33.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.723+0000 7f53137fe640 10 client.5838.objecter ms_dispatch 0x7f532c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:33.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.723+0000 7f53137fe640 3 client.5838.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.723+0000 7f53137fe640 3 client.5838.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.723+0000 7f53137fe640 20 client.5838.objecter dump_active .. 0 homeless 2024-01-20T11:22:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.723+0000 7f53137fe640 1 -- 172.21.15.110:0/3770349281 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5320054030 con 0x7f532c0b4d10 2024-01-20T11:22:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.723+0000 7f53137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.723+0000 7f53137fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:33.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.902+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f532c14c7f0 con 0x7f531806c8e0 2024-01-20T11:22:33.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.905+0000 7f53137fe640 1 -- 172.21.15.110:0/3770349281 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63890 (secure 0 0 0) 0x7f532403f440 con 0x7f531806c8e0 2024-01-20T11:22:33.906 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:33.907 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 10 librados: watch_flush enter 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 10 librados: watch_flush exit 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 20 client.5838.objecter shutdown clearing up homeless session... 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 10 client.5838.objecter successfully canceled tick 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f531806c8e0 msgr2=0x7f531806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f531806c8e0 0x7f531806eda0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5324035000 tx=0x7f5324030000 comp rx=0 tx=0).stop 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 10 monclient: shutdown 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 msgr2=0x7f532c140280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 0x7f532c140280 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f532000c440 tx=0x7f532000c900 comp rx=0 tx=0).stop 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 shutdown_connections 2024-01-20T11:22:33.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f531806c8e0 0x7f531806eda0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c145980 0x7f532c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c0b4d10 0x7f532c140280 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 --2- 172.21.15.110:0/3770349281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f532c0b17c0 0x7f532c146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 >> 172.21.15.110:0/3770349281 conn(0x7f532c066590 msgr2=0x7f532c144bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 shutdown_connections 2024-01-20T11:22:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 -- 172.21.15.110:0/3770349281 wait complete. 2024-01-20T11:22:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:33.907+0000 7f5331341640 1 librados: shutdown 2024-01-20T11:22:33.919 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-01-20T11:22:32.798854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002537"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3320000000000007,"5min":7.8490000000000002,"15min":12.856},"min":{"1min":3.444,"5min":3.444,"15min":3.444},"max":{"1min":17.597999999999999,"5min":21.064,"15min":100.05800000000001},"last":6.6449999999999996},{"interface":"front","average":{"1min":8.0310000000000006,"5min":8.234,"15min":13.922000000000001},"min":{"1min":4.8849999999999998,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":15.567,"5min":24.245000000000001,"15min":91.789000000000001},"last":5.0670000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5939999999999994,"5min":10.131,"15min":15.615},"min":{"1min":6.4560000000000004,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.071999999999999,"5min":26.027000000000001,"15min":98.700000000000003},"last":10.723000000000001},{"interface":"front","average":{"1min":9.5939999999999994,"5min":10.465,"15min":14.859},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":17.390999999999998,"5min":22.533999999999999,"15min":95.603999999999999},"last":7.5999999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":11.486000000000001,"5min":11.699999999999999,"15min":16.780999999999999},"min":{"1min":7.125,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":16.091000000000001,"5min":16.492999999999999,"15min":131.09700000000001},"last":11.523999999999999},{"interface":"front","average":{"1min":12.319000000000001,"5min":12.630000000000001,"15min":16.359999999999999},"min":{"1min":8.0280000000000005,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":18.013000000000002,"5min":28.405999999999999,"15min":105.43899999999999},"last":12.483000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":8.9550000000000001,"5min":11.041,"15min":18.286000000000001},"min":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":17.98,"5min":55.868000000000002,"15min":94.911000000000001},"last":13.339},{"interface":"front","average":{"1min":12.926,"5min":13.813000000000001,"15min":20.337},"min":{"1min":7.6769999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"max":{"1min":17.672000000000001,"5min":52.960000000000001,"15min":100.633},"last":9.6259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":13.295,"5min":14.619999999999999,"15min":20.215},"min":{"1min":9.6020000000000003,"5min":6.8070000000000004,"15min":6.8070000000000004},"max":{"1min":17.577000000000002,"5min":30,"15min":131.70599999999999},"last":13.968},{"interface":"front","average":{"1min":12.083,"5min":12.960000000000001,"15min":18.829000000000001},"min":{"1min":7.9809999999999999,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.911000000000001,"5min":23.395,"15min":107.962},"last":14.589}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":10.811999999999999,"5min":11.91,"15min":17.491},"min":{"1min":7.0190000000000001,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":19.302,"5min":29.283999999999999,"15min":96.911000000000001},"last":8.9209999999999994},{"interface":"front","average":{"1min":14.345000000000001,"5min":15.727,"15min":21.106000000000002},"min":{"1min":9.157,"5min":9.157,"15min":9.157},"max":{"1min":16.972000000000001,"5min":29.736999999999998,"15min":129.828},"last":15.818}]}]},{"osd":7,"up_from":21,"seq":90194313308,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":7.2050000000000001,"5min":7.6619999999999999,"15min":9.4760000000000009},"min":{"1min":4.2930000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":11.484999999999999,"5min":15.970000000000001,"15min":38.005000000000003},"last":5},{"interface":"front","average":{"1min":8.3010000000000002,"5min":8.9250000000000007,"15min":12.347},"min":{"1min":4.6950000000000003,"5min":3.9289999999999998,"15min":3.9289999999999998},"max":{"1min":14.521000000000001,"5min":16.138999999999999,"15min":76.718999999999994},"last":11.349}]},{"osd":1,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":8.2050000000000001,"5min":8.8369999999999997,"15min":10.047000000000001},"min":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":12.863,"5min":16.992000000000001,"15min":18.859000000000002},"last":12.183999999999999},{"interface":"front","average":{"1min":10.525,"5min":10.904999999999999,"15min":12.631},"min":{"1min":6.4409999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"max":{"1min":15.563000000000001,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.4459999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.715,"5min":12.081,"15min":15.691000000000001},"min":{"1min":7.8559999999999999,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":16.603999999999999,"5min":18.154,"15min":49.210000000000001},"last":10.105},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.132,"15min":15.151},"min":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":17.547999999999998,"5min":19.795999999999999,"15min":78.548000000000002},"last":13.249000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":10.791,"5min":11.33,"15min":14.087999999999999},"min":{"1min":6.992,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":17,"5min":18.934999999999999,"15min":55.771000000000001},"last":8.5299999999999994},{"interface":"front","average":{"1min":13.085000000000001,"5min":13.967000000000001,"15min":16.279},"min":{"1min":8.9600000000000009,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":17.478000000000002,"5min":32.773000000000003,"15min":49.829999999999998},"last":10.843}]},{"osd":4,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":11.467000000000001,"5min":12.744999999999999,"15min":16.405999999999999},"min":{"1min":6.7960000000000003,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":16.260000000000002,"5min":27.498000000000001,"15min":91.667000000000002},"last":16.306000000000001},{"interface":"front","average":{"1min":12.199999999999999,"5min":12.287000000000001,"15min":16.306999999999999},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.858000000000001,"5min":20.239999999999998,"15min":92.613},"last":15.391999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:21:34 2024","interfaces":[{"interface":"back","average":{"1min":13.019,"5min":13.861000000000001,"15min":17.463000000000001},"min":{"1min":9.1310000000000002,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.478000000000002,"5min":29.834,"15min":80.102000000000004},"last":17.036000000000001},{"interface":"front","average":{"1min":13.206,"5min":14.548,"15min":17.855},"min":{"1min":9.1880000000000006,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.960000000000001,"5min":25.001999999999999,"15min":53.398000000000003},"last":17.707000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313309,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.46},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":10.894}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":11.667999999999999},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":15.039999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":12.372},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":7.5910000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":14.298},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":8.4800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":9.0690000000000008},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":13.709}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":13.089},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":9.7739999999999991}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444122,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":10.978999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":9.4740000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":14.817},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.343}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.012},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":7.9550000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":13.605},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.416}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":14.202999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":12.1}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":16.951000000000001},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":15.452999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444123,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":7.7069999999999999,"5min":8.1549999999999994,"15min":9.2110000000000003},"min":{"1min":4.3680000000000003,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.042,"5min":18.547000000000001,"15min":18.547000000000001},"last":8.6379999999999999},{"interface":"front","average":{"1min":9.2789999999999999,"5min":8.8369999999999997,"15min":10.009},"min":{"1min":5.4809999999999999,"5min":5.1769999999999996,"15min":5.1150000000000002},"max":{"1min":15.800000000000001,"5min":16.992999999999999,"15min":30.439},"last":6.5519999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.532999999999999,"5min":11.273,"15min":12.718999999999999},"min":{"1min":5.6920000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":17.009,"5min":23.077999999999999,"15min":24.725000000000001},"last":12.215999999999999},{"interface":"front","average":{"1min":9.9990000000000006,"5min":10.662000000000001,"15min":12.619999999999999},"min":{"1min":5.4969999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.723000000000001,"5min":22.175999999999998,"15min":23.981999999999999},"last":13.113}]},{"osd":3,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":13.071,"15min":22.376999999999999},"min":{"1min":6.6479999999999997,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":21.178999999999998,"5min":31.931000000000001,"15min":259.483},"last":11.416},{"interface":"front","average":{"1min":9.3719999999999999,"5min":9.7590000000000003,"15min":17.678999999999998},"min":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.689,"5min":31.634,"15min":243.24199999999999},"last":5.8220000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":9.6609999999999996},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.743}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:21:24 2024","interfaces":[{"interface":"back","average":{"1min":10.599,"5min":10.91,"15min":11.824999999999999},"min":{"1min":6.1609999999999996,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.571000000000002,"5min":27.233000000000001,"15min":27.233000000000001},"last":7.9809999999999999},{"interface":"front","average":{"1min":13.35,"5min":13.808,"15min":15.122999999999999},"min":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":21.553000000000001,"5min":23.158000000000001,"15min":31.634},"last":15.612}]},{"osd":7,"last update":"Sat Jan 20 11:21:29 2024","interfaces":[{"interface":"back","average":{"1min":13.214,"5min":12.662000000000001,"15min":25.738},"min":{"1min":8.3710000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.175999999999998,"5min":28.774000000000001,"15min":344.98099999999999},"last":10.340999999999999},{"interface":"front","average":{"1min":14.756,"5min":15.06,"15min":26.710000000000001},"min":{"1min":9.6150000000000002,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":22.863,"5min":26.094000000000001,"15min":313.45999999999998},"last":14.494}]}]},{"osd":0,"up_from":17,"seq":73014444123,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.2430000000000003},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":11.881}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":6.3170000000000002},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":8.3059999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":9.3350000000000009},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":12.521000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":13.143000000000001},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":16.202000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":10.958},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":13.712}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.398999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":17.552}]}]},{"osd":6,"up_from":15,"seq":64424509532,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":12.244},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":5.5709999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":6.4690000000000003},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":9.9819999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":10.789},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":9.3190000000000008}]},{"osd":3,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":12.234,"5min":12.711,"15min":15.093999999999999},"min":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.699000000000002,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.885},{"interface":"front","average":{"1min":11.683,"5min":12.481999999999999,"15min":15.279999999999999},"min":{"1min":6.9240000000000004,"5min":6.8700000000000001,"15min":6.6900000000000004},"max":{"1min":16.468,"5min":19.268999999999998,"15min":84.524000000000001},"last":15.117000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":11.615},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":12.933999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:21:28 2024","interfaces":[{"interface":"back","average":{"1min":13.377000000000001,"5min":13.904999999999999,"15min":15.804},"min":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.521999999999998,"5min":18.527999999999999,"15min":25.303999999999998},"last":13.638999999999999},{"interface":"front","average":{"1min":12.75,"5min":13.504,"15min":15.316000000000001},"min":{"1min":7.4889999999999999,"5min":7.149,"15min":7.149},"max":{"1min":18.591000000000001,"5min":19.515000000000001,"15min":22.838999999999999},"last":8.7219999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:35.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~217s 2024-01-20T11:22:36.922 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.021+0000 7f40ce239640 10 monclient: get_monmap_and_config 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.021+0000 7f40ce239640 10 monclient: build_initial_monmap 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.021+0000 7f40ce239640 10 monclient: monmap: 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:37.022755+0000 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:37.022755+0000 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:37.022 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:37.023 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:37.023 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:37.023 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:37.023 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.021+0000 7f40ce239640 1 Processor -- start 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.021+0000 7f40ce239640 1 -- start start 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.021+0000 7f40ce239640 10 monclient: init 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f40c80a96f0 0x7f40c813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 10 monclient(hunting): picked mon.noname-c con 0x7f40c80a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40c813fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c8142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 10 monclient(hunting): picked mon.noname-b con 0x7f40c80a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c80b25c0 con 0x7f40c813fa10 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c80aaea0 con 0x7f40c80a96f0 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80ab0a0 con 0x7f40c80a9ad0 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.022+0000 7f40ce239640 10 monclient(hunting): authenticate will time out at 1800.373723s 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.023+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c8142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.023+0000 7f40c77fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f40c80a96f0 0x7f40c813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.023+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c8142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46980/0 (socket says 172.21.15.110:46980) 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.023+0000 7f40c7fff640 1 -- 172.21.15.110:0/628826468 learned_addr learned my addr 172.21.15.110:0/628826468 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.023+0000 7f40c7fff640 10 monclient(hunting): get_auth_request con 0x7f40c80a9ad0 auth_method 0 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.023+0000 7f40c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.023+0000 7f40c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.023+0000 7f40c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.024+0000 7f40c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.024+0000 7f40c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.024+0000 7f40c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.024+0000 7f40c77fe640 10 monclient(hunting): get_auth_request con 0x7f40c80a96f0 auth_method 0 2024-01-20T11:22:37.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.024+0000 7f40c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.024+0000 7f40c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.024+0000 7f40c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c7fff640 10 monclient(hunting): handle_auth_done global_id 5507 payload 274 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c7fff640 1 -- 172.21.15.110:0/628826468 >> v2:172.21.15.134:3300/0 conn(0x7f40c80a96f0 msgr2=0x7f40c813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c7fff640 1 --2- 172.21.15.110:0/628826468 >> v2:172.21.15.134:3300/0 conn(0x7f40c80a96f0 0x7f40c813f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c7fff640 1 -- 172.21.15.110:0/628826468 >> v1:172.21.15.134:6789/0 conn(0x7f40c813fa10 legacy=0x7f40c81423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c7fff640 1 monclient: found mon.noname-b 2024-01-20T11:22:37.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:37.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:37.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.025+0000 7f40c7fff640 1 -- 172.21.15.110:0/628826468 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c80b3fb0 con 0x7f40c80a9ad0 2024-01-20T11:22:37.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.027+0000 7f40c7fff640 1 --2- 172.21.15.110:0/628826468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c8142af0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009e50 tx=0x7f40c002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35a5505619e5f52f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 1 -- 172.21.15.110:0/628826468 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c002ca70 con 0x7f40c80a9ad0 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 10 monclient: dump: 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:37.032 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 10 monclient: _finish_auth 0 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 1 -- 172.21.15.110:0/628826468 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c002cbd0 con 0x7f40c80a9ad0 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 5 monclient: authenticate success, global_id 5507 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 1 -- 172.21.15.110:0/628826468 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c00343f0 con 0x7f40c80a9ad0 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 4 set_mon_vals no callback set 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40c67fc640 10 monclient: dump: 2024-01-20T11:22:37.033 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:37.034 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.034 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.034 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.034 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:37.034 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.034 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:37.034 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:37.034 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:37.034 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 10 monclient: shutdown 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 1 -- 172.21.15.110:0/628826468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 msgr2=0x7f40c8142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 1 --2- 172.21.15.110:0/628826468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c8142af0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009e50 tx=0x7f40c002bed0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 1 -- 172.21.15.110:0/628826468 shutdown_connections 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 1 --2- 172.21.15.110:0/628826468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c8142af0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 1 --2- 172.21.15.110:0/628826468 >> v2:172.21.15.134:3300/0 conn(0x7f40c80a96f0 0x7f40c813f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 1 -- 172.21.15.110:0/628826468 >> 172.21.15.110:0/628826468 conn(0x7f40c80b7bc0 msgr2=0x7f40c80b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 1 -- 172.21.15.110:0/628826468 shutdown_connections 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.031+0000 7f40ce239640 1 -- 172.21.15.110:0/628826468 wait complete. 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.032+0000 7f40ce239640 10 monclient: build_initial_monmap 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.032+0000 7f40ce239640 10 monclient: monmap: 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:37.038 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.032+0000 7f40ce239640 1 librados: starting msgr at 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.032+0000 7f40ce239640 1 librados: starting objecter 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.032+0000 7f40ce239640 1 Processor -- start 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 1 -- start start 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 1 librados: setting wanted keys 2024-01-20T11:22:37.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 1 librados: calling monclient init 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient: init 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40c80a96f0 0x7f40c8132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient(hunting): picked mon.noname-a con 0x7f40c80a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c81320b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient(hunting): picked mon.noname-c con 0x7f40c80a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40c813fe00 0x7f40c8140250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient(hunting): picked mon.noname-b con 0x7f40c813fe00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c8132f60 con 0x7f40c80a96f0 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c81325f0 con 0x7f40c813fe00 2024-01-20T11:22:37.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c81336b0 con 0x7f40c80a9ad0 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40ce239640 10 monclient(hunting): authenticate will time out at 1800.384750s 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.033+0000 7f40c6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c81320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40c80a96f0 0x7f40c8132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c81320b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46996/0 (socket says 172.21.15.110:46996) 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c6ffd640 1 -- 172.21.15.110:0/3010453251 learned_addr learned my addr 172.21.15.110:0/3010453251 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c6ffd640 10 monclient(hunting): get_auth_request con 0x7f40c80a9ad0 auth_method 0 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c77fe640 10 monclient(hunting): get_auth_request con 0x7f40c80a96f0 auth_method 0 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c7fff640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40c813fe00 0x7f40c8140250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.034+0000 7f40c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.035+0000 7f40c7fff640 10 monclient(hunting): get_auth_request con 0x7f40c813fe00 auth_method 0 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.035+0000 7f40c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.035+0000 7f40c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.035+0000 7f40c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.035+0000 7f40c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.035+0000 7f40c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.035+0000 7f40c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 10 monclient(hunting): handle_auth_done global_id 5841 payload 824 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 1 -- 172.21.15.110:0/3010453251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 msgr2=0x7f40c81320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c81320b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 1 -- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40c813fe00 msgr2=0x7f40c8140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40c813fe00 0x7f40c8140250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 1 monclient: found mon.noname-a 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:37.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.037+0000 7f40c77fe640 1 -- 172.21.15.110:0/3010453251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c0009b30 con 0x7f40c80a96f0 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.039+0000 7f40c77fe640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40c80a96f0 0x7f40c8132a20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f40b800e850 tx=0x7f40b800ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.039+0000 7f40affff640 10 client.?.objecter ms_handle_connect 0x7f40c80a96f0 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.039+0000 7f40affff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 1 -- 172.21.15.110:0/3010453251 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b80095b0 con 0x7f40c80a96f0 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 10 monclient: dump: 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:37.046 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 10 monclient: _finish_auth 0 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 10 monclient: _check_auth_tickets 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 1 -- 172.21.15.110:0/3010453251 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b800c8e0 con 0x7f40c80a96f0 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 1 -- 172.21.15.110:0/3010453251 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b800cc70 con 0x7f40c80a96f0 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40ce239640 5 monclient: authenticate success, global_id 5841 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40affff640 10 monclient: dump: 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:37.047 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40ce239640 10 monclient: _renew_subs 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40ce239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c81343f0 con 0x7f40c80a96f0 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40c5ffb640 4 set_mon_vals no callback set 2024-01-20T11:22:37.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40ce239640 10 client.5841.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:37.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40ce239640 10 monclient: _renew_subs 2024-01-20T11:22:37.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40ce239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:37.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c8140970 con 0x7f40c80a96f0 2024-01-20T11:22:37.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.043+0000 7f40ce239640 1 librados: init done 2024-01-20T11:22:37.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.044+0000 7f40ce239640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:37.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.044+0000 7f40ce239640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:37.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.044+0000 7f40ce239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:37.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.044+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c8075490 con 0x7f40c80a96f0 2024-01-20T11:22:37.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.061+0000 7f40affff640 1 -- 172.21.15.110:0/3010453251 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f40b8013470 con 0x7f40c80a96f0 2024-01-20T11:22:37.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.062+0000 7f40affff640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b406c860 0x7f40b406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.062+0000 7f40c6ffd640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b406c860 0x7f40b406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.062+0000 7f40c6ffd640 10 monclient: get_auth_request con 0x7f40b406c860 auth_method 0 2024-01-20T11:22:37.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.063+0000 7f40affff640 10 client.5841.objecter ms_handle_connect 0x7f40b406c860 2024-01-20T11:22:37.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.063+0000 7f40c6ffd640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b406c860 0x7f40b406ed20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f40c80b7450 tx=0x7f40bc009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:37.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.064+0000 7f40affff640 1 -- 172.21.15.110:0/3010453251 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f40b8058ae0 con 0x7f40c80a96f0 2024-01-20T11:22:37.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.064+0000 7f40affff640 10 client.5841.objecter ms_dispatch 0x7f40c8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:37.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.064+0000 7f40affff640 3 client.5841.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:37.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.064+0000 7f40affff640 3 client.5841.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:37.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.064+0000 7f40affff640 20 client.5841.objecter dump_active .. 0 homeless 2024-01-20T11:22:37.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.297+0000 7f40affff640 1 -- 172.21.15.110:0/3010453251 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f40b8054e20 con 0x7f40c80a96f0 2024-01-20T11:22:37.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.297+0000 7f40affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:37.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.297+0000 7f40affff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:37.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.482+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40c80b6320 con 0x7f40b406c860 2024-01-20T11:22:37.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.485+0000 7f40affff640 1 -- 172.21.15.110:0/3010453251 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63665 (secure 0 0 0) 0x7f40c80b6320 con 0x7f40b406c860 2024-01-20T11:22:37.485 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:37.486 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.486+0000 7f40ce239640 10 librados: watch_flush enter 2024-01-20T11:22:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.486+0000 7f40ce239640 10 librados: watch_flush exit 2024-01-20T11:22:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.486+0000 7f40ce239640 20 client.5841.objecter shutdown clearing up homeless session... 2024-01-20T11:22:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.486+0000 7f40ce239640 10 client.5841.objecter successfully canceled tick 2024-01-20T11:22:37.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b406c860 msgr2=0x7f40b406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b406c860 0x7f40b406ed20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f40c80b7450 tx=0x7f40bc009040 comp rx=0 tx=0).stop 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 10 monclient: shutdown 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40c80a96f0 msgr2=0x7f40c8132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40c80a96f0 0x7f40c8132a20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f40b800e850 tx=0x7f40b800ed10 comp rx=0 tx=0).stop 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 shutdown_connections 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f40b406c860 0x7f40b406ed20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40c813fe00 0x7f40c8140250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40c80a9ad0 0x7f40c81320b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 --2- 172.21.15.110:0/3010453251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40c80a96f0 0x7f40c8132a20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 >> 172.21.15.110:0/3010453251 conn(0x7f40c80b7bc0 msgr2=0x7f40c80b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 shutdown_connections 2024-01-20T11:22:37.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 -- 172.21.15.110:0/3010453251 wait complete. 2024-01-20T11:22:37.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.487+0000 7f40ce239640 1 librados: shutdown 2024-01-20T11:22:37.497 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-01-20T11:22:36.799829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002617"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280606,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":6.5979999999999999},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":4.9219999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.0340000000000007},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":11.183999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":10.111000000000001},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":13.084}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":15.163},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":14.25}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":12.356999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":18.001000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":16.733000000000001},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":16.021999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313309,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":4.8499999999999996},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.1859999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":11.275},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":5.8860000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":11.986000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":10.460000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":12.727},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":9.3059999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":13.391999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":16.303999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":17.32},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.281000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313309,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.46},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":10.894}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":11.667999999999999},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":15.039999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":12.372},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":7.5910000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":14.298},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":8.4800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":9.0690000000000008},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":13.709}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":13.089},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":9.7739999999999991}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444123,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.9299999999999997},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":8.8439999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":10.109},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.5169999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":15.56},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":11.356}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.544},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":12.606}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":11.929},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.5869999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":13.836},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.376000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444124,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":12.603},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":10.564}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.4649999999999999},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":14.619}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":9.4830000000000005},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":15.794}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":6.702},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.231}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":8.266},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.343999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":8.9209999999999994},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":17.007999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444124,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.5910000000000002},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.8040000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":6.2469999999999999},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":9.4649999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":10.135},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":7.4130000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":14.308},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":13.715}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":11.327999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":11.975}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":12.579000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":15.042}]}]},{"osd":6,"up_from":15,"seq":64424509533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.172000000000001},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":4.4900000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.6859999999999999},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":10.984}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.178000000000001},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":9.5030000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.853},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":8.7390000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":15.629},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":14.880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":16.446999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.478}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:37.499 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:37.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.599+0000 7f5c7c1e2640 10 monclient: get_monmap_and_config 2024-01-20T11:22:37.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.599+0000 7f5c7c1e2640 10 monclient: build_initial_monmap 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.599+0000 7f5c7c1e2640 10 monclient: monmap: 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:37.600571+0000 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:37.600571+0000 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:37.600 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.600+0000 7f5c7c1e2640 1 Processor -- start 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.600+0000 7f5c7c1e2640 1 -- start start 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.600+0000 7f5c7c1e2640 10 monclient: init 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c74145760 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c740a9d80 0x7f5c74145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5c740a9d80 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5c7413f8b0 0x7f5c740b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5c7413f8b0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c74146250 con 0x7f5c74145760 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c740ab050 con 0x7f5c740a9d80 2024-01-20T11:22:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c740b20c0 con 0x7f5c7413f8b0 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7c1e2640 10 monclient(hunting): authenticate will time out at 1800.951634s 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.601+0000 7f5c7977d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c740a9d80 0x7f5c74145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7a77f640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5c7413f8b0 0x7f5c740b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7977d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c740a9d80 0x7f5c74145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47646/0 (socket says 172.21.15.110:47646) 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7977d640 1 -- 172.21.15.110:0/1659512907 learned_addr learned my addr 172.21.15.110:0/1659512907 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7977d640 10 monclient(hunting): get_auth_request con 0x7f5c740a9d80 auth_method 0 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7977d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7977d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7977d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7977d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7977d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7977d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7a77f640 10 monclient(hunting): get_auth_request con 0x7f5c7413f8b0 auth_method 0 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7a77f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7a77f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.602+0000 7f5c7a77f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.603+0000 7f5c7a77f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.603+0000 7f5c7a77f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.603+0000 7f5c7a77f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.604+0000 7f5c7977d640 10 monclient(hunting): handle_auth_done global_id 5844 payload 274 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.604+0000 7f5c7977d640 1 -- 172.21.15.110:0/1659512907 >> v2:172.21.15.134:3300/0 conn(0x7f5c7413f8b0 msgr2=0x7f5c740b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.604+0000 7f5c7977d640 1 --2- 172.21.15.110:0/1659512907 >> v2:172.21.15.134:3300/0 conn(0x7f5c7413f8b0 0x7f5c740b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.604+0000 7f5c7977d640 1 -- 172.21.15.110:0/1659512907 >> v1:172.21.15.134:6789/0 conn(0x7f5c74145760 legacy=0x7f5c7413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.604+0000 7f5c7977d640 10 monclient: _finish_hunting 0 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.604+0000 7f5c7977d640 1 monclient: found mon.noname-a 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.604+0000 7f5c7977d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.604+0000 7f5c7977d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:37.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.604+0000 7f5c7977d640 1 -- 172.21.15.110:0/1659512907 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c740b2bb0 con 0x7f5c740a9d80 2024-01-20T11:22:37.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.606+0000 7f5c7977d640 1 --2- 172.21.15.110:0/1659512907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c740a9d80 0x7f5c74145b40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5c68009e50 tx=0x7f5c6802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418cb232f1dca3ee server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:37.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 1 -- 172.21.15.110:0/1659512907 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6802cb20 con 0x7f5c740a9d80 2024-01-20T11:22:37.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:37.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:37.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 10 monclient: dump: 2024-01-20T11:22:37.610 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 10 monclient: _finish_auth 0 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 10 monclient: _check_auth_tickets 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 1 -- 172.21.15.110:0/1659512907 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6802cc80 con 0x7f5c740a9d80 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:37.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 1 -- 172.21.15.110:0/1659512907 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680343f0 con 0x7f5c740a9d80 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c78f7c640 10 monclient: dump: 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c7c1e2640 5 monclient: authenticate success, global_id 5844 2024-01-20T11:22:37.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c7c1e2640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:37.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.610+0000 7f5c7c1e2640 4 set_mon_vals no callback set 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 10 monclient: shutdown 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/1659512907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c740a9d80 msgr2=0x7f5c74145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 1 --2- 172.21.15.110:0/1659512907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c740a9d80 0x7f5c74145b40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5c68009e50 tx=0x7f5c6802bf80 comp rx=0 tx=0).stop 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/1659512907 shutdown_connections 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 1 --2- 172.21.15.110:0/1659512907 >> v2:172.21.15.134:3300/0 conn(0x7f5c7413f8b0 0x7f5c740b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 1 --2- 172.21.15.110:0/1659512907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c740a9d80 0x7f5c74145b40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/1659512907 >> 172.21.15.110:0/1659512907 conn(0x7f5c740663b0 msgr2=0x7f5c740667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/1659512907 shutdown_connections 2024-01-20T11:22:37.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.611+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/1659512907 wait complete. 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.612+0000 7f5c7c1e2640 10 monclient: build_initial_monmap 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.612+0000 7f5c7c1e2640 10 monclient: monmap: 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.612+0000 7f5c7c1e2640 1 librados: starting msgr at 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.612+0000 7f5c7c1e2640 1 librados: starting objecter 2024-01-20T11:22:37.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.612+0000 7f5c7c1e2640 1 Processor -- start 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.612+0000 7f5c7c1e2640 1 -- start start 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.612+0000 7f5c7c1e2640 1 librados: setting wanted keys 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.612+0000 7f5c7c1e2640 1 librados: calling monclient init 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.612+0000 7f5c7c1e2640 10 monclient: init 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c740a9d80 0x7f5c74136640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5c740a9d80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c7413f8b0 0x7f5c74141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5c7413f8b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c74145760 0x7f5c741420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5c74145760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74138ac0 con 0x7f5c7413f8b0 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74138d30 con 0x7f5c740a9d80 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74142ce0 con 0x7f5c74145760 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7c1e2640 10 monclient(hunting): authenticate will time out at 1800.963529s 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7977d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c7413f8b0 0x7f5c74141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7977d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c7413f8b0 0x7f5c74141580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47658/0 (socket says 172.21.15.110:47658) 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7977d640 1 -- 172.21.15.110:0/4265448026 learned_addr learned my addr 172.21.15.110:0/4265448026 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7977d640 10 monclient(hunting): get_auth_request con 0x7f5c7413f8b0 auth_method 0 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7977d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7977d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7977d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.613+0000 7f5c7a77f640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c74145760 0x7f5c741420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c79f7e640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c740a9d80 0x7f5c74136640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7a77f640 10 monclient(hunting): get_auth_request con 0x7f5c74145760 auth_method 0 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7a77f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7a77f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7a77f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7977d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7977d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7977d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c79f7e640 10 monclient(hunting): get_auth_request con 0x7f5c740a9d80 auth_method 0 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c79f7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c79f7e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c79f7e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7a77f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7a77f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.614+0000 7f5c7a77f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.615+0000 7f5c79f7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.615+0000 7f5c79f7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.615+0000 7f5c79f7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 10 monclient(hunting): handle_auth_done global_id 5847 payload 824 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 1 -- 172.21.15.110:0/4265448026 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c74145760 msgr2=0x7f5c741420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c74145760 0x7f5c741420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 1 -- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c740a9d80 msgr2=0x7f5c74136640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c740a9d80 0x7f5c74136640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 10 monclient: _finish_hunting 0 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 1 monclient: found mon.noname-a 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:37.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.617+0000 7f5c7977d640 1 -- 172.21.15.110:0/4265448026 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c68009b30 con 0x7f5c7413f8b0 2024-01-20T11:22:37.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.619+0000 7f5c7977d640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c7413f8b0 0x7f5c74141580 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5c68000f80 tx=0x7f5c68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:37.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.619+0000 7f5c727fc640 10 client.?.objecter ms_handle_connect 0x7f5c7413f8b0 2024-01-20T11:22:37.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.619+0000 7f5c727fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:37.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.622+0000 7f5c727fc640 1 -- 172.21.15.110:0/4265448026 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68009510 con 0x7f5c7413f8b0 2024-01-20T11:22:37.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.622+0000 7f5c727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:37.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.622+0000 7f5c727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:37.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.622+0000 7f5c727fc640 10 monclient: dump: 2024-01-20T11:22:37.624 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c727fc640 10 monclient: _finish_auth 0 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c727fc640 10 monclient: _check_auth_tickets 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c727fc640 1 -- 172.21.15.110:0/4265448026 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c68007e60 con 0x7f5c7413f8b0 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 5 monclient: authenticate success, global_id 5847 2024-01-20T11:22:37.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c727fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c727fc640 1 -- 172.21.15.110:0/4265448026 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6803e690 con 0x7f5c7413f8b0 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c727fc640 10 monclient: dump: 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:37.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c73fff640 4 set_mon_vals no callback set 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 10 monclient: _renew_subs 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7413fd70 con 0x7f5c7413f8b0 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 10 client.5847.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 10 monclient: _renew_subs 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c741427d0 con 0x7f5c7413f8b0 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 1 librados: init done 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:37.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.623+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c74142ea0 con 0x7f5c7413f8b0 2024-01-20T11:22:37.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.642+0000 7f5c727fc640 1 -- 172.21.15.110:0/4265448026 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5c6803e7f0 con 0x7f5c7413f8b0 2024-01-20T11:22:37.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.642+0000 7f5c727fc640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6006c8b0 0x7f5c6006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:37.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.643+0000 7f5c79f7e640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6006c8b0 0x7f5c6006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:37.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.643+0000 7f5c79f7e640 10 monclient: get_auth_request con 0x7f5c6006c8b0 auth_method 0 2024-01-20T11:22:37.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.643+0000 7f5c79f7e640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6006c8b0 0x7f5c6006ed70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0016e0 tx=0x7f5c6c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:37.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.643+0000 7f5c727fc640 10 client.5847.objecter ms_handle_connect 0x7f5c6006c8b0 2024-01-20T11:22:37.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.644+0000 7f5c727fc640 1 -- 172.21.15.110:0/4265448026 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5c6807b1d0 con 0x7f5c7413f8b0 2024-01-20T11:22:37.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.644+0000 7f5c727fc640 10 client.5847.objecter ms_dispatch 0x7f5c74000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:37.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.644+0000 7f5c727fc640 3 client.5847.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:37.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.644+0000 7f5c727fc640 3 client.5847.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:37.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.644+0000 7f5c727fc640 20 client.5847.objecter dump_active .. 0 homeless 2024-01-20T11:22:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.876+0000 7f5c727fc640 1 -- 172.21.15.110:0/4265448026 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5c68077510 con 0x7f5c7413f8b0 2024-01-20T11:22:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.876+0000 7f5c727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:37.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:37.876+0000 7f5c727fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:38.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.047+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5c740b2200 con 0x7f5c6006c8b0 2024-01-20T11:22:38.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.050+0000 7f5c727fc640 1 -- 172.21.15.110:0/4265448026 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63665 (secure 0 0 0) 0x7f5c6c001cc0 con 0x7f5c6006c8b0 2024-01-20T11:22:38.050 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:38.051 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 10 librados: watch_flush enter 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 10 librados: watch_flush exit 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 20 client.5847.objecter shutdown clearing up homeless session... 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 10 client.5847.objecter successfully canceled tick 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6006c8b0 msgr2=0x7f5c6006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6006c8b0 0x7f5c6006ed70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0016e0 tx=0x7f5c6c009040 comp rx=0 tx=0).stop 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 10 monclient: shutdown 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c7413f8b0 msgr2=0x7f5c74141580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c7413f8b0 0x7f5c74141580 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5c68000f80 tx=0x7f5c68004a40 comp rx=0 tx=0).stop 2024-01-20T11:22:38.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 shutdown_connections 2024-01-20T11:22:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f5c6006c8b0 0x7f5c6006ed70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c74145760 0x7f5c741420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c7413f8b0 0x7f5c74141580 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 --2- 172.21.15.110:0/4265448026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c740a9d80 0x7f5c74136640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 >> 172.21.15.110:0/4265448026 conn(0x7f5c740663b0 msgr2=0x7f5c740b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 shutdown_connections 2024-01-20T11:22:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 -- 172.21.15.110:0/4265448026 wait complete. 2024-01-20T11:22:38.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.052+0000 7f5c7c1e2640 1 librados: shutdown 2024-01-20T11:22:38.063 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-01-20T11:22:36.799829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002617"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280606,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":6.5979999999999999},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":4.9219999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.0340000000000007},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":11.183999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":10.111000000000001},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":13.084}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":15.163},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":14.25}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":12.356999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":18.001000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":16.733000000000001},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":16.021999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313309,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":4.8499999999999996},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.1859999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":11.275},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":5.8860000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":11.986000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":10.460000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":12.727},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":9.3059999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":13.391999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":16.303999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":17.32},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.281000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313309,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.46},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":10.894}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":11.667999999999999},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":15.039999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":12.372},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":7.5910000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":14.298},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":8.4800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":9.0690000000000008},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":13.709}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":13.089},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":9.7739999999999991}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444123,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.9299999999999997},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":8.8439999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":10.109},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.5169999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.73,"5min":12.586,"15min":16.039000000000001},"min":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":20.829000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":15.56},{"interface":"front","average":{"1min":10.227,"5min":10.506,"15min":13.529},"min":{"1min":6.3300000000000001,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":16.702999999999999,"5min":17.103999999999999,"15min":52.905999999999999},"last":11.356}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":14.576000000000001,"5min":13.927,"15min":16.657},"min":{"1min":8.4019999999999992,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":20.393000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.544},{"interface":"front","average":{"1min":12.788,"5min":12.345000000000001,"15min":14.016},"min":{"1min":7.5890000000000004,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":21.629999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":12.606}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":11.929},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.5869999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:21:30 2024","interfaces":[{"interface":"back","average":{"1min":12.4,"5min":12.384,"15min":13.356},"min":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":16.466000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":13.836},{"interface":"front","average":{"1min":14.967000000000001,"5min":14.909000000000001,"15min":17.030000000000001},"min":{"1min":9.7200000000000006,"5min":9.7200000000000006,"15min":9.7200000000000006},"max":{"1min":21.884,"5min":21.884,"15min":28.148},"last":16.376000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444124,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":12.603},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":10.564}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.4649999999999999},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":14.619}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":9.4830000000000005},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":15.794}]},{"osd":4,"last update":"Sat Jan 20 11:21:35 2024","interfaces":[{"interface":"back","average":{"1min":11.629,"5min":12.244,"15min":13.609999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":18.896999999999998,"5min":25.541,"15min":25.541},"last":6.702},{"interface":"front","average":{"1min":13.178000000000001,"5min":13.816000000000001,"15min":14.83},"min":{"1min":8.3680000000000003,"5min":6.75,"15min":6.75},"max":{"1min":22.280000000000001,"5min":30.204999999999998,"15min":30.204999999999998},"last":13.231}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":8.266},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.343999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":8.9209999999999994},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":17.007999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444124,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.5910000000000002},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.8040000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":6.2469999999999999},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":9.4649999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":10.135},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":7.4130000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":14.308},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":13.715}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":11.327999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":11.975}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":12.579000000000001},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":15.042}]}]},{"osd":6,"up_from":15,"seq":64424509533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.172000000000001},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":4.4900000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.6859999999999999},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":10.984}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.178000000000001},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":9.5030000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.853},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":8.7390000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":15.629},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":14.880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":16.446999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.478}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:38.065 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:38.065 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:38.065 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:38.065 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:38.065 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:38.065 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:38.065 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:38.066 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:38.066 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:38.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.167+0000 7efdfb1a9640 10 monclient: get_monmap_and_config 2024-01-20T11:22:38.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.167+0000 7efdfb1a9640 10 monclient: build_initial_monmap 2024-01-20T11:22:38.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.167+0000 7efdfb1a9640 10 monclient: monmap: 2024-01-20T11:22:38.167 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:38.168036+0000 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:38.168036+0000 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:38.168 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 1 Processor -- start 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 1 -- start start 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient: init 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7efdf4071c60 0x7efdf4142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient(hunting): picked mon.noname-c con 0x7efdf4071c60 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdf40ac540 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efdf40b0930 0x7efdf4149510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient(hunting): picked mon.noname-a con 0x7efdf40b0930 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf40ad610 con 0x7efdf40ac540 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf4072040 con 0x7efdf40b0930 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf41426c0 con 0x7efdf4071c60 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.168+0000 7efdfb1a9640 10 monclient(hunting): authenticate will time out at 1801.518948s 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.169+0000 7efdf9746640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efdf40b0930 0x7efdf4149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.169+0000 7efdf9746640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efdf40b0930 0x7efdf4149510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47672/0 (socket says 172.21.15.110:47672) 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.169+0000 7efdf9746640 1 -- 172.21.15.110:0/2178661533 learned_addr learned my addr 172.21.15.110:0/2178661533 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.169+0000 7efdf9746640 10 monclient(hunting): get_auth_request con 0x7efdf40b0930 auth_method 0 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.169+0000 7efdf9746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.169+0000 7efdf9746640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.169+0000 7efdf9746640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.169+0000 7efdf8f45640 1 --2- 172.21.15.110:0/2178661533 >> v2:172.21.15.134:3300/0 conn(0x7efdf4071c60 0x7efdf4142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.170+0000 7efdf9746640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.170+0000 7efdf9746640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.170+0000 7efdf9746640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.170+0000 7efdf8f45640 10 monclient(hunting): get_auth_request con 0x7efdf4071c60 auth_method 0 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.170+0000 7efdf8f45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.170+0000 7efdf8f45640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:38.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.170+0000 7efdf8f45640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:38.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.171+0000 7efdf9746640 10 monclient(hunting): handle_auth_done global_id 5850 payload 274 2024-01-20T11:22:38.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.171+0000 7efdf9746640 1 -- 172.21.15.110:0/2178661533 >> v2:172.21.15.134:3300/0 conn(0x7efdf4071c60 msgr2=0x7efdf4142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:38.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.171+0000 7efdf9746640 1 --2- 172.21.15.110:0/2178661533 >> v2:172.21.15.134:3300/0 conn(0x7efdf4071c60 0x7efdf4142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.171+0000 7efdf9746640 1 -- 172.21.15.110:0/2178661533 >> v1:172.21.15.134:6789/0 conn(0x7efdf40ac540 legacy=0x7efdf40ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.171+0000 7efdf9746640 10 monclient: _finish_hunting 0 2024-01-20T11:22:38.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.171+0000 7efdf9746640 1 monclient: found mon.noname-a 2024-01-20T11:22:38.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.171+0000 7efdf9746640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:38.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.171+0000 7efdf9746640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:38.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.171+0000 7efdf9746640 1 -- 172.21.15.110:0/2178661533 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf40721b0 con 0x7efdf40b0930 2024-01-20T11:22:38.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.173+0000 7efdf9746640 1 --2- 172.21.15.110:0/2178661533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40b0930 0x7efdf4149510 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efdf40ada90 tx=0x7efdec02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70212fdc65176680 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:38.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.176+0000 7efdf37fe640 1 -- 172.21.15.110:0/2178661533 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec02cb20 con 0x7efdf40b0930 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.176+0000 7efdf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 10 monclient: dump: 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:38.179 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 1 -- 172.21.15.110:0/2178661533 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec02cc80 con 0x7efdf40b0930 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 5 monclient: authenticate success, global_id 5850 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 1 -- 172.21.15.110:0/2178661533 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec0343f0 con 0x7efdf40b0930 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 4 set_mon_vals no callback set 2024-01-20T11:22:38.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdf37fe640 10 monclient: dump: 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:38.181 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 10 monclient: shutdown 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2178661533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40b0930 msgr2=0x7efdf4149510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 1 --2- 172.21.15.110:0/2178661533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40b0930 0x7efdf4149510 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efdf40ada90 tx=0x7efdec02bf80 comp rx=0 tx=0).stop 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2178661533 shutdown_connections 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 1 --2- 172.21.15.110:0/2178661533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40b0930 0x7efdf4149510 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 1 --2- 172.21.15.110:0/2178661533 >> v2:172.21.15.134:3300/0 conn(0x7efdf4071c60 0x7efdf4142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2178661533 >> 172.21.15.110:0/2178661533 conn(0x7efdf40b3e60 msgr2=0x7efdf40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2178661533 shutdown_connections 2024-01-20T11:22:38.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.177+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2178661533 wait complete. 2024-01-20T11:22:38.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.178+0000 7efdfb1a9640 10 monclient: build_initial_monmap 2024-01-20T11:22:38.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.178+0000 7efdfb1a9640 10 monclient: monmap: 2024-01-20T11:22:38.186 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:38.186 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:38.186 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:38.186 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:38.186 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.178+0000 7efdfb1a9640 1 librados: starting msgr at 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.178+0000 7efdfb1a9640 1 librados: starting objecter 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.178+0000 7efdfb1a9640 1 Processor -- start 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.178+0000 7efdfb1a9640 1 -- start start 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.178+0000 7efdfb1a9640 1 librados: setting wanted keys 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.178+0000 7efdfb1a9640 1 librados: calling monclient init 2024-01-20T11:22:38.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.178+0000 7efdfb1a9640 10 monclient: init 2024-01-20T11:22:38.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:38.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:38.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdf4071c60 0x7efdf413ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient(hunting): picked mon.noname-b con 0x7efdf4071c60 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:38.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40ac540 0x7efdf41337b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient(hunting): picked mon.noname-a con 0x7efdf40ac540 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:38.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdf40b0930 0x7efdf4133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient(hunting): picked mon.noname-c con 0x7efdf40b0930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:38.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40aeb60 con 0x7efdf40ac540 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40ae640 con 0x7efdf4071c60 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4134230 con 0x7efdf40b0930 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.179+0000 7efdfb1a9640 10 monclient(hunting): authenticate will time out at 1801.529856s 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40ac540 0x7efdf41337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf9746640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdf40b0930 0x7efdf4133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf8f45640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdf4071c60 0x7efdf413ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40ac540 0x7efdf41337b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47680/0 (socket says 172.21.15.110:47680) 2024-01-20T11:22:38.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 1 -- 172.21.15.110:0/2138825270 learned_addr learned my addr 172.21.15.110:0/2138825270 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 10 monclient(hunting): get_auth_request con 0x7efdf40ac540 auth_method 0 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf9746640 10 monclient(hunting): get_auth_request con 0x7efdf40b0930 auth_method 0 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf9746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf9746640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf9746640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf8f45640 10 monclient(hunting): get_auth_request con 0x7efdf4071c60 auth_method 0 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf8f45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf8f45640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf8f45640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:38.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.180+0000 7efdf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.181+0000 7efdf8f45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.181+0000 7efdf8f45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.181+0000 7efdf8f45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.181+0000 7efdf9746640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.181+0000 7efdf9746640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.181+0000 7efdf9746640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 10 monclient(hunting): handle_auth_done global_id 5853 payload 824 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 1 -- 172.21.15.110:0/2138825270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdf40b0930 msgr2=0x7efdf4133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdf40b0930 0x7efdf4133cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 1 -- 172.21.15.110:0/2138825270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdf4071c60 msgr2=0x7efdf413ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdf4071c60 0x7efdf413ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:38.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 1 monclient: found mon.noname-a 2024-01-20T11:22:38.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:38.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:38.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.183+0000 7efdf3fff640 1 -- 172.21.15.110:0/2138825270 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdec009b30 con 0x7efdf40ac540 2024-01-20T11:22:38.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.186+0000 7efdf3fff640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40ac540 0x7efdf41337b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7efde000c400 tx=0x7efde000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:38.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.186+0000 7efdf17fa640 10 client.?.objecter ms_handle_connect 0x7efdf40ac540 2024-01-20T11:22:38.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.186+0000 7efdf17fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:38.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 1 -- 172.21.15.110:0/2138825270 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde00052d0 con 0x7efdf40ac540 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 10 monclient: dump: 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:38.194 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 10 monclient: _finish_auth 0 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 10 monclient: _check_auth_tickets 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 1 -- 172.21.15.110:0/2138825270 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde0005450 con 0x7efdf40ac540 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 1 -- 172.21.15.110:0/2138825270 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0007260 con 0x7efdf40ac540 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf17fa640 10 monclient: dump: 2024-01-20T11:22:38.195 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdfb1a9640 5 monclient: authenticate success, global_id 5853 2024-01-20T11:22:38.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdfb1a9640 10 monclient: _renew_subs 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdf2ffd640 4 set_mon_vals no callback set 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdfb1a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf40affd0 con 0x7efdf40ac540 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdfb1a9640 10 client.5853.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdfb1a9640 10 monclient: _renew_subs 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdfb1a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf4148870 con 0x7efdf40ac540 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.189+0000 7efdfb1a9640 1 librados: init done 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.190+0000 7efdfb1a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.190+0000 7efdfb1a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:38.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.190+0000 7efdfb1a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:38.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.190+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf4075490 con 0x7efdf40ac540 2024-01-20T11:22:38.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.426+0000 7efdf17fa640 1 -- 172.21.15.110:0/2138825270 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7efde00073c0 con 0x7efdf40ac540 2024-01-20T11:22:38.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.426+0000 7efdf17fa640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd806c8b0 0x7efdd806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.426+0000 7efdf8f45640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd806c8b0 0x7efdd806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.426+0000 7efdf8f45640 10 monclient: get_auth_request con 0x7efdd806c8b0 auth_method 0 2024-01-20T11:22:38.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.427+0000 7efdf8f45640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd806c8b0 0x7efdd806ed70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efde8006fd0 tx=0x7efde8008040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:38.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.427+0000 7efdf17fa640 10 client.5853.objecter ms_handle_connect 0x7efdd806c8b0 2024-01-20T11:22:38.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.440+0000 7efdf17fa640 1 -- 172.21.15.110:0/2138825270 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efde0007670 con 0x7efdf40ac540 2024-01-20T11:22:38.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.440+0000 7efdf17fa640 10 client.5853.objecter ms_dispatch 0x7efdf4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:38.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.440+0000 7efdf17fa640 3 client.5853.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:38.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.440+0000 7efdf17fa640 3 client.5853.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:38.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.440+0000 7efdf17fa640 20 client.5853.objecter dump_active .. 0 homeless 2024-01-20T11:22:38.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.441+0000 7efdf17fa640 1 -- 172.21.15.110:0/2138825270 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7efde0054190 con 0x7efdf40ac540 2024-01-20T11:22:38.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.441+0000 7efdf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:38.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.441+0000 7efdf17fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:38.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.679+0000 7efdfb1a9640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:38.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.679+0000 7efdfb1a9640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:38.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.679+0000 7efdfb1a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:38.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.679+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7efdf4072c50 con 0x7efdf40ac540 2024-01-20T11:22:38.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.684+0000 7efdf17fa640 1 -- 172.21.15.110:0/2138825270 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7efde0020520 con 0x7efdf40ac540 2024-01-20T11:22:38.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.684+0000 7efdf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:38.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.684+0000 7efdf17fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:38.685 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:38.685 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":493,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:38.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 10 librados: watch_flush enter 2024-01-20T11:22:38.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 10 librados: watch_flush exit 2024-01-20T11:22:38.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 20 client.5853.objecter shutdown clearing up homeless session... 2024-01-20T11:22:38.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 10 client.5853.objecter successfully canceled tick 2024-01-20T11:22:38.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd806c8b0 msgr2=0x7efdd806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd806c8b0 0x7efdd806ed70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efde8006fd0 tx=0x7efde8008040 comp rx=0 tx=0).stop 2024-01-20T11:22:38.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 10 monclient: shutdown 2024-01-20T11:22:38.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40ac540 msgr2=0x7efdf41337b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.686+0000 7efdfb1a9640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40ac540 0x7efdf41337b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7efde000c400 tx=0x7efde000c8c0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.687+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 shutdown_connections 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.687+0000 7efdfb1a9640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7efdd806c8b0 0x7efdd806ed70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.687+0000 7efdfb1a9640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdf40b0930 0x7efdf4133cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.687+0000 7efdfb1a9640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdf40ac540 0x7efdf41337b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.687+0000 7efdfb1a9640 1 --2- 172.21.15.110:0/2138825270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdf4071c60 0x7efdf413ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.687+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 >> 172.21.15.110:0/2138825270 conn(0x7efdf40b3e60 msgr2=0x7efdf40b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.687+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 shutdown_connections 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.687+0000 7efdfb1a9640 1 -- 172.21.15.110:0/2138825270 wait complete. 2024-01-20T11:22:38.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.687+0000 7efdfb1a9640 1 librados: shutdown 2024-01-20T11:22:38.697 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:38.697 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.795+0000 7f4cb21e2640 10 monclient: get_monmap_and_config 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.795+0000 7f4cb21e2640 10 monclient: build_initial_monmap 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.796+0000 7f4cb21e2640 10 monclient: monmap: 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:38.796598+0000 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:38.796598+0000 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:38.797 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.796+0000 7f4cb21e2640 1 Processor -- start 2024-01-20T11:22:38.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.796+0000 7f4cb21e2640 1 -- start start 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.796+0000 7f4cb21e2640 10 monclient: init 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cac145760 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cac0a9d80 0x7f4cac145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4cac0a9d80 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac13f8b0 0x7f4cac0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 10 monclient(hunting): picked mon.noname-b con 0x7f4cac13f8b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cac146250 con 0x7f4cac145760 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cac0ab050 con 0x7f4cac0a9d80 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0b20c0 con 0x7f4cac13f8b0 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cb21e2640 10 monclient(hunting): authenticate will time out at 1802.147596s 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.797+0000 7f4cabfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac13f8b0 0x7f4cac0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cac0a9d80 0x7f4cac145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cac0a9d80 0x7f4cac145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47688/0 (socket says 172.21.15.110:47688) 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 1 -- 172.21.15.110:0/2433349179 learned_addr learned my addr 172.21.15.110:0/2433349179 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 10 monclient(hunting): get_auth_request con 0x7f4cac0a9d80 auth_method 0 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4cabfff640 10 monclient(hunting): get_auth_request con 0x7f4cac13f8b0 auth_method 0 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4cabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4cabfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:38.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4cabfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:38.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:38.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:38.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4caaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:38.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4cabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:38.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4cabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:38.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.798+0000 7f4cabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:38.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.800+0000 7f4cabfff640 10 monclient(hunting): handle_auth_done global_id 5519 payload 274 2024-01-20T11:22:38.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.800+0000 7f4cabfff640 1 -- 172.21.15.110:0/2433349179 >> v2:172.21.15.110:3300/0 conn(0x7f4cac0a9d80 msgr2=0x7f4cac145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:38.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.800+0000 7f4cabfff640 1 --2- 172.21.15.110:0/2433349179 >> v2:172.21.15.110:3300/0 conn(0x7f4cac0a9d80 0x7f4cac145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.800+0000 7f4cabfff640 1 -- 172.21.15.110:0/2433349179 >> v1:172.21.15.110:6789/0 conn(0x7f4cac145760 legacy=0x7f4cac13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:38.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.800+0000 7f4cabfff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:38.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.800+0000 7f4cabfff640 1 monclient: found mon.noname-b 2024-01-20T11:22:38.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.800+0000 7f4cabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:38.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.800+0000 7f4cabfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:38.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.800+0000 7f4cabfff640 1 -- 172.21.15.110:0/2433349179 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0b2bb0 con 0x7f4cac13f8b0 2024-01-20T11:22:38.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.804+0000 7f4cabfff640 1 --2- 172.21.15.110:0/2433349179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac13f8b0 0x7f4cac0b2670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4009d90 tx=0x7f4ca402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f919bf47ae56f477 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:38.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 1 -- 172.21.15.110:0/2433349179 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca402cbc0 con 0x7f4cac13f8b0 2024-01-20T11:22:38.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:38.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:38.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 10 monclient: dump: 2024-01-20T11:22:38.808 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:38.808 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 10 monclient: _finish_auth 0 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 1 -- 172.21.15.110:0/2433349179 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca402cd20 con 0x7f4cac13f8b0 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4cb21e2640 5 monclient: authenticate success, global_id 5519 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4cb21e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 1 -- 172.21.15.110:0/2433349179 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4034660 con 0x7f4cac13f8b0 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4caa7fc640 10 monclient: dump: 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4cb21e2640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.808+0000 7f4cb21e2640 4 set_mon_vals no callback set 2024-01-20T11:22:38.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 10 monclient: shutdown 2024-01-20T11:22:38.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:38.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 1 -- 172.21.15.110:0/2433349179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac13f8b0 msgr2=0x7f4cac0b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 1 --2- 172.21.15.110:0/2433349179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac13f8b0 0x7f4cac0b2670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4009d90 tx=0x7f4ca402be10 comp rx=0 tx=0).stop 2024-01-20T11:22:38.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 1 -- 172.21.15.110:0/2433349179 shutdown_connections 2024-01-20T11:22:38.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 1 --2- 172.21.15.110:0/2433349179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac13f8b0 0x7f4cac0b2670 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 1 --2- 172.21.15.110:0/2433349179 >> v2:172.21.15.110:3300/0 conn(0x7f4cac0a9d80 0x7f4cac145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 1 -- 172.21.15.110:0/2433349179 >> 172.21.15.110:0/2433349179 conn(0x7f4cac0663b0 msgr2=0x7f4cac0667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 1 -- 172.21.15.110:0/2433349179 shutdown_connections 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.809+0000 7f4cb21e2640 1 -- 172.21.15.110:0/2433349179 wait complete. 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.810+0000 7f4cb21e2640 10 monclient: build_initial_monmap 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.810+0000 7f4cb21e2640 10 monclient: monmap: 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:38.815 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.816 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:38.816 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:38.816 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:38.816 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.810+0000 7f4cb21e2640 1 librados: starting msgr at 2024-01-20T11:22:38.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.810+0000 7f4cb21e2640 1 librados: starting objecter 2024-01-20T11:22:38.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 1 Processor -- start 2024-01-20T11:22:38.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 1 -- start start 2024-01-20T11:22:38.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 1 librados: setting wanted keys 2024-01-20T11:22:38.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 1 librados: calling monclient init 2024-01-20T11:22:38.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 10 monclient: init 2024-01-20T11:22:38.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:38.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:38.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac0a9d80 0x7f4cac13a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 10 monclient(hunting): picked mon.noname-b con 0x7f4cac0a9d80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:38.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.811+0000 7f4cb21e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cac13f8b0 0x7f4cac1428d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4cac13f8b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:38.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac145760 0x7f4cac1d7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:38.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 10 monclient(hunting): picked mon.noname-c con 0x7f4cac145760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:38.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac142e10 con 0x7f4cac13f8b0 2024-01-20T11:22:38.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac142440 con 0x7f4cac0a9d80 2024-01-20T11:22:38.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:38.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0b6310 con 0x7f4cac145760 2024-01-20T11:22:38.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:38.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cb21e2640 10 monclient(hunting): authenticate will time out at 1802.162235s 2024-01-20T11:22:38.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cabfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac145760 0x7f4cac1d7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cabfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac145760 0x7f4cac1d7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:47036/0 (socket says 172.21.15.110:47036) 2024-01-20T11:22:38.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cabfff640 1 -- 172.21.15.110:0/917679222 learned_addr learned my addr 172.21.15.110:0/917679222 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:38.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cabfff640 10 monclient(hunting): get_auth_request con 0x7f4cac145760 auth_method 0 2024-01-20T11:22:38.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:38.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cabfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:38.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cabfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:38.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.812+0000 7f4cab7fe640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac0a9d80 0x7f4cac13a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:38.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cab7fe640 10 monclient(hunting): get_auth_request con 0x7f4cac0a9d80 auth_method 0 2024-01-20T11:22:38.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:38.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cab7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:38.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:38.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:38.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:38.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:38.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:38.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:38.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.813+0000 7f4cab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:38.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 10 monclient(hunting): handle_auth_done global_id 5522 payload 824 2024-01-20T11:22:38.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 1 -- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac0a9d80 msgr2=0x7f4cac13a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac0a9d80 0x7f4cac13a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 1 -- 172.21.15.110:0/917679222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cac13f8b0 msgr2=0x7f4cac1428d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cac13f8b0 0x7f4cac1428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 1 monclient: found mon.noname-c 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.816+0000 7f4cabfff640 1 -- 172.21.15.110:0/917679222 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca4009a70 con 0x7f4cac145760 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.818+0000 7f4cabfff640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac145760 0x7f4cac1d7860 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4037040 tx=0x7f4ca4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.818+0000 7f4c93fff640 10 client.?.objecter ms_handle_connect 0x7f4cac145760 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.818+0000 7f4c93fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 1 -- 172.21.15.110:0/917679222 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4040650 con 0x7f4cac145760 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:38.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 10 monclient: dump: 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 10 monclient: _finish_auth 0 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 10 monclient: _check_auth_tickets 2024-01-20T11:22:38.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 1 -- 172.21.15.110:0/917679222 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca402ce90 con 0x7f4cac145760 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4cb21e2640 5 monclient: authenticate success, global_id 5522 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4cb21e2640 10 monclient: _renew_subs 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4cb21e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac1d7da0 con 0x7f4cac145760 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 1 -- 172.21.15.110:0/917679222 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca40409f0 con 0x7f4cac145760 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4c93fff640 10 monclient: dump: 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:38.829 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4ca9ffb640 4 set_mon_vals no callback set 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4cb21e2640 10 client.5522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4cb21e2640 10 monclient: _renew_subs 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4cb21e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac141b80 con 0x7f4cac145760 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.824+0000 7f4cb21e2640 1 librados: init done 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.825+0000 7f4cb21e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.825+0000 7f4cb21e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:38.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.825+0000 7f4cb21e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:38.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:38.825+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cac0b2200 con 0x7f4cac145760 2024-01-20T11:22:39.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.068+0000 7f4c93fff640 1 -- 172.21.15.110:0/917679222 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4ca4040b50 con 0x7f4cac145760 2024-01-20T11:22:39.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.068+0000 7f4c93fff640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9806c8b0 0x7f4c9806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:39.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.068+0000 7f4cab7fe640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9806c8b0 0x7f4c9806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:39.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.068+0000 7f4cab7fe640 10 monclient: get_auth_request con 0x7f4c9806c8b0 auth_method 0 2024-01-20T11:22:39.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.069+0000 7f4cab7fe640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9806c8b0 0x7f4c9806ed70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c006fd0 tx=0x7f4c9c009060 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:39.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.069+0000 7f4c93fff640 10 client.5522.objecter ms_handle_connect 0x7f4c9806c8b0 2024-01-20T11:22:39.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.080+0000 7f4c93fff640 1 -- 172.21.15.110:0/917679222 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4ca407b800 con 0x7f4cac145760 2024-01-20T11:22:39.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.080+0000 7f4c93fff640 10 client.5522.objecter ms_dispatch 0x7f4cac000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:39.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.080+0000 7f4c93fff640 3 client.5522.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:39.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.080+0000 7f4c93fff640 3 client.5522.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:39.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.080+0000 7f4c93fff640 20 client.5522.objecter dump_active .. 0 homeless 2024-01-20T11:22:39.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.080+0000 7f4c93fff640 1 -- 172.21.15.110:0/917679222 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4ca4077440 con 0x7f4cac145760 2024-01-20T11:22:39.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.080+0000 7f4c93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:39.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.080+0000 7f4c93fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:39.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.259+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4cac0b2200 con 0x7f4c9806c8b0 2024-01-20T11:22:39.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.262+0000 7f4c93fff640 1 -- 172.21.15.110:0/917679222 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63716 (secure 0 0 0) 0x7f4cac0b2200 con 0x7f4c9806c8b0 2024-01-20T11:22:39.262 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:39.263 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 10 librados: watch_flush enter 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 10 librados: watch_flush exit 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 20 client.5522.objecter shutdown clearing up homeless session... 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 10 client.5522.objecter successfully canceled tick 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9806c8b0 msgr2=0x7f4c9806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9806c8b0 0x7f4c9806ed70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c006fd0 tx=0x7f4c9c009060 comp rx=0 tx=0).stop 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 10 monclient: shutdown 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac145760 msgr2=0x7f4cac1d7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac145760 0x7f4cac1d7860 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4037040 tx=0x7f4ca4004a40 comp rx=0 tx=0).stop 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 shutdown_connections 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f4c9806c8b0 0x7f4c9806ed70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:39.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cac145760 0x7f4cac1d7860 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:39.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cac13f8b0 0x7f4cac1428d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:39.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 --2- 172.21.15.110:0/917679222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac0a9d80 0x7f4cac13a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:39.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 >> 172.21.15.110:0/917679222 conn(0x7f4cac0663b0 msgr2=0x7f4cac0b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:39.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 shutdown_connections 2024-01-20T11:22:39.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.264+0000 7f4cb21e2640 1 -- 172.21.15.110:0/917679222 wait complete. 2024-01-20T11:22:39.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:39.265+0000 7f4cb21e2640 1 librados: shutdown 2024-01-20T11:22:39.275 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-01-20T11:22:38.800261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002617"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.2839999999999998},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.7279999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":8.8109999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":13.225}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":16.472999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.9630000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":11.167999999999999},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":11.836}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.544},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.744}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":12.561},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":15.662000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313309,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":4.8499999999999996},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.1859999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":11.275},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":5.8860000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":11.986000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":10.460000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":12.727},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":9.3059999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":13.391999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":16.303999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":17.32},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.281000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313310,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":7.484,"5min":7.8109999999999999,"15min":14.221},"min":{"1min":4.867,"5min":3.8700000000000001,"15min":3.8700000000000001},"max":{"1min":13.314,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.6239999999999997},{"interface":"front","average":{"1min":8.4920000000000009,"5min":7.8369999999999997,"15min":12.801},"min":{"1min":5.024,"5min":3.9510000000000001,"15min":3.9510000000000001},"max":{"1min":15.65,"5min":17.398,"15min":144.16499999999999},"last":8.6419999999999995}]},{"osd":1,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":9.8829999999999991,"5min":10.025,"15min":14.423},"min":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":13.385,"5min":18.771000000000001,"15min":149.55000000000001},"last":4.7240000000000002},{"interface":"front","average":{"1min":11.122,"5min":11.478999999999999,"15min":17.501000000000001},"min":{"1min":7.9900000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"max":{"1min":16.77,"5min":19.655000000000001,"15min":155.91200000000001},"last":10.537000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":10.997,"5min":10.638,"15min":16.341999999999999},"min":{"1min":6.7000000000000002,"5min":5.1719999999999997,"15min":5.1719999999999997},"max":{"1min":16.210000000000001,"5min":18.369,"15min":145.553},"last":11.554},{"interface":"front","average":{"1min":11.436999999999999,"5min":11.561,"15min":16.536999999999999},"min":{"1min":7.0119999999999996,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":19.343,"5min":19.343,"15min":147.66300000000001},"last":9.6460000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":13.16,"15min":22.564},"min":{"1min":8.1929999999999996,"5min":8.1549999999999994,"15min":8.1549999999999994},"max":{"1min":16.256,"5min":20.449000000000002,"15min":284.00700000000001},"last":12.300000000000001},{"interface":"front","average":{"1min":12.77,"5min":12.891,"15min":23.111000000000001},"min":{"1min":7.8630000000000004,"5min":7.8470000000000004,"15min":7.8470000000000004},"max":{"1min":19.489000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":16.622}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":11.808,"5min":11.343999999999999,"15min":18.216000000000001},"min":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":18.204000000000001,"5min":18.204000000000001,"15min":146.744},"last":13.488},{"interface":"front","average":{"1min":14.523999999999999,"5min":13.821999999999999,"15min":19.326000000000001},"min":{"1min":11.166,"5min":8.5969999999999995,"15min":7.8099999999999996},"max":{"1min":17.888999999999999,"5min":19.210000000000001,"15min":150.542},"last":14.192}]},{"osd":7,"last update":"Sat Jan 20 11:21:32 2024","interfaces":[{"interface":"back","average":{"1min":15.616,"5min":15.573,"15min":21.297999999999998},"min":{"1min":10.816000000000001,"5min":10.516,"15min":10.516},"max":{"1min":20.244,"5min":21.661999999999999,"15min":159.179},"last":14.795},{"interface":"front","average":{"1min":13.528,"5min":13.975,"15min":20.945},"min":{"1min":11.077999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":17.818999999999999,"5min":19.161999999999999,"15min":157.35599999999999},"last":12.885999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444124,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":8.8919999999999995},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":10.210000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.1530000000000005},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":6.8739999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.548999999999999},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":5.6840000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.532},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":7.5620000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":14.382},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.5790000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.840999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":13.933}]}]},{"osd":1,"up_from":17,"seq":73014444125,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":5.3010000000000002},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.0359999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":12.279999999999999},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":11.557}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":14.023},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":9.343}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":7.9550000000000001},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":15.879}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":8.6579999999999995},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":13.367000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":10.603999999999999},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":14.654999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444125,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":4.4809999999999999},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":6.4699999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.6229999999999993},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":13.105},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":7.992}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":11.119999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":8.9459999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.755000000000001},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.423}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.481999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":10.411}]}]},{"osd":6,"up_from":15,"seq":64424509533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.172000000000001},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":4.4900000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.6859999999999999},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":10.984}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.178000000000001},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":9.5030000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.853},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":8.7390000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:21:36 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.509,"15min":16.149000000000001},"min":{"1min":8.8079999999999998,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":17.509,"5min":20.404,"15min":70.254999999999995},"last":15.629},{"interface":"front","average":{"1min":12.345000000000001,"5min":12.669,"15min":15.897},"min":{"1min":7.274,"5min":7.1079999999999997,"15min":7.1079999999999997},"max":{"1min":15.028,"5min":18.748000000000001,"15min":72.670000000000002},"last":14.880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":16.446999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.478}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:41.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~224s 2024-01-20T11:22:42.278 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:42.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.378+0000 7f1cf1989640 10 monclient: get_monmap_and_config 2024-01-20T11:22:42.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.378+0000 7f1cf1989640 10 monclient: build_initial_monmap 2024-01-20T11:22:42.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.378+0000 7f1cf1989640 10 monclient: monmap: 2024-01-20T11:22:42.378 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:42.379375+0000 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:42.379375+0000 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:42.379 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.379+0000 7f1cf1989640 1 Processor -- start 2024-01-20T11:22:42.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.379+0000 7f1cf1989640 1 -- start start 2024-01-20T11:22:42.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.379+0000 7f1cf1989640 10 monclient: init 2024-01-20T11:22:42.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1cec0b4d10 0x7f1cec0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 10 monclient(hunting): picked mon.noname-a con 0x7f1cec0b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1cec0b17c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1cec145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0aade0 con 0x7f1cec0b17c0 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec145460 con 0x7f1cec145980 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec0b50f0 con 0x7f1cec0b4d10 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1cf1989640 10 monclient(hunting): authenticate will time out at 1805.730340s 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.380+0000 7f1ceaffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1cec0b4d10 0x7f1cec0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.381+0000 7f1ceaffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1cec0b4d10 0x7f1cec0b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58560/0 (socket says 172.21.15.110:58560) 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.381+0000 7f1ceaffd640 1 -- 172.21.15.110:0/1907361099 learned_addr learned my addr 172.21.15.110:0/1907361099 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:42.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.381+0000 7f1ceaffd640 10 monclient(hunting): get_auth_request con 0x7f1cec0b4d10 auth_method 0 2024-01-20T11:22:42.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.381+0000 7f1ceaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:42.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.381+0000 7f1ceaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:42.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.381+0000 7f1ceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:42.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.381+0000 7f1ceaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:42.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.381+0000 7f1ceaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:42.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.381+0000 7f1ceaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:42.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.383+0000 7f1ceaffd640 10 monclient(hunting): handle_auth_done global_id 5859 payload 274 2024-01-20T11:22:42.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.383+0000 7f1ceaffd640 1 -- 172.21.15.110:0/1907361099 >> v1:172.21.15.134:6789/0 conn(0x7f1cec145980 legacy=0x7f1cec145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.383+0000 7f1ceaffd640 1 -- 172.21.15.110:0/1907361099 >> v1:172.21.15.110:6789/0 conn(0x7f1cec0b17c0 legacy=0x7f1cec0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:42.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.383+0000 7f1ceaffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:42.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.383+0000 7f1ceaffd640 1 monclient: found mon.noname-a 2024-01-20T11:22:42.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.383+0000 7f1ceaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:42.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.383+0000 7f1ceaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:42.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.383+0000 7f1ceaffd640 1 -- 172.21.15.110:0/1907361099 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0b2660 con 0x7f1cec0b4d10 2024-01-20T11:22:42.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.385+0000 7f1ceaffd640 1 --2- 172.21.15.110:0/1907361099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec0b4d10 0x7f1cec0b3da0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009870 tx=0x7f1cdc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65d00c44e498e8d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:42.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 1 -- 172.21.15.110:0/1907361099 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc031870 con 0x7f1cec0b4d10 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 10 monclient: dump: 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:42.390 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 10 monclient: _finish_auth 0 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 1 -- 172.21.15.110:0/1907361099 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc0319d0 con 0x7f1cec0b4d10 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cf1989640 5 monclient: authenticate success, global_id 5859 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cf1989640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cf1989640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 1 -- 172.21.15.110:0/1907361099 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc031d60 con 0x7f1cec0b4d10 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cf1989640 4 set_mon_vals no callback set 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cea7fc640 10 monclient: dump: 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cf1989640 10 monclient: shutdown 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cf1989640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cf1989640 1 -- 172.21.15.110:0/1907361099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec0b4d10 msgr2=0x7f1cec0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.389+0000 7f1cf1989640 1 --2- 172.21.15.110:0/1907361099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec0b4d10 0x7f1cec0b3da0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009870 tx=0x7f1cdc02b910 comp rx=0 tx=0).stop 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.390+0000 7f1cf1989640 1 -- 172.21.15.110:0/1907361099 shutdown_connections 2024-01-20T11:22:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.390+0000 7f1cf1989640 1 --2- 172.21.15.110:0/1907361099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec0b4d10 0x7f1cec0b3da0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.390+0000 7f1cf1989640 1 -- 172.21.15.110:0/1907361099 >> 172.21.15.110:0/1907361099 conn(0x7f1cec066590 msgr2=0x7f1cec0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:42.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.390+0000 7f1cf1989640 1 -- 172.21.15.110:0/1907361099 shutdown_connections 2024-01-20T11:22:42.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.390+0000 7f1cf1989640 1 -- 172.21.15.110:0/1907361099 wait complete. 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.390+0000 7f1cf1989640 10 monclient: build_initial_monmap 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.390+0000 7f1cf1989640 10 monclient: monmap: 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:42.396 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 librados: starting msgr at 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 librados: starting objecter 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 Processor -- start 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 -- start start 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 librados: setting wanted keys 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 librados: calling monclient init 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient: init 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cec0b17c0 0x7f1cec13b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient(hunting): picked mon.noname-b con 0x7f1cec0b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cec0b4d10 0x7f1cec13b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:42.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec0b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec145980 0x7f1cec13e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient(hunting): picked mon.noname-a con 0x7f1cec145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0abab0 con 0x7f1cec145980 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec13d6c0 con 0x7f1cec0b17c0 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0abcb0 con 0x7f1cec0b4d10 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.391+0000 7f1cf1989640 10 monclient(hunting): authenticate will time out at 1805.742132s 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ceb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec145980 0x7f1cec13e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ce27fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cec0b4d10 0x7f1cec13b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ceb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec145980 0x7f1cec13e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58572/0 (socket says 172.21.15.110:58572) 2024-01-20T11:22:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ceb7fe640 1 -- 172.21.15.110:0/4001550788 learned_addr learned my addr 172.21.15.110:0/4001550788 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ceb7fe640 10 monclient(hunting): get_auth_request con 0x7f1cec145980 auth_method 0 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ceb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ceb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ceaffd640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cec0b17c0 0x7f1cec13b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ce27fc640 10 monclient(hunting): get_auth_request con 0x7f1cec0b4d10 auth_method 0 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.392+0000 7f1ce27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ce27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ce27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceaffd640 10 monclient(hunting): get_auth_request con 0x7f1cec0b17c0 auth_method 0 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:42.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:42.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:42.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:42.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:42.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ce27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:42.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ce27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:42.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ce27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:42.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:42.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:42.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.393+0000 7f1ceaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.396+0000 7f1ceaffd640 10 monclient(hunting): handle_auth_done global_id 5956 payload 824 2024-01-20T11:22:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.396+0000 7f1ceaffd640 1 -- 172.21.15.110:0/4001550788 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cec0b4d10 msgr2=0x7f1cec13b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.396+0000 7f1ceaffd640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cec0b4d10 0x7f1cec13b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.397+0000 7f1ceaffd640 1 -- 172.21.15.110:0/4001550788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec145980 msgr2=0x7f1cec13e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.397+0000 7f1ceaffd640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec145980 0x7f1cec13e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.397+0000 7f1ceaffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.397+0000 7f1ceaffd640 1 monclient: found mon.noname-b 2024-01-20T11:22:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.397+0000 7f1ceaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.397+0000 7f1ceaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.397+0000 7f1ceaffd640 1 -- 172.21.15.110:0/4001550788 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc0094d0 con 0x7f1cec0b17c0 2024-01-20T11:22:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.399+0000 7f1ceaffd640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cec0b17c0 0x7f1cec13b050 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc033000 tx=0x7f1cdc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.399+0000 7f1ce8ff9640 10 client.?.objecter ms_handle_connect 0x7f1cec0b17c0 2024-01-20T11:22:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.399+0000 7f1ce8ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:42.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 1 -- 172.21.15.110:0/4001550788 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc03adf0 con 0x7f1cec0b17c0 2024-01-20T11:22:42.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:42.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:42.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 10 monclient: dump: 2024-01-20T11:22:42.404 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:42.404 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.404 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.404 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.404 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 10 monclient: _finish_auth 0 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 1 -- 172.21.15.110:0/4001550788 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc0443f0 con 0x7f1cec0b17c0 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1cf1989640 5 monclient: authenticate success, global_id 5956 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 1 -- 172.21.15.110:0/4001550788 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc0447a0 con 0x7f1cec0b17c0 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1cf1989640 10 monclient: _renew_subs 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1cf1989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:42.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec0abf90 con 0x7f1cec0b17c0 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce8ff9640 10 monclient: dump: 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1ce9ffb640 4 set_mon_vals no callback set 2024-01-20T11:22:42.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1cf1989640 10 client.5956.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:42.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1cf1989640 10 monclient: _renew_subs 2024-01-20T11:22:42.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1cf1989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:42.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec13e650 con 0x7f1cec0b17c0 2024-01-20T11:22:42.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.403+0000 7f1cf1989640 1 librados: init done 2024-01-20T11:22:42.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.404+0000 7f1cf1989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:42.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.404+0000 7f1cf1989640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:42.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.404+0000 7f1cf1989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:42.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.404+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec075490 con 0x7f1cec0b17c0 2024-01-20T11:22:42.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.422+0000 7f1ce8ff9640 1 -- 172.21.15.110:0/4001550788 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1cdc04d450 con 0x7f1cec0b17c0 2024-01-20T11:22:42.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.423+0000 7f1ce8ff9640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1ccc06c8b0 0x7f1ccc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:42.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.423+0000 7f1ce27fc640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1ccc06c8b0 0x7f1ccc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:42.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.423+0000 7f1ce27fc640 10 monclient: get_auth_request con 0x7f1ccc06c8b0 auth_method 0 2024-01-20T11:22:42.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.424+0000 7f1ce8ff9640 10 client.5956.objecter ms_handle_connect 0x7f1ccc06c8b0 2024-01-20T11:22:42.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.424+0000 7f1ce27fc640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1ccc06c8b0 0x7f1ccc06ed70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1cec137320 tx=0x7f1cd4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:42.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.650+0000 7f1ce8ff9640 1 -- 172.21.15.110:0/4001550788 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1cdc02d040 con 0x7f1cec0b17c0 2024-01-20T11:22:42.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.650+0000 7f1ce8ff9640 10 client.5956.objecter ms_dispatch 0x7f1cec000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:42.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.650+0000 7f1ce8ff9640 3 client.5956.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:42.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.650+0000 7f1ce8ff9640 3 client.5956.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:42.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.651+0000 7f1ce8ff9640 20 client.5956.objecter dump_active .. 0 homeless 2024-01-20T11:22:42.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.661+0000 7f1ce8ff9640 1 -- 172.21.15.110:0/4001550788 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1cdc076e50 con 0x7f1cec0b17c0 2024-01-20T11:22:42.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.661+0000 7f1ce8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:42.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.661+0000 7f1ce8ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:42.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.845+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1cec0a9cf0 con 0x7f1ccc06c8b0 2024-01-20T11:22:42.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.848+0000 7f1ce8ff9640 1 -- 172.21.15.110:0/4001550788 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63804 (secure 0 0 0) 0x7f1cd4001440 con 0x7f1ccc06c8b0 2024-01-20T11:22:42.849 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:42.849 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:42.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.850+0000 7f1cf1989640 10 librados: watch_flush enter 2024-01-20T11:22:42.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.850+0000 7f1cf1989640 10 librados: watch_flush exit 2024-01-20T11:22:42.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.850+0000 7f1cf1989640 20 client.5956.objecter shutdown clearing up homeless session... 2024-01-20T11:22:42.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.850+0000 7f1cf1989640 10 client.5956.objecter successfully canceled tick 2024-01-20T11:22:42.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1ccc06c8b0 msgr2=0x7f1ccc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1ccc06c8b0 0x7f1ccc06ed70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1cec137320 tx=0x7f1cd4009040 comp rx=0 tx=0).stop 2024-01-20T11:22:42.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 10 monclient: shutdown 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cec0b17c0 msgr2=0x7f1cec13b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cec0b17c0 0x7f1cec13b050 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc033000 tx=0x7f1cdc0046c0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 shutdown_connections 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1ccc06c8b0 0x7f1ccc06ed70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cec145980 0x7f1cec13e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cec0b4d10 0x7f1cec13b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 --2- 172.21.15.110:0/4001550788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1cec0b17c0 0x7f1cec13b050 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 >> 172.21.15.110:0/4001550788 conn(0x7f1cec066590 msgr2=0x7f1cec139a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 shutdown_connections 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 -- 172.21.15.110:0/4001550788 wait complete. 2024-01-20T11:22:42.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.851+0000 7f1cf1989640 1 librados: shutdown 2024-01-20T11:22:42.861 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-01-20T11:22:42.801366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002899"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.2839999999999998},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.7279999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":8.8109999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":13.225}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":16.472999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.9630000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":11.167999999999999},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":11.836}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.544},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.744}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":12.561},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":15.662000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313310,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":10.778},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.3360000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":5.7359999999999998},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":8.0470000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.266},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":12.343}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":15.954000000000001},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":14.670999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":13.993},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":10.413}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":12.926},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":16.661999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313311,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":13.784000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":6.5739999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":15.194000000000001},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":10.106}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":11.656000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":14.436999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.936999999999999},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.5540000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.9760000000000009},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.439}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":17.571000000000002},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.871}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444124,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":8.8919999999999995},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":10.210000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.1530000000000005},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":6.8739999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.548999999999999},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":5.6840000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.532},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":7.5620000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":14.382},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.5790000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.840999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":13.933}]}]},{"osd":1,"up_from":17,"seq":73014444125,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":5.3010000000000002},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.0359999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":12.279999999999999},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":11.557}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":14.023},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":9.343}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":7.9550000000000001},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":15.879}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":8.6579999999999995},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":13.367000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":10.603999999999999},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":14.654999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444125,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":4.4809999999999999},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":6.4699999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.6229999999999993},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":13.105},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":7.992}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":11.119999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":8.9459999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.755000000000001},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.423}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.481999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":10.411}]}]},{"osd":6,"up_from":15,"seq":64424509534,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":4.8929999999999998},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":6.9359999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":14.938000000000001},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":11.458}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":8.2430000000000003},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":12.337}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":13.045},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":8.9329999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":10.385999999999999},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.214}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":17.154},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.001999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:42.863 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.963+0000 7f20402fe640 10 monclient: get_monmap_and_config 2024-01-20T11:22:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.963+0000 7f20402fe640 10 monclient: build_initial_monmap 2024-01-20T11:22:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.963+0000 7f20402fe640 10 monclient: monmap: 2024-01-20T11:22:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:42.963905+0000 2024-01-20T11:22:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:42.963905+0000 2024-01-20T11:22:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.964 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:42.964 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:42.964 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:42.964 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:42.964 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:42.964 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.963+0000 7f20402fe640 1 Processor -- start 2024-01-20T11:22:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.963+0000 7f20402fe640 1 -- start start 2024-01-20T11:22:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.963+0000 7f20402fe640 10 monclient: init 2024-01-20T11:22:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f20380b4d10 0x7f20380b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 10 monclient(hunting): picked mon.noname-c con 0x7f20380b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20380b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2038145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20380aade0 con 0x7f2038145980 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2038145460 con 0x7f20380b17c0 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20380b50f0 con 0x7f20380b4d10 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.964+0000 7f20402fe640 10 monclient(hunting): authenticate will time out at 1806.314833s 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.965+0000 7f203e89b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2038145980 0x7f2038145e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46304/0 (socket says 172.21.15.110:46304) 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.965+0000 7f203e89b640 1 -- 172.21.15.110:0/1043061089 learned_addr learned my addr 172.21.15.110:0/1043061089 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.965+0000 7f203e09a640 1 --2- 172.21.15.110:0/1043061089 >> v2:172.21.15.134:3300/0 conn(0x7f20380b4d10 0x7f20380b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:42.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.966+0000 7f203e09a640 10 monclient(hunting): get_auth_request con 0x7f20380b4d10 auth_method 0 2024-01-20T11:22:42.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.966+0000 7f203e09a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.966+0000 7f203e09a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.966+0000 7f203e09a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.967+0000 7f203e09a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.967+0000 7f203e09a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.967+0000 7f203e09a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.967+0000 7f203d098640 1 -- 172.21.15.110:0/1043061089 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933465580 0 0) 0x7f20380aade0 con 0x7f2038145980 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.967+0000 7f203d098640 10 monclient(hunting): _negotiate 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.967+0000 7f203d098640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.967+0000 7f203d098640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.967+0000 7f203d098640 10 monclient(hunting): my global_id is 5865 2024-01-20T11:22:42.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.967+0000 7f203d098640 1 -- 172.21.15.110:0/1043061089 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2024003520 con 0x7f2038145980 2024-01-20T11:22:42.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.969+0000 7f203e09a640 10 monclient(hunting): handle_auth_done global_id 5959 payload 274 2024-01-20T11:22:42.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.969+0000 7f203e09a640 1 -- 172.21.15.110:0/1043061089 >> v1:172.21.15.134:6789/0 conn(0x7f20380b17c0 legacy=0x7f20380b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.969+0000 7f203e09a640 1 -- 172.21.15.110:0/1043061089 >> v1:172.21.15.110:6789/0 conn(0x7f2038145980 legacy=0x7f2038145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.969+0000 7f203e09a640 10 monclient: _finish_hunting 0 2024-01-20T11:22:42.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.969+0000 7f203e09a640 1 monclient: found mon.noname-c 2024-01-20T11:22:42.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.969+0000 7f203e09a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:42.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.969+0000 7f203e09a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:22:42.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.969+0000 7f203e09a640 1 -- 172.21.15.110:0/1043061089 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20380b2660 con 0x7f20380b4d10 2024-01-20T11:22:42.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.973+0000 7f203e09a640 1 --2- 172.21.15.110:0/1043061089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20380b4d10 0x7f20380b3da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f20340097a0 tx=0x7f203402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d9c82e3a63aef1f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 1 -- 172.21.15.110:0/1043061089 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2034031830 con 0x7f20380b4d10 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 10 monclient: dump: 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:42.977 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 10 monclient: _finish_auth 0 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 10 monclient: _check_auth_tickets 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 1 -- 172.21.15.110:0/1043061089 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2034031990 con 0x7f20380b4d10 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 1 -- 172.21.15.110:0/1043061089 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2034031d20 con 0x7f20380b4d10 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f20402fe640 5 monclient: authenticate success, global_id 5959 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f20402fe640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f20402fe640 4 set_mon_vals no callback set 2024-01-20T11:22:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f203d098640 10 monclient: dump: 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:42.979 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f20402fe640 10 monclient: shutdown 2024-01-20T11:22:42.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f20402fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f20402fe640 1 -- 172.21.15.110:0/1043061089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20380b4d10 msgr2=0x7f20380b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.977+0000 7f20402fe640 1 --2- 172.21.15.110:0/1043061089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20380b4d10 0x7f20380b3da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f20340097a0 tx=0x7f203402b8d0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.978+0000 7f20402fe640 1 -- 172.21.15.110:0/1043061089 shutdown_connections 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.978+0000 7f20402fe640 1 --2- 172.21.15.110:0/1043061089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20380b4d10 0x7f20380b3da0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.978+0000 7f20402fe640 1 -- 172.21.15.110:0/1043061089 >> 172.21.15.110:0/1043061089 conn(0x7f2038066590 msgr2=0x7f20380669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.978+0000 7f20402fe640 1 -- 172.21.15.110:0/1043061089 shutdown_connections 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.978+0000 7f20402fe640 1 -- 172.21.15.110:0/1043061089 wait complete. 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.978+0000 7f20402fe640 10 monclient: build_initial_monmap 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.978+0000 7f20402fe640 10 monclient: monmap: 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:42.983 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.978+0000 7f20402fe640 1 librados: starting msgr at 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.978+0000 7f20402fe640 1 librados: starting objecter 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 Processor -- start 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 -- start start 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 librados: setting wanted keys 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 librados: calling monclient init 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient: init 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20380b17c0 0x7f203813dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient(hunting): picked mon.noname-c con 0x7f20380b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20380b4d10 0x7f203813e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient(hunting): picked mon.noname-a con 0x7f20380b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2038145980 0x7f203813b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient(hunting): picked mon.noname-b con 0x7f2038145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203813ea80 con 0x7f20380b4d10 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20380ac0d0 con 0x7f2038145980 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2038137020 con 0x7f20380b17c0 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.979+0000 7f20402fe640 10 monclient(hunting): authenticate will time out at 1806.329835s 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203d899640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20380b4d10 0x7f203813e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2038145980 0x7f203813b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e09a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20380b17c0 0x7f203813dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2038145980 0x7f203813b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:37982/0 (socket says 172.21.15.110:37982) 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 1 -- 172.21.15.110:0/1660271789 learned_addr learned my addr 172.21.15.110:0/1660271789 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 10 monclient(hunting): get_auth_request con 0x7f2038145980 auth_method 0 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203d899640 10 monclient(hunting): get_auth_request con 0x7f20380b4d10 auth_method 0 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203d899640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203d899640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203d899640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e09a640 10 monclient(hunting): get_auth_request con 0x7f20380b17c0 auth_method 0 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e09a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e09a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e09a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:42.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:42.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.980+0000 7f203e89b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:42.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.981+0000 7f203d899640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:42.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.981+0000 7f203d899640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:42.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.981+0000 7f203d899640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:42.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.981+0000 7f203e09a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:42.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.981+0000 7f203e09a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:42.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.981+0000 7f203e09a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:42.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 10 monclient(hunting): handle_auth_done global_id 5965 payload 824 2024-01-20T11:22:42.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 1 -- 172.21.15.110:0/1660271789 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20380b17c0 msgr2=0x7f203813dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20380b17c0 0x7f203813dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 1 -- 172.21.15.110:0/1660271789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20380b4d10 msgr2=0x7f203813e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:42.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20380b4d10 0x7f203813e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:42.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 10 monclient: _finish_hunting 0 2024-01-20T11:22:42.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 1 monclient: found mon.noname-b 2024-01-20T11:22:42.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:42.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:42.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.983+0000 7f203e89b640 1 -- 172.21.15.110:0/1660271789 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2034009400 con 0x7f2038145980 2024-01-20T11:22:42.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.985+0000 7f203e89b640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2038145980 0x7f203813b1c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f203000c9f0 tx=0x7f203000ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:42.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.985+0000 7f202effd640 10 client.?.objecter ms_handle_connect 0x7f2038145980 2024-01-20T11:22:42.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.985+0000 7f202effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:42.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 1 -- 172.21.15.110:0/1660271789 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203000ab70 con 0x7f2038145980 2024-01-20T11:22:42.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:42.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:42.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 10 monclient: dump: 2024-01-20T11:22:42.990 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:42.990 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 10 monclient: _finish_auth 0 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 1 -- 172.21.15.110:0/1660271789 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203000acd0 con 0x7f2038145980 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:42.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 1 -- 172.21.15.110:0/1660271789 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2030012600 con 0x7f2038145980 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f202effd640 10 monclient: dump: 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:42.992 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f20402fe640 5 monclient: authenticate success, global_id 5965 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f20402fe640 10 monclient: _renew_subs 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f20402fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f203c897640 4 set_mon_vals no callback set 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203813ef30 con 0x7f2038145980 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f20402fe640 10 client.5965.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f20402fe640 10 monclient: _renew_subs 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f20402fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203813f210 con 0x7f2038145980 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.989+0000 7f20402fe640 1 librados: init done 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.990+0000 7f20402fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:42.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.990+0000 7f20402fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:42.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.990+0000 7f20402fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:42.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:42.990+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20380abf40 con 0x7f2038145980 2024-01-20T11:22:43.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.007+0000 7f202effd640 1 -- 172.21.15.110:0/1660271789 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2030007070 con 0x7f2038145980 2024-01-20T11:22:43.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.008+0000 7f202effd640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f201c06c860 0x7f201c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:43.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.008+0000 7f203e09a640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f201c06c860 0x7f201c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:43.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.008+0000 7f203e09a640 10 monclient: get_auth_request con 0x7f201c06c860 auth_method 0 2024-01-20T11:22:43.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.008+0000 7f202effd640 10 client.5965.objecter ms_handle_connect 0x7f201c06c860 2024-01-20T11:22:43.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.008+0000 7f203e09a640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f201c06c860 0x7f201c06ed20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2034036000 tx=0x7f203403c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:43.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.011+0000 7f202effd640 1 -- 172.21.15.110:0/1660271789 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2030058500 con 0x7f2038145980 2024-01-20T11:22:43.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.011+0000 7f202effd640 10 client.5965.objecter ms_dispatch 0x7f2038000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:43.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.011+0000 7f202effd640 3 client.5965.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:43.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.011+0000 7f202effd640 3 client.5965.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:43.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.011+0000 7f202effd640 20 client.5965.objecter dump_active .. 0 homeless 2024-01-20T11:22:43.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.244+0000 7f202effd640 1 -- 172.21.15.110:0/1660271789 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2030054840 con 0x7f2038145980 2024-01-20T11:22:43.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.244+0000 7f202effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:43.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.244+0000 7f202effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:43.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.413+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20380a9cf0 con 0x7f201c06c860 2024-01-20T11:22:43.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.416+0000 7f202effd640 1 -- 172.21.15.110:0/1660271789 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63804 (secure 0 0 0) 0x7f203403a420 con 0x7f201c06c860 2024-01-20T11:22:43.416 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:43.417 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 10 librados: watch_flush enter 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 10 librados: watch_flush exit 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 20 client.5965.objecter shutdown clearing up homeless session... 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 10 client.5965.objecter successfully canceled tick 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f201c06c860 msgr2=0x7f201c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f201c06c860 0x7f201c06ed20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2034036000 tx=0x7f203403c040 comp rx=0 tx=0).stop 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 10 monclient: shutdown 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2038145980 msgr2=0x7f203813b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:43.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2038145980 0x7f203813b1c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f203000c9f0 tx=0x7f203000ceb0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 shutdown_connections 2024-01-20T11:22:43.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f201c06c860 0x7f201c06ed20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2038145980 0x7f203813b1c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20380b4d10 0x7f203813e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 --2- 172.21.15.110:0/1660271789 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20380b17c0 0x7f203813dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 >> 172.21.15.110:0/1660271789 conn(0x7f2038066590 msgr2=0x7f2038145e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:43.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 shutdown_connections 2024-01-20T11:22:43.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 -- 172.21.15.110:0/1660271789 wait complete. 2024-01-20T11:22:43.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.418+0000 7f20402fe640 1 librados: shutdown 2024-01-20T11:22:43.428 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-01-20T11:22:42.801366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002899"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.2839999999999998},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.7279999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":8.8109999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":13.225}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":16.472999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.9630000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":11.167999999999999},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":11.836}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.544},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.744}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":12.561},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":15.662000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313310,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":10.778},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.3360000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":5.7359999999999998},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":8.0470000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.266},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":12.343}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":15.954000000000001},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":14.670999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":13.993},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":10.413}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":12.926},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":16.661999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313311,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":13.784000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":6.5739999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":15.194000000000001},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":10.106}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":11.656000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":14.436999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.936999999999999},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.5540000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.9760000000000009},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.439}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":17.571000000000002},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.871}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444124,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":8.8919999999999995},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":10.210000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.1530000000000005},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":6.8739999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.548999999999999},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":5.6840000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.532},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":7.5620000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":14.382},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.5790000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.840999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":13.933}]}]},{"osd":1,"up_from":17,"seq":73014444125,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":5.3010000000000002},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.0359999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":12.279999999999999},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":11.557}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":14.023},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":9.343}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":7.9550000000000001},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":15.879}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":8.6579999999999995},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":13.367000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":10.603999999999999},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":14.654999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444125,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":4.4809999999999999},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":6.4699999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.6229999999999993},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":13.105},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":7.992}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":11.119999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":8.9459999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.755000000000001},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.423}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.481999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":10.411}]}]},{"osd":6,"up_from":15,"seq":64424509534,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":4.8929999999999998},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":6.9359999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":14.938000000000001},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":11.458}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":8.2430000000000003},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":12.337}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":13.045},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":8.9329999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":10.385999999999999},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.214}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":17.154},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.001999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:43.430 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:43.430 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:43.430 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:43.430 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:43.430 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:43.430 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:43.430 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:43.430 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:43.431 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:43.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.529+0000 7f423316c640 10 monclient: get_monmap_and_config 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.529+0000 7f423316c640 10 monclient: build_initial_monmap 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.529+0000 7f423316c640 10 monclient: monmap: 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:43.529721+0000 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:43.529721+0000 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:43.529 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:43.530 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:43.530 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:43.530 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:43.530 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:43.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.529+0000 7f423316c640 1 Processor -- start 2024-01-20T11:22:43.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.529+0000 7f423316c640 1 -- start start 2024-01-20T11:22:43.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.529+0000 7f423316c640 10 monclient: init 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 10 monclient(hunting): picked mon.noname-b con 0x7f422c0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f422c13fa10 0x7f422c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 10 monclient(hunting): picked mon.noname-c con 0x7f422c13fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f422c0a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422c0b25c0 con 0x7f422c0a9ad0 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0aaea0 con 0x7f422c13fa10 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0ab0a0 con 0x7f422c0a96f0 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:43.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.530+0000 7f423316c640 10 monclient(hunting): authenticate will time out at 1806.880724s 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f4231709640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f422c0a9ad0 0x7f422c142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46318/0 (socket says 172.21.15.110:46318) 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f4231709640 1 -- 172.21.15.110:0/2500936276 learned_addr learned my addr 172.21.15.110:0/2500936276 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f4230f08640 1 --2- 172.21.15.110:0/2500936276 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f422bfff640 1 --2- 172.21.15.110:0/2500936276 >> v2:172.21.15.134:3300/0 conn(0x7f422c13fa10 0x7f422c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f4230f08640 10 monclient(hunting): get_auth_request con 0x7f422c0a96f0 auth_method 0 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f4230f08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f4230f08640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f4230f08640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f422bfff640 10 monclient(hunting): get_auth_request con 0x7f422c13fa10 auth_method 0 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f422bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f422bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f422bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.531+0000 7f4230f08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.532+0000 7f4230f08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:43.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.532+0000 7f4230f08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:43.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.532+0000 7f422bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:43.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.532+0000 7f422bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:43.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.532+0000 7f422bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:43.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f4230f08640 10 monclient(hunting): handle_auth_done global_id 5531 payload 274 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f422b7fe640 1 -- 172.21.15.110:0/2500936276 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647567891 0 0) 0x7f422c0b25c0 con 0x7f422c0a9ad0 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f4230f08640 1 -- 172.21.15.110:0/2500936276 >> v2:172.21.15.134:3300/0 conn(0x7f422c13fa10 msgr2=0x7f422c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f4230f08640 1 --2- 172.21.15.110:0/2500936276 >> v2:172.21.15.134:3300/0 conn(0x7f422c13fa10 0x7f422c1423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f4230f08640 1 -- 172.21.15.110:0/2500936276 >> v1:172.21.15.110:6789/0 conn(0x7f422c0a9ad0 legacy=0x7f422c142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f4230f08640 10 monclient: _finish_hunting 0 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f4230f08640 1 monclient: found mon.noname-b 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f4230f08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f4230f08640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f4230f08640 1 -- 172.21.15.110:0/2500936276 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c0b3fb0 con 0x7f422c0a96f0 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.533+0000 7f422b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:22:43.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.535+0000 7f4230f08640 1 --2- 172.21.15.110:0/2500936276 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c13f4d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f421c0097a0 tx=0x7f421c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=103681b2141fe5bc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:43.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 1 -- 172.21.15.110:0/2500936276 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c02e070 con 0x7f422c0a96f0 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 10 monclient: dump: 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:43.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 1 -- 172.21.15.110:0/2500936276 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c0315a0 con 0x7f422c0a96f0 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 1 -- 172.21.15.110:0/2500936276 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c031930 con 0x7f422c0a96f0 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f422b7fe640 10 monclient: dump: 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:43.540 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:43.541 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f423316c640 5 monclient: authenticate success, global_id 5531 2024-01-20T11:22:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f423316c640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.538+0000 7f423316c640 4 set_mon_vals no callback set 2024-01-20T11:22:43.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 10 monclient: shutdown 2024-01-20T11:22:43.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 1 -- 172.21.15.110:0/2500936276 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 msgr2=0x7f422c13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 1 --2- 172.21.15.110:0/2500936276 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c13f4d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f421c0097a0 tx=0x7f421c02b840 comp rx=0 tx=0).stop 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 1 -- 172.21.15.110:0/2500936276 shutdown_connections 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 1 --2- 172.21.15.110:0/2500936276 >> v2:172.21.15.134:3300/0 conn(0x7f422c13fa10 0x7f422c1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 1 --2- 172.21.15.110:0/2500936276 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c13f4d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 1 -- 172.21.15.110:0/2500936276 >> 172.21.15.110:0/2500936276 conn(0x7f422c0b7bc0 msgr2=0x7f422c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 1 -- 172.21.15.110:0/2500936276 shutdown_connections 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.539+0000 7f423316c640 1 -- 172.21.15.110:0/2500936276 wait complete. 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.540+0000 7f423316c640 10 monclient: build_initial_monmap 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.540+0000 7f423316c640 10 monclient: monmap: 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:43.546 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.540+0000 7f423316c640 1 librados: starting msgr at 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.540+0000 7f423316c640 1 librados: starting objecter 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.540+0000 7f423316c640 1 Processor -- start 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.540+0000 7f423316c640 1 -- start start 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.540+0000 7f423316c640 1 librados: setting wanted keys 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.540+0000 7f423316c640 1 librados: calling monclient init 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.540+0000 7f423316c640 10 monclient: init 2024-01-20T11:22:43.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c136770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient(hunting): picked mon.noname-c con 0x7f422c0a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f422c0a9ad0 0x7f422c13e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient(hunting): picked mon.noname-a con 0x7f422c0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f422c143300 0x7f422c1da470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient(hunting): picked mon.noname-b con 0x7f422c143300 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c13ecc0 con 0x7f422c0a9ad0 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c144880 con 0x7f422c143300 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c144b60 con 0x7f422c0a96f0 2024-01-20T11:22:43.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f423316c640 10 monclient(hunting): authenticate will time out at 1806.891580s 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f422bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f422c0a9ad0 0x7f422c13e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.541+0000 7f4230f08640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f422bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f422c0a9ad0 0x7f422c13e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58584/0 (socket says 172.21.15.110:58584) 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f422bfff640 1 -- 172.21.15.110:0/2228555538 learned_addr learned my addr 172.21.15.110:0/2228555538 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f422bfff640 10 monclient(hunting): get_auth_request con 0x7f422c0a9ad0 auth_method 0 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f422bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f422bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f422bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4230f08640 10 monclient(hunting): get_auth_request con 0x7f422c0a96f0 auth_method 0 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4230f08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4230f08640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4230f08640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:43.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4231709640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f422c143300 0x7f422c1da470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4231709640 10 monclient(hunting): get_auth_request con 0x7f422c143300 auth_method 0 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4231709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4231709640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4231709640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f422bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f422bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f422bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4230f08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4230f08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.542+0000 7f4230f08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.543+0000 7f4231709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.543+0000 7f4231709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:43.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.543+0000 7f4231709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 10 monclient(hunting): handle_auth_done global_id 5874 payload 824 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 1 -- 172.21.15.110:0/2228555538 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 msgr2=0x7f422c136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c136770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 1 -- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f422c143300 msgr2=0x7f422c1da470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f422c143300 0x7f422c1da470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 1 monclient: found mon.noname-a 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f422bfff640 1 -- 172.21.15.110:0/2228555538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c009400 con 0x7f422c0a9ad0 2024-01-20T11:22:43.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.545+0000 7f4230f08640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c136770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:22:43.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.547+0000 7f422bfff640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f422c0a9ad0 0x7f422c13e780 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f422000e840 tx=0x7f422000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:43.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.547+0000 7f42297fa640 10 client.?.objecter ms_handle_connect 0x7f422c0a9ad0 2024-01-20T11:22:43.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.547+0000 7f42297fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:43.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 1 -- 172.21.15.110:0/2228555538 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422000c730 con 0x7f422c0a9ad0 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 10 monclient: dump: 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:43.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 10 monclient: _finish_auth 0 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 10 monclient: _check_auth_tickets 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 1 -- 172.21.15.110:0/2228555538 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422000c890 con 0x7f422c0a9ad0 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 1 -- 172.21.15.110:0/2228555538 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422000cc40 con 0x7f422c0a9ad0 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f423316c640 5 monclient: authenticate success, global_id 5874 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f423316c640 10 monclient: _renew_subs 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f423316c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c143790 con 0x7f422c0a9ad0 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f422affd640 4 set_mon_vals no callback set 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f42297fa640 10 monclient: dump: 2024-01-20T11:22:43.555 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f423316c640 10 client.5874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f423316c640 10 monclient: _renew_subs 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f423316c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:43.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c1d4a30 con 0x7f422c0a9ad0 2024-01-20T11:22:43.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.551+0000 7f423316c640 1 librados: init done 2024-01-20T11:22:43.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.552+0000 7f423316c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:43.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.552+0000 7f423316c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:43.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.552+0000 7f423316c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:43.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.552+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c075490 con 0x7f422c0a9ad0 2024-01-20T11:22:43.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.798+0000 7f42297fa640 1 -- 172.21.15.110:0/2228555538 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4220013470 con 0x7f422c0a9ad0 2024-01-20T11:22:43.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.798+0000 7f42297fa640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f421806c860 0x7f421806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:43.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.798+0000 7f4230f08640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f421806c860 0x7f421806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:43.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.798+0000 7f4230f08640 10 monclient: get_auth_request con 0x7f421806c860 auth_method 0 2024-01-20T11:22:43.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.799+0000 7f42297fa640 10 client.5874.objecter ms_handle_connect 0x7f421806c860 2024-01-20T11:22:43.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.799+0000 7f4230f08640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f421806c860 0x7f421806ed20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f421c036000 tx=0x7f421c03d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:43.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.814+0000 7f42297fa640 1 -- 172.21.15.110:0/2228555538 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4220058a00 con 0x7f422c0a9ad0 2024-01-20T11:22:43.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.814+0000 7f42297fa640 10 client.5874.objecter ms_dispatch 0x7f422c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:43.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.814+0000 7f42297fa640 3 client.5874.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:43.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.814+0000 7f42297fa640 3 client.5874.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:43.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.814+0000 7f42297fa640 20 client.5874.objecter dump_active .. 0 homeless 2024-01-20T11:22:43.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.814+0000 7f42297fa640 1 -- 172.21.15.110:0/2228555538 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4220054d40 con 0x7f422c0a9ad0 2024-01-20T11:22:43.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.814+0000 7f42297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:43.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:43.814+0000 7f42297fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:44.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.077+0000 7f423316c640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:44.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.077+0000 7f423316c640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:44.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.077+0000 7f423316c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:44.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.078+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f422c069850 con 0x7f422c0a9ad0 2024-01-20T11:22:44.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.083+0000 7f42297fa640 1 -- 172.21.15.110:0/2228555538 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f4220021520 con 0x7f422c0a9ad0 2024-01-20T11:22:44.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.083+0000 7f42297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:44.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.083+0000 7f42297fa640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:44.083 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:44.083 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":498,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:44.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.084+0000 7f423316c640 10 librados: watch_flush enter 2024-01-20T11:22:44.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.084+0000 7f423316c640 10 librados: watch_flush exit 2024-01-20T11:22:44.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.084+0000 7f423316c640 20 client.5874.objecter shutdown clearing up homeless session... 2024-01-20T11:22:44.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.084+0000 7f423316c640 10 client.5874.objecter successfully canceled tick 2024-01-20T11:22:44.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.084+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f421806c860 msgr2=0x7f421806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:44.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f421806c860 0x7f421806ed20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f421c036000 tx=0x7f421c03d040 comp rx=0 tx=0).stop 2024-01-20T11:22:44.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 10 monclient: shutdown 2024-01-20T11:22:44.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:44.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f422c0a9ad0 msgr2=0x7f422c13e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f422c0a9ad0 0x7f422c13e780 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f422000e840 tx=0x7f422000ed00 comp rx=0 tx=0).stop 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 shutdown_connections 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f421806c860 0x7f421806ed20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f422c143300 0x7f422c1da470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f422c0a9ad0 0x7f422c13e780 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 --2- 172.21.15.110:0/2228555538 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f422c0a96f0 0x7f422c136770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 >> 172.21.15.110:0/2228555538 conn(0x7f422c0b7bc0 msgr2=0x7f422c0b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 shutdown_connections 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 -- 172.21.15.110:0/2228555538 wait complete. 2024-01-20T11:22:44.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.085+0000 7f423316c640 1 librados: shutdown 2024-01-20T11:22:44.096 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:44.096 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.195+0000 7f8519cb4640 10 monclient: get_monmap_and_config 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.195+0000 7f8519cb4640 10 monclient: build_initial_monmap 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.195+0000 7f8519cb4640 10 monclient: monmap: 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:44.196428+0000 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:44.196428+0000 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:44.197 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.196+0000 7f8519cb4640 1 Processor -- start 2024-01-20T11:22:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.196+0000 7f8519cb4640 1 -- start start 2024-01-20T11:22:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.196+0000 7f8519cb4640 10 monclient: init 2024-01-20T11:22:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f851406a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f851406ac40 0x7f85141455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 10 monclient(hunting): picked mon.noname-c con 0x7f851406ac40 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85140af730 0x7f8514145af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 10 monclient(hunting): picked mon.noname-b con 0x7f85140af730 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140afd30 con 0x7f851406a860 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8514146030 con 0x7f851406ac40 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85140b01c0 con 0x7f85140af730 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8519cb4640 10 monclient(hunting): authenticate will time out at 1807.547513s 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.197+0000 7f8513fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85140af730 0x7f8514145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f850bfff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f851406ac40 0x7f85141455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f85137fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f851406a860 0x7f851413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:43306/0 (socket says 172.21.15.110:43306) 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f85137fe640 1 -- 172.21.15.110:0/938081928 learned_addr learned my addr 172.21.15.110:0/938081928 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f8513fff640 10 monclient(hunting): get_auth_request con 0x7f85140af730 auth_method 0 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f8513fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f8513fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f8513fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f850bfff640 10 monclient(hunting): get_auth_request con 0x7f851406ac40 auth_method 0 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f850bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f850bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f850bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f8513fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f8513fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.198+0000 7f8513fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.199+0000 7f850bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.199+0000 7f850bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.199+0000 7f850bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.200+0000 7f8513fff640 10 monclient(hunting): handle_auth_done global_id 5537 payload 274 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.200+0000 7f8513fff640 1 -- 172.21.15.110:0/938081928 >> v2:172.21.15.134:3300/0 conn(0x7f851406ac40 msgr2=0x7f85141455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.200+0000 7f8513fff640 1 --2- 172.21.15.110:0/938081928 >> v2:172.21.15.134:3300/0 conn(0x7f851406ac40 0x7f85141455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.200+0000 7f8513fff640 1 -- 172.21.15.110:0/938081928 >> v1:172.21.15.134:6789/0 conn(0x7f851406a860 legacy=0x7f851413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.200+0000 7f8513fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.200+0000 7f8513fff640 1 monclient: found mon.noname-b 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.200+0000 7f8513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.200+0000 7f8513fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.200+0000 7f8513fff640 1 -- 172.21.15.110:0/938081928 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851413fb60 con 0x7f85140af730 2024-01-20T11:22:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.202+0000 7f8513fff640 1 --2- 172.21.15.110:0/938081928 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85140af730 0x7f8514145af0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f850c009d90 tx=0x7f850c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c557b4b96cd88509 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:44.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 1 -- 172.21.15.110:0/938081928 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c02cbc0 con 0x7f85140af730 2024-01-20T11:22:44.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:44.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:44.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 10 monclient: dump: 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:44.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 1 -- 172.21.15.110:0/938081928 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c02cd20 con 0x7f85140af730 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 1 -- 172.21.15.110:0/938081928 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c034660 con 0x7f85140af730 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8519cb4640 5 monclient: authenticate success, global_id 5537 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8519cb4640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8519cb4640 4 set_mon_vals no callback set 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.205+0000 7f8512ffd640 10 monclient: dump: 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:44.207 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:44.208 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:44.208 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:44.208 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:44.208 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:44.208 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:44.208 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:44.208 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:44.208 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:44.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 10 monclient: shutdown 2024-01-20T11:22:44.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:44.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 1 -- 172.21.15.110:0/938081928 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85140af730 msgr2=0x7f8514145af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:44.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 1 --2- 172.21.15.110:0/938081928 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85140af730 0x7f8514145af0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f850c009d90 tx=0x7f850c02be10 comp rx=0 tx=0).stop 2024-01-20T11:22:44.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 1 -- 172.21.15.110:0/938081928 shutdown_connections 2024-01-20T11:22:44.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 1 --2- 172.21.15.110:0/938081928 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85140af730 0x7f8514145af0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 1 --2- 172.21.15.110:0/938081928 >> v2:172.21.15.134:3300/0 conn(0x7f851406ac40 0x7f85141455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 1 -- 172.21.15.110:0/938081928 >> 172.21.15.110:0/938081928 conn(0x7f85140b47d0 msgr2=0x7f85140b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 1 -- 172.21.15.110:0/938081928 shutdown_connections 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.206+0000 7f8519cb4640 1 -- 172.21.15.110:0/938081928 wait complete. 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.207+0000 7f8519cb4640 10 monclient: build_initial_monmap 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.207+0000 7f8519cb4640 10 monclient: monmap: 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:44.213 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.207+0000 7f8519cb4640 1 librados: starting msgr at 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.207+0000 7f8519cb4640 1 librados: starting objecter 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.207+0000 7f8519cb4640 1 Processor -- start 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 1 -- start start 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 1 librados: setting wanted keys 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 1 librados: calling monclient init 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient: init 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851406a860 0x7f85141db0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient(hunting): picked mon.noname-a con 0x7f851406a860 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:44.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f851406ac40 0x7f85141db900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient(hunting): picked mon.noname-c con 0x7f851406ac40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85140af730 0x7f8514141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient(hunting): picked mon.noname-b con 0x7f85140af730 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85140715b0 con 0x7f851406a860 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85141396f0 con 0x7f85140af730 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85140717e0 con 0x7f851406ac40 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f8519cb4640 10 monclient(hunting): authenticate will time out at 1807.558732s 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.208+0000 7f850bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f851406ac40 0x7f85141db900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:44.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851406a860 0x7f85141db0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f8513fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85140af730 0x7f8514141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851406a860 0x7f85141db0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58594/0 (socket says 172.21.15.110:58594) 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 1 -- 172.21.15.110:0/2019852939 learned_addr learned my addr 172.21.15.110:0/2019852939 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 10 monclient(hunting): get_auth_request con 0x7f851406a860 auth_method 0 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f850bfff640 10 monclient(hunting): get_auth_request con 0x7f851406ac40 auth_method 0 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f850bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f850bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f850bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f8513fff640 10 monclient(hunting): get_auth_request con 0x7f85140af730 auth_method 0 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f8513fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:44.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f8513fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f8513fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f85137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f850bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f850bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.209+0000 7f850bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.210+0000 7f8513fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.210+0000 7f8513fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.210+0000 7f8513fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 10 monclient(hunting): handle_auth_done global_id 5877 payload 824 2024-01-20T11:22:44.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 1 -- 172.21.15.110:0/2019852939 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f851406ac40 msgr2=0x7f85141db900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:44.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f851406ac40 0x7f85141db900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 1 -- 172.21.15.110:0/2019852939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85140af730 msgr2=0x7f8514141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:44.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85140af730 0x7f8514141120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:44.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 1 monclient: found mon.noname-a 2024-01-20T11:22:44.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:44.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:44.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.212+0000 7f85137fe640 1 -- 172.21.15.110:0/2019852939 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c009a70 con 0x7f851406a860 2024-01-20T11:22:44.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.214+0000 7f85137fe640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851406a860 0x7f85141db0c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f850400c400 tx=0x7f850400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:44.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.214+0000 7f8510ff9640 10 client.?.objecter ms_handle_connect 0x7f851406a860 2024-01-20T11:22:44.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.214+0000 7f8510ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:44.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 1 -- 172.21.15.110:0/2019852939 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8504005ce0 con 0x7f851406a860 2024-01-20T11:22:44.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:44.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:44.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 10 monclient: dump: 2024-01-20T11:22:44.220 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 10 monclient: _finish_auth 0 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 1 -- 172.21.15.110:0/2019852939 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8504005e40 con 0x7f851406a860 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8519cb4640 5 monclient: authenticate success, global_id 5877 2024-01-20T11:22:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8519cb4640 10 monclient: _renew_subs 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8519cb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8514071ac0 con 0x7f851406a860 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 1 -- 172.21.15.110:0/2019852939 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85040072d0 con 0x7f851406a860 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8510ff9640 10 monclient: dump: 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f85127fc640 4 set_mon_vals no callback set 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8519cb4640 10 client.5877.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8519cb4640 10 monclient: _renew_subs 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8519cb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8514141660 con 0x7f851406a860 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.218+0000 7f8519cb4640 1 librados: init done 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.219+0000 7f8519cb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.219+0000 7f8519cb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.219+0000 7f8519cb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:44.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.219+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8514075490 con 0x7f851406a860 2024-01-20T11:22:44.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.465+0000 7f8510ff9640 1 -- 172.21.15.110:0/2019852939 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f85040074f0 con 0x7f851406a860 2024-01-20T11:22:44.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.465+0000 7f8510ff9640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84f406c8b0 0x7f84f406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:44.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.465+0000 7f850bfff640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84f406c8b0 0x7f84f406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:44.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.465+0000 7f850bfff640 10 monclient: get_auth_request con 0x7f84f406c8b0 auth_method 0 2024-01-20T11:22:44.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.466+0000 7f850bfff640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84f406c8b0 0x7f84f406ed70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8514142cb0 tx=0x7f84fc009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:44.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.466+0000 7f8510ff9640 10 client.5877.objecter ms_handle_connect 0x7f84f406c8b0 2024-01-20T11:22:44.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.478+0000 7f8510ff9640 1 -- 172.21.15.110:0/2019852939 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8504057ca0 con 0x7f851406a860 2024-01-20T11:22:44.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.479+0000 7f8510ff9640 10 client.5877.objecter ms_dispatch 0x7f8514000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:44.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.479+0000 7f8510ff9640 3 client.5877.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:44.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.479+0000 7f8510ff9640 3 client.5877.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:44.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.479+0000 7f8510ff9640 20 client.5877.objecter dump_active .. 0 homeless 2024-01-20T11:22:44.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.479+0000 7f8510ff9640 1 -- 172.21.15.110:0/2019852939 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8504053fe0 con 0x7f851406a860 2024-01-20T11:22:44.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.479+0000 7f8510ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:44.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.479+0000 7f8510ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:44.538 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:35.562 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:44.541 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:35.565 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:44.555 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:35.580 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:44.582 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:35.589 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:44.585 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:35.591 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:44.589 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:35.614 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:44.592 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:35.617 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:44.600 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:35.606 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:44.608 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:35.632 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:44.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.676+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f851413b5e0 con 0x7f84f406c8b0 2024-01-20T11:22:44.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.679+0000 7f8510ff9640 1 -- 172.21.15.110:0/2019852939 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63804 (secure 0 0 0) 0x7f851413b5e0 con 0x7f84f406c8b0 2024-01-20T11:22:44.679 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:44.680 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 10 librados: watch_flush enter 2024-01-20T11:22:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 10 librados: watch_flush exit 2024-01-20T11:22:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 20 client.5877.objecter shutdown clearing up homeless session... 2024-01-20T11:22:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 10 client.5877.objecter successfully canceled tick 2024-01-20T11:22:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84f406c8b0 msgr2=0x7f84f406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84f406c8b0 0x7f84f406ed70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8514142cb0 tx=0x7f84fc009190 comp rx=0 tx=0).stop 2024-01-20T11:22:44.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 10 monclient: shutdown 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851406a860 msgr2=0x7f85141db0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851406a860 0x7f85141db0c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f850400c400 tx=0x7f850400c8c0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.681+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 shutdown_connections 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.682+0000 7f8519cb4640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84f406c8b0 0x7f84f406ed70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.682+0000 7f8519cb4640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85140af730 0x7f8514141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.682+0000 7f8519cb4640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f851406ac40 0x7f85141db900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.682+0000 7f8519cb4640 1 --2- 172.21.15.110:0/2019852939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851406a860 0x7f85141db0c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.682+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 >> 172.21.15.110:0/2019852939 conn(0x7f85140b47d0 msgr2=0x7f85140b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.682+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 shutdown_connections 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.682+0000 7f8519cb4640 1 -- 172.21.15.110:0/2019852939 wait complete. 2024-01-20T11:22:44.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:44.682+0000 7f8519cb4640 1 librados: shutdown 2024-01-20T11:22:44.693 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-01-20T11:22:42.801366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002899"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.2839999999999998},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.7279999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":8.8109999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":13.225}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":16.472999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.9630000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":11.167999999999999},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":11.836}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.544},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.744}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":12.561},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":15.662000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313310,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":10.778},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.3360000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":5.7359999999999998},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":8.0470000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.266},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":12.343}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":15.954000000000001},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":14.670999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":13.993},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":10.413}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":12.926},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":16.661999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313311,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":13.784000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":6.5739999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":15.194000000000001},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":10.106}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":11.656000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":14.436999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.936999999999999},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.5540000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.9760000000000009},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.439}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":17.571000000000002},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.871}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444124,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":8.8919999999999995},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":10.210000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.1530000000000005},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":6.8739999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.548999999999999},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":5.6840000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.532},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":7.5620000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":14.382},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.5790000000000006}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.840999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":13.933}]}]},{"osd":1,"up_from":17,"seq":73014444125,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":5.3010000000000002},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.0359999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":12.279999999999999},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":11.557}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":14.023},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":9.343}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":7.9550000000000001},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":15.879}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":8.6579999999999995},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":13.367000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":10.603999999999999},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":14.654999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444125,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":4.4809999999999999},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":6.4699999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.6229999999999993},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":11.404,"5min":11.94,"15min":23.978999999999999},"min":{"1min":5.2389999999999999,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":17.901,"5min":17.901,"15min":358.738},"last":13.105},{"interface":"front","average":{"1min":10.679,"5min":10.849,"15min":12.452999999999999},"min":{"1min":6.4009999999999998,"5min":5.6870000000000003,"15min":5.6870000000000003},"max":{"1min":15.904,"5min":19.866,"15min":21.004999999999999},"last":7.992}]},{"osd":4,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":12.836,"5min":12.747999999999999,"15min":25.925000000000001},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":20.478000000000002,"5min":20.478000000000002,"15min":372.02300000000002},"last":11.119999999999999},{"interface":"front","average":{"1min":10.058,"5min":10.138,"15min":23.149999999999999},"min":{"1min":6.633,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":14.06,"5min":19.41,"15min":372.70800000000003},"last":8.9459999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.755000000000001},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.423}]},{"osd":7,"last update":"Sat Jan 20 11:21:37 2024","interfaces":[{"interface":"back","average":{"1min":14.904,"5min":14.977,"15min":16.68},"min":{"1min":10.98,"5min":9.2509999999999994,"15min":9.2509999999999994},"max":{"1min":18.472000000000001,"5min":19.030999999999999,"15min":27.385999999999999},"last":14.481999999999999},{"interface":"front","average":{"1min":11.574,"5min":11.768000000000001,"15min":12.231999999999999},"min":{"1min":7.2220000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":16.992999999999999,"5min":17.597999999999999,"15min":21.529},"last":10.411}]}]},{"osd":6,"up_from":15,"seq":64424509534,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":4.8929999999999998},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":6.9359999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":14.938000000000001},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":11.458}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":8.2430000000000003},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":12.337}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":13.045},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":8.9329999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":10.385999999999999},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.214}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":17.154},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.001999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:47.695 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:47.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.794+0000 7f850175a640 10 monclient: get_monmap_and_config 2024-01-20T11:22:47.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.794+0000 7f850175a640 10 monclient: build_initial_monmap 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.794+0000 7f850175a640 10 monclient: monmap: 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:47.795213+0000 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:47.795213+0000 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:47.796 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:47.796 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:47.796 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 1 Processor -- start 2024-01-20T11:22:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 1 -- start start 2024-01-20T11:22:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient: init 2024-01-20T11:22:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-01-20T11:22:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f84fc0b4d10 0x7f84fc0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient(hunting): picked mon.noname-c con 0x7f84fc0b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84fc0b17c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84fc145980 0x7f84fc145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient(hunting): picked mon.noname-a con 0x7f84fc145980 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc146370 con 0x7f84fc0b17c0 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84fc0b50f0 con 0x7f84fc145980 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84fc0aade0 con 0x7f84fc0b4d10 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.795+0000 7f850175a640 10 monclient(hunting): authenticate will time out at 1811.146136s 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.796+0000 7f84fb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84fc145980 0x7f84fc145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.796+0000 7f84faffd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f84fc0b4d10 0x7f84fc0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.796+0000 7f84fb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84fc145980 0x7f84fc145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58596/0 (socket says 172.21.15.110:58596) 2024-01-20T11:22:47.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.796+0000 7f84fb7fe640 1 -- 172.21.15.110:0/3245705225 learned_addr learned my addr 172.21.15.110:0/3245705225 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.796+0000 7f84fb7fe640 10 monclient(hunting): get_auth_request con 0x7f84fc145980 auth_method 0 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.796+0000 7f84fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.796+0000 7f84fb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.796+0000 7f84fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84faffd640 10 monclient(hunting): get_auth_request con 0x7f84fc0b4d10 auth_method 0 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84faffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84faffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:47.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.797+0000 7f84faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.799+0000 7f84fb7fe640 10 monclient(hunting): handle_auth_done global_id 5880 payload 274 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.799+0000 7f84fb7fe640 1 -- 172.21.15.110:0/3245705225 >> v2:172.21.15.134:3300/0 conn(0x7f84fc0b4d10 msgr2=0x7f84fc0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.799+0000 7f84fb7fe640 1 --2- 172.21.15.110:0/3245705225 >> v2:172.21.15.134:3300/0 conn(0x7f84fc0b4d10 0x7f84fc0b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.799+0000 7f84fb7fe640 1 -- 172.21.15.110:0/3245705225 >> v1:172.21.15.110:6789/0 conn(0x7f84fc0b17c0 legacy=0x7f84fc0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.799+0000 7f84fb7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.799+0000 7f84fb7fe640 1 monclient: found mon.noname-a 2024-01-20T11:22:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.799+0000 7f84fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.799+0000 7f84fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.799+0000 7f84fb7fe640 1 -- 172.21.15.110:0/3245705225 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc145460 con 0x7f84fc145980 2024-01-20T11:22:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.801+0000 7f84fb7fe640 1 --2- 172.21.15.110:0/3245705225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc145980 0x7f84fc145e30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f84fc0b8cd0 tx=0x7f84f402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f6cb4e6363bea0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 1 -- 172.21.15.110:0/3245705225 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f402c960 con 0x7f84fc145980 2024-01-20T11:22:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 10 monclient: dump: 2024-01-20T11:22:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:47.807 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 10 monclient: _finish_auth 0 2024-01-20T11:22:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:22:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 1 -- 172.21.15.110:0/3245705225 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f402cac0 con 0x7f84fc145980 2024-01-20T11:22:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f850175a640 5 monclient: authenticate success, global_id 5880 2024-01-20T11:22:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f850175a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 1 -- 172.21.15.110:0/3245705225 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f402ce50 con 0x7f84fc145980 2024-01-20T11:22:47.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f850175a640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:47.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f850175a640 4 set_mon_vals no callback set 2024-01-20T11:22:47.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:47.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:47.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.805+0000 7f84fa7fc640 10 monclient: dump: 2024-01-20T11:22:47.808 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:47.808 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:47.808 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:47.809 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:47.809 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:47.809 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:47.809 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:47.809 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:47.809 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:47.810 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:47.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~230s 2024-01-20T11:22:47.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 10 monclient: shutdown 2024-01-20T11:22:47.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:47.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 -- 172.21.15.110:0/3245705225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc145980 msgr2=0x7f84fc145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:47.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 --2- 172.21.15.110:0/3245705225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc145980 0x7f84fc145e30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f84fc0b8cd0 tx=0x7f84f402bdc0 comp rx=0 tx=0).stop 2024-01-20T11:22:47.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 -- 172.21.15.110:0/3245705225 shutdown_connections 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 --2- 172.21.15.110:0/3245705225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc145980 0x7f84fc145e30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 --2- 172.21.15.110:0/3245705225 >> v2:172.21.15.134:3300/0 conn(0x7f84fc0b4d10 0x7f84fc0b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 -- 172.21.15.110:0/3245705225 >> 172.21.15.110:0/3245705225 conn(0x7f84fc066590 msgr2=0x7f84fc0669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 -- 172.21.15.110:0/3245705225 shutdown_connections 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 -- 172.21.15.110:0/3245705225 wait complete. 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 10 monclient: build_initial_monmap 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 10 monclient: monmap: 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:47.819 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:47.820 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:47.820 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:47.820 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:47.820 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:47.820 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:47.820 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:47.820 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:47.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 librados: starting msgr at 2024-01-20T11:22:47.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.806+0000 7f850175a640 1 librados: starting objecter 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 Processor -- start 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 -- start start 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 librados: setting wanted keys 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 librados: calling monclient init 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient: init 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc0b17c0 0x7f84fc142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient(hunting): picked mon.noname-a con 0x7f84fc0b17c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:47.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84fc0b4d10 0x7f84fc146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:47.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient(hunting): picked mon.noname-c con 0x7f84fc0b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:47.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84fc145980 0x7f84fc1d7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:47.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient(hunting): picked mon.noname-b con 0x7f84fc145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:47.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:47.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc147120 con 0x7f84fc0b17c0 2024-01-20T11:22:47.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:47.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc147290 con 0x7f84fc145980 2024-01-20T11:22:47.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:47.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc147450 con 0x7f84fc0b4d10 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.807+0000 7f850175a640 10 monclient(hunting): authenticate will time out at 1811.157924s 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc0b17c0 0x7f84fc142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84f3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84fc0b4d10 0x7f84fc146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84fb7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84fc145980 0x7f84fc1d7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc0b17c0 0x7f84fc142ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58604/0 (socket says 172.21.15.110:58604) 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 1 -- 172.21.15.110:0/4192205875 learned_addr learned my addr 172.21.15.110:0/4192205875 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 10 monclient(hunting): get_auth_request con 0x7f84fc0b17c0 auth_method 0 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:47.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84f3fff640 10 monclient(hunting): get_auth_request con 0x7f84fc0b4d10 auth_method 0 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84f3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84fb7fe640 10 monclient(hunting): get_auth_request con 0x7f84fc145980 auth_method 0 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84fb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:47.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:47.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:47.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.808+0000 7f84faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:47.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.809+0000 7f84f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:47.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.809+0000 7f84f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:47.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.809+0000 7f84f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:47.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.809+0000 7f84fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:47.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.809+0000 7f84fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:47.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.809+0000 7f84fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:47.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 10 monclient(hunting): handle_auth_done global_id 5883 payload 824 2024-01-20T11:22:47.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 1 -- 172.21.15.110:0/4192205875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84fc0b4d10 msgr2=0x7f84fc146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:47.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84fc0b4d10 0x7f84fc146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:47.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 1 -- 172.21.15.110:0/4192205875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84fc145980 msgr2=0x7f84fc1d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:47.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84fc145980 0x7f84fc1d7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:47.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:47.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 1 monclient: found mon.noname-a 2024-01-20T11:22:47.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:47.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:47.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.811+0000 7f84faffd640 1 -- 172.21.15.110:0/4192205875 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f4009b30 con 0x7f84fc0b17c0 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.813+0000 7f84faffd640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc0b17c0 0x7f84fc142ee0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00e9b0 tx=0x7f84ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.813+0000 7f84f37fe640 10 client.?.objecter ms_handle_connect 0x7f84fc0b17c0 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.813+0000 7f84f37fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 1 -- 172.21.15.110:0/4192205875 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec009670 con 0x7f84fc0b17c0 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 10 monclient: dump: 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:47.827 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:47.828 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:47.828 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:47.828 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:47.828 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:47.828 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:47.828 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:47.828 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:47.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:47.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f850175a640 5 monclient: authenticate success, global_id 5883 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 1 -- 172.21.15.110:0/4192205875 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec00c9a0 con 0x7f84fc0b17c0 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 1 -- 172.21.15.110:0/4192205875 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec00cd50 con 0x7f84fc0b17c0 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f850175a640 10 monclient: _renew_subs 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f850175a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84fc1d7e90 con 0x7f84fc0b17c0 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f9ffb640 4 set_mon_vals no callback set 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f84f37fe640 10 monclient: dump: 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:47.829 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f850175a640 10 client.5883.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f850175a640 10 monclient: _renew_subs 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f850175a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84fc1d81e0 con 0x7f84fc0b17c0 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.817+0000 7f850175a640 1 librados: init done 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.818+0000 7f850175a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:47.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.818+0000 7f850175a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:47.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.818+0000 7f850175a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:47.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.818+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84fc13bd20 con 0x7f84fc0b17c0 2024-01-20T11:22:47.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.837+0000 7f84f37fe640 1 -- 172.21.15.110:0/4192205875 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f84ec013470 con 0x7f84fc0b17c0 2024-01-20T11:22:47.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.838+0000 7f84f37fe640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84dc06c860 0x7f84dc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:47.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.838+0000 7f84f3fff640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84dc06c860 0x7f84dc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:47.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.838+0000 7f84f3fff640 10 monclient: get_auth_request con 0x7f84dc06c860 auth_method 0 2024-01-20T11:22:47.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.838+0000 7f84f3fff640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84dc06c860 0x7f84dc06ed20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f84fc1d8a10 tx=0x7f84e4009060 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:47.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.839+0000 7f84f37fe640 10 client.5883.objecter ms_handle_connect 0x7f84dc06c860 2024-01-20T11:22:47.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.841+0000 7f84f37fe640 1 -- 172.21.15.110:0/4192205875 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f84ec058c20 con 0x7f84fc0b17c0 2024-01-20T11:22:47.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.841+0000 7f84f37fe640 10 client.5883.objecter ms_dispatch 0x7f84fc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:47.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.841+0000 7f84f37fe640 3 client.5883.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:47.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.841+0000 7f84f37fe640 3 client.5883.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:47.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:47.841+0000 7f84f37fe640 20 client.5883.objecter dump_active .. 0 homeless 2024-01-20T11:22:48.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.099+0000 7f84f37fe640 1 -- 172.21.15.110:0/4192205875 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f84ec054f60 con 0x7f84fc0b17c0 2024-01-20T11:22:48.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.099+0000 7f84f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:48.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.099+0000 7f84f37fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:48.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.275+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84fc075490 con 0x7f84dc06c860 2024-01-20T11:22:48.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.278+0000 7f84f37fe640 1 -- 172.21.15.110:0/4192205875 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63867 (secure 0 0 0) 0x7f84fc075490 con 0x7f84dc06c860 2024-01-20T11:22:48.279 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:48.279 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:48.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 10 librados: watch_flush enter 2024-01-20T11:22:48.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 10 librados: watch_flush exit 2024-01-20T11:22:48.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 20 client.5883.objecter shutdown clearing up homeless session... 2024-01-20T11:22:48.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 10 client.5883.objecter successfully canceled tick 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84dc06c860 msgr2=0x7f84dc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84dc06c860 0x7f84dc06ed20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f84fc1d8a10 tx=0x7f84e4009060 comp rx=0 tx=0).stop 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 10 monclient: shutdown 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc0b17c0 msgr2=0x7f84fc142ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.280+0000 7f850175a640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc0b17c0 0x7f84fc142ee0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00e9b0 tx=0x7f84ec00ee70 comp rx=0 tx=0).stop 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.281+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 shutdown_connections 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.281+0000 7f850175a640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f84dc06c860 0x7f84dc06ed20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.281+0000 7f850175a640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84fc145980 0x7f84fc1d7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.281+0000 7f850175a640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84fc0b4d10 0x7f84fc146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.281+0000 7f850175a640 1 --2- 172.21.15.110:0/4192205875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84fc0b17c0 0x7f84fc142ee0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.281+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 >> 172.21.15.110:0/4192205875 conn(0x7f84fc066590 msgr2=0x7f84fc0b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:48.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.281+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 shutdown_connections 2024-01-20T11:22:48.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.281+0000 7f850175a640 1 -- 172.21.15.110:0/4192205875 wait complete. 2024-01-20T11:22:48.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.281+0000 7f850175a640 1 librados: shutdown 2024-01-20T11:22:48.291 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-01-20T11:22:46.802139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002811"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280608,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":12.135},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":17.138999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.9260000000000002},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":7.9409999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":18.085000000000001},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":16.286999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.493},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":15.304}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":11.334},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":12.845000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":8.8010000000000002},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":10.683999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313311,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.4209999999999994},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.0119999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":6.6429999999999998},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.3200000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":10.845000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":12.218}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":11.552},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":10.254}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":12.896000000000001},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":15.323}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":13.523},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":16.079999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313311,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":13.784000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":6.5739999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":15.194000000000001},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":10.106}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":11.656000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":14.436999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.936999999999999},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.5540000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.9760000000000009},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.439}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":17.571000000000002},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.871}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444125,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":9.7219999999999995},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":13.333}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.1029999999999998},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":14.138}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":15.071999999999999},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":6.6950000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.858000000000001},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.632}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":11.509},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.423}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":8.8930000000000007},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":16.628}]}]},{"osd":1,"up_from":17,"seq":73014444126,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":7.0289999999999999},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":5.9269999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":10.135999999999999},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":12.885999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":11.537000000000001},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":13.635999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":7.6820000000000004},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":10.968999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":8.2569999999999997},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":15.119}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":9.4130000000000003},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":14.371}]}]},{"osd":0,"up_from":17,"seq":73014444126,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":7.0129999999999999},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":4.0439999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":8.9540000000000006},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":13.021000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":15.385},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.8469999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":10.369999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":11.468999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":14.506},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":16.745000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":17.378},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":12.204000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509535,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":12.222},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":6.024}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":6.4699999999999998},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.839}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":7.3970000000000002},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":14.073}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.885999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":10.247}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":10.945},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":9.6259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":8.8399999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.827999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:48.293 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.400+0000 7feef4eb4640 10 monclient: get_monmap_and_config 2024-01-20T11:22:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.400+0000 7feef4eb4640 10 monclient: build_initial_monmap 2024-01-20T11:22:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.400+0000 7feef4eb4640 10 monclient: monmap: 2024-01-20T11:22:48.400 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:48.401292+0000 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:48.401292+0000 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:48.401 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.401+0000 7feef4eb4640 1 Processor -- start 2024-01-20T11:22:48.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.401+0000 7feef4eb4640 1 -- start start 2024-01-20T11:22:48.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.401+0000 7feef4eb4640 10 monclient: init 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef006bb00 0x7feef00b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 10 monclient(hunting): picked mon.noname-b con 0x7feef006bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7feef00af790 0x7feef0145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 10 monclient(hunting): picked mon.noname-c con 0x7feef00af790 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feef00b0900 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef006c220 con 0x7feef00b0900 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef006bee0 con 0x7feef00af790 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef00ada30 con 0x7feef006bb00 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feef4eb4640 10 monclient(hunting): authenticate will time out at 1811.752274s 2024-01-20T11:22:48.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feeeed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7feef00b0900 0x7feef0145cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46348/0 (socket says 172.21.15.110:46348) 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feeeed9d640 1 -- 172.21.15.110:0/443913682 learned_addr learned my addr 172.21.15.110:0/443913682 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feeee59c640 1 --2- 172.21.15.110:0/443913682 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef006bb00 0x7feef00b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.402+0000 7feee7fff640 1 --2- 172.21.15.110:0/443913682 >> v2:172.21.15.134:3300/0 conn(0x7feef00af790 0x7feef0145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feee7fff640 10 monclient(hunting): get_auth_request con 0x7feef00af790 auth_method 0 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feee7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feee7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feeee59c640 10 monclient(hunting): get_auth_request con 0x7feef006bb00 auth_method 0 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feeee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feeee59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feeee59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feeee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feeee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feeee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:48.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feee7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:48.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feee7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:48.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.403+0000 7feee7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.404+0000 7feeedd9b640 1 -- 172.21.15.110:0/443913682 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464810894 0 0) 0x7feef006c220 con 0x7feef00b0900 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.404+0000 7feeedd9b640 10 monclient(hunting): _negotiate 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.404+0000 7feeedd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.404+0000 7feeedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.404+0000 7feeedd9b640 10 monclient(hunting): my global_id is 5886 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.404+0000 7feeedd9b640 1 -- 172.21.15.110:0/443913682 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feedc003520 con 0x7feef00b0900 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.405+0000 7feeee59c640 10 monclient(hunting): handle_auth_done global_id 5546 payload 274 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.405+0000 7feeee59c640 1 -- 172.21.15.110:0/443913682 >> v2:172.21.15.134:3300/0 conn(0x7feef00af790 msgr2=0x7feef0145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.405+0000 7feeee59c640 1 --2- 172.21.15.110:0/443913682 >> v2:172.21.15.134:3300/0 conn(0x7feef00af790 0x7feef0145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.405+0000 7feeee59c640 1 -- 172.21.15.110:0/443913682 >> v1:172.21.15.110:6789/0 conn(0x7feef00b0900 legacy=0x7feef0145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.405+0000 7feeee59c640 10 monclient: _finish_hunting 0 2024-01-20T11:22:48.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.405+0000 7feeee59c640 1 monclient: found mon.noname-b 2024-01-20T11:22:48.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.405+0000 7feeee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:48.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.405+0000 7feeee59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:48.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.405+0000 7feeee59c640 1 -- 172.21.15.110:0/443913682 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00add10 con 0x7feef006bb00 2024-01-20T11:22:48.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.407+0000 7feeee59c640 1 --2- 172.21.15.110:0/443913682 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef006bb00 0x7feef00b03c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feee00097a0 tx=0x7feee002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d54f6a8d4fcead server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:48.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 1 -- 172.21.15.110:0/443913682 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee002e070 con 0x7feef006bb00 2024-01-20T11:22:48.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:48.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 10 monclient: dump: 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 10 monclient: _finish_auth 0 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 10 monclient: _check_auth_tickets 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:48.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 1 -- 172.21.15.110:0/443913682 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee00315a0 con 0x7feef006bb00 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feef4eb4640 5 monclient: authenticate success, global_id 5546 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feef4eb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feef4eb4640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feef4eb4640 4 set_mon_vals no callback set 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 1 -- 172.21.15.110:0/443913682 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0031930 con 0x7feef006bb00 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.410+0000 7feeedd9b640 10 monclient: dump: 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.412 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:48.413 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.413 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:48.413 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:48.413 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:48.413 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 10 monclient: shutdown 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 1 -- 172.21.15.110:0/443913682 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef006bb00 msgr2=0x7feef00b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 1 --2- 172.21.15.110:0/443913682 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef006bb00 0x7feef00b03c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feee00097a0 tx=0x7feee002b840 comp rx=0 tx=0).stop 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 1 -- 172.21.15.110:0/443913682 shutdown_connections 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 1 --2- 172.21.15.110:0/443913682 >> v2:172.21.15.134:3300/0 conn(0x7feef00af790 0x7feef0145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 1 --2- 172.21.15.110:0/443913682 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef006bb00 0x7feef00b03c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 1 -- 172.21.15.110:0/443913682 >> 172.21.15.110:0/443913682 conn(0x7feef0066d60 msgr2=0x7feef00b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 1 -- 172.21.15.110:0/443913682 shutdown_connections 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.411+0000 7feef4eb4640 1 -- 172.21.15.110:0/443913682 wait complete. 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.412+0000 7feef4eb4640 10 monclient: build_initial_monmap 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.412+0000 7feef4eb4640 10 monclient: monmap: 2024-01-20T11:22:48.417 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:48.418 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:48.418 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:48.418 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:48.418 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:48.418 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.418 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:48.418 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 librados: starting msgr at 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 librados: starting objecter 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 Processor -- start 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 -- start start 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 librados: setting wanted keys 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 librados: calling monclient init 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient: init 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef006bb00 0x7feef00b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient(hunting): picked mon.noname-a con 0x7feef006bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:48.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef00af790 0x7feef011d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient(hunting): picked mon.noname-c con 0x7feef00af790 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:48.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feef00b0900 0x7feef013bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient(hunting): picked mon.noname-b con 0x7feef00b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:48.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:48.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef011de00 con 0x7feef006bb00 2024-01-20T11:22:48.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:48.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef011dae0 con 0x7feef00b0900 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef0071300 con 0x7feef00af790 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.413+0000 7feef4eb4640 10 monclient(hunting): authenticate will time out at 1811.764127s 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeee59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef006bb00 0x7feef00b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeeed9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feef00b0900 0x7feef013bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feee7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef00af790 0x7feef011d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeeed9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feef00b0900 0x7feef013bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:38060/0 (socket says 172.21.15.110:38060) 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeeed9d640 1 -- 172.21.15.110:0/1116222802 learned_addr learned my addr 172.21.15.110:0/1116222802 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeeed9d640 10 monclient(hunting): get_auth_request con 0x7feef00b0900 auth_method 0 2024-01-20T11:22:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:48.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeeed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeeed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeee59c640 10 monclient(hunting): get_auth_request con 0x7feef006bb00 auth_method 0 2024-01-20T11:22:48.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:48.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeee59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feeee59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feee7fff640 10 monclient(hunting): get_auth_request con 0x7feef00af790 auth_method 0 2024-01-20T11:22:48.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feee7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:48.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feee7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.414+0000 7feee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.415+0000 7feeeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.415+0000 7feeeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.415+0000 7feeeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.415+0000 7feeee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.415+0000 7feeee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.415+0000 7feeee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.415+0000 7feee7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.415+0000 7feee7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:48.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.415+0000 7feee7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:48.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.417+0000 7feeeed9d640 10 monclient(hunting): handle_auth_done global_id 5992 payload 824 2024-01-20T11:22:48.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.417+0000 7feeeed9d640 1 -- 172.21.15.110:0/1116222802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef00af790 msgr2=0x7feef011d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.417+0000 7feeeed9d640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef00af790 0x7feef011d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.417+0000 7feeeed9d640 1 -- 172.21.15.110:0/1116222802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef006bb00 msgr2=0x7feef00b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.417+0000 7feeeed9d640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef006bb00 0x7feef00b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.417+0000 7feeeed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:22:48.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.418+0000 7feeeed9d640 1 monclient: found mon.noname-b 2024-01-20T11:22:48.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.418+0000 7feeeed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.418+0000 7feeeed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.418+0000 7feeeed9d640 1 -- 172.21.15.110:0/1116222802 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0009400 con 0x7feef00b0900 2024-01-20T11:22:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.420+0000 7feeeed9d640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feef00b0900 0x7feef013bd10 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7feee800caa0 tx=0x7feee800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.420+0000 7feee6ffd640 10 client.?.objecter ms_handle_connect 0x7feef00b0900 2024-01-20T11:22:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.420+0000 7feee6ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 1 -- 172.21.15.110:0/1116222802 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee800ac00 con 0x7feef00b0900 2024-01-20T11:22:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 10 monclient: dump: 2024-01-20T11:22:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 1 -- 172.21.15.110:0/1116222802 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee800ad60 con 0x7feef00b0900 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 1 -- 172.21.15.110:0/1116222802 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8012620 con 0x7feef00b0900 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feef4eb4640 5 monclient: authenticate success, global_id 5992 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:48.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feee6ffd640 10 monclient: dump: 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:48.432 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feef4eb4640 10 monclient: _renew_subs 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feef4eb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feeed59a640 4 set_mon_vals no callback set 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef013c250 con 0x7feef00b0900 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feef4eb4640 10 client.5992.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feef4eb4640 10 monclient: _renew_subs 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feef4eb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef013c710 con 0x7feef00b0900 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.424+0000 7feef4eb4640 1 librados: init done 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.425+0000 7feef4eb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.425+0000 7feef4eb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.425+0000 7feef4eb4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:48.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.425+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef0075490 con 0x7feef00b0900 2024-01-20T11:22:48.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.668+0000 7feee6ffd640 1 -- 172.21.15.110:0/1116222802 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7feee8007070 con 0x7feef00b0900 2024-01-20T11:22:48.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.668+0000 7feee6ffd640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feed006c8b0 0x7feed006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.668+0000 7feeee59c640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feed006c8b0 0x7feed006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.668+0000 7feeee59c640 10 monclient: get_auth_request con 0x7feed006c8b0 auth_method 0 2024-01-20T11:22:48.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.669+0000 7feee6ffd640 10 client.5992.objecter ms_handle_connect 0x7feed006c8b0 2024-01-20T11:22:48.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.669+0000 7feeee59c640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feed006c8b0 0x7feed006ed70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7feee0036000 tx=0x7feee003d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:48.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.671+0000 7feee6ffd640 1 -- 172.21.15.110:0/1116222802 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feee8058630 con 0x7feef00b0900 2024-01-20T11:22:48.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.671+0000 7feee6ffd640 10 client.5992.objecter ms_dispatch 0x7feef0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:48.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.671+0000 7feee6ffd640 3 client.5992.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:48.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.671+0000 7feee6ffd640 3 client.5992.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:48.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.672+0000 7feee6ffd640 20 client.5992.objecter dump_active .. 0 homeless 2024-01-20T11:22:48.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.684+0000 7feee6ffd640 1 -- 172.21.15.110:0/1116222802 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7feee8054970 con 0x7feef00b0900 2024-01-20T11:22:48.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.684+0000 7feee6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:48.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.684+0000 7feee6ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:48.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.851+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feef006c910 con 0x7feed006c8b0 2024-01-20T11:22:48.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.854+0000 7feee6ffd640 1 -- 172.21.15.110:0/1116222802 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63850 (secure 0 0 0) 0x7feee00319d0 con 0x7feed006c8b0 2024-01-20T11:22:48.855 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:48.855 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:48.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 10 librados: watch_flush enter 2024-01-20T11:22:48.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 10 librados: watch_flush exit 2024-01-20T11:22:48.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 20 client.5992.objecter shutdown clearing up homeless session... 2024-01-20T11:22:48.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 10 client.5992.objecter successfully canceled tick 2024-01-20T11:22:48.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feed006c8b0 msgr2=0x7feed006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feed006c8b0 0x7feed006ed70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7feee0036000 tx=0x7feee003d040 comp rx=0 tx=0).stop 2024-01-20T11:22:48.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 10 monclient: shutdown 2024-01-20T11:22:48.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:48.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feef00b0900 msgr2=0x7feef013bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feef00b0900 0x7feef013bd10 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7feee800caa0 tx=0x7feee800cf60 comp rx=0 tx=0).stop 2024-01-20T11:22:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 shutdown_connections 2024-01-20T11:22:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7feed006c8b0 0x7feed006ed70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feef00b0900 0x7feef013bd10 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feef00af790 0x7feef011d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 --2- 172.21.15.110:0/1116222802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef006bb00 0x7feef00b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 >> 172.21.15.110:0/1116222802 conn(0x7feef0066d60 msgr2=0x7feef0144ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 shutdown_connections 2024-01-20T11:22:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 -- 172.21.15.110:0/1116222802 wait complete. 2024-01-20T11:22:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.856+0000 7feef4eb4640 1 librados: shutdown 2024-01-20T11:22:48.867 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-01-20T11:22:48.802575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002746"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":5.0149999999999997},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":7.1769999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":5.79},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":9.6630000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.637}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":12.725},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":14.407999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":11.016999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":13.592000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":15.298999999999999},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":10.475}]}]},{"osd":7,"up_from":21,"seq":90194313311,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.4209999999999994},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.0119999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":6.6429999999999998},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.3200000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":10.845000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":12.218}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":11.552},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":10.254}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":12.896000000000001},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":15.323}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":13.523},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":16.079999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313312,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0010000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":6.2389999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":12.635999999999999},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.2650000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":7.968},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.084}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":13.257999999999999},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":10.869}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.5779999999999994},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":11.455}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.805999999999999},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":14.48}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444126,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.1459999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.0629999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.718},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.927}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.898},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.5920000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.786},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":8.9760000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":10.263},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.6680000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.007999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.577999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444127,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":8.4160000000000004},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.0510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":9.1349999999999998},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":15.407999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":12.430999999999999},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":6.6059999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":9.7029999999999994},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":11.792999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":14.323},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.012}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":10.259},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":16.466999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444127,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.7519999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.1669999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.0839999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":11.859}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":10.691000000000001},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.2390000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":9.766},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":12.696999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":11.282999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":14.553000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":16.158000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":13.630000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509535,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":12.222},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":6.024}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":6.4699999999999998},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.839}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":7.3970000000000002},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":14.073}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.885999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":10.247}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":10.945},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":9.6259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":8.8399999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.827999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:48.868 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:48.869 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:48.869 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:48.869 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:48.869 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:48.869 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:48.869 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:48.869 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:48.870 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:48.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.968+0000 7f43b596c640 10 monclient: get_monmap_and_config 2024-01-20T11:22:48.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.968+0000 7f43b596c640 10 monclient: build_initial_monmap 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.968+0000 7f43b596c640 10 monclient: monmap: 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:48.969385+0000 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:48.969385+0000 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:48.969 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.969+0000 7f43b596c640 1 Processor -- start 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.969+0000 7f43b596c640 1 -- start start 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.969+0000 7f43b596c640 10 monclient: init 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f43b00b4d10 0x7f43b00b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 10 monclient(hunting): picked mon.noname-a con 0x7f43b00b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43b00b17c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43b0145980 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:48.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b00aade0 con 0x7f43b00b17c0 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b0145460 con 0x7f43b0145980 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b00b50f0 con 0x7f43b00b4d10 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43b596c640 10 monclient(hunting): authenticate will time out at 1812.320346s 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43aeffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f43b00b4d10 0x7f43b00b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43af7fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f43b0145980 0x7f43b0145e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:43308/0 (socket says 172.21.15.110:43308) 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.970+0000 7f43af7fe640 1 -- 172.21.15.110:0/549683497 learned_addr learned my addr 172.21.15.110:0/549683497 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.971+0000 7f43aeffd640 10 monclient(hunting): get_auth_request con 0x7f43b00b4d10 auth_method 0 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.971+0000 7f43aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.971+0000 7f43aeffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.971+0000 7f43aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.971+0000 7f43aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:48.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.971+0000 7f43aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:48.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.971+0000 7f43aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:48.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43adffb640 1 -- 172.21.15.110:0/549683497 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555518007 0 0) 0x7f43b0145460 con 0x7f43b0145980 2024-01-20T11:22:48.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43adffb640 10 monclient(hunting): _negotiate 2024-01-20T11:22:48.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43adffb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43adffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43adffb640 10 monclient(hunting): my global_id is 5995 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43adffb640 1 -- 172.21.15.110:0/549683497 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4398003520 con 0x7f43b0145980 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43aeffd640 10 monclient(hunting): handle_auth_done global_id 5892 payload 274 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43aeffd640 1 -- 172.21.15.110:0/549683497 >> v1:172.21.15.134:6789/0 conn(0x7f43b0145980 legacy=0x7f43b0145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43aeffd640 1 -- 172.21.15.110:0/549683497 >> v1:172.21.15.110:6789/0 conn(0x7f43b00b17c0 legacy=0x7f43b00b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43aeffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43aeffd640 1 monclient: found mon.noname-a 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:48.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.973+0000 7f43aeffd640 1 -- 172.21.15.110:0/549683497 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b00b2660 con 0x7f43b00b4d10 2024-01-20T11:22:48.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.975+0000 7f43aeffd640 1 --2- 172.21.15.110:0/549683497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b4d10 0x7f43b00b3da0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f43a00097a0 tx=0x7f43a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e24a94ff93d41b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 1 -- 172.21.15.110:0/549683497 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a002e070 con 0x7f43b00b4d10 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 10 monclient: dump: 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:48.979 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 10 monclient: _finish_auth 0 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 10 monclient: _check_auth_tickets 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 1 -- 172.21.15.110:0/549683497 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a00315a0 con 0x7f43b00b4d10 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 1 -- 172.21.15.110:0/549683497 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0031930 con 0x7f43b00b4d10 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43adffb640 10 monclient: dump: 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.980 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.981 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:48.981 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.981 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:48.981 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:48.981 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:48.981 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 5 monclient: authenticate success, global_id 5892 2024-01-20T11:22:48.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:48.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 4 set_mon_vals no callback set 2024-01-20T11:22:48.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 10 monclient: shutdown 2024-01-20T11:22:48.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:48.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 1 -- 172.21.15.110:0/549683497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b4d10 msgr2=0x7f43b00b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 1 --2- 172.21.15.110:0/549683497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b4d10 0x7f43b00b3da0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f43a00097a0 tx=0x7f43a002b840 comp rx=0 tx=0).stop 2024-01-20T11:22:48.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 1 -- 172.21.15.110:0/549683497 shutdown_connections 2024-01-20T11:22:48.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 1 --2- 172.21.15.110:0/549683497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b4d10 0x7f43b00b3da0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 1 -- 172.21.15.110:0/549683497 >> 172.21.15.110:0/549683497 conn(0x7f43b0066590 msgr2=0x7f43b00669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.979+0000 7f43b596c640 1 -- 172.21.15.110:0/549683497 shutdown_connections 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.980+0000 7f43b596c640 1 -- 172.21.15.110:0/549683497 wait complete. 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.980+0000 7f43b596c640 10 monclient: build_initial_monmap 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.980+0000 7f43b596c640 10 monclient: monmap: 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:48.985 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.981+0000 7f43b596c640 1 librados: starting msgr at 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.981+0000 7f43b596c640 1 librados: starting objecter 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.981+0000 7f43b596c640 1 Processor -- start 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.981+0000 7f43b596c640 1 -- start start 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.981+0000 7f43b596c640 1 librados: setting wanted keys 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.981+0000 7f43b596c640 1 librados: calling monclient init 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.981+0000 7f43b596c640 10 monclient: init 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b17c0 0x7f43b013dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient(hunting): picked mon.noname-a con 0x7f43b00b17c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f43b00b4d10 0x7f43b013afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient(hunting): picked mon.noname-c con 0x7f43b00b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43b0145980 0x7f43b013b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient(hunting): picked mon.noname-b con 0x7f43b0145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0136b60 con 0x7f43b00b17c0 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00ab9f0 con 0x7f43b0145980 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00abcd0 con 0x7f43b00b4d10 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43b596c640 10 monclient(hunting): authenticate will time out at 1812.332399s 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43af7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43b0145980 0x7f43b013b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b17c0 0x7f43b013dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43ae7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f43b00b4d10 0x7f43b013afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43af7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43b0145980 0x7f43b013b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:38072/0 (socket says 172.21.15.110:38072) 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b17c0 0x7f43b013dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58644/0 (socket says 172.21.15.110:58644) 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43af7fe640 1 -- 172.21.15.110:0/595933309 learned_addr learned my addr 172.21.15.110:0/595933309 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43af7fe640 10 monclient(hunting): get_auth_request con 0x7f43b0145980 auth_method 0 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43af7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43aeffd640 10 monclient(hunting): get_auth_request con 0x7f43b00b17c0 auth_method 0 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43aeffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.982+0000 7f43aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43ae7fc640 10 monclient(hunting): get_auth_request con 0x7f43b00b4d10 auth_method 0 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43ae7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:48.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.983+0000 7f43ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 10 monclient(hunting): handle_auth_done global_id 5998 payload 824 2024-01-20T11:22:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 1 -- 172.21.15.110:0/595933309 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f43b00b4d10 msgr2=0x7f43b013afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f43b00b4d10 0x7f43b013afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 1 -- 172.21.15.110:0/595933309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b17c0 msgr2=0x7f43b013dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b17c0 0x7f43b013dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:48.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 1 monclient: found mon.noname-b 2024-01-20T11:22:48.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:48.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:48.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.986+0000 7f43af7fe640 1 -- 172.21.15.110:0/595933309 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a0009400 con 0x7f43b0145980 2024-01-20T11:22:48.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.988+0000 7f43af7fe640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43b0145980 0x7f43b013b500 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f43a800c9f0 tx=0x7f43a800ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:48.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.988+0000 7f4397fff640 10 client.?.objecter ms_handle_connect 0x7f43b0145980 2024-01-20T11:22:48.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.988+0000 7f4397fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:48.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.991+0000 7f4397fff640 1 -- 172.21.15.110:0/595933309 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a800ab70 con 0x7f43b0145980 2024-01-20T11:22:48.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.991+0000 7f4397fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:48.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.991+0000 7f4397fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:48.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.991+0000 7f4397fff640 10 monclient: dump: 2024-01-20T11:22:48.992 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:48.992 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.992 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.992 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f4397fff640 10 monclient: _finish_auth 0 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f4397fff640 10 monclient: _check_auth_tickets 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f4397fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f4397fff640 1 -- 172.21.15.110:0/595933309 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a800acd0 con 0x7f43b0145980 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f4397fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f4397fff640 1 -- 172.21.15.110:0/595933309 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8012600 con 0x7f43b0145980 2024-01-20T11:22:48.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43b596c640 5 monclient: authenticate success, global_id 5998 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f4397fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f4397fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f4397fff640 10 monclient: dump: 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43b596c640 10 monclient: _renew_subs 2024-01-20T11:22:48.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43b596c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:48.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43acff9640 4 set_mon_vals no callback set 2024-01-20T11:22:48.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b00abfb0 con 0x7f43b0145980 2024-01-20T11:22:48.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43b596c640 10 client.5998.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:48.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43b596c640 10 monclient: _renew_subs 2024-01-20T11:22:48.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43b596c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:48.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b013ec30 con 0x7f43b0145980 2024-01-20T11:22:48.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.992+0000 7f43b596c640 1 librados: init done 2024-01-20T11:22:48.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.993+0000 7f43b596c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:48.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.993+0000 7f43b596c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:48.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.993+0000 7f43b596c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:48.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:48.993+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b013e950 con 0x7f43b0145980 2024-01-20T11:22:49.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.229+0000 7f4397fff640 1 -- 172.21.15.110:0/595933309 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f43a8007070 con 0x7f43b0145980 2024-01-20T11:22:49.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.229+0000 7f4397fff640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f439c06c8b0 0x7f439c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:49.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.229+0000 7f43aeffd640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f439c06c8b0 0x7f439c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:49.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.229+0000 7f43aeffd640 10 monclient: get_auth_request con 0x7f439c06c8b0 auth_method 0 2024-01-20T11:22:49.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.230+0000 7f43aeffd640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f439c06c8b0 0x7f439c06ed70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f43a0036000 tx=0x7f43a003d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:49.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.230+0000 7f4397fff640 10 client.5998.objecter ms_handle_connect 0x7f439c06c8b0 2024-01-20T11:22:49.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.232+0000 7f4397fff640 1 -- 172.21.15.110:0/595933309 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f43a80583e0 con 0x7f43b0145980 2024-01-20T11:22:49.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.232+0000 7f4397fff640 10 client.5998.objecter ms_dispatch 0x7f43b0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:49.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.232+0000 7f4397fff640 3 client.5998.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:49.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.232+0000 7f4397fff640 3 client.5998.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:49.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.232+0000 7f4397fff640 20 client.5998.objecter dump_active .. 0 homeless 2024-01-20T11:22:49.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.242+0000 7f4397fff640 1 -- 172.21.15.110:0/595933309 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f43a8054720 con 0x7f43b0145980 2024-01-20T11:22:49.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.242+0000 7f4397fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:49.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.242+0000 7f4397fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:49.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.490+0000 7f43b596c640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:49.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.490+0000 7f43b596c640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:49.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.490+0000 7f43b596c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:49.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.490+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f43b013e950 con 0x7f43b0145980 2024-01-20T11:22:49.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.496+0000 7f4397fff640 1 -- 172.21.15.110:0/595933309 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f43a8017520 con 0x7f43b0145980 2024-01-20T11:22:49.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.496+0000 7f4397fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:49.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.496+0000 7f4397fff640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:49.496 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:49.496 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":504,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 10 librados: watch_flush enter 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 10 librados: watch_flush exit 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 20 client.5998.objecter shutdown clearing up homeless session... 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 10 client.5998.objecter successfully canceled tick 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f439c06c8b0 msgr2=0x7f439c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f439c06c8b0 0x7f439c06ed70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f43a0036000 tx=0x7f43a003d040 comp rx=0 tx=0).stop 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 10 monclient: shutdown 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43b0145980 msgr2=0x7f43b013b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43b0145980 0x7f43b013b500 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f43a800c9f0 tx=0x7f43a800ceb0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 shutdown_connections 2024-01-20T11:22:49.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f439c06c8b0 0x7f439c06ed70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43b0145980 0x7f43b013b500 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f43b00b4d10 0x7f43b013afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 --2- 172.21.15.110:0/595933309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43b00b17c0 0x7f43b013dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 >> 172.21.15.110:0/595933309 conn(0x7f43b0066590 msgr2=0x7f43b0066970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 shutdown_connections 2024-01-20T11:22:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 -- 172.21.15.110:0/595933309 wait complete. 2024-01-20T11:22:49.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.498+0000 7f43b596c640 1 librados: shutdown 2024-01-20T11:22:49.509 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:49.509 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.613+0000 7f893a189640 10 monclient: get_monmap_and_config 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.613+0000 7f893a189640 10 monclient: build_initial_monmap 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.613+0000 7f893a189640 10 monclient: monmap: 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:49.613779+0000 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:49.613779+0000 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:49.614 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:49.614 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:49.614 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:49.614 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:49.614 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:49.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.613+0000 7f893a189640 1 Processor -- start 2024-01-20T11:22:49.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 1 -- start start 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient: init 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a96f0 0x7f893413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient(hunting): picked mon.noname-b con 0x7f89340a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f893413fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f89340a9ad0 0x7f8934142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient(hunting): picked mon.noname-c con 0x7f89340a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89340b25c0 con 0x7f893413fa10 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89340aaea0 con 0x7f89340a9ad0 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89340ab0a0 con 0x7f89340a96f0 2024-01-20T11:22:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.614+0000 7f893a189640 10 monclient(hunting): authenticate will time out at 1812.964790s 2024-01-20T11:22:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f89337fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a96f0 0x7f893413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f892affd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f893413fa10 0x7f89341423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58018/0 (socket says 172.21.15.110:58018) 2024-01-20T11:22:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f892affd640 1 -- 172.21.15.110:0/2960871569 learned_addr learned my addr 172.21.15.110:0/2960871569 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f8933fff640 1 --2- 172.21.15.110:0/2960871569 >> v2:172.21.15.134:3300/0 conn(0x7f89340a9ad0 0x7f8934142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f8933fff640 10 monclient(hunting): get_auth_request con 0x7f89340a9ad0 auth_method 0 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f8933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f8933fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f8933fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f89337fe640 10 monclient(hunting): get_auth_request con 0x7f89340a96f0 auth_method 0 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f89337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f89337fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f89337fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f8933fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f8933fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.615+0000 7f8933fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.616+0000 7f89337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.616+0000 7f89337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:49.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.616+0000 7f89337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.617+0000 7f8933fff640 10 monclient(hunting): handle_auth_done global_id 6001 payload 274 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.617+0000 7f8933fff640 1 -- 172.21.15.110:0/2960871569 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a96f0 msgr2=0x7f893413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.617+0000 7f8933fff640 1 --2- 172.21.15.110:0/2960871569 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a96f0 0x7f893413f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.617+0000 7f8933fff640 1 -- 172.21.15.110:0/2960871569 >> v1:172.21.15.110:6789/0 conn(0x7f893413fa10 legacy=0x7f89341423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.617+0000 7f8933fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.617+0000 7f8933fff640 1 monclient: found mon.noname-c 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.617+0000 7f8933fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.617+0000 7f8933fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.617+0000 7f8933fff640 1 -- 172.21.15.110:0/2960871569 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89340b3fb0 con 0x7f89340a9ad0 2024-01-20T11:22:49.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.619+0000 7f8933fff640 1 --2- 172.21.15.110:0/2960871569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89340a9ad0 0x7f8934142af0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f892c009d90 tx=0x7f892c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e32a0a982da844e server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:49.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 1 -- 172.21.15.110:0/2960871569 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c02cbc0 con 0x7f89340a9ad0 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 10 monclient: dump: 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 1 -- 172.21.15.110:0/2960871569 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892c02cd20 con 0x7f89340a9ad0 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 5 monclient: authenticate success, global_id 6001 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 1 -- 172.21.15.110:0/2960871569 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c034660 con 0x7f89340a9ad0 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 4 set_mon_vals no callback set 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f8932ffd640 10 monclient: dump: 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 10 monclient: shutdown 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 1 -- 172.21.15.110:0/2960871569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89340a9ad0 msgr2=0x7f8934142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 1 --2- 172.21.15.110:0/2960871569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89340a9ad0 0x7f8934142af0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f892c009d90 tx=0x7f892c02be10 comp rx=0 tx=0).stop 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 1 -- 172.21.15.110:0/2960871569 shutdown_connections 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 1 --2- 172.21.15.110:0/2960871569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89340a9ad0 0x7f8934142af0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 1 --2- 172.21.15.110:0/2960871569 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a96f0 0x7f893413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 1 -- 172.21.15.110:0/2960871569 >> 172.21.15.110:0/2960871569 conn(0x7f89340b7bc0 msgr2=0x7f89340b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 1 -- 172.21.15.110:0/2960871569 shutdown_connections 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.623+0000 7f893a189640 1 -- 172.21.15.110:0/2960871569 wait complete. 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.624+0000 7f893a189640 10 monclient: build_initial_monmap 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.624+0000 7f893a189640 10 monclient: monmap: 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:49.628 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.624+0000 7f893a189640 1 librados: starting msgr at 2024-01-20T11:22:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.624+0000 7f893a189640 1 librados: starting objecter 2024-01-20T11:22:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.624+0000 7f893a189640 1 Processor -- start 2024-01-20T11:22:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 1 -- start start 2024-01-20T11:22:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 1 librados: setting wanted keys 2024-01-20T11:22:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 1 librados: calling monclient init 2024-01-20T11:22:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient: init 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89340a96f0 0x7f893412ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient(hunting): picked mon.noname-a con 0x7f89340a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a9ad0 0x7f893413d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient(hunting): picked mon.noname-c con 0x7f89340a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8934141610 0x7f89341d8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient(hunting): picked mon.noname-b con 0x7f8934141610 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8934130de0 con 0x7f89340a96f0 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893413de00 con 0x7f8934141610 2024-01-20T11:22:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89341d8540 con 0x7f89340a9ad0 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f893a189640 10 monclient(hunting): authenticate will time out at 1812.975712s 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f89337fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89340a96f0 0x7f893412ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.625+0000 7f8933fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8934141610 0x7f89341d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f892affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a9ad0 0x7f893413d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f8933fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8934141610 0x7f89341d8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:56026/0 (socket says 172.21.15.110:56026) 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f8933fff640 1 -- 172.21.15.110:0/1300392808 learned_addr learned my addr 172.21.15.110:0/1300392808 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f8933fff640 10 monclient(hunting): get_auth_request con 0x7f8934141610 auth_method 0 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f8933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f8933fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f8933fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f89337fe640 10 monclient(hunting): get_auth_request con 0x7f89340a96f0 auth_method 0 2024-01-20T11:22:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f89337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f89337fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f89337fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f892affd640 10 monclient(hunting): get_auth_request con 0x7f89340a9ad0 auth_method 0 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f892affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f892affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f892affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f8933fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f8933fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f8933fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f89337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f89337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f89337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f892affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f892affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.626+0000 7f892affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 10 monclient(hunting): handle_auth_done global_id 6004 payload 824 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 1 -- 172.21.15.110:0/1300392808 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a9ad0 msgr2=0x7f893413d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a9ad0 0x7f893413d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 1 -- 172.21.15.110:0/1300392808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89340a96f0 msgr2=0x7f893412ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89340a96f0 0x7f893412ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 1 monclient: found mon.noname-b 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.629+0000 7f8933fff640 1 -- 172.21.15.110:0/1300392808 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c009a70 con 0x7f8934141610 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.631+0000 7f8933fff640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8934141610 0x7f89341d8000 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f892c009d90 tx=0x7f892c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.631+0000 7f8930ff9640 10 client.?.objecter ms_handle_connect 0x7f8934141610 2024-01-20T11:22:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.631+0000 7f8930ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 1 -- 172.21.15.110:0/1300392808 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c03e610 con 0x7f8934141610 2024-01-20T11:22:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 10 monclient: dump: 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 10 monclient: _finish_auth 0 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:22:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 1 -- 172.21.15.110:0/1300392808 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892c02ce90 con 0x7f8934141610 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 1 -- 172.21.15.110:0/1300392808 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c03e990 con 0x7f8934141610 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f893a189640 5 monclient: authenticate success, global_id 6004 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f8930ff9640 10 monclient: dump: 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:49.639 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f89327fc640 4 set_mon_vals no callback set 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f893a189640 10 monclient: _renew_subs 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f893a189640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89341d8700 con 0x7f8934141610 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f893a189640 10 client.6004.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f893a189640 10 monclient: _renew_subs 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f893a189640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:49.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89341d8b60 con 0x7f8934141610 2024-01-20T11:22:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.634+0000 7f893a189640 1 librados: init done 2024-01-20T11:22:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.635+0000 7f893a189640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.635+0000 7f893a189640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.635+0000 7f893a189640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:49.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.635+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89341d7dc0 con 0x7f8934141610 2024-01-20T11:22:49.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.873+0000 7f8930ff9640 1 -- 172.21.15.110:0/1300392808 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f892c03b050 con 0x7f8934141610 2024-01-20T11:22:49.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.873+0000 7f8930ff9640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f891406c8b0 0x7f891406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:49.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.873+0000 7f89337fe640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f891406c8b0 0x7f891406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:49.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.873+0000 7f89337fe640 10 monclient: get_auth_request con 0x7f891406c8b0 auth_method 0 2024-01-20T11:22:49.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.874+0000 7f8930ff9640 10 client.6004.objecter ms_handle_connect 0x7f891406c8b0 2024-01-20T11:22:49.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.874+0000 7f89337fe640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f891406c8b0 0x7f891406ed70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8924009b40 tx=0x7f8924009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:49.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.875+0000 7f8930ff9640 1 -- 172.21.15.110:0/1300392808 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f892c07a6c0 con 0x7f8934141610 2024-01-20T11:22:49.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.875+0000 7f8930ff9640 10 client.6004.objecter ms_dispatch 0x7f8934000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:49.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.875+0000 7f8930ff9640 3 client.6004.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:49.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.875+0000 7f8930ff9640 3 client.6004.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:49.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.875+0000 7f8930ff9640 20 client.6004.objecter dump_active .. 0 homeless 2024-01-20T11:22:49.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.885+0000 7f8930ff9640 1 -- 172.21.15.110:0/1300392808 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f892c076a00 con 0x7f8934141610 2024-01-20T11:22:49.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.886+0000 7f8930ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:49.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:49.886+0000 7f8930ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:50.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.051+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89340aa4a0 con 0x7f891406c8b0 2024-01-20T11:22:50.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.054+0000 7f8930ff9640 1 -- 172.21.15.110:0/1300392808 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63850 (secure 0 0 0) 0x7f89340aa4a0 con 0x7f891406c8b0 2024-01-20T11:22:50.054 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:50.054 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.055+0000 7f893a189640 10 librados: watch_flush enter 2024-01-20T11:22:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.055+0000 7f893a189640 10 librados: watch_flush exit 2024-01-20T11:22:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.055+0000 7f893a189640 20 client.6004.objecter shutdown clearing up homeless session... 2024-01-20T11:22:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.055+0000 7f893a189640 10 client.6004.objecter successfully canceled tick 2024-01-20T11:22:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f891406c8b0 msgr2=0x7f891406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f891406c8b0 0x7f891406ed70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8924009b40 tx=0x7f8924009190 comp rx=0 tx=0).stop 2024-01-20T11:22:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 10 monclient: shutdown 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8934141610 msgr2=0x7f89341d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8934141610 0x7f89341d8000 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f892c009d90 tx=0x7f892c004a40 comp rx=0 tx=0).stop 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 shutdown_connections 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f891406c8b0 0x7f891406ed70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8934141610 0x7f89341d8000 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89340a9ad0 0x7f893413d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 --2- 172.21.15.110:0/1300392808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89340a96f0 0x7f893412ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 >> 172.21.15.110:0/1300392808 conn(0x7f89340b7bc0 msgr2=0x7f8934137380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 shutdown_connections 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 -- 172.21.15.110:0/1300392808 wait complete. 2024-01-20T11:22:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:50.056+0000 7f893a189640 1 librados: shutdown 2024-01-20T11:22:50.066 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-01-20T11:22:48.802575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002746"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":5.0149999999999997},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":7.1769999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":5.79},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":9.6630000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.637}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":12.725},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":14.407999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":11.016999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":13.592000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":15.298999999999999},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":10.475}]}]},{"osd":7,"up_from":21,"seq":90194313311,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.4209999999999994},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.0119999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":6.6429999999999998},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.3200000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":10.845000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":12.218}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":11.552},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":10.254}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":12.896000000000001},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":15.323}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":13.523},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":16.079999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313312,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0010000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":6.2389999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":12.635999999999999},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.2650000000000006}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":7.968},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.084}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":13.257999999999999},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":10.869}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.5779999999999994},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":11.455}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.805999999999999},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":14.48}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444126,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.1459999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.0629999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.718},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.927}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.898},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.5920000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.786},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":8.9760000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":10.263},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.6680000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.007999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.577999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444127,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":8.4160000000000004},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.0510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":9.1349999999999998},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":15.407999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":12.430999999999999},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":6.6059999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":9.7029999999999994},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":11.792999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":14.323},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.012}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":10.259},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":16.466999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444127,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.7519999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.1669999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.0839999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":11.859}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":10.691000000000001},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.2390000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":9.766},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":12.696999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":11.282999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":14.553000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":16.158000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":13.630000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509535,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":12.222},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":6.024}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":6.4699999999999998},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.839}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":7.3970000000000002},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":14.073}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.885999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":10.247}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":10.945},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":9.6259999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":8.8399999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.827999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:53.069 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.170+0000 7ff4e5c2b640 10 monclient: get_monmap_and_config 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.170+0000 7ff4e5c2b640 10 monclient: build_initial_monmap 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.170+0000 7ff4e5c2b640 10 monclient: monmap: 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:53.170879+0000 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:53.170879+0000 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.170 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:53.171 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:53.171 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:53.171 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:53.171 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:53.171 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.170+0000 7ff4e5c2b640 1 Processor -- start 2024-01-20T11:22:53.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 1 -- start start 2024-01-20T11:22:53.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient: init 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff4e00a96f0 0x7ff4e013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4e00a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e013fa10 0x7ff4e01423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e013fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7ff4e00a9ad0 0x7ff4e0142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4e00a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e00aaea0 con 0x7ff4e00a96f0 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e00b25c0 con 0x7ff4e00a9ad0 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e00ab0a0 con 0x7ff4e013fa10 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.171+0000 7ff4e5c2b640 10 monclient(hunting): authenticate will time out at 1816.521829s 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4df7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff4e00a96f0 0x7ff4e013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4deffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e013fa10 0x7ff4e01423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4df7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff4e00a96f0 0x7ff4e013f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33240/0 (socket says 172.21.15.110:33240) 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4df7fe640 1 -- 172.21.15.110:0/1620217732 learned_addr learned my addr 172.21.15.110:0/1620217732 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4df7fe640 10 monclient(hunting): get_auth_request con 0x7ff4e00a96f0 auth_method 0 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4df7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4dffff640 1 --2- 172.21.15.110:0/1620217732 >> v2:172.21.15.134:3300/0 conn(0x7ff4e00a9ad0 0x7ff4e0142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4deffd640 10 monclient(hunting): get_auth_request con 0x7ff4e013fa10 auth_method 0 2024-01-20T11:22:53.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4deffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4deffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4dffff640 10 monclient(hunting): get_auth_request con 0x7ff4e00a9ad0 auth_method 0 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4dffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.172+0000 7ff4dffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.173+0000 7ff4df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.173+0000 7ff4df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.173+0000 7ff4df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.173+0000 7ff4deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.173+0000 7ff4deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.173+0000 7ff4deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.173+0000 7ff4dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:53.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.173+0000 7ff4dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:53.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.173+0000 7ff4dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:53.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 10 monclient(hunting): handle_auth_done global_id 5904 payload 274 2024-01-20T11:22:53.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 1 -- 172.21.15.110:0/1620217732 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e013fa10 msgr2=0x7ff4e01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:53.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 1 --2- 172.21.15.110:0/1620217732 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e013fa10 0x7ff4e01423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 1 -- 172.21.15.110:0/1620217732 >> v2:172.21.15.134:3300/0 conn(0x7ff4e00a9ad0 msgr2=0x7ff4e0142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:53.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 1 --2- 172.21.15.110:0/1620217732 >> v2:172.21.15.134:3300/0 conn(0x7ff4e00a9ad0 0x7ff4e0142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:53.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 1 monclient: found mon.noname-a 2024-01-20T11:22:53.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:53.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:53.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.174+0000 7ff4df7fe640 1 -- 172.21.15.110:0/1620217732 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e00b3fb0 con 0x7ff4e00a96f0 2024-01-20T11:22:53.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.176+0000 7ff4df7fe640 1 --2- 172.21.15.110:0/1620217732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e00a96f0 0x7ff4e013f4d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff4e00749e0 tx=0x7ff4d002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bed61a0eb95f69d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:53.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 1 -- 172.21.15.110:0/1620217732 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d00317c0 con 0x7ff4e00a96f0 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 10 monclient: dump: 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:53.180 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 10 monclient: _finish_auth 0 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 5 monclient: authenticate success, global_id 5904 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 1 -- 172.21.15.110:0/1620217732 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d0031920 con 0x7ff4e00a96f0 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 4 set_mon_vals no callback set 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 1 -- 172.21.15.110:0/1620217732 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0031cb0 con 0x7ff4e00a96f0 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4de7fc640 10 monclient: dump: 2024-01-20T11:22:53.181 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:53.182 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 10 monclient: shutdown 2024-01-20T11:22:53.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/1620217732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e00a96f0 msgr2=0x7ff4e013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/1620217732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e00a96f0 0x7ff4e013f4d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff4e00749e0 tx=0x7ff4d002b860 comp rx=0 tx=0).stop 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/1620217732 shutdown_connections 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/1620217732 >> v2:172.21.15.134:3300/0 conn(0x7ff4e00a9ad0 0x7ff4e0142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/1620217732 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e013fa10 0x7ff4e01423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/1620217732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e00a96f0 0x7ff4e013f4d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/1620217732 >> 172.21.15.110:0/1620217732 conn(0x7ff4e00b7bc0 msgr2=0x7ff4e00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/1620217732 shutdown_connections 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.180+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/1620217732 wait complete. 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.181+0000 7ff4e5c2b640 10 monclient: build_initial_monmap 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.181+0000 7ff4e5c2b640 10 monclient: monmap: 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:53.186 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.181+0000 7ff4e5c2b640 1 librados: starting msgr at 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.181+0000 7ff4e5c2b640 1 librados: starting objecter 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 Processor -- start 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 -- start start 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 librados: setting wanted keys 2024-01-20T11:22:53.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 librados: calling monclient init 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient: init 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e00a96f0 0x7ff4e013ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e00a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e00a9ad0 0x7ff4e0144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4e00a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e01cc320 0x7ff4e01dae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4e01cc320 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e01da9a0 con 0x7ff4e01cc320 2024-01-20T11:22:53.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e01cc770 con 0x7ff4e00a96f0 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e00abf00 con 0x7ff4e00a9ad0 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.182+0000 7ff4e5c2b640 10 monclient(hunting): authenticate will time out at 1816.532966s 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4dffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e01cc320 0x7ff4e01dae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4deffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e00a9ad0 0x7ff4e0144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4df7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e00a96f0 0x7ff4e013ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4dffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e01cc320 0x7ff4e01dae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33242/0 (socket says 172.21.15.110:33242) 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4dffff640 1 -- 172.21.15.110:0/2415336594 learned_addr learned my addr 172.21.15.110:0/2415336594 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4dffff640 10 monclient(hunting): get_auth_request con 0x7ff4e01cc320 auth_method 0 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:53.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4dffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4dffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4deffd640 10 monclient(hunting): get_auth_request con 0x7ff4e00a9ad0 auth_method 0 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4deffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4deffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4df7fe640 10 monclient(hunting): get_auth_request con 0x7ff4e00a96f0 auth_method 0 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4df7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.183+0000 7ff4df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.184+0000 7ff4dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.184+0000 7ff4dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.184+0000 7ff4dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.184+0000 7ff4deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.184+0000 7ff4deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:53.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.184+0000 7ff4deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:53.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.184+0000 7ff4df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:53.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.184+0000 7ff4df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:53.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.184+0000 7ff4df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:53.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 10 monclient(hunting): handle_auth_done global_id 5907 payload 824 2024-01-20T11:22:53.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 1 -- 172.21.15.110:0/2415336594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e00a9ad0 msgr2=0x7ff4e0144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e00a9ad0 0x7ff4e0144820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 1 -- 172.21.15.110:0/2415336594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e00a96f0 msgr2=0x7ff4e013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e00a96f0 0x7ff4e013ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 1 monclient: found mon.noname-a 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4dffff640 1 -- 172.21.15.110:0/2415336594 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d00094d0 con 0x7ff4e01cc320 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.186+0000 7ff4deffd640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e00a9ad0 0x7ff4e0144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.189+0000 7ff4dffff640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e01cc320 0x7ff4e01dae30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800eed0 tx=0x7ff4d800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.189+0000 7ff4c7fff640 10 client.?.objecter ms_handle_connect 0x7ff4e01cc320 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.189+0000 7ff4c7fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 1 -- 172.21.15.110:0/2415336594 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8010040 con 0x7ff4e01cc320 2024-01-20T11:22:53.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 10 monclient: dump: 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 10 monclient: _finish_auth 0 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 10 monclient: _check_auth_tickets 2024-01-20T11:22:53.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 1 -- 172.21.15.110:0/2415336594 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d8014400 con 0x7ff4e01cc320 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4e5c2b640 5 monclient: authenticate success, global_id 5907 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4e5c2b640 10 monclient: _renew_subs 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4e5c2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e00ac240 con 0x7ff4e01cc320 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 1 -- 172.21.15.110:0/2415336594 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d80147b0 con 0x7ff4e01cc320 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4c7fff640 10 monclient: dump: 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.195 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4e5c2b640 10 client.5907.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4e5c2b640 10 monclient: _renew_subs 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4dd7fa640 4 set_mon_vals no callback set 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4e5c2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e01ca960 con 0x7ff4e01cc320 2024-01-20T11:22:53.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.192+0000 7ff4e5c2b640 1 librados: init done 2024-01-20T11:22:53.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.193+0000 7ff4e5c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:53.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.193+0000 7ff4e5c2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:53.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.193+0000 7ff4e5c2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:53.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.193+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e0075490 con 0x7ff4e01cc320 2024-01-20T11:22:53.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.214+0000 7ff4c7fff640 1 -- 172.21.15.110:0/2415336594 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff4d8014910 con 0x7ff4e01cc320 2024-01-20T11:22:53.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.214+0000 7ff4c7fff640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c8b0 0x7ff4cc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.214+0000 7ff4df7fe640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c8b0 0x7ff4cc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.214+0000 7ff4df7fe640 10 monclient: get_auth_request con 0x7ff4cc06c8b0 auth_method 0 2024-01-20T11:22:53.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.215+0000 7ff4c7fff640 10 client.5907.objecter ms_handle_connect 0x7ff4cc06c8b0 2024-01-20T11:22:53.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.215+0000 7ff4df7fe640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c8b0 0x7ff4cc06ed70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0036000 tx=0x7ff4d003d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:53.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.217+0000 7ff4c7fff640 1 -- 172.21.15.110:0/2415336594 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff4d8059180 con 0x7ff4e01cc320 2024-01-20T11:22:53.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.217+0000 7ff4c7fff640 10 client.5907.objecter ms_dispatch 0x7ff4e0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:53.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.217+0000 7ff4c7fff640 3 client.5907.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:53.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.217+0000 7ff4c7fff640 3 client.5907.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:53.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.217+0000 7ff4c7fff640 20 client.5907.objecter dump_active .. 0 homeless 2024-01-20T11:22:53.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.446+0000 7ff4c7fff640 1 -- 172.21.15.110:0/2415336594 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff4d80554c0 con 0x7ff4e01cc320 2024-01-20T11:22:53.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.446+0000 7ff4c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:53.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.446+0000 7ff4c7fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:53.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.615+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4e00b4400 con 0x7ff4cc06c8b0 2024-01-20T11:22:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.619+0000 7ff4c7fff640 1 -- 172.21.15.110:0/2415336594 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63900 (secure 0 0 0) 0x7ff4d0031a90 con 0x7ff4cc06c8b0 2024-01-20T11:22:53.619 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:53.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 10 librados: watch_flush enter 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 10 librados: watch_flush exit 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 20 client.5907.objecter shutdown clearing up homeless session... 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 10 client.5907.objecter successfully canceled tick 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c8b0 msgr2=0x7ff4cc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c8b0 0x7ff4cc06ed70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0036000 tx=0x7ff4d003d040 comp rx=0 tx=0).stop 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 10 monclient: shutdown 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e01cc320 msgr2=0x7ff4e01dae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e01cc320 0x7ff4e01dae30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800eed0 tx=0x7ff4d800b480 comp rx=0 tx=0).stop 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 shutdown_connections 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ff4cc06c8b0 0x7ff4cc06ed70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e01cc320 0x7ff4e01dae30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4e00a9ad0 0x7ff4e0144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 --2- 172.21.15.110:0/2415336594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4e00a96f0 0x7ff4e013ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 >> 172.21.15.110:0/2415336594 conn(0x7ff4e00b7bc0 msgr2=0x7ff4e013bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:53.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 shutdown_connections 2024-01-20T11:22:53.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 -- 172.21.15.110:0/2415336594 wait complete. 2024-01-20T11:22:53.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.621+0000 7ff4e5c2b640 1 librados: shutdown 2024-01-20T11:22:53.632 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-01-20T11:22:52.803535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002631"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":5.0149999999999997},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":7.1769999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":5.79},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":9.6630000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.637}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":12.725},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":14.407999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":11.016999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":13.592000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":15.298999999999999},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":10.475}]}]},{"osd":7,"up_from":21,"seq":90194313312,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":10.045999999999999},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":5.9640000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":14.073},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":11.750999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":16.132000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":6.9790000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":10.403},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":11.106}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":7.6959999999999997},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":9.4039999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":15.273999999999999},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":13.362}]}]},{"osd":3,"up_from":21,"seq":90194313313,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":12.507},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":11.087}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":8.6539999999999999},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":6.5869999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":14.335000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":11.545999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":16.184000000000001},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.8959999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":15.294},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":17.369}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":12.170999999999999},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":7.2910000000000004}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444126,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.1459999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.0629999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.718},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.927}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.898},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.5920000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.786},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":8.9760000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":10.263},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.6680000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.007999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.577999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444127,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":8.4160000000000004},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.0510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":9.1349999999999998},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":15.407999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":12.430999999999999},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":6.6059999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":9.7029999999999994},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":11.792999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":14.323},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.012}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":10.259},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":16.466999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444127,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.7519999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.1669999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.0839999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":11.859}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":10.691000000000001},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.2390000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":9.766},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":12.696999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":11.282999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":14.553000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":16.158000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":13.630000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509536,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":11.003},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":6.2770000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.3499999999999996},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.377000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":8.9380000000000006},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":13.01}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.663},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":9.5259999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":14.382},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":13.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":16.077000000000002},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":7.9889999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:53.634 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:53.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.740+0000 7f940ecf2640 10 monclient: get_monmap_and_config 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.740+0000 7f940ecf2640 10 monclient: build_initial_monmap 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.740+0000 7f940ecf2640 10 monclient: monmap: 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:53.740692+0000 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:53.740692+0000 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:53.740 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:53.741 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:53.741 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.740+0000 7f940ecf2640 1 Processor -- start 2024-01-20T11:22:53.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.740+0000 7f940ecf2640 1 -- start start 2024-01-20T11:22:53.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.740+0000 7f940ecf2640 10 monclient: init 2024-01-20T11:22:53.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:53.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-01-20T11:22:53.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94080a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:53.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94080a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:53.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9408142af0 0x7f940814d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 10 monclient(hunting): picked mon.noname-b con 0x7f9408142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:53.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94080b25c0 con 0x7f94080a9ad0 2024-01-20T11:22:53.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9408142f50 con 0x7f94080a96f0 2024-01-20T11:22:53.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94080aaea0 con 0x7f9408142af0 2024-01-20T11:22:53.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:53.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f940ecf2640 10 monclient(hunting): authenticate will time out at 1817.091674s 2024-01-20T11:22:53.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f93fffff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f94080a9ad0 0x7f94081423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58022/0 (socket says 172.21.15.110:58022) 2024-01-20T11:22:53.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.741+0000 7f93fffff640 1 -- 172.21.15.110:0/3483358343 learned_addr learned my addr 172.21.15.110:0/3483358343 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:53.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.744+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926069714 0 0) 0x7f94080b25c0 con 0x7f94080a9ad0 2024-01-20T11:22:53.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.744+0000 7f93ff7fe640 10 monclient(hunting): _negotiate 2024-01-20T11:22:53.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.744+0000 7f93ff7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.744+0000 7f93ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:53.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.744+0000 7f93ff7fe640 10 monclient(hunting): my global_id is 5910 2024-01-20T11:22:53.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.744+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f0003520 con 0x7f94080a9ad0 2024-01-20T11:22:53.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1663880538 0 0) 0x7f93f0003520 con 0x7f94080a9ad0 2024-01-20T11:22:53.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9408142af0 msgr2=0x7f940814d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:53.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 --2- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9408142af0 0x7f940814d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 >> v1:172.21.15.134:6789/0 conn(0x7f94080a96f0 legacy=0x7f940813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:53.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 monclient: found mon.noname-a-legacy 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.110:6789/0 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940813fbe0 con 0x7f94080a9ad0 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2748620985 0 0) 0x7f93f8001670 con 0x7f94080a9ad0 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient: dump: 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.749 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 >> v1:172.21.15.110:6789/0 conn(0x7f94080a9ad0 legacy=0x7f94081423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 --2- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93f00039a0 0x7f93f0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient(hunting): picked mon.a con 0x7f93f00039a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 --2- 172.21.15.110:0/3483358343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00047c0 0x7f93f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient(hunting): picked mon.b con 0x7f93f00047c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 --2- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9408142af0 0x7f93f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient(hunting): picked mon.c con 0x7f9408142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94080aaea0 con 0x7f93f00039a0 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f00055c0 con 0x7f93f00047c0 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f000bec0 con 0x7f9408142af0 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:53.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.747+0000 7f93ff7fe640 10 monclient(hunting): _finish_auth 0 2024-01-20T11:22:53.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.750+0000 7f940ca8e640 1 --2- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93f00039a0 0x7f93f0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.750+0000 7f940ca8e640 10 monclient(hunting): get_auth_request con 0x7f93f00039a0 auth_method 0 2024-01-20T11:22:53.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.750+0000 7f940ca8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:53.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.750+0000 7f940ca8e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.750+0000 7f940ca8e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-20T11:22:53.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.751+0000 7f940ca8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:53.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.751+0000 7f940ca8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:53.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.751+0000 7f940ca8e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-20T11:22:53.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 10 monclient(hunting): handle_auth_done global_id 5910 payload 293 2024-01-20T11:22:53.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 1 -- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9408142af0 msgr2=0x7f93f000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:53.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 1 --2- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9408142af0 0x7f93f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 1 -- 172.21.15.110:0/3483358343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00047c0 msgr2=0x7f93f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 1 --2- 172.21.15.110:0/3483358343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00047c0 0x7f93f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 10 monclient: _finish_hunting 0 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 1 monclient: found mon.a 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 1 -- 172.21.15.110:0/3483358343 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f0003520 con 0x7f93f00039a0 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 10 monclient: _finish_auth 0 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 10 monclient: _check_auth_tickets 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ca8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ecf2640 5 monclient: authenticate success, global_id 5910 2024-01-20T11:22:53.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.753+0000 7f940ecf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:53.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.754+0000 7f940ca8e640 1 --2- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93f00039a0 0x7f93f0003da0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9404001650 tx=0x7f93f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f803f61bb82e0e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404005270 con 0x7f93f00039a0 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 10 monclient: dump: 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:53.758 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940402e070 con 0x7f93f00039a0 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f940ecf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f940ecf2640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f940ecf2640 4 set_mon_vals no callback set 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 1 -- 172.21.15.110:0/3483358343 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404003610 con 0x7f93f00039a0 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f93ff7fe640 10 monclient: dump: 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.759 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.760 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:53.760 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.760 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:53.760 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:53.760 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:53.760 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f940ecf2640 10 monclient: shutdown 2024-01-20T11:22:53.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f940ecf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:53.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f940ecf2640 1 -- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93f00039a0 msgr2=0x7f93f0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:53.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f940ecf2640 1 --2- 172.21.15.110:0/3483358343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f93f00039a0 0x7f93f0003da0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9404001650 tx=0x7f93f00057f0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.758+0000 7f940d28f640 1 -- 172.21.15.110:0/3483358343 reap_dead start 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.759+0000 7f940ecf2640 1 -- 172.21.15.110:0/3483358343 shutdown_connections 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.759+0000 7f940ecf2640 1 -- 172.21.15.110:0/3483358343 >> 172.21.15.110:0/3483358343 conn(0x7f94080b7bc0 msgr2=0x7f94080b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.759+0000 7f940ecf2640 1 -- 172.21.15.110:0/3483358343 shutdown_connections 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.759+0000 7f940ecf2640 1 -- 172.21.15.110:0/3483358343 wait complete. 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.759+0000 7f940ecf2640 10 monclient: build_initial_monmap 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.759+0000 7f940ecf2640 10 monclient: monmap: 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:53.764 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.759+0000 7f940ecf2640 1 librados: starting msgr at 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.759+0000 7f940ecf2640 1 librados: starting objecter 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 Processor -- start 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 -- start start 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 librados: setting wanted keys 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 librados: calling monclient init 2024-01-20T11:22:53.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient: init 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00039a0 0x7f940813e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient(hunting): picked mon.noname-b con 0x7f93f00039a0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94080ab5c0 0x7f9408134cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient(hunting): picked mon.noname-a con 0x7f94080ab5c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94080aba00 0x7f94081351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient(hunting): picked mon.noname-c con 0x7f94080aba00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9408135730 con 0x7f94080ab5c0 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940813b160 con 0x7f93f00039a0 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:53.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940813ad80 con 0x7f94080aba00 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.760+0000 7f940ecf2640 10 monclient(hunting): authenticate will time out at 1817.111025s 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.761+0000 7f940ca8e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00039a0 0x7f940813e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.761+0000 7f940ca8e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00039a0 0x7f940813e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:56054/0 (socket says 172.21.15.110:56054) 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.761+0000 7f940ca8e640 1 -- 172.21.15.110:0/1705176770 learned_addr learned my addr 172.21.15.110:0/1705176770 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.761+0000 7f940ca8e640 10 monclient(hunting): get_auth_request con 0x7f93f00039a0 auth_method 0 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.761+0000 7f940ca8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.761+0000 7f940ca8e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.761+0000 7f940ca8e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:53.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.762+0000 7f940ca8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.762+0000 7f940ca8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.762+0000 7f940ca8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.763+0000 7f93fffff640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94080ab5c0 0x7f9408134cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.763+0000 7f93fffff640 10 monclient(hunting): get_auth_request con 0x7f94080ab5c0 auth_method 0 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.763+0000 7f93fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.763+0000 7f93fffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.763+0000 7f93fffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.763+0000 7f93fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.763+0000 7f93fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.763+0000 7f93fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:53.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.764+0000 7f940d28f640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94080aba00 0x7f94081351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 10 monclient(hunting): handle_auth_done global_id 6013 payload 824 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 1 -- 172.21.15.110:0/1705176770 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94080aba00 msgr2=0x7f94081351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94080aba00 0x7f94081351f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 1 -- 172.21.15.110:0/1705176770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94080ab5c0 msgr2=0x7f9408134cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94080ab5c0 0x7f9408134cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 10 monclient: _finish_hunting 0 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 1 monclient: found mon.noname-b 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.765+0000 7f940ca8e640 1 -- 172.21.15.110:0/1705176770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94081da5e0 con 0x7f93f00039a0 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.768+0000 7f940ca8e640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00039a0 0x7f940813e7a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9404008fd0 tx=0x7f9404000fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.768+0000 7f93fd7fa640 10 client.?.objecter ms_handle_connect 0x7f93f00039a0 2024-01-20T11:22:53.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.768+0000 7f93fd7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:53.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.771+0000 7f93fd7fa640 1 -- 172.21.15.110:0/1705176770 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404007240 con 0x7f93f00039a0 2024-01-20T11:22:53.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.771+0000 7f93fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.771+0000 7f93fd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.771+0000 7f93fd7fa640 10 monclient: dump: 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.771+0000 7f93fd7fa640 10 monclient: _finish_auth 0 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.771+0000 7f93fd7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.771+0000 7f93fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f93fd7fa640 1 -- 172.21.15.110:0/1705176770 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94040073a0 con 0x7f93f00039a0 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 5 monclient: authenticate success, global_id 6013 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 10 monclient: _renew_subs 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:53.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94081da770 con 0x7f93f00039a0 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f93fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f93fd7fa640 1 -- 172.21.15.110:0/1705176770 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94040067b0 con 0x7f93f00039a0 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f93fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f93fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f93fd7fa640 10 monclient: dump: 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:53.775 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f93feffd640 4 set_mon_vals no callback set 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 10 client.6013.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 10 monclient: _renew_subs 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94081dac80 con 0x7f93f00039a0 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 1 librados: init done 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:53.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.772+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940813b320 con 0x7f93f00039a0 2024-01-20T11:22:53.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.791+0000 7f93fd7fa640 1 -- 172.21.15.110:0/1705176770 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9404030030 con 0x7f93f00039a0 2024-01-20T11:22:53.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.791+0000 7f93fd7fa640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93f406c8b0 0x7f93f406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:53.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.791+0000 7f93fffff640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93f406c8b0 0x7f93f406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:53.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.791+0000 7f93fffff640 10 monclient: get_auth_request con 0x7f93f406c8b0 auth_method 0 2024-01-20T11:22:53.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.792+0000 7f93fffff640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93f406c8b0 0x7f93f406ed70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f94080ac950 tx=0x7f93f8001a60 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:53.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:53.792+0000 7f93fd7fa640 10 client.6013.objecter ms_handle_connect 0x7f93f406c8b0 2024-01-20T11:22:54.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.014+0000 7f93fd7fa640 1 -- 172.21.15.110:0/1705176770 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9404078ed0 con 0x7f93f00039a0 2024-01-20T11:22:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.014+0000 7f93fd7fa640 10 client.6013.objecter ms_dispatch 0x7f9408000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.014+0000 7f93fd7fa640 3 client.6013.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.014+0000 7f93fd7fa640 3 client.6013.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:54.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.014+0000 7f93fd7fa640 20 client.6013.objecter dump_active .. 0 homeless 2024-01-20T11:22:54.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~236s 2024-01-20T11:22:54.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.025+0000 7f93fd7fa640 1 -- 172.21.15.110:0/1705176770 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9404075210 con 0x7f93f00039a0 2024-01-20T11:22:54.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.025+0000 7f93fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:54.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.025+0000 7f93fd7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:54.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.198+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f94080b4400 con 0x7f93f406c8b0 2024-01-20T11:22:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.201+0000 7f93fd7fa640 1 -- 172.21.15.110:0/1705176770 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63900 (secure 0 0 0) 0x7f93f800f9a0 con 0x7f93f406c8b0 2024-01-20T11:22:54.202 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 10 librados: watch_flush enter 2024-01-20T11:22:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 10 librados: watch_flush exit 2024-01-20T11:22:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 20 client.6013.objecter shutdown clearing up homeless session... 2024-01-20T11:22:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 10 client.6013.objecter successfully canceled tick 2024-01-20T11:22:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93f406c8b0 msgr2=0x7f93f406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93f406c8b0 0x7f93f406ed70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f94080ac950 tx=0x7f93f8001a60 comp rx=0 tx=0).stop 2024-01-20T11:22:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 10 monclient: shutdown 2024-01-20T11:22:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00039a0 msgr2=0x7f940813e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00039a0 0x7f940813e7a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9404008fd0 tx=0x7f9404000fb0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 shutdown_connections 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f93f406c8b0 0x7f93f406ed70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94080aba00 0x7f94081351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94080ab5c0 0x7f9408134cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 --2- 172.21.15.110:0/1705176770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93f00039a0 0x7f940813e7a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 >> 172.21.15.110:0/1705176770 conn(0x7f94080b7bc0 msgr2=0x7f94080b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 shutdown_connections 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 -- 172.21.15.110:0/1705176770 wait complete. 2024-01-20T11:22:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.203+0000 7f940ecf2640 1 librados: shutdown 2024-01-20T11:22:54.214 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-01-20T11:22:52.803535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002631"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":5.0149999999999997},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":7.1769999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":5.79},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":9.6630000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.637}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":12.725},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":14.407999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":11.016999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":13.592000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":15.298999999999999},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":10.475}]}]},{"osd":7,"up_from":21,"seq":90194313312,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":10.045999999999999},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":5.9640000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":14.073},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":11.750999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":16.132000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":6.9790000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":10.403},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":11.106}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":7.6959999999999997},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":9.4039999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":15.273999999999999},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":13.362}]}]},{"osd":3,"up_from":21,"seq":90194313313,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":12.507},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":11.087}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":8.6539999999999999},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":6.5869999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":14.335000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":11.545999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":16.184000000000001},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.8959999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":15.294},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":17.369}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":12.170999999999999},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":7.2910000000000004}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444126,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.1459999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.0629999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.718},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.927}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.898},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.5920000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.786},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":8.9760000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":10.263},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.6680000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.007999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.577999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444127,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":8.4160000000000004},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.0510000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":9.1349999999999998},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":15.407999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":12.430999999999999},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":6.6059999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":9.7029999999999994},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":11.792999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":14.323},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.012}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":10.259},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":16.466999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444127,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.7519999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.1669999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.0839999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":11.859}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":10.691000000000001},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.2390000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":9.766},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":12.696999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":11.282999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":14.553000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":16.158000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":13.630000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509536,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":11.003},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":6.2770000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.3499999999999996},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.377000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":8.9380000000000006},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":13.01}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.663},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":9.5259999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":14.382},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":13.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":16.077000000000002},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":7.9889999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:54.217 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:54.217 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:54.218 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:54.218 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:54.218 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:54.218 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:54.218 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:54.218 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:54.218 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:54.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.319+0000 7f723653a640 10 monclient: get_monmap_and_config 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.319+0000 7f723653a640 10 monclient: build_initial_monmap 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.319+0000 7f723653a640 10 monclient: monmap: 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:54.320529+0000 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:54.320529+0000 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:54.320 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:54.321 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:54.321 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:54.321 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:54.321 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.320+0000 7f723653a640 1 Processor -- start 2024-01-20T11:22:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.320+0000 7f723653a640 1 -- start start 2024-01-20T11:22:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.320+0000 7f723653a640 10 monclient: init 2024-01-20T11:22:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-01-20T11:22:54.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f723006a860 0x7f723013f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient(hunting): picked mon.noname-c con 0x7f723006a860 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f723006ac40 0x7f72301455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient(hunting): picked mon.noname-a con 0x7f723006ac40 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723013f990 0x7f7230145af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient(hunting): picked mon.noname-b con 0x7f723013f990 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72300afd30 con 0x7f723006ac40 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7230146030 con 0x7f723006a860 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72300b01c0 con 0x7f723013f990 2024-01-20T11:22:54.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f723653a640 10 monclient(hunting): authenticate will time out at 1817.671492s 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.321+0000 7f722f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f723006ac40 0x7f72301455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f7234ad7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723013f990 0x7f7230145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722ffff640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f723006a860 0x7f723013f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f723006ac40 0x7f72301455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33262/0 (socket says 172.21.15.110:33262) 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722f7fe640 1 -- 172.21.15.110:0/2337159306 learned_addr learned my addr 172.21.15.110:0/2337159306 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722f7fe640 10 monclient(hunting): get_auth_request con 0x7f723006ac40 auth_method 0 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f7234ad7640 10 monclient(hunting): get_auth_request con 0x7f723013f990 auth_method 0 2024-01-20T11:22:54.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f7234ad7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f7234ad7640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f7234ad7640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722ffff640 10 monclient(hunting): get_auth_request con 0x7f723006a860 auth_method 0 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f7234ad7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f7234ad7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f7234ad7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.322+0000 7f722ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 10 monclient(hunting): handle_auth_done global_id 5916 payload 274 2024-01-20T11:22:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 1 -- 172.21.15.110:0/2337159306 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723013f990 msgr2=0x7f7230145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 1 --2- 172.21.15.110:0/2337159306 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723013f990 0x7f7230145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 1 -- 172.21.15.110:0/2337159306 >> v2:172.21.15.134:3300/0 conn(0x7f723006a860 msgr2=0x7f723013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:54.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 1 --2- 172.21.15.110:0/2337159306 >> v2:172.21.15.134:3300/0 conn(0x7f723006a860 0x7f723013f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 1 monclient: found mon.noname-a 2024-01-20T11:22:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.324+0000 7f722f7fe640 1 -- 172.21.15.110:0/2337159306 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72300723e0 con 0x7f723006ac40 2024-01-20T11:22:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.326+0000 7f722f7fe640 1 --2- 172.21.15.110:0/2337159306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006ac40 0x7f72301455b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f723006e390 tx=0x7f722402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3ce2ef47bbee63 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 1 -- 172.21.15.110:0/2337159306 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72240317e0 con 0x7f723006ac40 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 10 monclient: dump: 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 10 monclient: _finish_auth 0 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 1 -- 172.21.15.110:0/2337159306 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7224031940 con 0x7f723006ac40 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 1 -- 172.21.15.110:0/2337159306 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224031cd0 con 0x7f723006ac40 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f722effd640 10 monclient: dump: 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f723653a640 5 monclient: authenticate success, global_id 5916 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f723653a640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:54.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.329+0000 7f723653a640 4 set_mon_vals no callback set 2024-01-20T11:22:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 10 monclient: shutdown 2024-01-20T11:22:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 1 -- 172.21.15.110:0/2337159306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006ac40 msgr2=0x7f72301455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 1 --2- 172.21.15.110:0/2337159306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006ac40 0x7f72301455b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f723006e390 tx=0x7f722402b880 comp rx=0 tx=0).stop 2024-01-20T11:22:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 1 -- 172.21.15.110:0/2337159306 shutdown_connections 2024-01-20T11:22:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 1 --2- 172.21.15.110:0/2337159306 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723013f990 0x7f7230145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 1 --2- 172.21.15.110:0/2337159306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006ac40 0x7f72301455b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 1 --2- 172.21.15.110:0/2337159306 >> v2:172.21.15.134:3300/0 conn(0x7f723006a860 0x7f723013f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 1 -- 172.21.15.110:0/2337159306 >> 172.21.15.110:0/2337159306 conn(0x7f72300b47d0 msgr2=0x7f72300b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 1 -- 172.21.15.110:0/2337159306 shutdown_connections 2024-01-20T11:22:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.330+0000 7f723653a640 1 -- 172.21.15.110:0/2337159306 wait complete. 2024-01-20T11:22:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.331+0000 7f723653a640 10 monclient: build_initial_monmap 2024-01-20T11:22:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.331+0000 7f723653a640 10 monclient: monmap: 2024-01-20T11:22:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:54.339 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.331+0000 7f723653a640 1 librados: starting msgr at 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.331+0000 7f723653a640 1 librados: starting objecter 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.331+0000 7f723653a640 1 Processor -- start 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.331+0000 7f723653a640 1 -- start start 2024-01-20T11:22:54.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.331+0000 7f723653a640 1 librados: setting wanted keys 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.331+0000 7f723653a640 1 librados: calling monclient init 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.331+0000 7f723653a640 10 monclient: init 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006a860 0x7f723006f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient(hunting): picked mon.noname-a con 0x7f723006a860 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723006ac40 0x7f72301387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient(hunting): picked mon.noname-c con 0x7f723006ac40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f723013f990 0x7f7230138ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient(hunting): picked mon.noname-b con 0x7f723013f990 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7230139220 con 0x7f723006a860 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72300711b0 con 0x7f723013f990 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7230071490 con 0x7f723006ac40 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f723653a640 10 monclient(hunting): authenticate will time out at 1817.682715s 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.332+0000 7f722ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006a860 0x7f723006f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723006ac40 0x7f72301387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006a860 0x7f723006f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33268/0 (socket says 172.21.15.110:33268) 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f7234ad7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f723013f990 0x7f7230138ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722ffff640 1 -- 172.21.15.110:0/2356155548 learned_addr learned my addr 172.21.15.110:0/2356155548 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:54.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722ffff640 10 monclient(hunting): get_auth_request con 0x7f723006a860 auth_method 0 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f7234ad7640 10 monclient(hunting): get_auth_request con 0x7f723013f990 auth_method 0 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f7234ad7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f7234ad7640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f7234ad7640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722f7fe640 10 monclient(hunting): get_auth_request con 0x7f723006ac40 auth_method 0 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f7234ad7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f7234ad7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f7234ad7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.333+0000 7f722f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 10 monclient(hunting): handle_auth_done global_id 6019 payload 824 2024-01-20T11:22:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 1 -- 172.21.15.110:0/2356155548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723006ac40 msgr2=0x7f72301387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723006ac40 0x7f72301387a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 1 -- 172.21.15.110:0/2356155548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006a860 msgr2=0x7f723006f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006a860 0x7f723006f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 10 monclient: _finish_hunting 0 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 1 monclient: found mon.noname-b 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.336+0000 7f7234ad7640 1 -- 172.21.15.110:0/2356155548 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72240094d0 con 0x7f723013f990 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.338+0000 7f7234ad7640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f723013f990 0x7f7230138ce0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f722800eb70 tx=0x7f722800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.338+0000 7f722cff9640 10 client.?.objecter ms_handle_connect 0x7f723013f990 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.338+0000 7f722cff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 1 -- 172.21.15.110:0/2356155548 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722800cd20 con 0x7f723013f990 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 10 monclient: dump: 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 10 monclient: _finish_auth 0 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 10 monclient: _check_auth_tickets 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 1 -- 172.21.15.110:0/2356155548 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722800f040 con 0x7f723013f990 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 1 -- 172.21.15.110:0/2356155548 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72280146c0 con 0x7f723013f990 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722cff9640 10 monclient: dump: 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f723653a640 5 monclient: authenticate success, global_id 6019 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f723653a640 10 monclient: _renew_subs 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f723653a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72300706b0 con 0x7f723013f990 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f722e7fc640 4 set_mon_vals no callback set 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f723653a640 10 client.6019.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f723653a640 10 monclient: _renew_subs 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f723653a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723006f910 con 0x7f723013f990 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.342+0000 7f723653a640 1 librados: init done 2024-01-20T11:22:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.343+0000 7f723653a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.343+0000 7f723653a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.343+0000 7f723653a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.343+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72301d70e0 con 0x7f723013f990 2024-01-20T11:22:54.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.581+0000 7f722cff9640 1 -- 172.21.15.110:0/2356155548 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7228014820 con 0x7f723013f990 2024-01-20T11:22:54.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.581+0000 7f722cff9640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721c06c8b0 0x7f721c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.582+0000 7f722ffff640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721c06c8b0 0x7f721c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.582+0000 7f722ffff640 10 monclient: get_auth_request con 0x7f721c06c8b0 auth_method 0 2024-01-20T11:22:54.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.582+0000 7f722cff9640 10 client.6019.objecter ms_handle_connect 0x7f721c06c8b0 2024-01-20T11:22:54.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.582+0000 7f722ffff640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721c06c8b0 0x7f721c06ed70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7220009c00 tx=0x7f7220009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:54.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.585+0000 7f722cff9640 1 -- 172.21.15.110:0/2356155548 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f72280591a0 con 0x7f723013f990 2024-01-20T11:22:54.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.585+0000 7f722cff9640 10 client.6019.objecter ms_dispatch 0x7f7230000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:54.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.585+0000 7f722cff9640 3 client.6019.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:54.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.585+0000 7f722cff9640 3 client.6019.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:54.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.585+0000 7f722cff9640 20 client.6019.objecter dump_active .. 0 homeless 2024-01-20T11:22:54.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.597+0000 7f722cff9640 1 -- 172.21.15.110:0/2356155548 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f72280554e0 con 0x7f723013f990 2024-01-20T11:22:54.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.597+0000 7f722cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:54.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.597+0000 7f722cff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:54.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.837+0000 7f723653a640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:22:54.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.837+0000 7f723653a640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:54.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.837+0000 7f723653a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:54.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.837+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f72301d70e0 con 0x7f723013f990 2024-01-20T11:22:54.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.843+0000 7f722cff9640 1 -- 172.21.15.110:0/2356155548 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f72301d70e0 con 0x7f723013f990 2024-01-20T11:22:54.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.843+0000 7f722cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:22:54.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.843+0000 7f722cff9640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:22:54.843 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:54.843 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":509,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 10 librados: watch_flush enter 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 10 librados: watch_flush exit 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 20 client.6019.objecter shutdown clearing up homeless session... 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 10 client.6019.objecter successfully canceled tick 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721c06c8b0 msgr2=0x7f721c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721c06c8b0 0x7f721c06ed70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7220009c00 tx=0x7f7220009190 comp rx=0 tx=0).stop 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 10 monclient: shutdown 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f723013f990 msgr2=0x7f7230138ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f723013f990 0x7f7230138ce0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f722800eb70 tx=0x7f722800c490 comp rx=0 tx=0).stop 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 shutdown_connections 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f721c06c8b0 0x7f721c06ed70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f723013f990 0x7f7230138ce0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f723006ac40 0x7f72301387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 --2- 172.21.15.110:0/2356155548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f723006a860 0x7f723006f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 >> 172.21.15.110:0/2356155548 conn(0x7f72300b47d0 msgr2=0x7f72300b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:54.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 shutdown_connections 2024-01-20T11:22:54.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 -- 172.21.15.110:0/2356155548 wait complete. 2024-01-20T11:22:54.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.845+0000 7f723653a640 1 librados: shutdown 2024-01-20T11:22:54.856 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:22:54.856 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:54.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.954+0000 7f66f1b9e640 10 monclient: get_monmap_and_config 2024-01-20T11:22:54.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.954+0000 7f66f1b9e640 10 monclient: build_initial_monmap 2024-01-20T11:22:54.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.954+0000 7f66f1b9e640 10 monclient: monmap: 2024-01-20T11:22:54.954 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:54.954 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.954 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:54.955084+0000 2024-01-20T11:22:54.954 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:54.955084+0000 2024-01-20T11:22:54.955 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:54.955 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.955 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:54.955 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:54.955 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:54.955 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:54.955 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:54.955 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 1 Processor -- start 2024-01-20T11:22:54.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 1 -- start start 2024-01-20T11:22:54.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient: init 2024-01-20T11:22:54.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:54.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-01-20T11:22:54.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f66ec06f920 0x7f66ec06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient(hunting): picked mon.noname-c con 0x7f66ec06f920 addr v2:172.21.15.134:3300/0 2024-01-20T11:22:54.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f66ec13d230 0x7f66ec13d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient(hunting): picked mon.noname-a con 0x7f66ec13d230 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66ec143950 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec0715b0 con 0x7f66ec143950 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec071890 con 0x7f66ec13d230 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec071b70 con 0x7f66ec06f920 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.955+0000 7f66f1b9e640 10 monclient(hunting): authenticate will time out at 1818.306055s 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.956+0000 7f66eaffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f66ec13d230 0x7f66ec13d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.956+0000 7f66ebfff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f66ec143950 0x7f66ec13fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:33620/0 (socket says 172.21.15.110:33620) 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.956+0000 7f66ebfff640 1 -- 172.21.15.110:0/498973279 learned_addr learned my addr 172.21.15.110:0/498973279 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.956+0000 7f66eaffd640 10 monclient(hunting): get_auth_request con 0x7f66ec13d230 auth_method 0 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.956+0000 7f66eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.956+0000 7f66eaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.956+0000 7f66eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.956+0000 7f66eb7fe640 1 --2- 172.21.15.110:0/498973279 >> v2:172.21.15.134:3300/0 conn(0x7f66ec06f920 0x7f66ec06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.957+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.957+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.957+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.957+0000 7f66eb7fe640 10 monclient(hunting): get_auth_request con 0x7f66ec06f920 auth_method 0 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.957+0000 7f66eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.957+0000 7f66eb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.957+0000 7f66eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eaffd640 10 monclient(hunting): handle_auth_done global_id 5922 payload 274 2024-01-20T11:22:54.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eaffd640 1 -- 172.21.15.110:0/498973279 >> v2:172.21.15.134:3300/0 conn(0x7f66ec06f920 msgr2=0x7f66ec06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:54.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eaffd640 1 --2- 172.21.15.110:0/498973279 >> v2:172.21.15.134:3300/0 conn(0x7f66ec06f920 0x7f66ec06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eaffd640 1 -- 172.21.15.110:0/498973279 >> v1:172.21.15.134:6789/0 conn(0x7f66ec143950 legacy=0x7f66ec13fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eaffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:54.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eaffd640 1 monclient: found mon.noname-a 2024-01-20T11:22:54.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:54.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:54.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.958+0000 7f66eaffd640 1 -- 172.21.15.110:0/498973279 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec143430 con 0x7f66ec13d230 2024-01-20T11:22:54.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.960+0000 7f66eaffd640 1 --2- 172.21.15.110:0/498973279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 0x7f66ec13d680 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f66ec067350 tx=0x7f66e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65542b3f5cabff6 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:54.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 1 -- 172.21.15.110:0/498973279 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e002e070 con 0x7f66ec13d230 2024-01-20T11:22:54.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 10 monclient: dump: 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 10 monclient: _finish_auth 0 2024-01-20T11:22:54.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 1 -- 172.21.15.110:0/498973279 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e00315a0 con 0x7f66ec13d230 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 1 -- 172.21.15.110:0/498973279 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0031930 con 0x7f66ec13d230 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66ea7fc640 10 monclient: dump: 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 5 monclient: authenticate success, global_id 5922 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 4 set_mon_vals no callback set 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 10 monclient: shutdown 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 1 -- 172.21.15.110:0/498973279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 msgr2=0x7f66ec13d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 1 --2- 172.21.15.110:0/498973279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 0x7f66ec13d680 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f66ec067350 tx=0x7f66e002b840 comp rx=0 tx=0).stop 2024-01-20T11:22:54.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 1 -- 172.21.15.110:0/498973279 shutdown_connections 2024-01-20T11:22:54.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 1 --2- 172.21.15.110:0/498973279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 0x7f66ec13d680 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.964+0000 7f66f1b9e640 1 --2- 172.21.15.110:0/498973279 >> v2:172.21.15.134:3300/0 conn(0x7f66ec06f920 0x7f66ec06fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.965+0000 7f66f1b9e640 1 -- 172.21.15.110:0/498973279 >> 172.21.15.110:0/498973279 conn(0x7f66ec066d60 msgr2=0x7f66ec0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:54.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.965+0000 7f66f1b9e640 1 -- 172.21.15.110:0/498973279 shutdown_connections 2024-01-20T11:22:54.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.965+0000 7f66f1b9e640 1 -- 172.21.15.110:0/498973279 wait complete. 2024-01-20T11:22:54.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.965+0000 7f66f1b9e640 10 monclient: build_initial_monmap 2024-01-20T11:22:54.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.965+0000 7f66f1b9e640 10 monclient: monmap: 2024-01-20T11:22:54.974 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:54.974 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:54.974 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:54.974 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:54.974 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:54.974 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.965+0000 7f66f1b9e640 1 librados: starting msgr at 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.965+0000 7f66f1b9e640 1 librados: starting objecter 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 Processor -- start 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 -- start start 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 librados: setting wanted keys 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 librados: calling monclient init 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient: init 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:54.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ec06f920 0x7f66ec13dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient(hunting): picked mon.noname-c con 0x7f66ec06f920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 0x7f66ec1d5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient(hunting): picked mon.noname-a con 0x7f66ec13d230 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ec143950 0x7f66ec1d5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient(hunting): picked mon.noname-b con 0x7f66ec143950 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec1c98f0 con 0x7f66ec13d230 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec1ccf80 con 0x7f66ec143950 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:54.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec1d63f0 con 0x7f66ec06f920 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.966+0000 7f66f1b9e640 10 monclient(hunting): authenticate will time out at 1818.317033s 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eaffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 0x7f66ec1d5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eb7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ec06f920 0x7f66ec13dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66ebfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ec143950 0x7f66ec1d5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eaffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 0x7f66ec1d5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33284/0 (socket says 172.21.15.110:33284) 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eaffd640 1 -- 172.21.15.110:0/2526762823 learned_addr learned my addr 172.21.15.110:0/2526762823 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eaffd640 10 monclient(hunting): get_auth_request con 0x7f66ec13d230 auth_method 0 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eaffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66ebfff640 10 monclient(hunting): get_auth_request con 0x7f66ec143950 auth_method 0 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66ebfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eb7fe640 10 monclient(hunting): get_auth_request con 0x7f66ec06f920 auth_method 0 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.967+0000 7f66eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.968+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.968+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.968+0000 7f66eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.968+0000 7f66ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.968+0000 7f66ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.968+0000 7f66ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.968+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.968+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.968+0000 7f66eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 10 monclient(hunting): handle_auth_done global_id 5925 payload 824 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 1 -- 172.21.15.110:0/2526762823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ec06f920 msgr2=0x7f66ec13dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ec06f920 0x7f66ec13dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 1 -- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ec143950 msgr2=0x7f66ec1d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ec143950 0x7f66ec1d5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 10 monclient: _finish_hunting 0 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 1 monclient: found mon.noname-a 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:54.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:54.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66eaffd640 1 -- 172.21.15.110:0/2526762823 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e00093a0 con 0x7f66ec13d230 2024-01-20T11:22:54.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.970+0000 7f66ebfff640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ec143950 0x7f66ec1d5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:22:54.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.973+0000 7f66eaffd640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 0x7f66ec1d5970 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f66e0000c00 tx=0x7f66e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:54.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.973+0000 7f66d3fff640 10 client.?.objecter ms_handle_connect 0x7f66ec13d230 2024-01-20T11:22:54.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.973+0000 7f66d3fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:54.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 1 -- 172.21.15.110:0/2526762823 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0032840 con 0x7f66ec13d230 2024-01-20T11:22:54.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:54.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:54.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 10 monclient: dump: 2024-01-20T11:22:54.982 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:54.982 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.982 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.982 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.982 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:54.982 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.982 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 10 monclient: _finish_auth 0 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 10 monclient: _check_auth_tickets 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66f1b9e640 5 monclient: authenticate success, global_id 5925 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 1 -- 172.21.15.110:0/2526762823 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e00329c0 con 0x7f66ec13d230 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 1 -- 172.21.15.110:0/2526762823 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0032d70 con 0x7f66ec13d230 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66f1b9e640 10 monclient: _renew_subs 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66f1b9e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:54.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec1ccba0 con 0x7f66ec13d230 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66d3fff640 10 monclient: dump: 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:54.984 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66e9ffb640 4 set_mon_vals no callback set 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66f1b9e640 10 client.5925.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66f1b9e640 10 monclient: _renew_subs 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66f1b9e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec134980 con 0x7f66ec13d230 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.976+0000 7f66f1b9e640 1 librados: init done 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.977+0000 7f66f1b9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.977+0000 7f66f1b9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.977+0000 7f66f1b9e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:54.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.977+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ec1d5690 con 0x7f66ec13d230 2024-01-20T11:22:54.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.997+0000 7f66d3fff640 1 -- 172.21.15.110:0/2526762823 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f66e004e400 con 0x7f66ec13d230 2024-01-20T11:22:54.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.997+0000 7f66d3fff640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f66d806c8b0 0x7f66d806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.997+0000 7f66eb7fe640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f66d806c8b0 0x7f66d806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.997+0000 7f66eb7fe640 10 monclient: get_auth_request con 0x7f66d806c8b0 auth_method 0 2024-01-20T11:22:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.998+0000 7f66eb7fe640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f66d806c8b0 0x7f66d806ed70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f66dc009b40 tx=0x7f66dc009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:54.998+0000 7f66d3fff640 10 client.5925.objecter ms_handle_connect 0x7f66d806c8b0 2024-01-20T11:22:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.233+0000 7f66d3fff640 1 -- 172.21.15.110:0/2526762823 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f66e007a1a0 con 0x7f66ec13d230 2024-01-20T11:22:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.233+0000 7f66d3fff640 10 client.5925.objecter ms_dispatch 0x7f66ec000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.233+0000 7f66d3fff640 3 client.5925.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.233+0000 7f66d3fff640 3 client.5925.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.233+0000 7f66d3fff640 20 client.5925.objecter dump_active .. 0 homeless 2024-01-20T11:22:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.233+0000 7f66d3fff640 1 -- 172.21.15.110:0/2526762823 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f66e00764e0 con 0x7f66ec13d230 2024-01-20T11:22:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.233+0000 7f66d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.233+0000 7f66d3fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:55.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.402+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66ec1d5690 con 0x7f66d806c8b0 2024-01-20T11:22:55.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.405+0000 7f66d3fff640 1 -- 172.21.15.110:0/2526762823 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63899 (secure 0 0 0) 0x7f66ec1d5690 con 0x7f66d806c8b0 2024-01-20T11:22:55.405 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:55.406 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:55.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 10 librados: watch_flush enter 2024-01-20T11:22:55.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 10 librados: watch_flush exit 2024-01-20T11:22:55.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 20 client.5925.objecter shutdown clearing up homeless session... 2024-01-20T11:22:55.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 10 client.5925.objecter successfully canceled tick 2024-01-20T11:22:55.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f66d806c8b0 msgr2=0x7f66d806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:55.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f66d806c8b0 0x7f66d806ed70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f66dc009b40 tx=0x7f66dc009190 comp rx=0 tx=0).stop 2024-01-20T11:22:55.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 10 monclient: shutdown 2024-01-20T11:22:55.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:55.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 msgr2=0x7f66ec1d5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:55.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 0x7f66ec1d5970 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f66e0000c00 tx=0x7f66e00046c0 comp rx=0 tx=0).stop 2024-01-20T11:22:55.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 shutdown_connections 2024-01-20T11:22:55.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f66d806c8b0 0x7f66d806ed70 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:55.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ec143950 0x7f66ec1d5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:55.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ec13d230 0x7f66ec1d5970 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:55.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 --2- 172.21.15.110:0/2526762823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ec06f920 0x7f66ec13dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:55.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 >> 172.21.15.110:0/2526762823 conn(0x7f66ec066d60 msgr2=0x7f66ec0b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:55.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 shutdown_connections 2024-01-20T11:22:55.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 -- 172.21.15.110:0/2526762823 wait complete. 2024-01-20T11:22:55.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:55.407+0000 7f66f1b9e640 1 librados: shutdown 2024-01-20T11:22:55.419 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-01-20T11:22:54.804147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002781"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280610,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":5.7149999999999999},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":10.811}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.3559999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":8.766}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":6.2919999999999998},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":8.0860000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":7.524},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":13.653}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.053000000000001},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.625999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":15.746},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":11.776999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313312,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":10.045999999999999},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":5.9640000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":14.073},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":11.750999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":16.132000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":6.9790000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":10.403},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":11.106}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":7.6959999999999997},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":9.4039999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":15.273999999999999},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":13.362}]}]},{"osd":3,"up_from":21,"seq":90194313313,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":12.507},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":11.087}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":8.6539999999999999},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":6.5869999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":14.335000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":11.545999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":16.184000000000001},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":7.8959999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":15.294},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":17.369}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":12.170999999999999},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":7.2910000000000004}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444127,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.3209999999999997},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":5.2380000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":7.0499999999999998},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":11.209}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.911},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":8.1920000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":13.269},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":12.622999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":13.906000000000001},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.0459999999999994}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":9.7140000000000004},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.375}]}]},{"osd":1,"up_from":17,"seq":73014444128,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.2110000000000003},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":8.2439999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.1870000000000003},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":9.4390000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":8.8960000000000008},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":17.923999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":10.500999999999999},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":11.519}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":13.374000000000001},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":12.632999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":14.276999999999999},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":15.375}]}]},{"osd":0,"up_from":17,"seq":73014444128,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":4.96},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.4269999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":11.266999999999999},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.7560000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":9.109},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":11.869999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":16.675999999999998},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":12.582000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":10.542999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":13.404}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":18.562000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":8.0250000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509536,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":11.003},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":6.2770000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":7.3499999999999996},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.377000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":8.9380000000000006},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":13.01}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.663},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":9.5259999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":14.382},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":13.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":16.077000000000002},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":7.9889999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:58.422 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:58.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.523+0000 7f351497c640 10 monclient: get_monmap_and_config 2024-01-20T11:22:58.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.523+0000 7f351497c640 10 monclient: build_initial_monmap 2024-01-20T11:22:58.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.523+0000 7f351497c640 10 monclient: monmap: 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:58.524353+0000 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:58.524353+0000 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:58.524 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:58.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.524+0000 7f351497c640 1 Processor -- start 2024-01-20T11:22:58.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.524+0000 7f351497c640 1 -- start start 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.524+0000 7f351497c640 10 monclient: init 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f351006bb00 0x7f35100b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 10 monclient(hunting): picked mon.noname-b con 0x7f351006bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35100af790 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35100b0900 0x7f3510145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 10 monclient(hunting): picked mon.noname-a con 0x7f35100b0900 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351006c220 con 0x7f35100af790 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351006bee0 con 0x7f35100b0900 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35100ada30 con 0x7f351006bb00 2024-01-20T11:22:58.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f351497c640 10 monclient(hunting): authenticate will time out at 1821.875301s 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f350e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f351006bb00 0x7f35100b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.525+0000 7f350ed9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35100b0900 0x7f3510145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350ed9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35100b0900 0x7f3510145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33286/0 (socket says 172.21.15.110:33286) 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350ed9d640 1 -- 172.21.15.110:0/3976208795 learned_addr learned my addr 172.21.15.110:0/3976208795 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350ed9d640 10 monclient(hunting): get_auth_request con 0x7f35100b0900 auth_method 0 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f351006bb00 0x7f35100b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56982/0 (socket says 172.21.15.110:56982) 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350e59c640 10 monclient(hunting): get_auth_request con 0x7f351006bb00 auth_method 0 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:58.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:58.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:58.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:58.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:58.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:58.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:58.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:58.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.526+0000 7f350e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.528+0000 7f350e59c640 10 monclient(hunting): handle_auth_done global_id 5576 payload 274 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.528+0000 7f350e59c640 1 -- 172.21.15.110:0/3976208795 >> v2:172.21.15.110:3300/0 conn(0x7f35100b0900 msgr2=0x7f3510145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.528+0000 7f350e59c640 1 --2- 172.21.15.110:0/3976208795 >> v2:172.21.15.110:3300/0 conn(0x7f35100b0900 0x7f3510145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.528+0000 7f350e59c640 1 -- 172.21.15.110:0/3976208795 >> v1:172.21.15.110:6789/0 conn(0x7f35100af790 legacy=0x7f3510145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.528+0000 7f350e59c640 10 monclient: _finish_hunting 0 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.528+0000 7f350e59c640 1 monclient: found mon.noname-b 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.528+0000 7f350e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.528+0000 7f350e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.528+0000 7f350e59c640 1 -- 172.21.15.110:0/3976208795 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35100add10 con 0x7f351006bb00 2024-01-20T11:22:58.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.530+0000 7f350e59c640 1 --2- 172.21.15.110:0/3976208795 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f351006bb00 0x7f35100b03c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f35000097a0 tx=0x7f350002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=343d80a5929c9939 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:58.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 1 -- 172.21.15.110:0/3976208795 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350002f050 con 0x7f351006bb00 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 10 monclient: dump: 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 10 monclient: _finish_auth 0 2024-01-20T11:22:58.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 10 monclient: _check_auth_tickets 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 1 -- 172.21.15.110:0/3976208795 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500031700 con 0x7f351006bb00 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f351497c640 5 monclient: authenticate success, global_id 5576 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f351497c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f351497c640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f351497c640 4 set_mon_vals no callback set 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 1 -- 172.21.15.110:0/3976208795 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500031a90 con 0x7f351006bb00 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.533+0000 7f350dd9b640 10 monclient: dump: 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:58.535 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:58.536 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:58.536 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:58.536 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:58.536 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:58.536 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:58.536 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:58.536 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:58.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 10 monclient: shutdown 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 1 -- 172.21.15.110:0/3976208795 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f351006bb00 msgr2=0x7f35100b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 1 --2- 172.21.15.110:0/3976208795 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f351006bb00 0x7f35100b03c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f35000097a0 tx=0x7f350002b7a0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 1 -- 172.21.15.110:0/3976208795 shutdown_connections 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 1 --2- 172.21.15.110:0/3976208795 >> v2:172.21.15.110:3300/0 conn(0x7f35100b0900 0x7f3510145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 1 --2- 172.21.15.110:0/3976208795 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f351006bb00 0x7f35100b03c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 1 -- 172.21.15.110:0/3976208795 >> 172.21.15.110:0/3976208795 conn(0x7f3510066d60 msgr2=0x7f35100b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 1 -- 172.21.15.110:0/3976208795 shutdown_connections 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.534+0000 7f351497c640 1 -- 172.21.15.110:0/3976208795 wait complete. 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.535+0000 7f351497c640 10 monclient: build_initial_monmap 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.535+0000 7f351497c640 10 monclient: monmap: 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:58.540 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.535+0000 7f351497c640 1 librados: starting msgr at 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.535+0000 7f351497c640 1 librados: starting objecter 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 Processor -- start 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 -- start start 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 librados: setting wanted keys 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 librados: calling monclient init 2024-01-20T11:22:58.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient: init 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f351006bb00 0x7f351013eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient(hunting): picked mon.noname-b con 0x7f351006bb00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35100af790 0x7f35101dac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient(hunting): picked mon.noname-a con 0x7f35100af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35100b0900 0x7f351011fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient(hunting): picked mon.noname-c con 0x7f35100b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351013f550 con 0x7f35100af790 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351013f6c0 con 0x7f351006bb00 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35101db170 con 0x7f35100b0900 2024-01-20T11:22:58.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.536+0000 7f351497c640 10 monclient(hunting): authenticate will time out at 1821.886879s 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f3507fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35100af790 0x7f35101dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f350ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35100b0900 0x7f351011fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f3507fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35100af790 0x7f35101dac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33296/0 (socket says 172.21.15.110:33296) 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f350ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35100b0900 0x7f351011fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:56986/0 (socket says 172.21.15.110:56986) 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f3507fff640 1 -- 172.21.15.110:0/1226244663 learned_addr learned my addr 172.21.15.110:0/1226244663 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f3507fff640 10 monclient(hunting): get_auth_request con 0x7f35100af790 auth_method 0 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f3507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f3507fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f3507fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f350ed9d640 10 monclient(hunting): get_auth_request con 0x7f35100b0900 auth_method 0 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f350ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f350ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f350ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:58.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.537+0000 7f350e59c640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f351006bb00 0x7f351013eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f3507fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f3507fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f3507fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350e59c640 10 monclient(hunting): get_auth_request con 0x7f351006bb00 auth_method 0 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:58.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.538+0000 7f350e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:58.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.540+0000 7f350ed9d640 10 monclient(hunting): handle_auth_done global_id 5579 payload 824 2024-01-20T11:22:58.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.540+0000 7f350ed9d640 1 -- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f351006bb00 msgr2=0x7f351013eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:58.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.540+0000 7f350ed9d640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f351006bb00 0x7f351013eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.540+0000 7f350ed9d640 1 -- 172.21.15.110:0/1226244663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35100af790 msgr2=0x7f35101dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.540+0000 7f350ed9d640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35100af790 0x7f35101dac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.541+0000 7f350ed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.541+0000 7f350ed9d640 1 monclient: found mon.noname-c 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.541+0000 7f350ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.541+0000 7f350ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.541+0000 7f350ed9d640 1 -- 172.21.15.110:0/1226244663 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3500009400 con 0x7f35100b0900 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.543+0000 7f350ed9d640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35100b0900 0x7f351011fcb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f350800eba0 tx=0x7f350800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.543+0000 7f3506ffd640 10 client.?.objecter ms_handle_connect 0x7f35100b0900 2024-01-20T11:22:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.543+0000 7f3506ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:58.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 1 -- 172.21.15.110:0/1226244663 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350800cb20 con 0x7f35100b0900 2024-01-20T11:22:58.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 10 monclient: dump: 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:58.548 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 10 monclient: _finish_auth 0 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 1 -- 172.21.15.110:0/1226244663 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350800f040 con 0x7f35100b0900 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 1 -- 172.21.15.110:0/1226244663 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350800cea0 con 0x7f35100b0900 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f3506ffd640 10 monclient: dump: 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f351497c640 5 monclient: authenticate success, global_id 5579 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f351497c640 10 monclient: _renew_subs 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f351497c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35101201f0 con 0x7f35100b0900 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.546+0000 7f350d59a640 4 set_mon_vals no callback set 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.547+0000 7f351497c640 10 client.5579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.547+0000 7f351497c640 10 monclient: _renew_subs 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.547+0000 7f351497c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.547+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35101206b0 con 0x7f35100b0900 2024-01-20T11:22:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.547+0000 7f351497c640 1 librados: init done 2024-01-20T11:22:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.547+0000 7f351497c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.547+0000 7f351497c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.547+0000 7f351497c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:22:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.547+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35101db7b0 con 0x7f35100b0900 2024-01-20T11:22:58.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.794+0000 7f3506ffd640 1 -- 172.21.15.110:0/1226244663 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3508003990 con 0x7f35100b0900 2024-01-20T11:22:58.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.794+0000 7f3506ffd640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f34f006c8b0 0x7f34f006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:58.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.795+0000 7f350e59c640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f34f006c8b0 0x7f34f006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:58.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.795+0000 7f350e59c640 10 monclient: get_auth_request con 0x7f34f006c8b0 auth_method 0 2024-01-20T11:22:58.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.796+0000 7f350e59c640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f34f006c8b0 0x7f34f006ed70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3500035000 tx=0x7f3500008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:58.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.796+0000 7f3506ffd640 10 client.5579.objecter ms_handle_connect 0x7f34f006c8b0 2024-01-20T11:22:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.808+0000 7f3506ffd640 1 -- 172.21.15.110:0/1226244663 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f35080592a0 con 0x7f35100b0900 2024-01-20T11:22:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.808+0000 7f3506ffd640 10 client.5579.objecter ms_dispatch 0x7f3510000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.808+0000 7f3506ffd640 3 client.5579.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.808+0000 7f3506ffd640 3 client.5579.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.808+0000 7f3506ffd640 20 client.5579.objecter dump_active .. 0 homeless 2024-01-20T11:22:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.808+0000 7f3506ffd640 1 -- 172.21.15.110:0/1226244663 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f35080555e0 con 0x7f35100b0900 2024-01-20T11:22:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.808+0000 7f3506ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.808+0000 7f3506ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.979+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f351006c910 con 0x7f34f006c8b0 2024-01-20T11:22:58.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.982+0000 7f3506ffd640 1 -- 172.21.15.110:0/1226244663 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63864 (secure 0 0 0) 0x7f3500042070 con 0x7f34f006c8b0 2024-01-20T11:22:58.983 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:58.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.984+0000 7f351497c640 10 librados: watch_flush enter 2024-01-20T11:22:58.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.984+0000 7f351497c640 10 librados: watch_flush exit 2024-01-20T11:22:58.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.984+0000 7f351497c640 20 client.5579.objecter shutdown clearing up homeless session... 2024-01-20T11:22:58.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.984+0000 7f351497c640 10 client.5579.objecter successfully canceled tick 2024-01-20T11:22:58.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f34f006c8b0 msgr2=0x7f34f006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:58.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f34f006c8b0 0x7f34f006ed70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3500035000 tx=0x7f3500008030 comp rx=0 tx=0).stop 2024-01-20T11:22:58.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 10 monclient: shutdown 2024-01-20T11:22:58.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35100b0900 msgr2=0x7f351011fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35100b0900 0x7f351011fcb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f350800eba0 tx=0x7f350800c490 comp rx=0 tx=0).stop 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 shutdown_connections 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f34f006c8b0 0x7f34f006ed70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35100b0900 0x7f351011fcb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35100af790 0x7f35101dac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 --2- 172.21.15.110:0/1226244663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f351006bb00 0x7f351013eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 >> 172.21.15.110:0/1226244663 conn(0x7f3510066d60 msgr2=0x7f35100b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 shutdown_connections 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 -- 172.21.15.110:0/1226244663 wait complete. 2024-01-20T11:22:58.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:58.985+0000 7f351497c640 1 librados: shutdown 2024-01-20T11:22:58.995 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-01-20T11:22:58.805066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280611,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":3.5430000000000001},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":8.2750000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":14.324999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":6.5730000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.385999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.6449999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.176},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":11.445}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":15.101000000000001},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":10.776999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":12.615},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":9.3200000000000003}]}]},{"osd":7,"up_from":21,"seq":90194313313,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.444},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.3119999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":6.2320000000000002},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.7010000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.981999999999999},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":10.584}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.3579999999999997},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":15.127000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":13.869999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.175000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":14.295},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":12.335000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313314,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.6970000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":7.7489999999999997},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.6920000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.308999999999999},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.446999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.923},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":13.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.3079999999999998},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.039}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":15.102},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":11.798}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444128,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":6.952},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":6.4379999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":5.3600000000000003},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.5060000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.425000000000001},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":13.378}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.542999999999999},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":13.923}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":16.405999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":8.5549999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.382},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.128}]}]},{"osd":1,"up_from":17,"seq":73014444129,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":10.532},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":5.5839999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":13.316000000000001},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":6.1909999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":14.116},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":7.593}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.1609999999999996},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.897}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.2279999999999998},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.635}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":9.8569999999999993},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":12.233000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444129,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":9.9299999999999997},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":5.0730000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.1529999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":16.361999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":18.335999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":10.772}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":11.654999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":8.0079999999999991}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":17.52},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.638}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":13.539999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":8.6660000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":12.42},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.7910000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.9169999999999998},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":13.224}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":8.5830000000000002},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":11.071}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":13.847},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":9.7059999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":16.065000000000001},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":15.262}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":11.763999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.499000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:58.997 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:22:59.045 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:50.070 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:59.049 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:50.074 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:59.058 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:50.064 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:59.059 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:50.084 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:59.061 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:50.067 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:59.061 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:50.086 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:59.071 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:50.096 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:59.082 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:50.088 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:59.083 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:50.107 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:22:59.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.108+0000 7f6f5d24b640 10 monclient: get_monmap_and_config 2024-01-20T11:22:59.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.108+0000 7f6f5d24b640 10 monclient: build_initial_monmap 2024-01-20T11:22:59.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.108+0000 7f6f5d24b640 10 monclient: monmap: 2024-01-20T11:22:59.108 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:59.108 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.108 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:59.109340+0000 2024-01-20T11:22:59.109 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:59.109340+0000 2024-01-20T11:22:59.109 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:59.109 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.109 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:59.109 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:59.109 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:59.109 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:59.109 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:59.109 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.109+0000 7f6f5d24b640 1 Processor -- start 2024-01-20T11:22:59.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.109+0000 7f6f5d24b640 1 -- start start 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.109+0000 7f6f5d24b640 10 monclient: init 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f580a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 0x7f6f581423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 10 monclient(hunting): picked mon.noname-b con 0x7f6f580a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f58142920 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f580b25c0 con 0x7f6f58142920 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5813fbe0 con 0x7f6f580a96f0 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580aaea0 con 0x7f6f580a9ad0 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:59.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.110+0000 7f6f5d24b640 10 monclient(hunting): authenticate will time out at 1822.460401s 2024-01-20T11:22:59.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.121+0000 7f6f4ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 0x7f6f581423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.123+0000 7f6f5759e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6f58142920 0x7f6f5814c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58036/0 (socket says 172.21.15.110:58036) 2024-01-20T11:22:59.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.123+0000 7f6f5759e640 1 -- 172.21.15.110:0/512550144 learned_addr learned my addr 172.21.15.110:0/512550144 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.126+0000 7f6f4ffff640 10 monclient(hunting): get_auth_request con 0x7f6f580a9ad0 auth_method 0 2024-01-20T11:22:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.126+0000 7f6f4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.126+0000 7f6f4ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.126+0000 7f6f4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:59.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.162+0000 7f6f4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:59.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.162+0000 7f6f4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:59.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.162+0000 7f6f4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:59.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.179+0000 7f6f4ffff640 10 monclient(hunting): handle_auth_done global_id 5582 payload 274 2024-01-20T11:22:59.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.179+0000 7f6f4ffff640 1 -- 172.21.15.110:0/512550144 >> v1:172.21.15.134:6789/0 conn(0x7f6f580a96f0 legacy=0x7f6f5813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.179+0000 7f6f4ffff640 1 -- 172.21.15.110:0/512550144 >> v1:172.21.15.110:6789/0 conn(0x7f6f58142920 legacy=0x7f6f5814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.179+0000 7f6f4ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:22:59.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.179+0000 7f6f4ffff640 1 monclient: found mon.noname-b 2024-01-20T11:22:59.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.179+0000 7f6f4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:59.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.179+0000 7f6f4ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:22:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.179+0000 7f6f4ffff640 1 -- 172.21.15.110:0/512550144 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f580ab0a0 con 0x7f6f580a9ad0 2024-01-20T11:22:59.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.234+0000 7f6f4ffff640 1 --2- 172.21.15.110:0/512550144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 0x7f6f581423e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6f400097d0 tx=0x7f6f4002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25212680b2498c07 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 1 -- 172.21.15.110:0/512550144 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40031800 con 0x7f6f580a9ad0 2024-01-20T11:22:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 10 monclient: dump: 2024-01-20T11:22:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.292 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 10 monclient: _finish_auth 0 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 10 monclient: _check_auth_tickets 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 1 -- 172.21.15.110:0/512550144 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f40031960 con 0x7f6f580a9ad0 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 1 -- 172.21.15.110:0/512550144 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40031cf0 con 0x7f6f580a9ad0 2024-01-20T11:22:59.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5d24b640 5 monclient: authenticate success, global_id 5582 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5d24b640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5d24b640 4 set_mon_vals no callback set 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.292+0000 7f6f5659c640 10 monclient: dump: 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:59.294 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.293+0000 7f6f5d24b640 10 monclient: shutdown 2024-01-20T11:22:59.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.293+0000 7f6f5d24b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.293+0000 7f6f5d24b640 1 -- 172.21.15.110:0/512550144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 msgr2=0x7f6f581423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.293+0000 7f6f5d24b640 1 --2- 172.21.15.110:0/512550144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 0x7f6f581423e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6f400097d0 tx=0x7f6f4002b8a0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.293+0000 7f6f5d24b640 1 -- 172.21.15.110:0/512550144 shutdown_connections 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.293+0000 7f6f5d24b640 1 --2- 172.21.15.110:0/512550144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 0x7f6f581423e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.293+0000 7f6f5d24b640 1 -- 172.21.15.110:0/512550144 >> 172.21.15.110:0/512550144 conn(0x7f6f580b7bc0 msgr2=0x7f6f580b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.293+0000 7f6f5d24b640 1 -- 172.21.15.110:0/512550144 shutdown_connections 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.293+0000 7f6f5d24b640 1 -- 172.21.15.110:0/512550144 wait complete. 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.294+0000 7f6f5d24b640 10 monclient: build_initial_monmap 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.294+0000 7f6f5d24b640 10 monclient: monmap: 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:59.297 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.294+0000 7f6f5d24b640 1 librados: starting msgr at 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.294+0000 7f6f5d24b640 1 librados: starting objecter 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.294+0000 7f6f5d24b640 1 Processor -- start 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.294+0000 7f6f5d24b640 1 -- start start 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.294+0000 7f6f5d24b640 1 librados: setting wanted keys 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.294+0000 7f6f5d24b640 1 librados: calling monclient init 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.294+0000 7f6f5d24b640 10 monclient: init 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:59.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f580a96f0 0x7f6f58139660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient(hunting): picked mon.noname-b con 0x7f6f580a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 0x7f6f580adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient(hunting): picked mon.noname-c con 0x7f6f580a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f58142920 0x7f6f58136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient(hunting): picked mon.noname-a con 0x7f6f58142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580ae110 con 0x7f6f58142920 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580ad2f0 con 0x7f6f580a96f0 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580ae750 con 0x7f6f580a9ad0 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:59.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.295+0000 7f6f5d24b640 10 monclient(hunting): authenticate will time out at 1822.645497s 2024-01-20T11:22:59.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.297+0000 7f6f5759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f58142920 0x7f6f58136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:59.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.299+0000 7f6f5759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f58142920 0x7f6f58136cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33304/0 (socket says 172.21.15.110:33304) 2024-01-20T11:22:59.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.299+0000 7f6f5759e640 1 -- 172.21.15.110:0/1707997262 learned_addr learned my addr 172.21.15.110:0/1707997262 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:59.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.299+0000 7f6f5759e640 10 monclient(hunting): get_auth_request con 0x7f6f58142920 auth_method 0 2024-01-20T11:22:59.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.299+0000 7f6f5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:59.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.299+0000 7f6f5759e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:59.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.299+0000 7f6f5759e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:59.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.300+0000 7f6f5759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:59.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.300+0000 7f6f5759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:59.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.300+0000 7f6f5759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:59.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.301+0000 7f6f56d9d640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f580a96f0 0x7f6f58139660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:59.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 10 monclient(hunting): handle_auth_done global_id 5934 payload 824 2024-01-20T11:22:59.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 1 -- 172.21.15.110:0/1707997262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 msgr2=0x7f6f580adaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 0x7f6f580adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 1 -- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f580a96f0 msgr2=0x7f6f58139660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f580a96f0 0x7f6f58139660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 10 monclient: _finish_hunting 0 2024-01-20T11:22:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 1 monclient: found mon.noname-a 2024-01-20T11:22:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.303+0000 7f6f5759e640 1 -- 172.21.15.110:0/1707997262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f40009400 con 0x7f6f58142920 2024-01-20T11:22:59.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.309+0000 7f6f5759e640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f58142920 0x7f6f58136cd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000caa0 tx=0x7f6f5000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:59.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.309+0000 7f6f4f7fe640 10 client.?.objecter ms_handle_connect 0x7f6f58142920 2024-01-20T11:22:59.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.309+0000 7f6f4f7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f4f7fe640 1 -- 172.21.15.110:0/1707997262 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5000ac00 con 0x7f6f58142920 2024-01-20T11:22:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f4f7fe640 10 monclient: dump: 2024-01-20T11:22:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f4f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f4f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f5d24b640 5 monclient: authenticate success, global_id 5934 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f5d24b640 10 monclient: _renew_subs 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f5d24b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f58137210 con 0x7f6f58142920 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f5d24b640 10 client.5934.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f5d24b640 10 monclient: _renew_subs 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f5d24b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f581376e0 con 0x7f6f58142920 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f5d24b640 1 librados: init done 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f4f7fe640 1 -- 172.21.15.110:0/1707997262 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5000ad60 con 0x7f6f58142920 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.317+0000 7f6f55d9b640 4 set_mon_vals no callback set 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.318+0000 7f6f5d24b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.318+0000 7f6f5d24b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.318+0000 7f6f5d24b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.318+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f58075490 con 0x7f6f58142920 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.318+0000 7f6f4f7fe640 1 -- 172.21.15.110:0/1707997262 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5001c640 con 0x7f6f58142920 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.318+0000 7f6f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.318+0000 7f6f4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.318+0000 7f6f4f7fe640 10 monclient: dump: 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:59.321 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.598+0000 7f6f4f7fe640 1 -- 172.21.15.110:0/1707997262 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6f5001c7a0 con 0x7f6f58142920 2024-01-20T11:22:59.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.598+0000 7f6f4f7fe640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f3806c890 0x7f6f3806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:59.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.598+0000 7f6f56d9d640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f3806c890 0x7f6f3806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:59.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.598+0000 7f6f56d9d640 10 monclient: get_auth_request con 0x7f6f3806c890 auth_method 0 2024-01-20T11:22:59.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.599+0000 7f6f4f7fe640 10 client.5934.objecter ms_handle_connect 0x7f6f3806c890 2024-01-20T11:22:59.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.599+0000 7f6f56d9d640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f3806c890 0x7f6f3806ed50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6f480015a0 tx=0x7f6f48009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:59.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.612+0000 7f6f4f7fe640 1 -- 172.21.15.110:0/1707997262 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6f50058840 con 0x7f6f58142920 2024-01-20T11:22:59.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.612+0000 7f6f4f7fe640 10 client.5934.objecter ms_dispatch 0x7f6f58000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:22:59.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.612+0000 7f6f4f7fe640 3 client.5934.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:22:59.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.612+0000 7f6f4f7fe640 3 client.5934.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:22:59.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.612+0000 7f6f4f7fe640 20 client.5934.objecter dump_active .. 0 homeless 2024-01-20T11:22:59.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.612+0000 7f6f4f7fe640 1 -- 172.21.15.110:0/1707997262 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6f50054b80 con 0x7f6f58142920 2024-01-20T11:22:59.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.612+0000 7f6f4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:59.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.612+0000 7f6f4f7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:22:59.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.787+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6f580b4400 con 0x7f6f3806c890 2024-01-20T11:22:59.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.790+0000 7f6f4f7fe640 1 -- 172.21.15.110:0/1707997262 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63864 (secure 0 0 0) 0x7f6f4800d4a0 con 0x7f6f3806c890 2024-01-20T11:22:59.790 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:22:59.791 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:22:59.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 10 librados: watch_flush enter 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 10 librados: watch_flush exit 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 20 client.5934.objecter shutdown clearing up homeless session... 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 10 client.5934.objecter successfully canceled tick 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f3806c890 msgr2=0x7f6f3806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f3806c890 0x7f6f3806ed50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6f480015a0 tx=0x7f6f48009040 comp rx=0 tx=0).stop 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 10 monclient: shutdown 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f58142920 msgr2=0x7f6f58136cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f58142920 0x7f6f58136cd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000caa0 tx=0x7f6f5000cf60 comp rx=0 tx=0).stop 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 shutdown_connections 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f6f3806c890 0x7f6f3806ed50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f58142920 0x7f6f58136cd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f580a9ad0 0x7f6f580adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 --2- 172.21.15.110:0/1707997262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f580a96f0 0x7f6f58139660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 >> 172.21.15.110:0/1707997262 conn(0x7f6f580b7bc0 msgr2=0x7f6f5813b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 shutdown_connections 2024-01-20T11:22:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 -- 172.21.15.110:0/1707997262 wait complete. 2024-01-20T11:22:59.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.792+0000 7f6f5d24b640 1 librados: shutdown 2024-01-20T11:22:59.803 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-01-20T11:22:58.805066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280611,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":3.5430000000000001},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":8.2750000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":14.324999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":6.5730000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.385999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.6449999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.176},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":11.445}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":15.101000000000001},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":10.776999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":12.615},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":9.3200000000000003}]}]},{"osd":7,"up_from":21,"seq":90194313313,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.444},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.3119999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":6.2320000000000002},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.7010000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.981999999999999},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":10.584}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.3579999999999997},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":15.127000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":13.869999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.175000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":14.295},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":12.335000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313314,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.6970000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":7.7489999999999997},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.6920000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.308999999999999},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.446999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.923},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":13.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.3079999999999998},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.039}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":15.102},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":11.798}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444128,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":6.952},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":6.4379999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":5.3600000000000003},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.5060000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.425000000000001},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":13.378}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.542999999999999},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":13.923}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":16.405999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":8.5549999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.382},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.128}]}]},{"osd":1,"up_from":17,"seq":73014444129,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":10.532},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":5.5839999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":13.316000000000001},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":6.1909999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":14.116},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":7.593}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.1609999999999996},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.897}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.2279999999999998},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.635}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":9.8569999999999993},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":12.233000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444129,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":9.9299999999999997},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":5.0730000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.1529999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":16.361999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":18.335999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":10.772}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":11.654999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":8.0079999999999991}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":17.52},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.638}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":13.539999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":8.6660000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":12.42},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.7910000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.9169999999999998},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":13.224}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":8.5830000000000002},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":11.071}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":13.847},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":9.7059999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":16.065000000000001},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":15.262}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":11.763999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.499000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:22:59.804 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:22:59.804 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:22:59.805 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:22:59.805 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:22:59.805 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:22:59.805 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:22:59.805 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:22:59.805 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:22:59.805 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:22:59.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.902+0000 7f81cd701640 10 monclient: get_monmap_and_config 2024-01-20T11:22:59.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.902+0000 7f81cd701640 10 monclient: build_initial_monmap 2024-01-20T11:22:59.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.902+0000 7f81cd701640 10 monclient: monmap: 2024-01-20T11:22:59.902 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:59.902 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:22:59.902716+0000 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:22:59.902716+0000 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.902+0000 7f81cd701640 1 Processor -- start 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.902+0000 7f81cd701640 1 -- start start 2024-01-20T11:22:59.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.902+0000 7f81cd701640 10 monclient: init 2024-01-20T11:22:59.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:59.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-01-20T11:22:59.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81c80a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:22:59.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81c80a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:22:59.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f81c8142af0 0x7f81c814d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:59.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 10 monclient(hunting): picked mon.noname-a con 0x7f81c8142af0 addr v2:172.21.15.110:3300/0 2024-01-20T11:22:59.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c80b25c0 con 0x7f81c80a9ad0 2024-01-20T11:22:59.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c8142f50 con 0x7f81c80a96f0 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c80aaea0 con 0x7f81c8142af0 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81cd701640 10 monclient(hunting): authenticate will time out at 1823.253488s 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81c77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f81c8142af0 0x7f81c814d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81c6ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f81c80a96f0 0x7f81c813f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:57398/0 (socket says 172.21.15.110:57398) 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.903+0000 7f81c6ffd640 1 -- 172.21.15.110:0/2740675104 learned_addr learned my addr 172.21.15.110:0/2740675104 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.904+0000 7f81c77fe640 10 monclient(hunting): get_auth_request con 0x7f81c8142af0 auth_method 0 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.904+0000 7f81c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.904+0000 7f81c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.904+0000 7f81c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:59.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.904+0000 7f81c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.904+0000 7f81c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.904+0000 7f81c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c5ffb640 1 -- 172.21.15.110:0/2740675104 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441635328 0 0) 0x7f81c0001110 con 0x7f81c80a96f0 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c5ffb640 10 monclient(hunting): _negotiate 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c5ffb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c5ffb640 10 monclient(hunting): my global_id is 6034 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c5ffb640 1 -- 172.21.15.110:0/2740675104 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b0003520 con 0x7f81c80a96f0 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c77fe640 10 monclient(hunting): handle_auth_done global_id 5937 payload 274 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c77fe640 1 -- 172.21.15.110:0/2740675104 >> v1:172.21.15.134:6789/0 conn(0x7f81c80a96f0 legacy=0x7f81c813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c77fe640 1 -- 172.21.15.110:0/2740675104 >> v1:172.21.15.110:6789/0 conn(0x7f81c80a9ad0 legacy=0x7f81c81423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:22:59.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:22:59.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c77fe640 1 monclient: found mon.noname-a 2024-01-20T11:22:59.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:59.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:22:59.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.906+0000 7f81c77fe640 1 -- 172.21.15.110:0/2740675104 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c813fbe0 con 0x7f81c8142af0 2024-01-20T11:22:59.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.908+0000 7f81c77fe640 1 --2- 172.21.15.110:0/2740675104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 0x7f81c814d860 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f81b80097a0 tx=0x7f81b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0229f9a81b1ea34 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:59.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 1 -- 172.21.15.110:0/2740675104 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b802e070 con 0x7f81c8142af0 2024-01-20T11:22:59.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:59.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:22:59.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 10 monclient: dump: 2024-01-20T11:22:59.912 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:59.912 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.912 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 10 monclient: _finish_auth 0 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 1 -- 172.21.15.110:0/2740675104 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b80315a0 con 0x7f81c8142af0 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:59.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 1 -- 172.21.15.110:0/2740675104 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8031930 con 0x7f81c8142af0 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81cd701640 5 monclient: authenticate success, global_id 5937 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81cd701640 10 monclient: get_monmap_and_config success 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81cd701640 4 set_mon_vals no callback set 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.912+0000 7f81c5ffb640 10 monclient: dump: 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.914 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:59.915 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:59.915 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:59.915 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 10 monclient: shutdown 2024-01-20T11:22:59.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:22:59.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 1 -- 172.21.15.110:0/2740675104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 msgr2=0x7f81c814d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 1 --2- 172.21.15.110:0/2740675104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 0x7f81c814d860 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f81b80097a0 tx=0x7f81b802b840 comp rx=0 tx=0).stop 2024-01-20T11:22:59.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 1 -- 172.21.15.110:0/2740675104 shutdown_connections 2024-01-20T11:22:59.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 1 --2- 172.21.15.110:0/2740675104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 0x7f81c814d860 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 1 -- 172.21.15.110:0/2740675104 >> 172.21.15.110:0/2740675104 conn(0x7f81c80b7bc0 msgr2=0x7f81c80b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 1 -- 172.21.15.110:0/2740675104 shutdown_connections 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 1 -- 172.21.15.110:0/2740675104 wait complete. 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 10 monclient: build_initial_monmap 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.913+0000 7f81cd701640 10 monclient: monmap: 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:22:59.920 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 librados: starting msgr at 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 librados: starting objecter 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 Processor -- start 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 -- start start 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 librados: setting wanted keys 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 librados: calling monclient init 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient: init 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient: _reopen_session rank -1 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81c80a96f0 0x7f81c8133260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:59.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient(hunting): picked mon.noname-c con 0x7f81c80a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81c80a9ad0 0x7f81c8131010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient(hunting): picked mon.noname-b con 0x7f81c80a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 0x7f81c8131550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient(hunting): picked mon.noname-a con 0x7f81c8142af0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8131a90 con 0x7f81c8142af0 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c813cb90 con 0x7f81c80a9ad0 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient(hunting): start opening mon connection 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8132230 con 0x7f81c80a96f0 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient(hunting): _renew_subs 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.914+0000 7f81cd701640 10 monclient(hunting): authenticate will time out at 1823.265132s 2024-01-20T11:22:59.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 0x7f81c8131550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c67fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81c80a9ad0 0x7f81c8131010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81c80a96f0 0x7f81c8133260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 0x7f81c8131550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53856/0 (socket says 172.21.15.110:53856) 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c77fe640 1 -- 172.21.15.110:0/3555688254 learned_addr learned my addr 172.21.15.110:0/3555688254 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c77fe640 10 monclient(hunting): get_auth_request con 0x7f81c8142af0 auth_method 0 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c67fc640 10 monclient(hunting): get_auth_request con 0x7f81c80a9ad0 auth_method 0 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c67fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:59.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c6ffd640 10 monclient(hunting): get_auth_request con 0x7f81c80a96f0 auth_method 0 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.915+0000 7f81c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.916+0000 7f81c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.916+0000 7f81c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.916+0000 7f81c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.916+0000 7f81c67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.916+0000 7f81c67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.916+0000 7f81c67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.916+0000 7f81c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.916+0000 7f81c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:22:59.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.916+0000 7f81c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.918+0000 7f81c67fc640 10 monclient(hunting): handle_auth_done global_id 6037 payload 824 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.918+0000 7f81c67fc640 1 -- 172.21.15.110:0/3555688254 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81c80a96f0 msgr2=0x7f81c8133260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.918+0000 7f81c67fc640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81c80a96f0 0x7f81c8133260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.918+0000 7f81c67fc640 1 -- 172.21.15.110:0/3555688254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 msgr2=0x7f81c8131550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.918+0000 7f81c67fc640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 0x7f81c8131550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.918+0000 7f81c67fc640 10 monclient: _finish_hunting 0 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.919+0000 7f81c67fc640 1 monclient: found mon.noname-b 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.919+0000 7f81c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.919+0000 7f81c67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:22:59.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.919+0000 7f81c67fc640 1 -- 172.21.15.110:0/3555688254 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b8009400 con 0x7f81c80a9ad0 2024-01-20T11:22:59.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.921+0000 7f81c67fc640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81c80a9ad0 0x7f81c8131010 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00c750 tx=0x7f81bc00cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:22:59.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.921+0000 7f81af7fe640 10 client.?.objecter ms_handle_connect 0x7f81c80a9ad0 2024-01-20T11:22:59.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.921+0000 7f81af7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:22:59.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 1 -- 172.21.15.110:0/3555688254 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc003b80 con 0x7f81c80a9ad0 2024-01-20T11:22:59.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:59.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:22:59.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 10 monclient: dump: 2024-01-20T11:22:59.927 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 10 monclient: _finish_auth 0 2024-01-20T11:22:59.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81cd701640 5 monclient: authenticate success, global_id 6037 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 1 -- 172.21.15.110:0/3555688254 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc003d00 con 0x7f81c80a9ad0 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 1 -- 172.21.15.110:0/3555688254 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0052b0 con 0x7f81c80a9ad0 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81cd701640 10 monclient: _renew_subs 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81cd701640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c8132510 con 0x7f81c80a9ad0 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81af7fe640 10 monclient: dump: 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:22:59.929 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81c57fa640 4 set_mon_vals no callback set 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81cd701640 10 client.6037.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81cd701640 10 monclient: _renew_subs 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81cd701640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:59.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c813fe00 con 0x7f81c80a9ad0 2024-01-20T11:22:59.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.924+0000 7f81cd701640 1 librados: init done 2024-01-20T11:22:59.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.925+0000 7f81cd701640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:59.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.925+0000 7f81cd701640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:22:59.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.925+0000 7f81cd701640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:22:59.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:22:59.925+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c813cd90 con 0x7f81c80a9ad0 2024-01-20T11:23:00.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.166+0000 7f81af7fe640 1 -- 172.21.15.110:0/3555688254 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f81bc005410 con 0x7f81c80a9ad0 2024-01-20T11:23:00.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.166+0000 7f81af7fe640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f81b406c8b0 0x7f81b406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:00.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.167+0000 7f81c6ffd640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f81b406c8b0 0x7f81b406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:00.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.167+0000 7f81c6ffd640 10 monclient: get_auth_request con 0x7f81b406c8b0 auth_method 0 2024-01-20T11:23:00.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.167+0000 7f81c6ffd640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f81b406c8b0 0x7f81b406ed70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f81c0005880 tx=0x7f81c0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:00.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.167+0000 7f81af7fe640 10 client.6037.objecter ms_handle_connect 0x7f81b406c8b0 2024-01-20T11:23:00.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.169+0000 7f81af7fe640 1 -- 172.21.15.110:0/3555688254 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f81bc0580c0 con 0x7f81c80a9ad0 2024-01-20T11:23:00.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.169+0000 7f81af7fe640 10 client.6037.objecter ms_dispatch 0x7f81c8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:00.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.169+0000 7f81af7fe640 3 client.6037.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:00.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.169+0000 7f81af7fe640 3 client.6037.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:00.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.169+0000 7f81af7fe640 20 client.6037.objecter dump_active .. 0 homeless 2024-01-20T11:23:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.180+0000 7f81af7fe640 1 -- 172.21.15.110:0/3555688254 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f81bc054400 con 0x7f81c80a9ad0 2024-01-20T11:23:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.180+0000 7f81af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.180+0000 7f81af7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:00.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~242s 2024-01-20T11:23:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.429+0000 7f81cd701640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.429+0000 7f81cd701640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.429+0000 7f81cd701640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:00.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.429+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f81c813cd90 con 0x7f81c80a9ad0 2024-01-20T11:23:00.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.435+0000 7f81af7fe640 1 -- 172.21.15.110:0/3555688254 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f81c813cd90 con 0x7f81c80a9ad0 2024-01-20T11:23:00.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.435+0000 7f81af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:00.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.435+0000 7f81af7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:00.435 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:00.435 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":514,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:00.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 10 librados: watch_flush enter 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 10 librados: watch_flush exit 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 20 client.6037.objecter shutdown clearing up homeless session... 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 10 client.6037.objecter successfully canceled tick 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f81b406c8b0 msgr2=0x7f81b406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f81b406c8b0 0x7f81b406ed70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f81c0005880 tx=0x7f81c0009040 comp rx=0 tx=0).stop 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 10 monclient: shutdown 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81c80a9ad0 msgr2=0x7f81c8131010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81c80a9ad0 0x7f81c8131010 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00c750 tx=0x7f81bc00cc10 comp rx=0 tx=0).stop 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 shutdown_connections 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f81b406c8b0 0x7f81b406ed70 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81c8142af0 0x7f81c8131550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81c80a9ad0 0x7f81c8131010 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 --2- 172.21.15.110:0/3555688254 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81c80a96f0 0x7f81c8133260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 >> 172.21.15.110:0/3555688254 conn(0x7f81c80b7bc0 msgr2=0x7f81c813f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:00.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 shutdown_connections 2024-01-20T11:23:00.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 -- 172.21.15.110:0/3555688254 wait complete. 2024-01-20T11:23:00.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.437+0000 7f81cd701640 1 librados: shutdown 2024-01-20T11:23:00.448 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:00.449 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:00.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.550+0000 7f84ad725640 10 monclient: get_monmap_and_config 2024-01-20T11:23:00.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.550+0000 7f84ad725640 10 monclient: build_initial_monmap 2024-01-20T11:23:00.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.550+0000 7f84ad725640 10 monclient: monmap: 2024-01-20T11:23:00.550 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:00.550 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:00.551107+0000 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:00.551107+0000 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:00.551 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:00.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 1 Processor -- start 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 1 -- start start 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient: init 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 0x7f84a813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient(hunting): picked mon.noname-b con 0x7f84a80a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f84a813fa10 0x7f84a81423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient(hunting): picked mon.noname-c con 0x7f84a813fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84a80a9ad0 0x7f84a8142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient(hunting): picked mon.noname-a con 0x7f84a80a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a80aaea0 con 0x7f84a80a9ad0 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:00.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a80b25c0 con 0x7f84a813fa10 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80ab0a0 con 0x7f84a80a96f0 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.551+0000 7f84ad725640 10 monclient(hunting): authenticate will time out at 1823.902026s 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84a80a9ad0 0x7f84a8142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f849e7fc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f84a813fa10 0x7f84a81423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 0x7f84a813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84a80a9ad0 0x7f84a8142920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53868/0 (socket says 172.21.15.110:53868) 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a77fe640 1 -- 172.21.15.110:0/583603959 learned_addr learned my addr 172.21.15.110:0/583603959 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f849e7fc640 1 --2- 172.21.15.110:0/583603959 >> v2:172.21.15.134:3300/0 conn(0x7f84a813fa10 0x7f84a81423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:34050/0 (socket says 172.21.15.110:34050) 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a77fe640 10 monclient(hunting): get_auth_request con 0x7f84a80a9ad0 auth_method 0 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:00.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f849e7fc640 10 monclient(hunting): get_auth_request con 0x7f84a813fa10 auth_method 0 2024-01-20T11:23:00.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f849e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:00.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f849e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:00.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f849e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:00.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a6ffd640 10 monclient(hunting): get_auth_request con 0x7f84a80a96f0 auth_method 0 2024-01-20T11:23:00.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:00.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:00.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.552+0000 7f84a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:00.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.553+0000 7f849e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:00.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.553+0000 7f849e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:00.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.553+0000 7f849e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:00.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.553+0000 7f84a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:00.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.553+0000 7f84a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:00.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.553+0000 7f84a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:00.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.553+0000 7f84a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:00.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.553+0000 7f84a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:00.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.553+0000 7f84a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 10 monclient(hunting): handle_auth_done global_id 6040 payload 274 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 1 -- 172.21.15.110:0/583603959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 msgr2=0x7f84a813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 1 --2- 172.21.15.110:0/583603959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 0x7f84a813f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 1 -- 172.21.15.110:0/583603959 >> v2:172.21.15.110:3300/0 conn(0x7f84a80a9ad0 msgr2=0x7f84a8142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 1 --2- 172.21.15.110:0/583603959 >> v2:172.21.15.110:3300/0 conn(0x7f84a80a9ad0 0x7f84a8142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 10 monclient: _finish_hunting 0 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 1 monclient: found mon.noname-c 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.554+0000 7f849e7fc640 1 -- 172.21.15.110:0/583603959 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a80b3fb0 con 0x7f84a813fa10 2024-01-20T11:23:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.556+0000 7f849e7fc640 1 --2- 172.21.15.110:0/583603959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a813fa10 0x7f84a81423e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f84a80726e0 tx=0x7f849002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=170a0b78ccc69543 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 1 -- 172.21.15.110:0/583603959 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849002f050 con 0x7f84a813fa10 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 10 monclient: dump: 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 1 -- 172.21.15.110:0/583603959 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8490031700 con 0x7f84a813fa10 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84ad725640 5 monclient: authenticate success, global_id 6040 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84ad725640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84ad725640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 1 -- 172.21.15.110:0/583603959 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490031a90 con 0x7f84a813fa10 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84ad725640 4 set_mon_vals no callback set 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.560+0000 7f84a67fc640 10 monclient: dump: 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 10 monclient: shutdown 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 1 -- 172.21.15.110:0/583603959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a813fa10 msgr2=0x7f84a81423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:00.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 1 --2- 172.21.15.110:0/583603959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a813fa10 0x7f84a81423e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f84a80726e0 tx=0x7f849002b7a0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 1 -- 172.21.15.110:0/583603959 shutdown_connections 2024-01-20T11:23:00.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 1 --2- 172.21.15.110:0/583603959 >> v2:172.21.15.110:3300/0 conn(0x7f84a80a9ad0 0x7f84a8142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 1 --2- 172.21.15.110:0/583603959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a813fa10 0x7f84a81423e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 1 --2- 172.21.15.110:0/583603959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 0x7f84a813f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 1 -- 172.21.15.110:0/583603959 >> 172.21.15.110:0/583603959 conn(0x7f84a80b7bc0 msgr2=0x7f84a80b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:00.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 1 -- 172.21.15.110:0/583603959 shutdown_connections 2024-01-20T11:23:00.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 1 -- 172.21.15.110:0/583603959 wait complete. 2024-01-20T11:23:00.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 10 monclient: build_initial_monmap 2024-01-20T11:23:00.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.561+0000 7f84ad725640 10 monclient: monmap: 2024-01-20T11:23:00.567 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:00.567 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:00.567 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:00.567 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:00.567 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:00.567 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.562+0000 7f84ad725640 1 librados: starting msgr at 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.562+0000 7f84ad725640 1 librados: starting objecter 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.562+0000 7f84ad725640 1 Processor -- start 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.562+0000 7f84ad725640 1 -- start start 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.562+0000 7f84ad725640 1 librados: setting wanted keys 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.562+0000 7f84ad725640 1 librados: calling monclient init 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.562+0000 7f84ad725640 10 monclient: init 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.562+0000 7f84ad725640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:00.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 0x7f84a81d7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 10 monclient(hunting): picked mon.noname-c con 0x7f84a80a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84a80a9ad0 0x7f84a81d77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 10 monclient(hunting): picked mon.noname-a con 0x7f84a80a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a80abff0 0x7f84a80ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 10 monclient(hunting): picked mon.noname-b con 0x7f84a80abff0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a81d7d10 con 0x7f84a80a9ad0 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80ad760 con 0x7f84a80abff0 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a80ad9d0 con 0x7f84a80a96f0 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:00.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84ad725640 10 monclient(hunting): authenticate will time out at 1823.913278s 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f849e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84a80a9ad0 0x7f84a81d77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 0x7f84a81d7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a80abff0 0x7f84a80ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f849e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84a80a9ad0 0x7f84a81d77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53874/0 (socket says 172.21.15.110:53874) 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f849e7fc640 1 -- 172.21.15.110:0/1399349251 learned_addr learned my addr 172.21.15.110:0/1399349251 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f849e7fc640 10 monclient(hunting): get_auth_request con 0x7f84a80a9ad0 auth_method 0 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f849e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f849e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f849e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a77fe640 10 monclient(hunting): get_auth_request con 0x7f84a80abff0 auth_method 0 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a6ffd640 10 monclient(hunting): get_auth_request con 0x7f84a80a96f0 auth_method 0 2024-01-20T11:23:00.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.563+0000 7f84a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.564+0000 7f849e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.564+0000 7f849e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.564+0000 7f849e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.564+0000 7f84a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.564+0000 7f84a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.564+0000 7f84a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.564+0000 7f84a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.564+0000 7f84a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.564+0000 7f84a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 10 monclient(hunting): handle_auth_done global_id 6043 payload 824 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 1 -- 172.21.15.110:0/1399349251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 msgr2=0x7f84a81d7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:00.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 0x7f84a81d7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 1 -- 172.21.15.110:0/1399349251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84a80a9ad0 msgr2=0x7f84a81d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:00.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84a80a9ad0 0x7f84a81d77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:00.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 1 monclient: found mon.noname-b 2024-01-20T11:23:00.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:00.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:00.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.566+0000 7f84a77fe640 1 -- 172.21.15.110:0/1399349251 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490009400 con 0x7f84a80abff0 2024-01-20T11:23:00.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.569+0000 7f84a77fe640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a80abff0 0x7f84a80ac4a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84a000efc0 tx=0x7f84a000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:00.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.569+0000 7f849ffff640 10 client.?.objecter ms_handle_connect 0x7f84a80abff0 2024-01-20T11:23:00.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.569+0000 7f849ffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 1 -- 172.21.15.110:0/1399349251 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a000cd20 con 0x7f84a80abff0 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 10 monclient: dump: 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:00.574 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 10 monclient: _finish_auth 0 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 1 -- 172.21.15.110:0/1399349251 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a000f040 con 0x7f84a80abff0 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 1 -- 172.21.15.110:0/1399349251 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0003bd0 con 0x7f84a80abff0 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 5 monclient: authenticate success, global_id 6043 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 10 monclient: _renew_subs 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a81d6d70 con 0x7f84a80abff0 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84a5ffb640 4 set_mon_vals no callback set 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f849ffff640 10 monclient: dump: 2024-01-20T11:23:00.575 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 10 client.6043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 10 monclient: _renew_subs 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a80ac9e0 con 0x7f84a80abff0 2024-01-20T11:23:00.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 1 librados: init done 2024-01-20T11:23:00.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.573+0000 7f84ad725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:00.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.574+0000 7f84ad725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:00.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.574+0000 7f84ad725640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:00.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.574+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a8075490 con 0x7f84a80abff0 2024-01-20T11:23:00.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.592+0000 7f849ffff640 1 -- 172.21.15.110:0/1399349251 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f84a0003d30 con 0x7f84a80abff0 2024-01-20T11:23:00.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.593+0000 7f849ffff640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f848806c8b0 0x7f848806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:00.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.593+0000 7f84a6ffd640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f848806c8b0 0x7f848806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:00.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.593+0000 7f84a6ffd640 10 monclient: get_auth_request con 0x7f848806c8b0 auth_method 0 2024-01-20T11:23:00.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.593+0000 7f84a6ffd640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f848806c8b0 0x7f848806ed70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8498009ac0 tx=0x7f8498009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:00.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.593+0000 7f849ffff640 10 client.6043.objecter ms_handle_connect 0x7f848806c8b0 2024-01-20T11:23:00.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.813+0000 7f849ffff640 1 -- 172.21.15.110:0/1399349251 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f84a0059540 con 0x7f84a80abff0 2024-01-20T11:23:00.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.813+0000 7f849ffff640 10 client.6043.objecter ms_dispatch 0x7f84a8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:00.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.813+0000 7f849ffff640 3 client.6043.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:00.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.813+0000 7f849ffff640 3 client.6043.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:00.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.813+0000 7f849ffff640 20 client.6043.objecter dump_active .. 0 homeless 2024-01-20T11:23:00.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.823+0000 7f849ffff640 1 -- 172.21.15.110:0/1399349251 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f84a0055880 con 0x7f84a80abff0 2024-01-20T11:23:00.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.823+0000 7f849ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:00.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.823+0000 7f849ffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:00.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.991+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84a80b4400 con 0x7f848806c8b0 2024-01-20T11:23:00.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.994+0000 7f849ffff640 1 -- 172.21.15.110:0/1399349251 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63864 (secure 0 0 0) 0x7f84980119e0 con 0x7f848806c8b0 2024-01-20T11:23:00.994 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:00.994 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:00.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.995+0000 7f84ad725640 10 librados: watch_flush enter 2024-01-20T11:23:00.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 10 librados: watch_flush exit 2024-01-20T11:23:00.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 20 client.6043.objecter shutdown clearing up homeless session... 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 10 client.6043.objecter successfully canceled tick 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f848806c8b0 msgr2=0x7f848806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f848806c8b0 0x7f848806ed70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8498009ac0 tx=0x7f8498009190 comp rx=0 tx=0).stop 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 10 monclient: shutdown 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a80abff0 msgr2=0x7f84a80ac4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a80abff0 0x7f84a80ac4a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84a000efc0 tx=0x7f84a000c490 comp rx=0 tx=0).stop 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 shutdown_connections 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f848806c8b0 0x7f848806ed70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84a80abff0 0x7f84a80ac4a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84a80a9ad0 0x7f84a81d77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 --2- 172.21.15.110:0/1399349251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84a80a96f0 0x7f84a81d7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:00.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 >> 172.21.15.110:0/1399349251 conn(0x7f84a80b7bc0 msgr2=0x7f84a80b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:00.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 shutdown_connections 2024-01-20T11:23:00.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 -- 172.21.15.110:0/1399349251 wait complete. 2024-01-20T11:23:00.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:00.996+0000 7f84ad725640 1 librados: shutdown 2024-01-20T11:23:01.006 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-01-20T11:23:00.805479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002904"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280611,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":3.5430000000000001},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":8.2750000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":14.324999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":6.5730000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.385999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.6449999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.176},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":11.445}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":15.101000000000001},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":10.776999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":12.615},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":9.3200000000000003}]}]},{"osd":7,"up_from":21,"seq":90194313313,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.444},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.3119999999999994}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":6.2320000000000002},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.7010000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.981999999999999},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":10.584}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.3579999999999997},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":15.127000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":13.869999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.175000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":14.295},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":12.335000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313314,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.6970000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":7.7489999999999997},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.6920000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.308999999999999},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.446999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.923},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":13.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.3079999999999998},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.039}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":15.102},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":11.798}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444128,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":6.952},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":6.4379999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":5.3600000000000003},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.5060000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.425000000000001},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":13.378}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.542999999999999},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":13.923}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":16.405999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":8.5549999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.382},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.128}]}]},{"osd":1,"up_from":17,"seq":73014444129,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":10.532},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":5.5839999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":13.316000000000001},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":6.1909999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":14.116},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":7.593}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.1609999999999996},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.897}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.2279999999999998},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.635}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":9.8569999999999993},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":12.233000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444129,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":9.9299999999999997},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":5.0730000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.1529999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":16.361999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":18.335999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":10.772}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":11.654999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":8.0079999999999991}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":17.52},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.638}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":13.539999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":8.6660000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":12.42},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.7910000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.9169999999999998},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":13.224}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":8.5830000000000002},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":11.071}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":13.847},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":9.7059999999999995}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":16.065000000000001},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":15.262}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":11.763999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":14.499000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:04.009 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.111+0000 7f92b7fd0640 10 monclient: get_monmap_and_config 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.111+0000 7f92b7fd0640 10 monclient: build_initial_monmap 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.111+0000 7f92b7fd0640 10 monclient: monmap: 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:04.111766+0000 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:04.111766+0000 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:04.111 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:04.112 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.111+0000 7f92b7fd0640 1 Processor -- start 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.111+0000 7f92b7fd0640 1 -- start start 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.111+0000 7f92b7fd0640 10 monclient: init 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92b00a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92b00a9ad0 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92b0142af0 0x7f92b014d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f92b0142af0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b00b25c0 con 0x7f92b00a9ad0 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b0142f50 con 0x7f92b00a96f0 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b00aaea0 con 0x7f92b0142af0 2024-01-20T11:23:04.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:04.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.112+0000 7f92b7fd0640 10 monclient(hunting): authenticate will time out at 1827.462771s 2024-01-20T11:23:04.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.113+0000 7f92b656d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92b0142af0 0x7f92b014d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:04.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.113+0000 7f92b5d6c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f92b00a96f0 0x7f92b013f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:57410/0 (socket says 172.21.15.110:57410) 2024-01-20T11:23:04.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.113+0000 7f92b5d6c640 1 -- 172.21.15.110:0/3257243818 learned_addr learned my addr 172.21.15.110:0/3257243818 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:04.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.113+0000 7f92b656d640 10 monclient(hunting): get_auth_request con 0x7f92b0142af0 auth_method 0 2024-01-20T11:23:04.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.113+0000 7f92b656d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:04.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.113+0000 7f92b656d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:04.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.113+0000 7f92b656d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:04.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.114+0000 7f92b656d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:04.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.114+0000 7f92b656d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:04.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.114+0000 7f92b656d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:04.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.115+0000 7f92b656d640 10 monclient(hunting): handle_auth_done global_id 5949 payload 274 2024-01-20T11:23:04.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.115+0000 7f92b656d640 1 -- 172.21.15.110:0/3257243818 >> v1:172.21.15.134:6789/0 conn(0x7f92b00a96f0 legacy=0x7f92b013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.115+0000 7f92b656d640 1 -- 172.21.15.110:0/3257243818 >> v1:172.21.15.110:6789/0 conn(0x7f92b00a9ad0 legacy=0x7f92b01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:04.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.115+0000 7f92b656d640 10 monclient: _finish_hunting 0 2024-01-20T11:23:04.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.115+0000 7f92b656d640 1 monclient: found mon.noname-a 2024-01-20T11:23:04.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.115+0000 7f92b656d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:04.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.115+0000 7f92b656d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:04.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.115+0000 7f92b656d640 1 -- 172.21.15.110:0/3257243818 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b013fbe0 con 0x7f92b0142af0 2024-01-20T11:23:04.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.117+0000 7f92b656d640 1 --2- 172.21.15.110:0/3257243818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b0142af0 0x7f92b014d860 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009d90 tx=0x7f92a802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53dd7a021375a733 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:04.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 1 -- 172.21.15.110:0/3257243818 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a802cbd0 con 0x7f92b0142af0 2024-01-20T11:23:04.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:04.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 10 monclient: dump: 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 10 monclient: _finish_auth 0 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 10 monclient: _check_auth_tickets 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:04.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 1 -- 172.21.15.110:0/3257243818 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a802cd30 con 0x7f92b0142af0 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 1 -- 172.21.15.110:0/3257243818 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a802cea0 con 0x7f92b0142af0 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b4d6a640 10 monclient: dump: 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:04.123 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b7fd0640 5 monclient: authenticate success, global_id 5949 2024-01-20T11:23:04.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b7fd0640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:04.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.121+0000 7f92b7fd0640 4 set_mon_vals no callback set 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.122+0000 7f92b7fd0640 10 monclient: shutdown 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.122+0000 7f92b7fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.122+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3257243818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b0142af0 msgr2=0x7f92b014d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.122+0000 7f92b7fd0640 1 --2- 172.21.15.110:0/3257243818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b0142af0 0x7f92b014d860 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009d90 tx=0x7f92a802be20 comp rx=0 tx=0).stop 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.122+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3257243818 shutdown_connections 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.122+0000 7f92b7fd0640 1 --2- 172.21.15.110:0/3257243818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b0142af0 0x7f92b014d860 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.122+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3257243818 >> 172.21.15.110:0/3257243818 conn(0x7f92b00b7bc0 msgr2=0x7f92b00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.122+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3257243818 shutdown_connections 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.122+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3257243818 wait complete. 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.123+0000 7f92b7fd0640 10 monclient: build_initial_monmap 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.123+0000 7f92b7fd0640 10 monclient: monmap: 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:04.128 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.123+0000 7f92b7fd0640 1 librados: starting msgr at 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.123+0000 7f92b7fd0640 1 librados: starting objecter 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.123+0000 7f92b7fd0640 1 Processor -- start 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.123+0000 7f92b7fd0640 1 -- start start 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.123+0000 7f92b7fd0640 1 librados: setting wanted keys 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.123+0000 7f92b7fd0640 1 librados: calling monclient init 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.123+0000 7f92b7fd0640 10 monclient: init 2024-01-20T11:23:04.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92b00a96f0 0x7f92b01323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f92b00a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b00a9ad0 0x7f92b012ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f92b00a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92b0142af0 0x7f92b013e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f92b0142af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b0141ef0 con 0x7f92b00a9ad0 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b0131700 con 0x7f92b00a96f0 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b01304d0 con 0x7f92b0142af0 2024-01-20T11:23:04.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b7fd0640 10 monclient(hunting): authenticate will time out at 1827.474439s 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b556b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b00a9ad0 0x7f92b012ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b656d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92b0142af0 0x7f92b013e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b556b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b00a9ad0 0x7f92b012ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53884/0 (socket says 172.21.15.110:53884) 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b556b640 1 -- 172.21.15.110:0/3067019059 learned_addr learned my addr 172.21.15.110:0/3067019059 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b556b640 10 monclient(hunting): get_auth_request con 0x7f92b00a9ad0 auth_method 0 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b556b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b556b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.124+0000 7f92b556b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b5d6c640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92b00a96f0 0x7f92b01323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b656d640 10 monclient(hunting): get_auth_request con 0x7f92b0142af0 auth_method 0 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b656d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b656d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b656d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b5d6c640 10 monclient(hunting): get_auth_request con 0x7f92b00a96f0 auth_method 0 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b5d6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:04.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b5d6c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b5d6c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b556b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b556b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b556b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b656d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b656d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b656d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b5d6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b5d6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.125+0000 7f92b5d6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 10 monclient(hunting): handle_auth_done global_id 5952 payload 824 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 1 -- 172.21.15.110:0/3067019059 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92b0142af0 msgr2=0x7f92b013e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92b0142af0 0x7f92b013e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 1 -- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92b00a96f0 msgr2=0x7f92b01323c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92b00a96f0 0x7f92b01323c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 10 monclient: _finish_hunting 0 2024-01-20T11:23:04.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 1 monclient: found mon.noname-a 2024-01-20T11:23:04.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:04.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:04.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.128+0000 7f92b556b640 1 -- 172.21.15.110:0/3067019059 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a8009a70 con 0x7f92b00a9ad0 2024-01-20T11:23:04.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.130+0000 7f92b556b640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b00a9ad0 0x7f92b012ff90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f92a000c430 tx=0x7f92a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:04.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.130+0000 7f92a67fc640 10 client.?.objecter ms_handle_connect 0x7f92b00a9ad0 2024-01-20T11:23:04.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.130+0000 7f92a67fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:04.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 1 -- 172.21.15.110:0/3067019059 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a00035f0 con 0x7f92b00a9ad0 2024-01-20T11:23:04.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:04.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:04.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 10 monclient: dump: 2024-01-20T11:23:04.134 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 1 -- 172.21.15.110:0/3067019059 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a0003750 con 0x7f92b00a9ad0 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 1 -- 172.21.15.110:0/3067019059 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a0003b00 con 0x7f92b00a9ad0 2024-01-20T11:23:04.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a67fc640 10 monclient: dump: 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92b7fd0640 5 monclient: authenticate success, global_id 5952 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92b7fd0640 10 monclient: _renew_subs 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92a7fff640 4 set_mon_vals no callback set 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92b7fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:04.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b01319d0 con 0x7f92b00a9ad0 2024-01-20T11:23:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92b7fd0640 10 client.5952.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92b7fd0640 10 monclient: _renew_subs 2024-01-20T11:23:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92b7fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b01419d0 con 0x7f92b00a9ad0 2024-01-20T11:23:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.133+0000 7f92b7fd0640 1 librados: init done 2024-01-20T11:23:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.134+0000 7f92b7fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.134+0000 7f92b7fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.134+0000 7f92b7fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:04.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.134+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b0141bc0 con 0x7f92b00a9ad0 2024-01-20T11:23:04.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.152+0000 7f92a67fc640 1 -- 172.21.15.110:0/3067019059 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f92a0003c60 con 0x7f92b00a9ad0 2024-01-20T11:23:04.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.152+0000 7f92a67fc640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f929406c8b0 0x7f929406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.153+0000 7f92b5d6c640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f929406c8b0 0x7f929406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:04.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.153+0000 7f92b5d6c640 10 monclient: get_auth_request con 0x7f929406c8b0 auth_method 0 2024-01-20T11:23:04.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.153+0000 7f92b5d6c640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f929406c8b0 0x7f929406ed70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f92ac007fd0 tx=0x7f92ac007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:04.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.153+0000 7f92a67fc640 10 client.5952.objecter ms_handle_connect 0x7f929406c8b0 2024-01-20T11:23:04.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.154+0000 7f92a67fc640 1 -- 172.21.15.110:0/3067019059 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f92a00010e0 con 0x7f92b00a9ad0 2024-01-20T11:23:04.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.154+0000 7f92a67fc640 10 client.5952.objecter ms_dispatch 0x7f92b0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:04.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.154+0000 7f92a67fc640 3 client.5952.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:04.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.154+0000 7f92a67fc640 3 client.5952.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:04.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.155+0000 7f92a67fc640 20 client.5952.objecter dump_active .. 0 homeless 2024-01-20T11:23:04.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.400+0000 7f92a67fc640 1 -- 172.21.15.110:0/3067019059 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f92a0054380 con 0x7f92b00a9ad0 2024-01-20T11:23:04.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.400+0000 7f92a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:04.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.400+0000 7f92a67fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:04.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.577+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92b00b4400 con 0x7f929406c8b0 2024-01-20T11:23:04.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.580+0000 7f92a67fc640 1 -- 172.21.15.110:0/3067019059 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63839 (secure 0 0 0) 0x7f92ac001460 con 0x7f929406c8b0 2024-01-20T11:23:04.581 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 10 librados: watch_flush enter 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 10 librados: watch_flush exit 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 20 client.5952.objecter shutdown clearing up homeless session... 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 10 client.5952.objecter successfully canceled tick 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f929406c8b0 msgr2=0x7f929406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f929406c8b0 0x7f929406ed70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f92ac007fd0 tx=0x7f92ac007e80 comp rx=0 tx=0).stop 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 10 monclient: shutdown 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b00a9ad0 msgr2=0x7f92b012ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b00a9ad0 0x7f92b012ff90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f92a000c430 tx=0x7f92a000c8f0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 shutdown_connections 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f929406c8b0 0x7f929406ed70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92b0142af0 0x7f92b013e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92b00a9ad0 0x7f92b012ff90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 --2- 172.21.15.110:0/3067019059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92b00a96f0 0x7f92b01323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 >> 172.21.15.110:0/3067019059 conn(0x7f92b00b7bc0 msgr2=0x7f92b00b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 shutdown_connections 2024-01-20T11:23:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 -- 172.21.15.110:0/3067019059 wait complete. 2024-01-20T11:23:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.582+0000 7f92b7fd0640 1 librados: shutdown 2024-01-20T11:23:04.593 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-01-20T11:23:02.805884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002770"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280611,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":3.5430000000000001},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":8.2750000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":14.324999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":6.5730000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.385999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.6449999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.176},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":11.445}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":15.101000000000001},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":10.776999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":12.615},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":9.3200000000000003}]}]},{"osd":7,"up_from":21,"seq":90194313314,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.2590000000000003},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":9.1050000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":7.4870000000000001},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.135}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":9.8919999999999995},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.803000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":6.5899999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":11.929}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":15.202},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.276999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":14.48},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.875}]}]},{"osd":3,"up_from":21,"seq":90194313315,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.6970000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":7.7489999999999997},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.6920000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.308999999999999},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.446999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.923},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":13.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.3079999999999998},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.039}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":15.102},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":11.798}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444128,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":6.952},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":6.4379999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":5.3600000000000003},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.5060000000000002}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.425000000000001},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":13.378}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.542999999999999},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":13.923}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":16.405999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":8.5549999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.382},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.128}]}]},{"osd":1,"up_from":17,"seq":73014444129,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":10.532},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":5.5839999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":13.316000000000001},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":6.1909999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":14.116},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":7.593}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.1609999999999996},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.897}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.2279999999999998},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.635}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":9.8569999999999993},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":12.233000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444129,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":9.9299999999999997},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":5.0730000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.1529999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":16.361999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":18.335999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":10.772}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":11.654999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":8.0079999999999991}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":17.52},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.638}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":13.539999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":8.6660000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509538,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":4.1059999999999999},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.8700000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":15.849},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":7.516}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":9.0749999999999993},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":10.49}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.7010000000000005},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.183999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":12.566000000000001},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":15.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":13.847},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.162000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:04.594 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:04.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.695+0000 7f4266e7c640 10 monclient: get_monmap_and_config 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.695+0000 7f4266e7c640 10 monclient: build_initial_monmap 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.695+0000 7f4266e7c640 10 monclient: monmap: 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:04.696379+0000 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:04.696379+0000 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:04.696 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:04.697 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.696+0000 7f4266e7c640 1 Processor -- start 2024-01-20T11:23:04.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.696+0000 7f4266e7c640 1 -- start start 2024-01-20T11:23:04.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.696+0000 7f4266e7c640 10 monclient: init 2024-01-20T11:23:04.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42600a96f0 0x7f426013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 10 monclient(hunting): picked mon.noname-b con 0x7f42600a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f426013fa10 0x7f42601423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 10 monclient(hunting): picked mon.noname-a con 0x7f426013fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42600a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42600b25c0 con 0x7f42600a9ad0 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42600aaea0 con 0x7f426013fa10 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42600ab0a0 con 0x7f42600a96f0 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4266e7c640 10 monclient(hunting): authenticate will time out at 1828.047309s 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4264c18640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42600a96f0 0x7f426013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4265419640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f42600a9ad0 0x7f4260142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:57424/0 (socket says 172.21.15.110:57424) 2024-01-20T11:23:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.697+0000 7f4265419640 1 -- 172.21.15.110:0/289762243 learned_addr learned my addr 172.21.15.110:0/289762243 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:04.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.698+0000 7f4264c18640 10 monclient(hunting): get_auth_request con 0x7f42600a96f0 auth_method 0 2024-01-20T11:23:04.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.698+0000 7f4264c18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:04.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.698+0000 7f4264c18640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:04.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.698+0000 7f4264c18640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:04.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.698+0000 7f4264c18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:04.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.698+0000 7f4264c18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:04.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.698+0000 7f4264c18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.700+0000 7f4264c18640 10 monclient(hunting): handle_auth_done global_id 5597 payload 274 2024-01-20T11:23:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.700+0000 7f4264c18640 1 -- 172.21.15.110:0/289762243 >> v2:172.21.15.110:3300/0 conn(0x7f426013fa10 msgr2=0x7f42601423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.700+0000 7f4264c18640 1 --2- 172.21.15.110:0/289762243 >> v2:172.21.15.110:3300/0 conn(0x7f426013fa10 0x7f42601423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.700+0000 7f4264c18640 1 -- 172.21.15.110:0/289762243 >> v1:172.21.15.134:6789/0 conn(0x7f42600a9ad0 legacy=0x7f4260142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.700+0000 7f4264c18640 10 monclient: _finish_hunting 0 2024-01-20T11:23:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.700+0000 7f4264c18640 1 monclient: found mon.noname-b 2024-01-20T11:23:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.700+0000 7f4264c18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.700+0000 7f4264c18640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.700+0000 7f4264c18640 1 -- 172.21.15.110:0/289762243 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42600b3fb0 con 0x7f42600a96f0 2024-01-20T11:23:04.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.701+0000 7f4264c18640 1 --2- 172.21.15.110:0/289762243 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42600a96f0 0x7f426013f4d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f42540097a0 tx=0x7f425402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57883a1c031f872 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:04.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 1 -- 172.21.15.110:0/289762243 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425402e070 con 0x7f42600a96f0 2024-01-20T11:23:04.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:04.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 10 monclient: dump: 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 1 -- 172.21.15.110:0/289762243 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42540315a0 con 0x7f42600a96f0 2024-01-20T11:23:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 1 -- 172.21.15.110:0/289762243 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254031930 con 0x7f42600a96f0 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f425f7fe640 10 monclient: dump: 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f4266e7c640 5 monclient: authenticate success, global_id 5597 2024-01-20T11:23:04.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f4266e7c640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:04.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.705+0000 7f4266e7c640 4 set_mon_vals no callback set 2024-01-20T11:23:04.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 10 monclient: shutdown 2024-01-20T11:23:04.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:04.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 1 -- 172.21.15.110:0/289762243 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42600a96f0 msgr2=0x7f426013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 1 --2- 172.21.15.110:0/289762243 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42600a96f0 0x7f426013f4d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f42540097a0 tx=0x7f425402b840 comp rx=0 tx=0).stop 2024-01-20T11:23:04.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 1 -- 172.21.15.110:0/289762243 shutdown_connections 2024-01-20T11:23:04.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 1 --2- 172.21.15.110:0/289762243 >> v2:172.21.15.110:3300/0 conn(0x7f426013fa10 0x7f42601423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 1 --2- 172.21.15.110:0/289762243 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42600a96f0 0x7f426013f4d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 1 -- 172.21.15.110:0/289762243 >> 172.21.15.110:0/289762243 conn(0x7f42600b7bc0 msgr2=0x7f42600b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 1 -- 172.21.15.110:0/289762243 shutdown_connections 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.706+0000 7f4266e7c640 1 -- 172.21.15.110:0/289762243 wait complete. 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.707+0000 7f4266e7c640 10 monclient: build_initial_monmap 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.707+0000 7f4266e7c640 10 monclient: monmap: 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.707+0000 7f4266e7c640 1 librados: starting msgr at 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.707+0000 7f4266e7c640 1 librados: starting objecter 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.707+0000 7f4266e7c640 1 Processor -- start 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 1 -- start start 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 1 librados: setting wanted keys 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 1 librados: calling monclient init 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient: init 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42600a96f0 0x7f4260138180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient(hunting): picked mon.noname-b con 0x7f42600a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42600a9ad0 0x7f42601d7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient(hunting): picked mon.noname-a con 0x7f42600a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4260145860 0x7f42601d7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient(hunting): picked mon.noname-c con 0x7f4260145860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42601392e0 con 0x7f42600a9ad0 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42601d73b0 con 0x7f42600a96f0 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4260139530 con 0x7f4260145860 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.708+0000 7f4266e7c640 10 monclient(hunting): authenticate will time out at 1828.058963s 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4265419640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4260145860 0x7f42601d7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4264c18640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42600a96f0 0x7f4260138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4264c18640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42600a96f0 0x7f4260138180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:34066/0 (socket says 172.21.15.110:34066) 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4264c18640 1 -- 172.21.15.110:0/341667491 learned_addr learned my addr 172.21.15.110:0/341667491 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4264c18640 10 monclient(hunting): get_auth_request con 0x7f42600a96f0 auth_method 0 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4264c18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4264c18640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4264c18640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4265419640 10 monclient(hunting): get_auth_request con 0x7f4260145860 auth_method 0 2024-01-20T11:23:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4265419640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4265419640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.709+0000 7f4265419640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.710+0000 7f4265419640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.710+0000 7f4265419640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.710+0000 7f4265419640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.710+0000 7f4264c18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.710+0000 7f4264c18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.710+0000 7f4264c18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.712+0000 7f4265419640 10 monclient(hunting): handle_auth_done global_id 5600 payload 824 2024-01-20T11:23:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.712+0000 7f4265419640 1 -- 172.21.15.110:0/341667491 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42600a96f0 msgr2=0x7f4260138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.712+0000 7f4265419640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42600a96f0 0x7f4260138180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.712+0000 7f4265419640 1 -- 172.21.15.110:0/341667491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42600a9ad0 msgr2=0x7f42601d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.712+0000 7f4265419640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42600a9ad0 0x7f42601d7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:04.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.713+0000 7f4265419640 10 monclient: _finish_hunting 0 2024-01-20T11:23:04.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.713+0000 7f4265419640 1 monclient: found mon.noname-c 2024-01-20T11:23:04.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.713+0000 7f4265419640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:04.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.713+0000 7f4265419640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:23:04.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.713+0000 7f4265419640 1 -- 172.21.15.110:0/341667491 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4254009400 con 0x7f4260145860 2024-01-20T11:23:04.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.715+0000 7f4265419640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4260145860 0x7f42601d7dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f425800caa0 tx=0x7f425800cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:04.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.715+0000 7f425d7fa640 10 client.?.objecter ms_handle_connect 0x7f4260145860 2024-01-20T11:23:04.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.715+0000 7f425d7fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:04.717 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:55.739 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:04.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f425d7fa640 1 -- 172.21.15.110:0/341667491 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425800ac00 con 0x7f4260145860 2024-01-20T11:23:04.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f425d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:04.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f425d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:04.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f425d7fa640 10 monclient: dump: 2024-01-20T11:23:04.719 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:04.719 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.719 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.719 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f425d7fa640 10 monclient: _finish_auth 0 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f425d7fa640 10 monclient: _check_auth_tickets 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f425d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f4266e7c640 5 monclient: authenticate success, global_id 5600 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f4266e7c640 10 monclient: _renew_subs 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f4266e7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260146300 con 0x7f4260145860 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f4266e7c640 10 client.5600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f4266e7c640 10 monclient: _renew_subs 2024-01-20T11:23:04.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f4266e7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:04.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42601434e0 con 0x7f4260145860 2024-01-20T11:23:04.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.719+0000 7f4266e7c640 1 librados: init done 2024-01-20T11:23:04.721 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:55.743 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:04.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.720+0000 7f4266e7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:04.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.720+0000 7f4266e7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:04.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.720+0000 7f4266e7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:04.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.720+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42601d7580 con 0x7f4260145860 2024-01-20T11:23:04.732 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:55.757 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:04.766 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:55.773 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:04.767 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:55.774 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:04.773 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:55.798 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:04.777 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:08:55.784 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:04.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.778+0000 7f425d7fa640 1 -- 172.21.15.110:0/341667491 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4258001b20 con 0x7f4260145860 2024-01-20T11:23:04.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.778+0000 7f425d7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:04.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.778+0000 7f425d7fa640 1 -- 172.21.15.110:0/341667491 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425801c440 con 0x7f4260145860 2024-01-20T11:23:04.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.778+0000 7f425d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:04.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.778+0000 7f425d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.778+0000 7f425d7fa640 10 monclient: dump: 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:04.778+0000 7f425effd640 4 set_mon_vals no callback set 2024-01-20T11:23:04.780 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:55.804 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:05.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.029+0000 7f425d7fa640 1 -- 172.21.15.110:0/341667491 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4258012400 con 0x7f4260145860 2024-01-20T11:23:05.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.029+0000 7f425d7fa640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f424406c890 0x7f424406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.030+0000 7f4264c18640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f424406c890 0x7f424406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.030+0000 7f4264c18640 10 monclient: get_auth_request con 0x7f424406c890 auth_method 0 2024-01-20T11:23:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.030+0000 7f4264c18640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f424406c890 0x7f424406ed50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4254000c00 tx=0x7f425403d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:05.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.030+0000 7f425d7fa640 10 client.5600.objecter ms_handle_connect 0x7f424406c890 2024-01-20T11:23:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.046+0000 7f425d7fa640 1 -- 172.21.15.110:0/341667491 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f42580588b0 con 0x7f4260145860 2024-01-20T11:23:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.046+0000 7f425d7fa640 10 client.5600.objecter ms_dispatch 0x7f4260000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.046+0000 7f425d7fa640 3 client.5600.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.046+0000 7f425d7fa640 3 client.5600.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.046+0000 7f425d7fa640 20 client.5600.objecter dump_active .. 0 homeless 2024-01-20T11:23:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.046+0000 7f425d7fa640 1 -- 172.21.15.110:0/341667491 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4258054bf0 con 0x7f4260145860 2024-01-20T11:23:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.046+0000 7f425d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:05.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.046+0000 7f425d7fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:05.051 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:56.075 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:05.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.235+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4260075490 con 0x7f424406c890 2024-01-20T11:23:05.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.238+0000 7f425d7fa640 1 -- 172.21.15.110:0/341667491 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63874 (secure 0 0 0) 0x7f4260075490 con 0x7f424406c890 2024-01-20T11:23:05.239 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:05.239 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:05.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 10 librados: watch_flush enter 2024-01-20T11:23:05.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 10 librados: watch_flush exit 2024-01-20T11:23:05.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 20 client.5600.objecter shutdown clearing up homeless session... 2024-01-20T11:23:05.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 10 client.5600.objecter successfully canceled tick 2024-01-20T11:23:05.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f424406c890 msgr2=0x7f424406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:05.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f424406c890 0x7f424406ed50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4254000c00 tx=0x7f425403d040 comp rx=0 tx=0).stop 2024-01-20T11:23:05.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 10 monclient: shutdown 2024-01-20T11:23:05.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:05.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4260145860 msgr2=0x7f42601d7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4260145860 0x7f42601d7dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f425800caa0 tx=0x7f425800cf60 comp rx=0 tx=0).stop 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 shutdown_connections 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f424406c890 0x7f424406ed50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4260145860 0x7f42601d7dd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42600a9ad0 0x7f42601d7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.240+0000 7f4266e7c640 1 --2- 172.21.15.110:0/341667491 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42600a96f0 0x7f4260138180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.241+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 >> 172.21.15.110:0/341667491 conn(0x7f42600b7bc0 msgr2=0x7f42600b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.241+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 shutdown_connections 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.241+0000 7f4266e7c640 1 -- 172.21.15.110:0/341667491 wait complete. 2024-01-20T11:23:05.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.241+0000 7f4266e7c640 1 librados: shutdown 2024-01-20T11:23:05.254 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-01-20T11:23:04.806329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002794"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":6.9409999999999998},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.5110000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":11.246},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":6.125}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":9.2810000000000006},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.952999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.388},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":13.779}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":8.0920000000000005},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":13.037000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":14.711},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":8.7430000000000003}]}]},{"osd":7,"up_from":21,"seq":90194313314,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.2590000000000003},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":9.1050000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":7.4870000000000001},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.135}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":9.8919999999999995},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.803000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":6.5899999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":11.929}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":15.202},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.276999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":14.48},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.875}]}]},{"osd":3,"up_from":21,"seq":90194313315,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.6970000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":7.7489999999999997},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.6920000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.308999999999999},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.446999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.923},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":13.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.3079999999999998},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.039}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":15.102},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":11.798}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444129,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":8.4269999999999996},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":6.3049999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":12.941000000000001},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.151999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.210000000000001},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.1600000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.765000000000001},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":9.8629999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":11.423},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.9669999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":14.131},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.404999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444130,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.9699999999999998},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":6.2089999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.7169999999999996},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":8.7789999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":13.319000000000001},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":9.4199999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.978999999999999},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":11.361000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":14.06},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":10.670999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":15.699999999999999},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":12.622999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444130,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.79},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":7.1980000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.4030000000000005},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":9.9109999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":7.8280000000000003},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":8.9009999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":11.832000000000001},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":14.156000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":10.541},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.366}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":12.948},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":14.727}]}]},{"osd":6,"up_from":15,"seq":64424509538,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":4.1059999999999999},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.8700000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":15.849},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":7.516}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":9.0749999999999993},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":10.49}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.7010000000000005},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.183999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":12.566000000000001},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":15.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":13.847},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.162000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:05.256 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:05.256 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:18:55.786781+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:18:55.786781+0000', 'last_peered': '2024-01-20T11:18:55.786781+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:18:55.786781+0000', 'last_undegraded': '2024-01-20T11:18:55.786781+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:05.256 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:05.256 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:05.256 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:05.256 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:05.256 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:05.256 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:05.257 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.356+0000 7fd257c07640 10 monclient: get_monmap_and_config 2024-01-20T11:23:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.356+0000 7fd257c07640 10 monclient: build_initial_monmap 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.356+0000 7fd257c07640 10 monclient: monmap: 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:05.357353+0000 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:05.357353+0000 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:05.357 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:05.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.357+0000 7fd257c07640 1 Processor -- start 2024-01-20T11:23:05.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.357+0000 7fd257c07640 1 -- start start 2024-01-20T11:23:05.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.357+0000 7fd257c07640 10 monclient: init 2024-01-20T11:23:05.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:05.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd250071c60 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2500ac540 0x7fd2500ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 10 monclient(hunting): picked mon.noname-a con 0x7fd2500ac540 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2500b0930 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2500ad610 con 0x7fd250071c60 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2500aba00 con 0x7fd2500b0930 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd250072040 con 0x7fd2500ac540 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd257c07640 10 monclient(hunting): authenticate will time out at 1828.708323s 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd2551a2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2500ac540 0x7fd2500ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd2561a4640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd2500b0930 0x7fd250149510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:57438/0 (socket says 172.21.15.110:57438) 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.358+0000 7fd2561a4640 1 -- 172.21.15.110:0/1915377814 learned_addr learned my addr 172.21.15.110:0/1915377814 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.359+0000 7fd2551a2640 10 monclient(hunting): get_auth_request con 0x7fd2500ac540 auth_method 0 2024-01-20T11:23:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.359+0000 7fd2551a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.359+0000 7fd2551a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.359+0000 7fd2551a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.359+0000 7fd2551a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.359+0000 7fd2551a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.359+0000 7fd2551a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.360+0000 7fd2549a1640 1 -- 172.21.15.110:0/1915377814 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875805522 0 0) 0x7fd2500aba00 con 0x7fd2500b0930 2024-01-20T11:23:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.360+0000 7fd2549a1640 10 monclient(hunting): _negotiate 2024-01-20T11:23:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.360+0000 7fd2549a1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.360+0000 7fd2549a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.360+0000 7fd2549a1640 10 monclient(hunting): my global_id is 6055 2024-01-20T11:23:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.360+0000 7fd2549a1640 1 -- 172.21.15.110:0/1915377814 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd23c005730 con 0x7fd2500b0930 2024-01-20T11:23:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.361+0000 7fd2551a2640 10 monclient(hunting): handle_auth_done global_id 5955 payload 274 2024-01-20T11:23:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.361+0000 7fd2551a2640 1 -- 172.21.15.110:0/1915377814 >> v1:172.21.15.134:6789/0 conn(0x7fd2500b0930 legacy=0x7fd250149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:05.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.361+0000 7fd2551a2640 1 -- 172.21.15.110:0/1915377814 >> v1:172.21.15.110:6789/0 conn(0x7fd250071c60 legacy=0x7fd250142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:05.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.361+0000 7fd2551a2640 10 monclient: _finish_hunting 0 2024-01-20T11:23:05.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.361+0000 7fd2551a2640 1 monclient: found mon.noname-a 2024-01-20T11:23:05.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.361+0000 7fd2551a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:05.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.361+0000 7fd2551a2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:05.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.361+0000 7fd2551a2640 1 -- 172.21.15.110:0/1915377814 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2500721b0 con 0x7fd2500ac540 2024-01-20T11:23:05.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.363+0000 7fd2551a2640 1 --2- 172.21.15.110:0/1915377814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 0x7fd2500ab4c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd2400097a0 tx=0x7fd24002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f7f52648230af5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 1 -- 172.21.15.110:0/1915377814 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240031830 con 0x7fd2500ac540 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 10 monclient: dump: 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 10 monclient: _finish_auth 0 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 10 monclient: _check_auth_tickets 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 1 -- 172.21.15.110:0/1915377814 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240031990 con 0x7fd2500ac540 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 1 -- 172.21.15.110:0/1915377814 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240031d20 con 0x7fd2500ac540 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd2549a1640 10 monclient: dump: 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd257c07640 5 monclient: authenticate success, global_id 5955 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd257c07640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.367+0000 7fd257c07640 4 set_mon_vals no callback set 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 10 monclient: shutdown 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 1 -- 172.21.15.110:0/1915377814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 msgr2=0x7fd2500ab4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 1 --2- 172.21.15.110:0/1915377814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 0x7fd2500ab4c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd2400097a0 tx=0x7fd24002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 1 -- 172.21.15.110:0/1915377814 shutdown_connections 2024-01-20T11:23:05.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 1 --2- 172.21.15.110:0/1915377814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 0x7fd2500ab4c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 1 -- 172.21.15.110:0/1915377814 >> 172.21.15.110:0/1915377814 conn(0x7fd2500b3e60 msgr2=0x7fd2500b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:05.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 1 -- 172.21.15.110:0/1915377814 shutdown_connections 2024-01-20T11:23:05.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 1 -- 172.21.15.110:0/1915377814 wait complete. 2024-01-20T11:23:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 10 monclient: build_initial_monmap 2024-01-20T11:23:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.368+0000 7fd257c07640 10 monclient: monmap: 2024-01-20T11:23:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 librados: starting msgr at 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 librados: starting objecter 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 Processor -- start 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 -- start start 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 librados: setting wanted keys 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 librados: calling monclient init 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient: init 2024-01-20T11:23:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd250071c60 0x7fd25013e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient(hunting): picked mon.noname-b con 0x7fd250071c60 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 0x7fd2501c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient(hunting): picked mon.noname-a con 0x7fd2500ac540 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2500b0930 0x7fd25013f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient(hunting): picked mon.noname-c con 0x7fd2500b0930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25006d9c0 con 0x7fd2500ac540 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25013e7c0 con 0x7fd250071c60 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25006dc10 con 0x7fd2500b0930 2024-01-20T11:23:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.369+0000 7fd257c07640 10 monclient(hunting): authenticate will time out at 1828.720067s 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2551a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 0x7fd2501c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2561a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2500b0930 0x7fd25013f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2559a3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd250071c60 0x7fd25013e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2551a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 0x7fd2501c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53940/0 (socket says 172.21.15.110:53940) 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2551a2640 1 -- 172.21.15.110:0/3639824228 learned_addr learned my addr 172.21.15.110:0/3639824228 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2551a2640 10 monclient(hunting): get_auth_request con 0x7fd2500ac540 auth_method 0 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2551a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2551a2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2551a2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2561a4640 10 monclient(hunting): get_auth_request con 0x7fd2500b0930 auth_method 0 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2561a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2561a4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2561a4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2559a3640 10 monclient(hunting): get_auth_request con 0x7fd250071c60 auth_method 0 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2559a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2559a3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.370+0000 7fd2559a3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.371+0000 7fd2551a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.371+0000 7fd2551a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.371+0000 7fd2551a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.371+0000 7fd2559a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.371+0000 7fd2559a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.371+0000 7fd2559a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.371+0000 7fd2561a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.371+0000 7fd2561a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.371+0000 7fd2561a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 10 monclient(hunting): handle_auth_done global_id 6058 payload 824 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 1 -- 172.21.15.110:0/3639824228 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2500b0930 msgr2=0x7fd25013f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2500b0930 0x7fd25013f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 1 -- 172.21.15.110:0/3639824228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 msgr2=0x7fd2501c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 0x7fd2501c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 10 monclient: _finish_hunting 0 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 1 monclient: found mon.noname-b 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.373+0000 7fd2559a3640 1 -- 172.21.15.110:0/3639824228 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd240009400 con 0x7fd250071c60 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.376+0000 7fd2559a3640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd250071c60 0x7fd25013e280 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00c700 tx=0x7fd24c00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.376+0000 7fd2467fc640 10 client.?.objecter ms_handle_connect 0x7fd250071c60 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.376+0000 7fd2467fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 1 -- 172.21.15.110:0/3639824228 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c003d80 con 0x7fd250071c60 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 10 monclient: dump: 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 1 -- 172.21.15.110:0/3639824228 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24c00ce10 con 0x7fd250071c60 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 1 -- 172.21.15.110:0/3639824228 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c0052b0 con 0x7fd250071c60 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd2467fc640 10 monclient: dump: 2024-01-20T11:23:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd257c07640 5 monclient: authenticate success, global_id 6058 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd257c07640 10 monclient: _renew_subs 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd257c07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25013ead0 con 0x7fd250071c60 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd247fff640 4 set_mon_vals no callback set 2024-01-20T11:23:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd257c07640 10 client.6058.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd257c07640 10 monclient: _renew_subs 2024-01-20T11:23:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd257c07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25013f720 con 0x7fd250071c60 2024-01-20T11:23:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.379+0000 7fd257c07640 1 librados: init done 2024-01-20T11:23:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.380+0000 7fd257c07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.380+0000 7fd257c07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.380+0000 7fd257c07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.380+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25013ed20 con 0x7fd250071c60 2024-01-20T11:23:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.620+0000 7fd2467fc640 1 -- 172.21.15.110:0/3639824228 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd24c005410 con 0x7fd250071c60 2024-01-20T11:23:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.620+0000 7fd2467fc640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd23406c8b0 0x7fd23406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.620+0000 7fd2551a2640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd23406c8b0 0x7fd23406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.621+0000 7fd2551a2640 10 monclient: get_auth_request con 0x7fd23406c8b0 auth_method 0 2024-01-20T11:23:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.621+0000 7fd2467fc640 10 client.6058.objecter ms_handle_connect 0x7fd23406c8b0 2024-01-20T11:23:05.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.621+0000 7fd2551a2640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd23406c8b0 0x7fd23406ed70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd24002b910 tx=0x7fd24003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.625+0000 7fd2467fc640 1 -- 172.21.15.110:0/3639824228 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd24c057d40 con 0x7fd250071c60 2024-01-20T11:23:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.625+0000 7fd2467fc640 10 client.6058.objecter ms_dispatch 0x7fd250000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.625+0000 7fd2467fc640 3 client.6058.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.625+0000 7fd2467fc640 3 client.6058.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:05.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.625+0000 7fd2467fc640 20 client.6058.objecter dump_active .. 0 homeless 2024-01-20T11:23:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.635+0000 7fd2467fc640 1 -- 172.21.15.110:0/3639824228 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd24c054080 con 0x7fd250071c60 2024-01-20T11:23:05.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.635+0000 7fd2467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:05.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.635+0000 7fd2467fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:05.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.872+0000 7fd257c07640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:05.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.872+0000 7fd257c07640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:05.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.872+0000 7fd257c07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:05.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.872+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd25013ed20 con 0x7fd250071c60 2024-01-20T11:23:05.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.878+0000 7fd2467fc640 1 -- 172.21.15.110:0/3639824228 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fd25013ed20 con 0x7fd250071c60 2024-01-20T11:23:05.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.878+0000 7fd2467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:05.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.878+0000 7fd2467fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:05.879 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:05.879 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":520,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:05.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 10 librados: watch_flush enter 2024-01-20T11:23:05.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 10 librados: watch_flush exit 2024-01-20T11:23:05.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 20 client.6058.objecter shutdown clearing up homeless session... 2024-01-20T11:23:05.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 10 client.6058.objecter successfully canceled tick 2024-01-20T11:23:05.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd23406c8b0 msgr2=0x7fd23406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:05.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd23406c8b0 0x7fd23406ed70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd24002b910 tx=0x7fd24003c040 comp rx=0 tx=0).stop 2024-01-20T11:23:05.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 10 monclient: shutdown 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd250071c60 msgr2=0x7fd25013e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd250071c60 0x7fd25013e280 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00c700 tx=0x7fd24c00cbc0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 shutdown_connections 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd23406c8b0 0x7fd23406ed70 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2500b0930 0x7fd25013f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2500ac540 0x7fd2501c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 --2- 172.21.15.110:0/3639824228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd250071c60 0x7fd25013e280 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 >> 172.21.15.110:0/3639824228 conn(0x7fd2500b3e60 msgr2=0x7fd2500b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 shutdown_connections 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 -- 172.21.15.110:0/3639824228 wait complete. 2024-01-20T11:23:05.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.880+0000 7fd257c07640 1 librados: shutdown 2024-01-20T11:23:05.891 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:05.891 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.989+0000 7fdd1ffe9640 10 monclient: get_monmap_and_config 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.989+0000 7fdd1ffe9640 10 monclient: build_initial_monmap 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.989+0000 7fdd1ffe9640 10 monclient: monmap: 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:05.990551+0000 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:05.990551+0000 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:05.990 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.990+0000 7fdd1ffe9640 1 Processor -- start 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.990+0000 7fdd1ffe9640 1 -- start start 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.990+0000 7fdd1ffe9640 10 monclient: init 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd1806b380 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806e430 0x7fdd181435d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1806e430 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdd1813d600 0x7fdd18143b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1813d600 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd18071de0 con 0x7fdd1806b380 2024-01-20T11:23:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd1806b760 con 0x7fdd1813d600 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1806e0a0 con 0x7fdd1806e430 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1ffe9640 10 monclient(hunting): authenticate will time out at 1829.341560s 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1e586640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdd1813d600 0x7fdd18143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.991+0000 7fdd1d584640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806e430 0x7fdd181435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1dd85640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fdd1806b380 0x7fdd1813cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:57450/0 (socket says 172.21.15.110:57450) 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1dd85640 1 -- 172.21.15.110:0/2738445047 learned_addr learned my addr 172.21.15.110:0/2738445047 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1e586640 10 monclient(hunting): get_auth_request con 0x7fdd1813d600 auth_method 0 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1e586640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1e586640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1e586640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1d584640 10 monclient(hunting): get_auth_request con 0x7fdd1806e430 auth_method 0 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1d584640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1d584640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1d584640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1e586640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1e586640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1e586640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1d584640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1d584640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:05.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.992+0000 7fdd1d584640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:05.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1cd83640 1 -- 172.21.15.110:0/2738445047 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939039261 0 0) 0x7fdd18071de0 con 0x7fdd1806b380 2024-01-20T11:23:05.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1cd83640 10 monclient(hunting): _negotiate 2024-01-20T11:23:05.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1cd83640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:05.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1cd83640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1cd83640 10 monclient(hunting): my global_id is 6061 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1cd83640 1 -- 172.21.15.110:0/2738445047 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd04003520 con 0x7fdd1806b380 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1e586640 10 monclient(hunting): handle_auth_done global_id 5961 payload 274 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1e586640 1 -- 172.21.15.110:0/2738445047 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806e430 msgr2=0x7fdd181435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1e586640 1 --2- 172.21.15.110:0/2738445047 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806e430 0x7fdd181435d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1e586640 1 -- 172.21.15.110:0/2738445047 >> v1:172.21.15.134:6789/0 conn(0x7fdd1806b380 legacy=0x7fdd1813cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1e586640 10 monclient: _finish_hunting 0 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1e586640 1 monclient: found mon.noname-a 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1e586640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1e586640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.994+0000 7fdd1e586640 1 -- 172.21.15.110:0/2738445047 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd18070ef0 con 0x7fdd1813d600 2024-01-20T11:23:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:05.996+0000 7fdd1e586640 1 --2- 172.21.15.110:0/2738445047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd1813d600 0x7fdd18143b10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009dc0 tx=0x7fdd1002be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c985f830cebdeeb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 1 -- 172.21.15.110:0/2738445047 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1002cbb0 con 0x7fdd1813d600 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 10 monclient: dump: 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:06.000 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 10 monclient: _finish_auth 0 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 10 monclient: _check_auth_tickets 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 1 -- 172.21.15.110:0/2738445047 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1002cd10 con 0x7fdd1813d600 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 1 -- 172.21.15.110:0/2738445047 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10034660 con 0x7fdd1813d600 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 5 monclient: authenticate success, global_id 5961 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 4 set_mon_vals no callback set 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1cd83640 10 monclient: dump: 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:06.001 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:06.002 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 10 monclient: shutdown 2024-01-20T11:23:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/2738445047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd1813d600 msgr2=0x7fdd18143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 1 --2- 172.21.15.110:0/2738445047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd1813d600 0x7fdd18143b10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009dc0 tx=0x7fdd1002be00 comp rx=0 tx=0).stop 2024-01-20T11:23:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/2738445047 shutdown_connections 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 1 --2- 172.21.15.110:0/2738445047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd1813d600 0x7fdd18143b10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 1 --2- 172.21.15.110:0/2738445047 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806e430 0x7fdd181435d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/2738445047 >> 172.21.15.110:0/2738445047 conn(0x7fdd1806a5f0 msgr2=0x7fdd1806aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/2738445047 shutdown_connections 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.000+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/2738445047 wait complete. 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.001+0000 7fdd1ffe9640 10 monclient: build_initial_monmap 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.001+0000 7fdd1ffe9640 10 monclient: monmap: 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 librados: starting msgr at 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 librados: starting objecter 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 Processor -- start 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 -- start start 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 librados: setting wanted keys 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 librados: calling monclient init 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient: init 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806b380 0x7fdd18137e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1806b380 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd1806e430 0x7fdd1806e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:06.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1806e430 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd1813d600 0x7fdd1806f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1813d600 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1806ef50 con 0x7fdd1813d600 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd18070690 con 0x7fdd1806e430 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd18139250 con 0x7fdd1806b380 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.002+0000 7fdd1ffe9640 10 monclient(hunting): authenticate will time out at 1829.353040s 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1d584640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd1806e430 0x7fdd1806e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1e586640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd1813d600 0x7fdd1806f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1dd85640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806b380 0x7fdd18137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1d584640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd1806e430 0x7fdd1806e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:34088/0 (socket says 172.21.15.110:34088) 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1d584640 1 -- 172.21.15.110:0/1541352261 learned_addr learned my addr 172.21.15.110:0/1541352261 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1d584640 10 monclient(hunting): get_auth_request con 0x7fdd1806e430 auth_method 0 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1d584640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1d584640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1d584640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1e586640 10 monclient(hunting): get_auth_request con 0x7fdd1813d600 auth_method 0 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1e586640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1e586640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1e586640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1dd85640 10 monclient(hunting): get_auth_request con 0x7fdd1806b380 auth_method 0 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1dd85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1dd85640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.003+0000 7fdd1dd85640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.004+0000 7fdd1d584640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.004+0000 7fdd1d584640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.004+0000 7fdd1d584640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.004+0000 7fdd1e586640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.004+0000 7fdd1e586640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.004+0000 7fdd1e586640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.004+0000 7fdd1dd85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.004+0000 7fdd1dd85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.004+0000 7fdd1dd85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 10 monclient(hunting): handle_auth_done global_id 6064 payload 824 2024-01-20T11:23:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 1 -- 172.21.15.110:0/1541352261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806b380 msgr2=0x7fdd18137e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806b380 0x7fdd18137e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 1 -- 172.21.15.110:0/1541352261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd1813d600 msgr2=0x7fdd1806f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd1813d600 0x7fdd1806f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 10 monclient: _finish_hunting 0 2024-01-20T11:23:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 1 monclient: found mon.noname-b 2024-01-20T11:23:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.007+0000 7fdd1d584640 1 -- 172.21.15.110:0/1541352261 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd10009a70 con 0x7fdd1806e430 2024-01-20T11:23:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.009+0000 7fdd1d584640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd1806e430 0x7fdd1806e8a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800e980 tx=0x7fdd0800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.009+0000 7fdd0e7fc640 10 client.?.objecter ms_handle_connect 0x7fdd1806e430 2024-01-20T11:23:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.009+0000 7fdd0e7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:06.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 1 -- 172.21.15.110:0/1541352261 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08009670 con 0x7fdd1806e430 2024-01-20T11:23:06.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:06.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:06.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 10 monclient: dump: 2024-01-20T11:23:06.013 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:06.013 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:06.013 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 1 -- 172.21.15.110:0/1541352261 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0800ca00 con 0x7fdd1806e430 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 1 -- 172.21.15.110:0/1541352261 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0800cd90 con 0x7fdd1806e430 2024-01-20T11:23:06.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd1ffe9640 5 monclient: authenticate success, global_id 6064 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd1ffe9640 10 monclient: _renew_subs 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd1ffe9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd18139530 con 0x7fdd1806e430 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0ffff640 4 set_mon_vals no callback set 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd0e7fc640 10 monclient: dump: 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:06.015 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:06.016 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:06.016 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:06.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd1ffe9640 10 client.6064.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:06.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd1ffe9640 10 monclient: _renew_subs 2024-01-20T11:23:06.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd1ffe9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:06.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd181da950 con 0x7fdd1806e430 2024-01-20T11:23:06.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.013+0000 7fdd1ffe9640 1 librados: init done 2024-01-20T11:23:06.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.014+0000 7fdd1ffe9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:06.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.014+0000 7fdd1ffe9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:06.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.014+0000 7fdd1ffe9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:06.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.014+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd18075490 con 0x7fdd1806e430 2024-01-20T11:23:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.256+0000 7fdd0e7fc640 1 -- 172.21.15.110:0/1541352261 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdd08013470 con 0x7fdd1806e430 2024-01-20T11:23:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.256+0000 7fdd0e7fc640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdcfc06c8e0 0x7fdcfc06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.257+0000 7fdd1dd85640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdcfc06c8e0 0x7fdcfc06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.257+0000 7fdd1dd85640 10 monclient: get_auth_request con 0x7fdcfc06c8e0 auth_method 0 2024-01-20T11:23:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.257+0000 7fdd1dd85640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdcfc06c8e0 0x7fdcfc06eda0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdd14005880 tx=0x7fdd14009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:06.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.257+0000 7fdd0e7fc640 10 client.6064.objecter ms_handle_connect 0x7fdcfc06c8e0 2024-01-20T11:23:06.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.261+0000 7fdd0e7fc640 1 -- 172.21.15.110:0/1541352261 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdd08058790 con 0x7fdd1806e430 2024-01-20T11:23:06.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.261+0000 7fdd0e7fc640 10 client.6064.objecter ms_dispatch 0x7fdd18000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:06.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.261+0000 7fdd0e7fc640 3 client.6064.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:06.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.261+0000 7fdd0e7fc640 3 client.6064.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:06.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.261+0000 7fdd0e7fc640 20 client.6064.objecter dump_active .. 0 homeless 2024-01-20T11:23:06.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.274+0000 7fdd0e7fc640 1 -- 172.21.15.110:0/1541352261 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fdd08054ad0 con 0x7fdd1806e430 2024-01-20T11:23:06.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.274+0000 7fdd0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:06.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.274+0000 7fdd0e7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:06.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~248s 2024-01-20T11:23:06.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.453+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd18067b10 con 0x7fdcfc06c8e0 2024-01-20T11:23:06.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.455+0000 7fdd0e7fc640 1 -- 172.21.15.110:0/1541352261 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63874 (secure 0 0 0) 0x7fdd18067b10 con 0x7fdcfc06c8e0 2024-01-20T11:23:06.456 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:06.456 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:06.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.457+0000 7fdd1ffe9640 10 librados: watch_flush enter 2024-01-20T11:23:06.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.457+0000 7fdd1ffe9640 10 librados: watch_flush exit 2024-01-20T11:23:06.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.457+0000 7fdd1ffe9640 20 client.6064.objecter shutdown clearing up homeless session... 2024-01-20T11:23:06.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.457+0000 7fdd1ffe9640 10 client.6064.objecter successfully canceled tick 2024-01-20T11:23:06.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdcfc06c8e0 msgr2=0x7fdcfc06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:06.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdcfc06c8e0 0x7fdcfc06eda0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdd14005880 tx=0x7fdd14009040 comp rx=0 tx=0).stop 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 10 monclient: shutdown 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd1806e430 msgr2=0x7fdd1806e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd1806e430 0x7fdd1806e8a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800e980 tx=0x7fdd0800ee40 comp rx=0 tx=0).stop 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 shutdown_connections 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fdcfc06c8e0 0x7fdcfc06eda0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd1813d600 0x7fdd1806f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd1806e430 0x7fdd1806e8a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 --2- 172.21.15.110:0/1541352261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdd1806b380 0x7fdd18137e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 >> 172.21.15.110:0/1541352261 conn(0x7fdd1806a5f0 msgr2=0x7fdd180b3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 shutdown_connections 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 -- 172.21.15.110:0/1541352261 wait complete. 2024-01-20T11:23:06.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:06.458+0000 7fdd1ffe9640 1 librados: shutdown 2024-01-20T11:23:06.470 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-01-20T11:23:04.806329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002794"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:18:55.786781+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:18:55.786781+0000","last_peered":"2024-01-20T11:18:55.786781+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:18:55.786781+0000","last_undegraded":"2024-01-20T11:18:55.786781+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":6.9409999999999998},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.5110000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":11.246},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":6.125}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":9.2810000000000006},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.952999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.388},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":13.779}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":8.0920000000000005},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":13.037000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":14.711},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":8.7430000000000003}]}]},{"osd":7,"up_from":21,"seq":90194313314,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.2590000000000003},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":9.1050000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":7.4870000000000001},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.135}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":9.8919999999999995},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.803000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":6.5899999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":11.929}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":15.202},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.276999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":14.48},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.875}]}]},{"osd":3,"up_from":21,"seq":90194313315,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":6.6970000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.7119999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":7.7489999999999997},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.6920000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.308999999999999},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.446999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.923},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":13.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":8.3079999999999998},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.039}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":15.102},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":11.798}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444129,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":8.4269999999999996},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":6.3049999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":12.941000000000001},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":10.151999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.210000000000001},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.1600000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.765000000000001},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":9.8629999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":11.423},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.9669999999999996}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":14.131},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.404999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444130,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.9699999999999998},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":6.2089999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.7169999999999996},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":8.7789999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":13.319000000000001},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":9.4199999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.978999999999999},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":11.361000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":14.06},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":10.670999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":15.699999999999999},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":12.622999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444130,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.79},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":7.1980000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.4030000000000005},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":9.9109999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":7.8280000000000003},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":8.9009999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":11.832000000000001},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":14.156000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":10.541},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.366}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":12.948},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":14.727}]}]},{"osd":6,"up_from":15,"seq":64424509538,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":4.1059999999999999},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.8700000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":15.849},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":7.516}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":9.0749999999999993},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":10.49}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.7010000000000005},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.183999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":12.566000000000001},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":15.081}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":13.847},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.162000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:09.473 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:09.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.586+0000 7fe5ff210640 10 monclient: get_monmap_and_config 2024-01-20T11:23:09.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.586+0000 7fe5ff210640 10 monclient: build_initial_monmap 2024-01-20T11:23:09.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.586+0000 7fe5ff210640 10 monclient: monmap: 2024-01-20T11:23:09.586 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:09.586 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:09.587217+0000 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:09.587217+0000 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:09.587 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 1 Processor -- start 2024-01-20T11:23:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 1 -- start start 2024-01-20T11:23:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 10 monclient: init 2024-01-20T11:23:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-01-20T11:23:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe5f80a96f0 0x7fe5f813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 10 monclient(hunting): picked mon.noname-a con 0x7fe5f80a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f813fa10 0x7fe5f81423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 10 monclient(hunting): picked mon.noname-b con 0x7fe5f813fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5f80a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.587+0000 7fe5ff210640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f80b25c0 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.588+0000 7fe5ff210640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.588+0000 7fe5ff210640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f80aaea0 con 0x7fe5f80a96f0 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.588+0000 7fe5ff210640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.588+0000 7fe5ff210640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80ab0a0 con 0x7fe5f813fa10 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.588+0000 7fe5ff210640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.588+0000 7fe5ff210640 10 monclient(hunting): authenticate will time out at 1832.938202s 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.588+0000 7fe5f7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f813fa10 0x7fe5f81423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.588+0000 7fe5fd7ad640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fe5f80a9ad0 0x7fe5f8142920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:58538/0 (socket says 172.21.15.110:58538) 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.588+0000 7fe5fd7ad640 1 -- 172.21.15.110:0/3532676891 learned_addr learned my addr 172.21.15.110:0/3532676891 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.589+0000 7fe5f7fff640 10 monclient(hunting): get_auth_request con 0x7fe5f813fa10 auth_method 0 2024-01-20T11:23:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.589+0000 7fe5f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.589+0000 7fe5f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.589+0000 7fe5f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.589+0000 7fe5f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.589+0000 7fe5f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.589+0000 7fe5f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.591+0000 7fe5f7fff640 10 monclient(hunting): handle_auth_done global_id 5612 payload 274 2024-01-20T11:23:09.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.591+0000 7fe5f7fff640 1 -- 172.21.15.110:0/3532676891 >> v2:172.21.15.110:3300/0 conn(0x7fe5f80a96f0 msgr2=0x7fe5f813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.591+0000 7fe5f7fff640 1 --2- 172.21.15.110:0/3532676891 >> v2:172.21.15.110:3300/0 conn(0x7fe5f80a96f0 0x7fe5f813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.591+0000 7fe5f7fff640 1 -- 172.21.15.110:0/3532676891 >> v1:172.21.15.134:6789/0 conn(0x7fe5f80a9ad0 legacy=0x7fe5f8142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.591+0000 7fe5f7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.591+0000 7fe5f7fff640 1 monclient: found mon.noname-b 2024-01-20T11:23:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.591+0000 7fe5f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.591+0000 7fe5f7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.591+0000 7fe5f7fff640 1 -- 172.21.15.110:0/3532676891 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f80b3fb0 con 0x7fe5f813fa10 2024-01-20T11:23:09.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.593+0000 7fe5f7fff640 1 --2- 172.21.15.110:0/3532676891 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f813fa10 0x7fe5f81423e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40097a0 tx=0x7fe5e402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b46a6b819bc91e2f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.597+0000 7fe5f77fe640 1 -- 172.21.15.110:0/3532676891 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e402e070 con 0x7fe5f813fa10 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.597+0000 7fe5f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.597+0000 7fe5f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.597+0000 7fe5f77fe640 10 monclient: dump: 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:09.597 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:09.598 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.597+0000 7fe5f77fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.597+0000 7fe5f77fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.597+0000 7fe5f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.597+0000 7fe5ff210640 5 monclient: authenticate success, global_id 5612 2024-01-20T11:23:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.597+0000 7fe5ff210640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.645+0000 7fe5f77fe640 1 -- 172.21.15.110:0/3532676891 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e40315a0 con 0x7fe5f813fa10 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.645+0000 7fe5f77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.645+0000 7fe5f77fe640 1 -- 172.21.15.110:0/3532676891 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4031930 con 0x7fe5f813fa10 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.645+0000 7fe5ff210640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.645+0000 7fe5ff210640 4 set_mon_vals no callback set 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.645+0000 7fe5f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.645+0000 7fe5f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.645+0000 7fe5f77fe640 10 monclient: dump: 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:09.646 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:09.646 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:09.646 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:09.646 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:09.646 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:09.646 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:09.646 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:09.646 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:09.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 10 monclient: shutdown 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 1 -- 172.21.15.110:0/3532676891 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f813fa10 msgr2=0x7fe5f81423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 1 --2- 172.21.15.110:0/3532676891 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f813fa10 0x7fe5f81423e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40097a0 tx=0x7fe5e402b840 comp rx=0 tx=0).stop 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 1 -- 172.21.15.110:0/3532676891 shutdown_connections 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 1 --2- 172.21.15.110:0/3532676891 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f813fa10 0x7fe5f81423e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 1 --2- 172.21.15.110:0/3532676891 >> v2:172.21.15.110:3300/0 conn(0x7fe5f80a96f0 0x7fe5f813f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 1 -- 172.21.15.110:0/3532676891 >> 172.21.15.110:0/3532676891 conn(0x7fe5f80b7bc0 msgr2=0x7fe5f80b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 1 -- 172.21.15.110:0/3532676891 shutdown_connections 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.646+0000 7fe5ff210640 1 -- 172.21.15.110:0/3532676891 wait complete. 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.647+0000 7fe5ff210640 10 monclient: build_initial_monmap 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.647+0000 7fe5ff210640 10 monclient: monmap: 2024-01-20T11:23:09.649 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.647+0000 7fe5ff210640 1 librados: starting msgr at 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.647+0000 7fe5ff210640 1 librados: starting objecter 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.647+0000 7fe5ff210640 1 Processor -- start 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.647+0000 7fe5ff210640 1 -- start start 2024-01-20T11:23:09.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.647+0000 7fe5ff210640 1 librados: setting wanted keys 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.647+0000 7fe5ff210640 1 librados: calling monclient init 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.647+0000 7fe5ff210640 10 monclient: init 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5f80a96f0 0x7fe5f813aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient(hunting): picked mon.noname-a con 0x7fe5f80a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f80a9ad0 0x7fe5f8138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient(hunting): picked mon.noname-c con 0x7fe5f80a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5f8138890 0x7fe5f81dae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient(hunting): picked mon.noname-b con 0x7fe5f8138890 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:09.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80abef0 con 0x7fe5f80a96f0 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80ac160 con 0x7fe5f8138890 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80ac440 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5ff210640 10 monclient(hunting): authenticate will time out at 1832.998436s 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.648+0000 7fe5f7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f80a9ad0 0x7fe5f8138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.649+0000 7fe5f7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f80a9ad0 0x7fe5f8138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:48722/0 (socket says 172.21.15.110:48722) 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.649+0000 7fe5f7fff640 1 -- 172.21.15.110:0/313030160 learned_addr learned my addr 172.21.15.110:0/313030160 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.649+0000 7fe5f7fff640 10 monclient(hunting): get_auth_request con 0x7fe5f80a9ad0 auth_method 0 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.649+0000 7fe5f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:09.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.649+0000 7fe5f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:09.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.649+0000 7fe5f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:09.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.649+0000 7fe5f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:09.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.649+0000 7fe5f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:09.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.649+0000 7fe5f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:09.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.650+0000 7fe5fd7ad640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5f8138890 0x7fe5f81dae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:09.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 10 monclient(hunting): handle_auth_done global_id 5615 payload 824 2024-01-20T11:23:09.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 1 -- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5f8138890 msgr2=0x7fe5f81dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5f8138890 0x7fe5f81dae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 1 -- 172.21.15.110:0/313030160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5f80a96f0 msgr2=0x7fe5f813aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5f80a96f0 0x7fe5f813aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 1 monclient: found mon.noname-c 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.652+0000 7fe5f7fff640 1 -- 172.21.15.110:0/313030160 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4009400 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.654+0000 7fe5f7fff640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f80a9ad0 0x7fe5f8138350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4034040 tx=0x7fe5e40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.654+0000 7fe5f57fa640 10 client.?.objecter ms_handle_connect 0x7fe5f80a9ad0 2024-01-20T11:23:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.654+0000 7fe5f57fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:09.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.663+0000 7fe5f57fa640 1 -- 172.21.15.110:0/313030160 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4032640 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:09.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:09.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 10 monclient: dump: 2024-01-20T11:23:09.664 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:09.664 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:09.664 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 10 monclient: _finish_auth 0 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 10 monclient: _check_auth_tickets 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 1 -- 172.21.15.110:0/313030160 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e40327c0 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5ff210640 5 monclient: authenticate success, global_id 5615 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5ff210640 10 monclient: _renew_subs 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5ff210640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:09.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f80ac720 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 1 -- 172.21.15.110:0/313030160 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4032b70 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f57fa640 10 monclient: dump: 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5f6ffd640 4 set_mon_vals no callback set 2024-01-20T11:23:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5ff210640 10 client.5615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5ff210640 10 monclient: _renew_subs 2024-01-20T11:23:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5ff210640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f8138d20 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.664+0000 7fe5ff210640 1 librados: init done 2024-01-20T11:23:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.665+0000 7fe5ff210640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.665+0000 7fe5ff210640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.665+0000 7fe5ff210640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.665+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f813bb80 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.958+0000 7fe5f57fa640 1 -- 172.21.15.110:0/313030160 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe5e4032cd0 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.958+0000 7fe5f57fa640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5dc06c8b0 0x7fe5dc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.959+0000 7fe5fcfac640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5dc06c8b0 0x7fe5dc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.959+0000 7fe5fcfac640 10 monclient: get_auth_request con 0x7fe5dc06c8b0 auth_method 0 2024-01-20T11:23:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.959+0000 7fe5fcfac640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5dc06c8b0 0x7fe5dc06ed70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009710 tx=0x7fe5ec006c90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.959+0000 7fe5f57fa640 10 client.5615.objecter ms_handle_connect 0x7fe5dc06c8b0 2024-01-20T11:23:09.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.971+0000 7fe5f57fa640 1 -- 172.21.15.110:0/313030160 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe5e407a440 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.971+0000 7fe5f57fa640 10 client.5615.objecter ms_dispatch 0x7fe5f8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:09.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.971+0000 7fe5f57fa640 3 client.5615.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:09.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.971+0000 7fe5f57fa640 3 client.5615.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:09.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.971+0000 7fe5f57fa640 20 client.5615.objecter dump_active .. 0 homeless 2024-01-20T11:23:09.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.971+0000 7fe5f57fa640 1 -- 172.21.15.110:0/313030160 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe5e4076080 con 0x7fe5f80a9ad0 2024-01-20T11:23:09.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.971+0000 7fe5f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:09.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:09.971+0000 7fe5f57fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:10.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.148+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5f813bb80 con 0x7fe5dc06c8b0 2024-01-20T11:23:10.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.151+0000 7fe5f57fa640 1 -- 172.21.15.110:0/313030160 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63780 (secure 0 0 0) 0x7fe5f813bb80 con 0x7fe5dc06c8b0 2024-01-20T11:23:10.152 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:10.152 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 10 librados: watch_flush enter 2024-01-20T11:23:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 10 librados: watch_flush exit 2024-01-20T11:23:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 20 client.5615.objecter shutdown clearing up homeless session... 2024-01-20T11:23:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 10 client.5615.objecter successfully canceled tick 2024-01-20T11:23:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5dc06c8b0 msgr2=0x7fe5dc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5dc06c8b0 0x7fe5dc06ed70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009710 tx=0x7fe5ec006c90 comp rx=0 tx=0).stop 2024-01-20T11:23:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 10 monclient: shutdown 2024-01-20T11:23:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:10.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f80a9ad0 msgr2=0x7fe5f8138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f80a9ad0 0x7fe5f8138350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4034040 tx=0x7fe5e40046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 shutdown_connections 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe5dc06c8b0 0x7fe5dc06ed70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5f8138890 0x7fe5f81dae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5f80a9ad0 0x7fe5f8138350 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 --2- 172.21.15.110:0/313030160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5f80a96f0 0x7fe5f813aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 >> 172.21.15.110:0/313030160 conn(0x7fe5f80b7bc0 msgr2=0x7fe5f80b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 shutdown_connections 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 -- 172.21.15.110:0/313030160 wait complete. 2024-01-20T11:23:10.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.153+0000 7fe5ff210640 1 librados: shutdown 2024-01-20T11:23:10.164 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-01-20T11:23:08.807218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280613,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":11.887},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.4119999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.8759999999999994},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":12.584}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":11.095000000000001},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":13.279}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.944000000000001},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":6.3330000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.593},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.816000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":7.4829999999999997},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":17.497}]}]},{"osd":7,"up_from":21,"seq":90194313315,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.9370000000000003},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":5.6980000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":9.6370000000000005},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.606999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":13.916},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.279999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.7199999999999998},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":16.134}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":10.214},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.208}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":14.667},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":8.9190000000000005}]}]},{"osd":3,"up_from":21,"seq":90194313316,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":9.1210000000000004},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":4.0730000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":7.9189999999999996},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":11.42}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":8.3040000000000003},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":10.753}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":12.016},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.779}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":10.036},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":12.573}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.273},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.875999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444130,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.7859999999999996},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.6440000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.2949999999999999},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.9749999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.766999999999999},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":13.977}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.372},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":16.597999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":15.622999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":10.379}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.992000000000001},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":13.061}]}]},{"osd":1,"up_from":17,"seq":73014444131,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":7.3239999999999998},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":6.5460000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":12.817},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.129}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":15.542},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":8.6699999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":14.914999999999999},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":12.276999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":16.27},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.064}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":17.57},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":13.975}]}]},{"osd":0,"up_from":17,"seq":73014444131,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.3239999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.1050000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":6.9189999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":10.923999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":8.9429999999999996},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":10.298}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":9.6069999999999993},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":11.51}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.683},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":13.076000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.359999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":15.574999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.792999999999999},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.9260000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":13.788},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":6.6219999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.987},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":12.606999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.211},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.500999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":15.804},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":7.3079999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":13.17},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":9.4209999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:10.165 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:10.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.264+0000 7fd4bfec9640 10 monclient: get_monmap_and_config 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.264+0000 7fd4bfec9640 10 monclient: build_initial_monmap 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.264+0000 7fd4bfec9640 10 monclient: monmap: 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:10.265190+0000 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:10.265190+0000 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 1 Processor -- start 2024-01-20T11:23:10.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 1 -- start start 2024-01-20T11:23:10.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 10 monclient: init 2024-01-20T11:23:10.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:10.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-01-20T11:23:10.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4b8066620 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:10.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b806e2a0 0x7fd4b81456a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 10 monclient(hunting): picked mon.noname-b con 0x7fd4b806e2a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4b8145be0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b806dfe0 con 0x7fd4b8066620 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b80ae130 con 0x7fd4b8145be0 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b8066a00 con 0x7fd4b806e2a0 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.265+0000 7fd4bfec9640 10 monclient(hunting): authenticate will time out at 1833.616148s 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.266+0000 7fd4bdc65640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd4b8066620 0x7fd4b813f240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48000/0 (socket says 172.21.15.110:48000) 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.266+0000 7fd4bdc65640 1 -- 172.21.15.110:0/3244864579 learned_addr learned my addr 172.21.15.110:0/3244864579 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.266+0000 7fd4bd464640 1 --2- 172.21.15.110:0/3244864579 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b806e2a0 0x7fd4b81456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.266+0000 7fd4bd464640 10 monclient(hunting): get_auth_request con 0x7fd4b806e2a0 auth_method 0 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.266+0000 7fd4bd464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.266+0000 7fd4bd464640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.266+0000 7fd4bd464640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.267+0000 7fd4bd464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:10.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.267+0000 7fd4bd464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:10.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.267+0000 7fd4bd464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 1 -- 172.21.15.110:0/3244864579 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386734656 0 0) 0x7fd4b806dfe0 con 0x7fd4b8066620 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 10 monclient(hunting): _negotiate 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 10 monclient(hunting): my global_id is 5967 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 1 -- 172.21.15.110:0/3244864579 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a4003520 con 0x7fd4b8066620 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 1 -- 172.21.15.110:0/3244864579 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733347199 0 0) 0x7fd4b80ae130 con 0x7fd4b8145be0 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 10 monclient(hunting): _negotiate 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 10 monclient(hunting): my global_id is 6067 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bcc63640 1 -- 172.21.15.110:0/3244864579 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4b806dfe0 con 0x7fd4b8145be0 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bd464640 10 monclient(hunting): handle_auth_done global_id 5618 payload 274 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bd464640 1 -- 172.21.15.110:0/3244864579 >> v1:172.21.15.134:6789/0 conn(0x7fd4b8145be0 legacy=0x7fd4b814c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bd464640 1 -- 172.21.15.110:0/3244864579 >> v1:172.21.15.110:6789/0 conn(0x7fd4b8066620 legacy=0x7fd4b813f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bd464640 10 monclient: _finish_hunting 0 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bd464640 1 monclient: found mon.noname-b 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bd464640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bd464640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.269+0000 7fd4bd464640 1 -- 172.21.15.110:0/3244864579 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b80ade00 con 0x7fd4b806e2a0 2024-01-20T11:23:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.271+0000 7fd4bd464640 1 --2- 172.21.15.110:0/3244864579 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b806e2a0 0x7fd4b81456a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80097d0 tx=0x7fd4a802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3175d561986b157 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:10.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 1 -- 172.21.15.110:0/3244864579 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8031800 con 0x7fd4b806e2a0 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 10 monclient: dump: 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:10.275 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 10 monclient: _finish_auth 0 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 10 monclient: _check_auth_tickets 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 1 -- 172.21.15.110:0/3244864579 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a8031960 con 0x7fd4b806e2a0 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 1 -- 172.21.15.110:0/3244864579 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8031cf0 con 0x7fd4b806e2a0 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 5 monclient: authenticate success, global_id 5618 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 4 set_mon_vals no callback set 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bcc63640 10 monclient: dump: 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.276 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.277 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.277 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:10.277 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.277 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:10.277 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:10.277 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:10.277 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 10 monclient: shutdown 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 1 -- 172.21.15.110:0/3244864579 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b806e2a0 msgr2=0x7fd4b81456a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 1 --2- 172.21.15.110:0/3244864579 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b806e2a0 0x7fd4b81456a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80097d0 tx=0x7fd4a802b8a0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 1 -- 172.21.15.110:0/3244864579 shutdown_connections 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 1 --2- 172.21.15.110:0/3244864579 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b806e2a0 0x7fd4b81456a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 1 -- 172.21.15.110:0/3244864579 >> 172.21.15.110:0/3244864579 conn(0x7fd4b80b09e0 msgr2=0x7fd4b80b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 1 -- 172.21.15.110:0/3244864579 shutdown_connections 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.275+0000 7fd4bfec9640 1 -- 172.21.15.110:0/3244864579 wait complete. 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.276+0000 7fd4bfec9640 10 monclient: build_initial_monmap 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.276+0000 7fd4bfec9640 10 monclient: monmap: 2024-01-20T11:23:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.276+0000 7fd4bfec9640 1 librados: starting msgr at 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.276+0000 7fd4bfec9640 1 librados: starting objecter 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 Processor -- start 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 -- start start 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 librados: setting wanted keys 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 librados: calling monclient init 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient: init 2024-01-20T11:23:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4b8066620 0x7fd4b811bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient(hunting): picked mon.noname-b con 0x7fd4b8066620 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b8145be0 0x7fd4b81427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient(hunting): picked mon.noname-a con 0x7fd4b8145be0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b81d77e0 0x7fd4b81d7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient(hunting): picked mon.noname-c con 0x7fd4b81d77e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b81d7350 con 0x7fd4b8145be0 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b811d7c0 con 0x7fd4b8066620 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b81d81d0 con 0x7fd4b81d77e0 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.277+0000 7fd4bfec9640 10 monclient(hunting): authenticate will time out at 1833.627916s 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.278+0000 7fd4bdc65640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4b8066620 0x7fd4b811bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.279+0000 7fd4bd464640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b8145be0 0x7fd4b81427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.279+0000 7fd4bd464640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b8145be0 0x7fd4b81427e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47486/0 (socket says 172.21.15.110:47486) 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.279+0000 7fd4bd464640 1 -- 172.21.15.110:0/774094881 learned_addr learned my addr 172.21.15.110:0/774094881 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.279+0000 7fd4bd464640 10 monclient(hunting): get_auth_request con 0x7fd4b8145be0 auth_method 0 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.279+0000 7fd4bd464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.279+0000 7fd4bd464640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.279+0000 7fd4bd464640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4bdc65640 10 monclient(hunting): get_auth_request con 0x7fd4b8066620 auth_method 0 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4bdc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4bdc65640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4bdc65640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4bd464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4bd464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4bd464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4be466640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b81d77e0 0x7fd4b81d7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4be466640 10 monclient(hunting): get_auth_request con 0x7fd4b81d77e0 auth_method 0 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4be466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4be466640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.280+0000 7fd4be466640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.281+0000 7fd4bdc65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.281+0000 7fd4bdc65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.281+0000 7fd4bdc65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.281+0000 7fd4be466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.281+0000 7fd4be466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.281+0000 7fd4be466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 10 monclient(hunting): handle_auth_done global_id 5970 payload 824 2024-01-20T11:23:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 1 -- 172.21.15.110:0/774094881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b81d77e0 msgr2=0x7fd4b81d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b81d77e0 0x7fd4b81d7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 1 -- 172.21.15.110:0/774094881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4b8066620 msgr2=0x7fd4b811bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4b8066620 0x7fd4b811bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 10 monclient: _finish_hunting 0 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 1 monclient: found mon.noname-a 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.283+0000 7fd4bd464640 1 -- 172.21.15.110:0/774094881 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a8009400 con 0x7fd4b8145be0 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.285+0000 7fd4bd464640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b8145be0 0x7fd4b81427e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd4a802beb0 tx=0x7fd4a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.285+0000 7fd4ae7fc640 10 client.?.objecter ms_handle_connect 0x7fd4b8145be0 2024-01-20T11:23:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.285+0000 7fd4ae7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 1 -- 172.21.15.110:0/774094881 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a803bc70 con 0x7fd4b8145be0 2024-01-20T11:23:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 10 monclient: dump: 2024-01-20T11:23:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 1 -- 172.21.15.110:0/774094881 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a803bdd0 con 0x7fd4b8145be0 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 1 -- 172.21.15.110:0/774094881 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8045630 con 0x7fd4b8145be0 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4ae7fc640 10 monclient: dump: 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4bfec9640 5 monclient: authenticate success, global_id 5970 2024-01-20T11:23:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4bfec9640 10 monclient: _renew_subs 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4bfec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b811c4c0 con 0x7fd4b8145be0 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4af7fe640 4 set_mon_vals no callback set 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4bfec9640 10 client.5970.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4bfec9640 10 monclient: _renew_subs 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4bfec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b8142d20 con 0x7fd4b8145be0 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.288+0000 7fd4bfec9640 1 librados: init done 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.289+0000 7fd4bfec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.289+0000 7fd4bfec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.289+0000 7fd4bfec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.289+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b811c650 con 0x7fd4b8145be0 2024-01-20T11:23:10.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.307+0000 7fd4ae7fc640 1 -- 172.21.15.110:0/774094881 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd4a804e450 con 0x7fd4b8145be0 2024-01-20T11:23:10.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.308+0000 7fd4ae7fc640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd49c06c8b0 0x7fd49c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.308+0000 7fd4bdc65640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd49c06c8b0 0x7fd49c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.308+0000 7fd4bdc65640 10 monclient: get_auth_request con 0x7fd49c06c8b0 auth_method 0 2024-01-20T11:23:10.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.309+0000 7fd4bdc65640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd49c06c8b0 0x7fd49c06ed70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4000f60 tx=0x7fd4b4007d70 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:10.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.309+0000 7fd4ae7fc640 10 client.5970.objecter ms_handle_connect 0x7fd49c06c8b0 2024-01-20T11:23:10.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.540+0000 7fd4ae7fc640 1 -- 172.21.15.110:0/774094881 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd4a802e070 con 0x7fd4b8145be0 2024-01-20T11:23:10.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.540+0000 7fd4ae7fc640 10 client.5970.objecter ms_dispatch 0x7fd4b8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:10.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.540+0000 7fd4ae7fc640 3 client.5970.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:10.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.540+0000 7fd4ae7fc640 3 client.5970.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:10.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.541+0000 7fd4ae7fc640 20 client.5970.objecter dump_active .. 0 homeless 2024-01-20T11:23:10.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.541+0000 7fd4ae7fc640 1 -- 172.21.15.110:0/774094881 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd4a8076d00 con 0x7fd4b8145be0 2024-01-20T11:23:10.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.541+0000 7fd4ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:10.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.541+0000 7fd4ae7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:10.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.707+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4b811c650 con 0x7fd49c06c8b0 2024-01-20T11:23:10.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.710+0000 7fd4ae7fc640 1 -- 172.21.15.110:0/774094881 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63780 (secure 0 0 0) 0x7fd4b811c650 con 0x7fd49c06c8b0 2024-01-20T11:23:10.710 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:10.711 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:10.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.711+0000 7fd4bfec9640 10 librados: watch_flush enter 2024-01-20T11:23:10.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 10 librados: watch_flush exit 2024-01-20T11:23:10.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 20 client.5970.objecter shutdown clearing up homeless session... 2024-01-20T11:23:10.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 10 client.5970.objecter successfully canceled tick 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd49c06c8b0 msgr2=0x7fd49c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd49c06c8b0 0x7fd49c06ed70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4000f60 tx=0x7fd4b4007d70 comp rx=0 tx=0).stop 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 10 monclient: shutdown 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b8145be0 msgr2=0x7fd4b81427e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b8145be0 0x7fd4b81427e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd4a802beb0 tx=0x7fd4a80046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 shutdown_connections 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd49c06c8b0 0x7fd49c06ed70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4b81d77e0 0x7fd4b81d7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b8145be0 0x7fd4b81427e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 --2- 172.21.15.110:0/774094881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4b8066620 0x7fd4b811bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 >> 172.21.15.110:0/774094881 conn(0x7fd4b80b09e0 msgr2=0x7fd4b813f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 shutdown_connections 2024-01-20T11:23:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 -- 172.21.15.110:0/774094881 wait complete. 2024-01-20T11:23:10.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.712+0000 7fd4bfec9640 1 librados: shutdown 2024-01-20T11:23:10.722 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-01-20T11:23:08.807218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280613,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":11.887},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.4119999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.8759999999999994},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":12.584}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":11.095000000000001},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":13.279}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.944000000000001},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":6.3330000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.593},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.816000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":7.4829999999999997},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":17.497}]}]},{"osd":7,"up_from":21,"seq":90194313315,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.9370000000000003},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":5.6980000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":9.6370000000000005},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.606999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":13.916},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.279999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.7199999999999998},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":16.134}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":10.214},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.208}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":14.667},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":8.9190000000000005}]}]},{"osd":3,"up_from":21,"seq":90194313316,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":9.1210000000000004},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":4.0730000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":7.9189999999999996},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":11.42}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":8.3040000000000003},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":10.753}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":12.016},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.779}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":10.036},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":12.573}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.273},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.875999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444130,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.7859999999999996},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.6440000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.2949999999999999},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.9749999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.766999999999999},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":13.977}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.372},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":16.597999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":15.622999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":10.379}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.992000000000001},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":13.061}]}]},{"osd":1,"up_from":17,"seq":73014444131,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":7.3239999999999998},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":6.5460000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":12.817},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.129}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":15.542},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":8.6699999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":14.914999999999999},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":12.276999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":16.27},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.064}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":17.57},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":13.975}]}]},{"osd":0,"up_from":17,"seq":73014444131,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.3239999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.1050000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":6.9189999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":10.923999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":8.9429999999999996},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":10.298}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":9.6069999999999993},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":11.51}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.683},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":13.076000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.359999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":15.574999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.792999999999999},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.9260000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":13.788},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":6.6219999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.987},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":12.606999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.211},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.500999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":15.804},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":7.3079999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":13.17},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":9.4209999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:10.724 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:10.724 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:10.724 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:10.724 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:10.724 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:10.724 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:10.724 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:10.724 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:10.725 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.828+0000 7f9453aec640 10 monclient: get_monmap_and_config 2024-01-20T11:23:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.828+0000 7f9453aec640 10 monclient: build_initial_monmap 2024-01-20T11:23:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.828+0000 7f9453aec640 10 monclient: monmap: 2024-01-20T11:23:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:10.829010+0000 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:10.829010+0000 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.828+0000 7f9453aec640 1 Processor -- start 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.829+0000 7f9453aec640 1 -- start start 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.829+0000 7f9453aec640 10 monclient: init 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f944c145760 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f944c0a9d80 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f944c13f8b0 0x7f944c145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 10 monclient(hunting): picked mon.noname-a con 0x7f944c13f8b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944c146250 con 0x7f944c0a9d80 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944c0b20c0 con 0x7f944c145760 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944c0ab050 con 0x7f944c13f8b0 2024-01-20T11:23:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9453aec640 10 monclient(hunting): authenticate will time out at 1834.180380s 2024-01-20T11:23:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9451888640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f944c145760 0x7f944c13f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:58552/0 (socket says 172.21.15.110:58552) 2024-01-20T11:23:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.830+0000 7f9451888640 1 -- 172.21.15.110:0/858066873 learned_addr learned my addr 172.21.15.110:0/858066873 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.831+0000 7f9452089640 1 --2- 172.21.15.110:0/858066873 >> v2:172.21.15.110:3300/0 conn(0x7f944c13f8b0 0x7f944c145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.831+0000 7f9452089640 10 monclient(hunting): get_auth_request con 0x7f944c13f8b0 auth_method 0 2024-01-20T11:23:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.831+0000 7f9452089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.831+0000 7f9452089640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.831+0000 7f9452089640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.832+0000 7f9452089640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.832+0000 7f9452089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.832+0000 7f9452089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9450886640 1 -- 172.21.15.110:0/858066873 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724453524 0 0) 0x7f944c0b20c0 con 0x7f944c145760 2024-01-20T11:23:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9450886640 10 monclient(hunting): _negotiate 2024-01-20T11:23:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9450886640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9450886640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9450886640 10 monclient(hunting): my global_id is 6073 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9450886640 1 -- 172.21.15.110:0/858066873 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9438003520 con 0x7f944c145760 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9452089640 10 monclient(hunting): handle_auth_done global_id 5973 payload 274 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9452089640 1 -- 172.21.15.110:0/858066873 >> v1:172.21.15.134:6789/0 conn(0x7f944c145760 legacy=0x7f944c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9452089640 1 -- 172.21.15.110:0/858066873 >> v1:172.21.15.110:6789/0 conn(0x7f944c0a9d80 legacy=0x7f944c0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9452089640 10 monclient: _finish_hunting 0 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9452089640 1 monclient: found mon.noname-a 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9452089640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9452089640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.833+0000 7f9452089640 1 -- 172.21.15.110:0/858066873 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c0b2d80 con 0x7f944c13f8b0 2024-01-20T11:23:10.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.835+0000 7f9452089640 1 --2- 172.21.15.110:0/858066873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c13f8b0 0x7f944c145b40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9444009d90 tx=0x7f944402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d62e2cdb8a569c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 1 -- 172.21.15.110:0/858066873 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944402cc60 con 0x7f944c13f8b0 2024-01-20T11:23:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 10 monclient: dump: 2024-01-20T11:23:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 10 monclient: _finish_auth 0 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 10 monclient: _check_auth_tickets 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 1 -- 172.21.15.110:0/858066873 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944402cdc0 con 0x7f944c13f8b0 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9453aec640 5 monclient: authenticate success, global_id 5973 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9453aec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9453aec640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9453aec640 4 set_mon_vals no callback set 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 1 -- 172.21.15.110:0/858066873 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444034660 con 0x7f944c13f8b0 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.839+0000 7f9450886640 10 monclient: dump: 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:10.842 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.840+0000 7f9453aec640 10 monclient: shutdown 2024-01-20T11:23:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.840+0000 7f9453aec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.840+0000 7f9453aec640 1 -- 172.21.15.110:0/858066873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c13f8b0 msgr2=0x7f944c145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.840+0000 7f9453aec640 1 --2- 172.21.15.110:0/858066873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c13f8b0 0x7f944c145b40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9444009d90 tx=0x7f944402beb0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.840+0000 7f9453aec640 1 -- 172.21.15.110:0/858066873 shutdown_connections 2024-01-20T11:23:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.840+0000 7f9453aec640 1 --2- 172.21.15.110:0/858066873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c13f8b0 0x7f944c145b40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.840+0000 7f9453aec640 1 -- 172.21.15.110:0/858066873 >> 172.21.15.110:0/858066873 conn(0x7f944c0663b0 msgr2=0x7f944c0667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.840+0000 7f9453aec640 1 -- 172.21.15.110:0/858066873 shutdown_connections 2024-01-20T11:23:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.840+0000 7f9453aec640 1 -- 172.21.15.110:0/858066873 wait complete. 2024-01-20T11:23:10.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.841+0000 7f9453aec640 10 monclient: build_initial_monmap 2024-01-20T11:23:10.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.841+0000 7f9453aec640 10 monclient: monmap: 2024-01-20T11:23:10.846 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.841+0000 7f9453aec640 1 librados: starting msgr at 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.841+0000 7f9453aec640 1 librados: starting objecter 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.841+0000 7f9453aec640 1 Processor -- start 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.841+0000 7f9453aec640 1 -- start start 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.841+0000 7f9453aec640 1 librados: setting wanted keys 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.841+0000 7f9453aec640 1 librados: calling monclient init 2024-01-20T11:23:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.841+0000 7f9453aec640 10 monclient: init 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f944c0a9d80 0x7f944c140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient(hunting): picked mon.noname-b con 0x7f944c0a9d80 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f944c13f8b0 0x7f944c136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient(hunting): picked mon.noname-c con 0x7f944c13f8b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c145760 0x7f944c138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient(hunting): picked mon.noname-a con 0x7f944c145760 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c141470 con 0x7f944c145760 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c1384f0 con 0x7f944c0a9d80 2024-01-20T11:23:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944c1387d0 con 0x7f944c13f8b0 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9453aec640 10 monclient(hunting): authenticate will time out at 1834.192364s 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9452089640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c145760 0x7f944c138bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9451087640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f944c13f8b0 0x7f944c136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9451888640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f944c0a9d80 0x7f944c140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9452089640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c145760 0x7f944c138bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47502/0 (socket says 172.21.15.110:47502) 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9452089640 1 -- 172.21.15.110:0/1035079469 learned_addr learned my addr 172.21.15.110:0/1035079469 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9452089640 10 monclient(hunting): get_auth_request con 0x7f944c145760 auth_method 0 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9452089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9452089640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.842+0000 7f9452089640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451087640 10 monclient(hunting): get_auth_request con 0x7f944c13f8b0 auth_method 0 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451087640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451087640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451087640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451888640 10 monclient(hunting): get_auth_request con 0x7f944c0a9d80 auth_method 0 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451888640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451888640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9452089640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9452089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9452089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451888640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451888640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451888640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451087640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451087640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.843+0000 7f9451087640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 10 monclient(hunting): handle_auth_done global_id 6076 payload 824 2024-01-20T11:23:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 1 -- 172.21.15.110:0/1035079469 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f944c13f8b0 msgr2=0x7f944c136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f944c13f8b0 0x7f944c136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 1 -- 172.21.15.110:0/1035079469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c145760 msgr2=0x7f944c138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c145760 0x7f944c138bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 10 monclient: _finish_hunting 0 2024-01-20T11:23:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 1 monclient: found mon.noname-b 2024-01-20T11:23:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.846+0000 7f9451888640 1 -- 172.21.15.110:0/1035079469 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9444009a70 con 0x7f944c0a9d80 2024-01-20T11:23:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.848+0000 7f9451888640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f944c0a9d80 0x7f944c140f30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f944800c400 tx=0x7f944800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:10.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.848+0000 7f94427fc640 10 client.?.objecter ms_handle_connect 0x7f944c0a9d80 2024-01-20T11:23:10.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.848+0000 7f94427fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 1 -- 172.21.15.110:0/1035079469 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448005ce0 con 0x7f944c0a9d80 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 10 monclient: dump: 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:10.853 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 1 -- 172.21.15.110:0/1035079469 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9448005e40 con 0x7f944c0a9d80 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 1 -- 172.21.15.110:0/1035079469 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94480072f0 con 0x7f944c0a9d80 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f94427fc640 10 monclient: dump: 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 5 monclient: authenticate success, global_id 6076 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 10 monclient: _renew_subs 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944c1374f0 con 0x7f944c0a9d80 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.852+0000 7f9443fff640 4 set_mon_vals no callback set 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 10 client.6076.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 10 monclient: _renew_subs 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944c142410 con 0x7f944c0a9d80 2024-01-20T11:23:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 1 librados: init done 2024-01-20T11:23:10.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:10.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:10.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:10.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.853+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944c075490 con 0x7f944c0a9d80 2024-01-20T11:23:10.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.873+0000 7f94427fc640 1 -- 172.21.15.110:0/1035079469 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9448007450 con 0x7f944c0a9d80 2024-01-20T11:23:10.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.874+0000 7f94427fc640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f943006c8b0 0x7f943006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:10.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.874+0000 7f9451087640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f943006c8b0 0x7f943006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:10.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.874+0000 7f9451087640 10 monclient: get_auth_request con 0x7f943006c8b0 auth_method 0 2024-01-20T11:23:10.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.874+0000 7f94427fc640 10 client.6076.objecter ms_handle_connect 0x7f943006c8b0 2024-01-20T11:23:10.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.874+0000 7f9451087640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f943006c8b0 0x7f943006ed70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f944c0b6e10 tx=0x7f943c009060 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:10.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.876+0000 7f94427fc640 1 -- 172.21.15.110:0/1035079469 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9448057ef0 con 0x7f944c0a9d80 2024-01-20T11:23:10.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.876+0000 7f94427fc640 10 client.6076.objecter ms_dispatch 0x7f944c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:10.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.876+0000 7f94427fc640 3 client.6076.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:10.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.876+0000 7f94427fc640 3 client.6076.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:10.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:10.876+0000 7f94427fc640 20 client.6076.objecter dump_active .. 0 homeless 2024-01-20T11:23:11.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.110+0000 7f94427fc640 1 -- 172.21.15.110:0/1035079469 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9448054230 con 0x7f944c0a9d80 2024-01-20T11:23:11.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.110+0000 7f94427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:11.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.110+0000 7f94427fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:11.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.382+0000 7f9453aec640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:11.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.382+0000 7f9453aec640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:11.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.382+0000 7f9453aec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:11.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.382+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f944c137680 con 0x7f944c0a9d80 2024-01-20T11:23:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.388+0000 7f94427fc640 1 -- 172.21.15.110:0/1035079469 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f9448017080 con 0x7f944c0a9d80 2024-01-20T11:23:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.388+0000 7f94427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.388+0000 7f94427fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:11.388 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:11.388 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":525,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 10 librados: watch_flush enter 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 10 librados: watch_flush exit 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 20 client.6076.objecter shutdown clearing up homeless session... 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 10 client.6076.objecter successfully canceled tick 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f943006c8b0 msgr2=0x7f943006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f943006c8b0 0x7f943006ed70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f944c0b6e10 tx=0x7f943c009060 comp rx=0 tx=0).stop 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 10 monclient: shutdown 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f944c0a9d80 msgr2=0x7f944c140f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f944c0a9d80 0x7f944c140f30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f944800c400 tx=0x7f944800c8c0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 shutdown_connections 2024-01-20T11:23:11.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f943006c8b0 0x7f943006ed70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f944c145760 0x7f944c138bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f944c13f8b0 0x7f944c136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 --2- 172.21.15.110:0/1035079469 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f944c0a9d80 0x7f944c140f30 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 >> 172.21.15.110:0/1035079469 conn(0x7f944c0663b0 msgr2=0x7f944c13ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:11.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 shutdown_connections 2024-01-20T11:23:11.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 -- 172.21.15.110:0/1035079469 wait complete. 2024-01-20T11:23:11.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.389+0000 7f9453aec640 1 librados: shutdown 2024-01-20T11:23:11.400 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:11.400 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.499+0000 7f3495d2c640 10 monclient: get_monmap_and_config 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.499+0000 7f3495d2c640 10 monclient: build_initial_monmap 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.499+0000 7f3495d2c640 10 monclient: monmap: 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:11.499782+0000 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:11.499782+0000 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:11.499 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:11.500 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:11.500 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:11.500 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:11.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.499+0000 7f3495d2c640 1 Processor -- start 2024-01-20T11:23:11.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.499+0000 7f3495d2c640 1 -- start start 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.499+0000 7f3495d2c640 10 monclient: init 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f34900a96f0 0x7f349013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 10 monclient(hunting): picked mon.noname-c con 0x7f34900a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f349013fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f34900a9ad0 0x7f3490142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 10 monclient(hunting): picked mon.noname-a con 0x7f34900a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34900b25c0 con 0x7f349013fa10 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34900aaea0 con 0x7f34900a9ad0 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34900ab0a0 con 0x7f34900a96f0 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.500+0000 7f3495d2c640 10 monclient(hunting): authenticate will time out at 1834.850734s 2024-01-20T11:23:11.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f34900a96f0 0x7f349013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f34900a9ad0 0x7f3490142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f34900a96f0 0x7f349013f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:35548/0 (socket says 172.21.15.110:35548) 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 1 -- 172.21.15.110:0/2599134346 learned_addr learned my addr 172.21.15.110:0/2599134346 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 10 monclient(hunting): get_auth_request con 0x7f34900a96f0 auth_method 0 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348ffff640 10 monclient(hunting): get_auth_request con 0x7f34900a9ad0 auth_method 0 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:11.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.501+0000 7f348f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:11.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.502+0000 7f348ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:11.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.502+0000 7f348ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:11.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.502+0000 7f348ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:11.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.503+0000 7f348f7fe640 10 monclient(hunting): handle_auth_done global_id 6079 payload 274 2024-01-20T11:23:11.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.503+0000 7f348f7fe640 1 -- 172.21.15.110:0/2599134346 >> v2:172.21.15.110:3300/0 conn(0x7f34900a9ad0 msgr2=0x7f3490142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:11.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.503+0000 7f348f7fe640 1 --2- 172.21.15.110:0/2599134346 >> v2:172.21.15.110:3300/0 conn(0x7f34900a9ad0 0x7f3490142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.503+0000 7f348f7fe640 1 -- 172.21.15.110:0/2599134346 >> v1:172.21.15.110:6789/0 conn(0x7f349013fa10 legacy=0x7f34901423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:11.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.503+0000 7f348f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:11.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.503+0000 7f348f7fe640 1 monclient: found mon.noname-c 2024-01-20T11:23:11.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.503+0000 7f348f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:11.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.503+0000 7f348f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:23:11.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.503+0000 7f348f7fe640 1 -- 172.21.15.110:0/2599134346 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34900b3fb0 con 0x7f34900a96f0 2024-01-20T11:23:11.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.505+0000 7f348f7fe640 1 --2- 172.21.15.110:0/2599134346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 0x7f349013f4d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f34900b5720 tx=0x7f348402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adbfb6fb5410569d server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:11.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 1 -- 172.21.15.110:0/2599134346 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34840317e0 con 0x7f34900a96f0 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 10 monclient: dump: 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 1 -- 172.21.15.110:0/2599134346 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3484031940 con 0x7f34900a96f0 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f3495d2c640 5 monclient: authenticate success, global_id 6079 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f3495d2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f3495d2c640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f3495d2c640 4 set_mon_vals no callback set 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 1 -- 172.21.15.110:0/2599134346 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484031cd0 con 0x7f34900a96f0 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.508+0000 7f348e7fc640 10 monclient: dump: 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:11.511 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:11.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 10 monclient: shutdown 2024-01-20T11:23:11.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:11.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 1 -- 172.21.15.110:0/2599134346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 msgr2=0x7f349013f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 1 --2- 172.21.15.110:0/2599134346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 0x7f349013f4d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f34900b5720 tx=0x7f348402b880 comp rx=0 tx=0).stop 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 1 -- 172.21.15.110:0/2599134346 shutdown_connections 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 1 --2- 172.21.15.110:0/2599134346 >> v2:172.21.15.110:3300/0 conn(0x7f34900a9ad0 0x7f3490142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 1 --2- 172.21.15.110:0/2599134346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 0x7f349013f4d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 1 -- 172.21.15.110:0/2599134346 >> 172.21.15.110:0/2599134346 conn(0x7f34900b7bc0 msgr2=0x7f34900b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 1 -- 172.21.15.110:0/2599134346 shutdown_connections 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.509+0000 7f3495d2c640 1 -- 172.21.15.110:0/2599134346 wait complete. 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.510+0000 7f3495d2c640 10 monclient: build_initial_monmap 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.510+0000 7f3495d2c640 10 monclient: monmap: 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:11.516 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.510+0000 7f3495d2c640 1 librados: starting msgr at 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.510+0000 7f3495d2c640 1 librados: starting objecter 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.510+0000 7f3495d2c640 1 Processor -- start 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 1 -- start start 2024-01-20T11:23:11.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 1 librados: setting wanted keys 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 1 librados: calling monclient init 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient: init 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 0x7f349013eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient(hunting): picked mon.noname-b con 0x7f34900a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34900a9ad0 0x7f3490134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient(hunting): picked mon.noname-a con 0x7f34900a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3490134c20 0x7f34901da470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient(hunting): picked mon.noname-c con 0x7f3490134c20 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:11.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3490134770 con 0x7f34900a9ad0 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34901351d0 con 0x7f34900a96f0 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34901da9b0 con 0x7f3490134c20 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.511+0000 7f3495d2c640 10 monclient(hunting): authenticate will time out at 1834.862130s 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348f7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 0x7f349013eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34900a9ad0 0x7f3490134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348f7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 0x7f349013eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:35564/0 (socket says 172.21.15.110:35564) 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348f7fe640 1 -- 172.21.15.110:0/1295849918 learned_addr learned my addr 172.21.15.110:0/1295849918 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348f7fe640 10 monclient(hunting): get_auth_request con 0x7f34900a96f0 auth_method 0 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:11.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348effd640 10 monclient(hunting): get_auth_request con 0x7f34900a9ad0 auth_method 0 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.512+0000 7f348effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.513+0000 7f348f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.513+0000 7f348f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.513+0000 7f348f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.513+0000 7f348effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.513+0000 7f348effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.513+0000 7f348effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.514+0000 7f348ffff640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3490134c20 0x7f34901da470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.514+0000 7f348ffff640 10 monclient(hunting): get_auth_request con 0x7f3490134c20 auth_method 0 2024-01-20T11:23:11.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.514+0000 7f348ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.514+0000 7f348ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.514+0000 7f348ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 10 monclient(hunting): handle_auth_done global_id 6082 payload 824 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 1 -- 172.21.15.110:0/1295849918 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3490134c20 msgr2=0x7f34901da470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3490134c20 0x7f34901da470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 1 -- 172.21.15.110:0/1295849918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34900a9ad0 msgr2=0x7f3490134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:11.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34900a9ad0 0x7f3490134230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:11.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 1 monclient: found mon.noname-b 2024-01-20T11:23:11.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:11.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:11.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.515+0000 7f348f7fe640 1 -- 172.21.15.110:0/1295849918 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34840094d0 con 0x7f34900a96f0 2024-01-20T11:23:11.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.517+0000 7f348f7fe640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 0x7f349013eb20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3484000c00 tx=0x7f34840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:11.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.517+0000 7f3477fff640 10 client.?.objecter ms_handle_connect 0x7f34900a96f0 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.517+0000 7f3477fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 1 -- 172.21.15.110:0/1295849918 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484032880 con 0x7f34900a96f0 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 10 monclient: dump: 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:11.525 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 10 monclient: _finish_auth 0 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 1 -- 172.21.15.110:0/1295849918 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34840329e0 con 0x7f34900a96f0 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 1 -- 172.21.15.110:0/1295849918 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484032d90 con 0x7f34900a96f0 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3495d2c640 5 monclient: authenticate success, global_id 6082 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3495d2c640 10 monclient: _renew_subs 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3495d2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34901dac30 con 0x7f34900a96f0 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f348dffb640 4 set_mon_vals no callback set 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:11.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3477fff640 10 monclient: dump: 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3495d2c640 10 client.6082.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3495d2c640 10 monclient: _renew_subs 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3495d2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:11.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3490131df0 con 0x7f34900a96f0 2024-01-20T11:23:11.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.521+0000 7f3495d2c640 1 librados: init done 2024-01-20T11:23:11.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.522+0000 7f3495d2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:11.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.522+0000 7f3495d2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:11.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.522+0000 7f3495d2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:11.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.522+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3490136920 con 0x7f34900a96f0 2024-01-20T11:23:11.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.540+0000 7f3477fff640 1 -- 172.21.15.110:0/1295849918 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f34840453f0 con 0x7f34900a96f0 2024-01-20T11:23:11.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.541+0000 7f3477fff640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f347c06c860 0x7f347c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.541+0000 7f348effd640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f347c06c860 0x7f347c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.541+0000 7f348effd640 10 monclient: get_auth_request con 0x7f347c06c860 auth_method 0 2024-01-20T11:23:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.541+0000 7f348effd640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f347c06c860 0x7f347c06ed20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3490132e70 tx=0x7f3480006f90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.541+0000 7f3477fff640 10 client.6082.objecter ms_handle_connect 0x7f347c06c860 2024-01-20T11:23:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.762+0000 7f3477fff640 1 -- 172.21.15.110:0/1295849918 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f348407a6f0 con 0x7f34900a96f0 2024-01-20T11:23:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.762+0000 7f3477fff640 10 client.6082.objecter ms_dispatch 0x7f3490000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.762+0000 7f3477fff640 3 client.6082.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.762+0000 7f3477fff640 3 client.6082.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.762+0000 7f3477fff640 20 client.6082.objecter dump_active .. 0 homeless 2024-01-20T11:23:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.772+0000 7f3477fff640 1 -- 172.21.15.110:0/1295849918 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f3484076a30 con 0x7f34900a96f0 2024-01-20T11:23:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.772+0000 7f3477fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.772+0000 7f3477fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.941+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34901db100 con 0x7f347c06c860 2024-01-20T11:23:11.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.944+0000 7f3477fff640 1 -- 172.21.15.110:0/1295849918 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63780 (secure 0 0 0) 0x7f3480009520 con 0x7f347c06c860 2024-01-20T11:23:11.944 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:11.944 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:11.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 10 librados: watch_flush enter 2024-01-20T11:23:11.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 10 librados: watch_flush exit 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 20 client.6082.objecter shutdown clearing up homeless session... 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 10 client.6082.objecter successfully canceled tick 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f347c06c860 msgr2=0x7f347c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f347c06c860 0x7f347c06ed20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3490132e70 tx=0x7f3480006f90 comp rx=0 tx=0).stop 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 10 monclient: shutdown 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 msgr2=0x7f349013eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 0x7f349013eb20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3484000c00 tx=0x7f34840046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 shutdown_connections 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f347c06c860 0x7f347c06ed20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3490134c20 0x7f34901da470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34900a9ad0 0x7f3490134230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 --2- 172.21.15.110:0/1295849918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34900a96f0 0x7f349013eb20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 >> 172.21.15.110:0/1295849918 conn(0x7f34900b7bc0 msgr2=0x7f34900b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 shutdown_connections 2024-01-20T11:23:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 -- 172.21.15.110:0/1295849918 wait complete. 2024-01-20T11:23:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:11.946+0000 7f3495d2c640 1 librados: shutdown 2024-01-20T11:23:11.956 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-01-20T11:23:10.807694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280613,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":11.887},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.4119999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.8759999999999994},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":12.584}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":11.095000000000001},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":13.279}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.944000000000001},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":6.3330000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.593},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.816000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":7.4829999999999997},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":17.497}]}]},{"osd":7,"up_from":21,"seq":90194313315,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":6.9370000000000003},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":5.6980000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":9.6370000000000005},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.606999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":13.916},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.279999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.7199999999999998},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":16.134}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":10.214},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.208}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":14.667},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":8.9190000000000005}]}]},{"osd":3,"up_from":21,"seq":90194313316,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":9.1210000000000004},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":4.0730000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":7.9189999999999996},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":11.42}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":8.3040000000000003},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":10.753}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":12.016},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.779}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":10.036},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":12.573}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.273},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.875999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444130,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.7859999999999996},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.6440000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.2949999999999999},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":8.9749999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":11.766999999999999},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":13.977}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.372},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":16.597999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":15.622999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":10.379}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":10.992000000000001},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":13.061}]}]},{"osd":1,"up_from":17,"seq":73014444131,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":7.3239999999999998},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":6.5460000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":12.817},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.129}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":15.542},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":8.6699999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":14.914999999999999},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":12.276999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":16.27},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":11.064}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":17.57},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":13.975}]}]},{"osd":0,"up_from":17,"seq":73014444131,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.3239999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.1050000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":6.9189999999999996},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":10.923999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":8.9429999999999996},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":10.298}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":9.6069999999999993},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":11.51}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.683},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":13.076000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.359999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":15.574999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.792999999999999},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":7.9260000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":13.788},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":6.6219999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.987},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":12.606999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.211},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.500999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":15.804},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":7.3079999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":13.17},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":9.4209999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:12.648 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~255s 2024-01-20T11:23:14.959 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:15.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.057+0000 7f1e37c46640 10 monclient: get_monmap_and_config 2024-01-20T11:23:15.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.057+0000 7f1e37c46640 10 monclient: build_initial_monmap 2024-01-20T11:23:15.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.057+0000 7f1e37c46640 10 monclient: monmap: 2024-01-20T11:23:15.059 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:15.059 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.059 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:15.059181+0000 2024-01-20T11:23:15.059 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:15.059181+0000 2024-01-20T11:23:15.059 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:15.059 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.060 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:15.060 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:15.060 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:15.060 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:15.060 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:15.060 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 1 Processor -- start 2024-01-20T11:23:15.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 1 -- start start 2024-01-20T11:23:15.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient: init 2024-01-20T11:23:15.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:15.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-01-20T11:23:15.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1e300b4d10 0x7f1e300b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient(hunting): picked mon.noname-a con 0x7f1e300b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e300b17c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e30145980 0x7f1e30145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient(hunting): picked mon.noname-b con 0x7f1e30145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e30146370 con 0x7f1e300b17c0 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e300b50f0 con 0x7f1e300b4d10 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e300aade0 con 0x7f1e30145980 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.058+0000 7f1e37c46640 10 monclient(hunting): authenticate will time out at 1838.410156s 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e361e3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e30145980 0x7f1e30145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e359e2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1e300b4d10 0x7f1e300b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e359e2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1e300b4d10 0x7f1e300b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47534/0 (socket says 172.21.15.110:47534) 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e359e2640 1 -- 172.21.15.110:0/1447366504 learned_addr learned my addr 172.21.15.110:0/1447366504 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e359e2640 10 monclient(hunting): get_auth_request con 0x7f1e300b4d10 auth_method 0 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e359e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e359e2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e359e2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e361e3640 10 monclient(hunting): get_auth_request con 0x7f1e30145980 auth_method 0 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e361e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e361e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.059+0000 7f1e361e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.060+0000 7f1e359e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.060+0000 7f1e359e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.060+0000 7f1e359e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.060+0000 7f1e361e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.060+0000 7f1e361e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.060+0000 7f1e361e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.062+0000 7f1e359e2640 10 monclient(hunting): handle_auth_done global_id 5985 payload 274 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.062+0000 7f1e359e2640 1 -- 172.21.15.110:0/1447366504 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e30145980 msgr2=0x7f1e30145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.062+0000 7f1e359e2640 1 --2- 172.21.15.110:0/1447366504 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e30145980 0x7f1e30145e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.062+0000 7f1e359e2640 1 -- 172.21.15.110:0/1447366504 >> v1:172.21.15.110:6789/0 conn(0x7f1e300b17c0 legacy=0x7f1e300b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.062+0000 7f1e359e2640 10 monclient: _finish_hunting 0 2024-01-20T11:23:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.062+0000 7f1e359e2640 1 monclient: found mon.noname-a 2024-01-20T11:23:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.062+0000 7f1e359e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.062+0000 7f1e359e2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.062+0000 7f1e359e2640 1 -- 172.21.15.110:0/1447366504 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30145460 con 0x7f1e300b4d10 2024-01-20T11:23:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.064+0000 7f1e359e2640 1 --2- 172.21.15.110:0/1447366504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e300b4d10 0x7f1e300b3da0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009870 tx=0x7f1e2402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494b93f35c8fb0ab server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 1 -- 172.21.15.110:0/1447366504 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e240317c0 con 0x7f1e300b4d10 2024-01-20T11:23:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 10 monclient: dump: 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 10 monclient: _finish_auth 0 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 10 monclient: _check_auth_tickets 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 1 -- 172.21.15.110:0/1447366504 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e24031920 con 0x7f1e300b4d10 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e37c46640 5 monclient: authenticate success, global_id 5985 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e37c46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e37c46640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e37c46640 4 set_mon_vals no callback set 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 1 -- 172.21.15.110:0/1447366504 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24031cb0 con 0x7f1e300b4d10 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.068+0000 7f1e349e0640 10 monclient: dump: 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:15.072 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 10 monclient: shutdown 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 1 -- 172.21.15.110:0/1447366504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e300b4d10 msgr2=0x7f1e300b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 1 --2- 172.21.15.110:0/1447366504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e300b4d10 0x7f1e300b3da0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009870 tx=0x7f1e2402b860 comp rx=0 tx=0).stop 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 1 -- 172.21.15.110:0/1447366504 shutdown_connections 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 1 --2- 172.21.15.110:0/1447366504 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e30145980 0x7f1e30145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 1 --2- 172.21.15.110:0/1447366504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e300b4d10 0x7f1e300b3da0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 1 -- 172.21.15.110:0/1447366504 >> 172.21.15.110:0/1447366504 conn(0x7f1e30066590 msgr2=0x7f1e300669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 1 -- 172.21.15.110:0/1447366504 shutdown_connections 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.069+0000 7f1e37c46640 1 -- 172.21.15.110:0/1447366504 wait complete. 2024-01-20T11:23:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.070+0000 7f1e37c46640 10 monclient: build_initial_monmap 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.070+0000 7f1e37c46640 10 monclient: monmap: 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.070+0000 7f1e37c46640 1 librados: starting msgr at 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.070+0000 7f1e37c46640 1 librados: starting objecter 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.070+0000 7f1e37c46640 1 Processor -- start 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 1 -- start start 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 1 librados: setting wanted keys 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 1 librados: calling monclient init 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient: init 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e300b17c0 0x7f1e30142270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient(hunting): picked mon.noname-b con 0x7f1e300b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e300b4d10 0x7f1e3013fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient(hunting): picked mon.noname-c con 0x7f1e300b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e30145980 0x7f1e30140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient(hunting): picked mon.noname-a con 0x7f1e30145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30143180 con 0x7f1e30145980 2024-01-20T11:23:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30143610 con 0x7f1e300b17c0 2024-01-20T11:23:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30146e30 con 0x7f1e300b4d10 2024-01-20T11:23:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e37c46640 10 monclient(hunting): authenticate will time out at 1838.422723s 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.071+0000 7f1e361e3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e30145980 0x7f1e30140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e351e1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e300b4d10 0x7f1e3013fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e361e3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e30145980 0x7f1e30140240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47540/0 (socket says 172.21.15.110:47540) 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e361e3640 1 -- 172.21.15.110:0/1742022065 learned_addr learned my addr 172.21.15.110:0/1742022065 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e361e3640 10 monclient(hunting): get_auth_request con 0x7f1e30145980 auth_method 0 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e361e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e361e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e361e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e359e2640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e300b17c0 0x7f1e30142270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e351e1640 10 monclient(hunting): get_auth_request con 0x7f1e300b4d10 auth_method 0 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e351e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e351e1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e351e1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e359e2640 10 monclient(hunting): get_auth_request con 0x7f1e300b17c0 auth_method 0 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e359e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e359e2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e359e2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e361e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e361e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e361e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e351e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e351e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.072+0000 7f1e351e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.073+0000 7f1e359e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.073+0000 7f1e359e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.073+0000 7f1e359e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 10 monclient(hunting): handle_auth_done global_id 5988 payload 824 2024-01-20T11:23:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 1 -- 172.21.15.110:0/1742022065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e300b4d10 msgr2=0x7f1e3013fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e300b4d10 0x7f1e3013fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 1 -- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e300b17c0 msgr2=0x7f1e30142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e300b17c0 0x7f1e30142270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 10 monclient: _finish_hunting 0 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 1 monclient: found mon.noname-a 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.075+0000 7f1e361e3640 1 -- 172.21.15.110:0/1742022065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e240094d0 con 0x7f1e30145980 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.078+0000 7f1e361e3640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e30145980 0x7f1e30140240 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800ec10 tx=0x7f1e2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.078+0000 7f1e227fc640 10 client.?.objecter ms_handle_connect 0x7f1e30145980 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.078+0000 7f1e227fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 1 -- 172.21.15.110:0/1742022065 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2800cd20 con 0x7f1e30145980 2024-01-20T11:23:15.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 10 monclient: dump: 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:15.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 1 -- 172.21.15.110:0/1742022065 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2800f040 con 0x7f1e30145980 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 1 -- 172.21.15.110:0/1742022065 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e280146c0 con 0x7f1e30145980 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e227fc640 10 monclient: dump: 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e37c46640 5 monclient: authenticate success, global_id 5988 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e37c46640 10 monclient: _renew_subs 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e23fff640 4 set_mon_vals no callback set 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e37c46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e30147110 con 0x7f1e30145980 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e37c46640 10 client.5988.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e37c46640 10 monclient: _renew_subs 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e37c46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e30140780 con 0x7f1e30145980 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.081+0000 7f1e37c46640 1 librados: init done 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.082+0000 7f1e37c46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.082+0000 7f1e37c46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.082+0000 7f1e37c46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.082+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e301432f0 con 0x7f1e30145980 2024-01-20T11:23:15.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.323+0000 7f1e227fc640 1 -- 172.21.15.110:0/1742022065 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1e28014a30 con 0x7f1e30145980 2024-01-20T11:23:15.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.323+0000 7f1e227fc640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1e1406c8b0 0x7f1e1406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.324+0000 7f1e359e2640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1e1406c8b0 0x7f1e1406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.324+0000 7f1e359e2640 10 monclient: get_auth_request con 0x7f1e1406c8b0 auth_method 0 2024-01-20T11:23:15.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.324+0000 7f1e359e2640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1e1406c8b0 0x7f1e1406ed70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1e24036000 tx=0x7f1e2403d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:15.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.324+0000 7f1e227fc640 10 client.5988.objecter ms_handle_connect 0x7f1e1406c8b0 2024-01-20T11:23:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.336+0000 7f1e227fc640 1 -- 172.21.15.110:0/1742022065 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1e2800ce80 con 0x7f1e30145980 2024-01-20T11:23:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.336+0000 7f1e227fc640 10 client.5988.objecter ms_dispatch 0x7f1e30000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.336+0000 7f1e227fc640 3 client.5988.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.336+0000 7f1e227fc640 3 client.5988.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.336+0000 7f1e227fc640 20 client.5988.objecter dump_active .. 0 homeless 2024-01-20T11:23:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.336+0000 7f1e227fc640 1 -- 172.21.15.110:0/1742022065 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1e280559a0 con 0x7f1e30145980 2024-01-20T11:23:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.336+0000 7f1e227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.336+0000 7f1e227fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:15.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.504+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1e30075490 con 0x7f1e1406c8b0 2024-01-20T11:23:15.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.508+0000 7f1e227fc640 1 -- 172.21.15.110:0/1742022065 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63780 (secure 0 0 0) 0x7f1e30075490 con 0x7f1e1406c8b0 2024-01-20T11:23:15.509 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:15.509 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:15.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.509+0000 7f1e37c46640 10 librados: watch_flush enter 2024-01-20T11:23:15.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.509+0000 7f1e37c46640 10 librados: watch_flush exit 2024-01-20T11:23:15.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.509+0000 7f1e37c46640 20 client.5988.objecter shutdown clearing up homeless session... 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.509+0000 7f1e37c46640 10 client.5988.objecter successfully canceled tick 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.509+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1e1406c8b0 msgr2=0x7f1e1406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.509+0000 7f1e37c46640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1e1406c8b0 0x7f1e1406ed70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1e24036000 tx=0x7f1e2403d040 comp rx=0 tx=0).stop 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.509+0000 7f1e37c46640 10 monclient: shutdown 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e30145980 msgr2=0x7f1e30140240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e30145980 0x7f1e30140240 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800ec10 tx=0x7f1e2800c490 comp rx=0 tx=0).stop 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 shutdown_connections 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f1e1406c8b0 0x7f1e1406ed70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e30145980 0x7f1e30140240 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e300b4d10 0x7f1e3013fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 --2- 172.21.15.110:0/1742022065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e300b17c0 0x7f1e30142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 >> 172.21.15.110:0/1742022065 conn(0x7f1e30066590 msgr2=0x7f1e301380f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 shutdown_connections 2024-01-20T11:23:15.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 -- 172.21.15.110:0/1742022065 wait complete. 2024-01-20T11:23:15.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.510+0000 7f1e37c46640 1 librados: shutdown 2024-01-20T11:23:15.522 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-01-20T11:23:14.808623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002739"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":6.0599999999999996},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":6.6120000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":8.3000000000000007},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":7.8410000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":11.509},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":9.984}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":9.0990000000000002},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":13.554}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":15.861000000000001},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":10.657}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":15.214},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":16.599}]}]},{"osd":7,"up_from":21,"seq":90194313316,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.5229999999999997},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":11.542999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":6.7750000000000004},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.515000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":14.023},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.425000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":8.1379999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":16.192}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":9.3079999999999998},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":15.253}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":17.239000000000001},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":10.196}]}]},{"osd":3,"up_from":21,"seq":90194313317,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.3840000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":4.7300000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":8.1329999999999991},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":11.378}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":9.1300000000000008},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":9.9510000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":14.657999999999999},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.746}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":12.166},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":10.615}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":16.945},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.676}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444131,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":10.401},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":8.0690000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.8780000000000001},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.212}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.965},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":7.7110000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.563000000000001},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.019}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":11.597},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.173}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":9.7650000000000006},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":14.779999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444132,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":4.9039999999999999},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":8.234}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.3760000000000003},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.407999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":12.459},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":14.41}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.794},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":13.682}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.7230000000000008},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":15.414}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":9.0839999999999996},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":16.216000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444132,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":7.1589999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":4.2320000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.2829999999999995},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":10.047000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":13.125},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":14.638}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":14.053000000000001},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":8.6180000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":15.407},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":17.568999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":18.556999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.972}]}]},{"osd":6,"up_from":15,"seq":64424509540,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":9.3339999999999996},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.6969999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":6.7190000000000003},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":10.385}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":15.52},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":14.124000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.148999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":16.373000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":17.288},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":9.0099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":12.692},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":13.380000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:15.524 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:15.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.623+0000 7fd65c489640 10 monclient: get_monmap_and_config 2024-01-20T11:23:15.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.623+0000 7fd65c489640 10 monclient: build_initial_monmap 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.623+0000 7fd65c489640 10 monclient: monmap: 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:15.625003+0000 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:15.625003+0000 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:15.625 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.623+0000 7fd65c489640 1 Processor -- start 2024-01-20T11:23:15.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 1 -- start start 2024-01-20T11:23:15.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient: init 2024-01-20T11:23:15.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd654066620 0x7fd65413f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient(hunting): picked mon.noname-a con 0x7fd654066620 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd65406e2a0 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd65413f780 0x7fd654145db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient(hunting): picked mon.noname-c con 0x7fd65413f780 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6541462f0 con 0x7fd65406e2a0 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd654066a00 con 0x7fd654066620 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd65406dfe0 con 0x7fd65413f780 2024-01-20T11:23:15.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.624+0000 7fd65c489640 10 monclient(hunting): authenticate will time out at 1838.975961s 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65a225640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd654066620 0x7fd65413f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65aa26640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fd65413f780 0x7fd654145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65a225640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd654066620 0x7fd65413f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47554/0 (socket says 172.21.15.110:47554) 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65a225640 1 -- 172.21.15.110:0/1728071715 learned_addr learned my addr 172.21.15.110:0/1728071715 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65a225640 10 monclient(hunting): get_auth_request con 0x7fd654066620 auth_method 0 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65a225640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65a225640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65a225640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65aa26640 10 monclient(hunting): get_auth_request con 0x7fd65413f780 auth_method 0 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65aa26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65aa26640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.625+0000 7fd65aa26640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.626+0000 7fd65a225640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.626+0000 7fd65a225640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.626+0000 7fd65a225640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.626+0000 7fd65aa26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.626+0000 7fd65aa26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.626+0000 7fd65aa26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.627+0000 7fd65a225640 10 monclient(hunting): handle_auth_done global_id 5991 payload 274 2024-01-20T11:23:15.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.627+0000 7fd65a225640 1 -- 172.21.15.110:0/1728071715 >> v2:172.21.15.134:3300/0 conn(0x7fd65413f780 msgr2=0x7fd654145db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:15.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.627+0000 7fd65a225640 1 --2- 172.21.15.110:0/1728071715 >> v2:172.21.15.134:3300/0 conn(0x7fd65413f780 0x7fd654145db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.627+0000 7fd65a225640 1 -- 172.21.15.110:0/1728071715 >> v1:172.21.15.110:6789/0 conn(0x7fd65406e2a0 legacy=0x7fd6541456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:15.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.627+0000 7fd65a225640 10 monclient: _finish_hunting 0 2024-01-20T11:23:15.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.627+0000 7fd65a225640 1 monclient: found mon.noname-a 2024-01-20T11:23:15.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.627+0000 7fd65a225640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:15.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.627+0000 7fd65a225640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.627+0000 7fd65a225640 1 -- 172.21.15.110:0/1728071715 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6540ae130 con 0x7fd654066620 2024-01-20T11:23:15.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.629+0000 7fd65a225640 1 --2- 172.21.15.110:0/1728071715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd654066620 0x7fd65413f240 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd6540b0990 tx=0x7fd65002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ed3caf7ea40d8b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 1 -- 172.21.15.110:0/1728071715 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6500317e0 con 0x7fd654066620 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 10 monclient: dump: 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.635 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 10 monclient: _finish_auth 0 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 10 monclient: _check_auth_tickets 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 1 -- 172.21.15.110:0/1728071715 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd650031940 con 0x7fd654066620 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 5 monclient: authenticate success, global_id 5991 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:15.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 1 -- 172.21.15.110:0/1728071715 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd650031cd0 con 0x7fd654066620 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd659223640 10 monclient: dump: 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.637 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:15.638 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:15.638 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:15.638 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:15.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 4 set_mon_vals no callback set 2024-01-20T11:23:15.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 10 monclient: shutdown 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 1 -- 172.21.15.110:0/1728071715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd654066620 msgr2=0x7fd65413f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 1 --2- 172.21.15.110:0/1728071715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd654066620 0x7fd65413f240 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd6540b0990 tx=0x7fd65002b880 comp rx=0 tx=0).stop 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 1 -- 172.21.15.110:0/1728071715 shutdown_connections 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 1 --2- 172.21.15.110:0/1728071715 >> v2:172.21.15.134:3300/0 conn(0x7fd65413f780 0x7fd654145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 1 --2- 172.21.15.110:0/1728071715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd654066620 0x7fd65413f240 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 1 -- 172.21.15.110:0/1728071715 >> 172.21.15.110:0/1728071715 conn(0x7fd6540b09e0 msgr2=0x7fd6540b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 1 -- 172.21.15.110:0/1728071715 shutdown_connections 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.634+0000 7fd65c489640 1 -- 172.21.15.110:0/1728071715 wait complete. 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.635+0000 7fd65c489640 10 monclient: build_initial_monmap 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.635+0000 7fd65c489640 10 monclient: monmap: 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:15.642 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.635+0000 7fd65c489640 1 librados: starting msgr at 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.635+0000 7fd65c489640 1 librados: starting objecter 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.635+0000 7fd65c489640 1 Processor -- start 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.635+0000 7fd65c489640 1 -- start start 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.635+0000 7fd65c489640 1 librados: setting wanted keys 2024-01-20T11:23:15.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.635+0000 7fd65c489640 1 librados: calling monclient init 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.635+0000 7fd65c489640 10 monclient: init 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd654066620 0x7fd654136a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient(hunting): picked mon.noname-c con 0x7fd654066620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65413f780 0x7fd65413b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient(hunting): picked mon.noname-b con 0x7fd65413f780 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6541371e0 0x7fd65413be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient(hunting): picked mon.noname-a con 0x7fd6541371e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd654137c20 con 0x7fd6541371e0 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd654137790 con 0x7fd65413f780 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:15.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd654138b30 con 0x7fd654066620 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65c489640 10 monclient(hunting): authenticate will time out at 1838.987643s 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.636+0000 7fd65aa26640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6541371e0 0x7fd65413be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65a225640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd654066620 0x7fd654136a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd659a24640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65413f780 0x7fd65413b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65aa26640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6541371e0 0x7fd65413be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47560/0 (socket says 172.21.15.110:47560) 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65aa26640 1 -- 172.21.15.110:0/3172881307 learned_addr learned my addr 172.21.15.110:0/3172881307 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65aa26640 10 monclient(hunting): get_auth_request con 0x7fd6541371e0 auth_method 0 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65aa26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65aa26640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65aa26640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd659a24640 10 monclient(hunting): get_auth_request con 0x7fd65413f780 auth_method 0 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd659a24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd659a24640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd659a24640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65a225640 10 monclient(hunting): get_auth_request con 0x7fd654066620 auth_method 0 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65a225640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65a225640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65a225640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65aa26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65aa26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd65aa26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd659a24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.637+0000 7fd659a24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.638+0000 7fd659a24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.638+0000 7fd65a225640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.638+0000 7fd65a225640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.638+0000 7fd65a225640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:15.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 10 monclient(hunting): handle_auth_done global_id 5994 payload 824 2024-01-20T11:23:15.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 1 -- 172.21.15.110:0/3172881307 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd654066620 msgr2=0x7fd654136a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:15.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd654066620 0x7fd654136a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 1 -- 172.21.15.110:0/3172881307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65413f780 msgr2=0x7fd65413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:15.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65413f780 0x7fd65413b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:15.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 10 monclient: _finish_hunting 0 2024-01-20T11:23:15.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 1 monclient: found mon.noname-a 2024-01-20T11:23:15.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:15.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.640+0000 7fd65aa26640 1 -- 172.21.15.110:0/3172881307 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6500094d0 con 0x7fd6541371e0 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.643+0000 7fd65aa26640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6541371e0 0x7fd65413be80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00efc0 tx=0x7fd64c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.643+0000 7fd64affd640 10 client.?.objecter ms_handle_connect 0x7fd6541371e0 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.643+0000 7fd64affd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 1 -- 172.21.15.110:0/3172881307 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c009ba0 con 0x7fd6541371e0 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 10 monclient: dump: 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.649 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 10 monclient: _finish_auth 0 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 1 -- 172.21.15.110:0/3172881307 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64c00f040 con 0x7fd6541371e0 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd65c489640 5 monclient: authenticate success, global_id 5994 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd65c489640 10 monclient: _renew_subs 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd65c489640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd654136f60 con 0x7fd6541371e0 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 1 -- 172.21.15.110:0/3172881307 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c0146c0 con 0x7fd6541371e0 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd64affd640 10 monclient: dump: 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:15.651 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd658a22640 4 set_mon_vals no callback set 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd65c489640 10 client.5994.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd65c489640 10 monclient: _renew_subs 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd65c489640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd654142c20 con 0x7fd6541371e0 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.646+0000 7fd65c489640 1 librados: init done 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.647+0000 7fd65c489640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.647+0000 7fd65c489640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.647+0000 7fd65c489640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.647+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd654075490 con 0x7fd6541371e0 2024-01-20T11:23:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.666+0000 7fd64affd640 1 -- 172.21.15.110:0/3172881307 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd64c014820 con 0x7fd6541371e0 2024-01-20T11:23:15.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.666+0000 7fd64affd640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd63806c8e0 0x7fd63806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:15.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.666+0000 7fd65a225640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd63806c8e0 0x7fd63806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:15.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.666+0000 7fd65a225640 10 monclient: get_auth_request con 0x7fd63806c8e0 auth_method 0 2024-01-20T11:23:15.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.667+0000 7fd65a225640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd63806c8e0 0x7fd63806eda0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd650036000 tx=0x7fd65003d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:15.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.667+0000 7fd64affd640 10 client.5994.objecter ms_handle_connect 0x7fd63806c8e0 2024-01-20T11:23:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.669+0000 7fd64affd640 1 -- 172.21.15.110:0/3172881307 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd64c059480 con 0x7fd6541371e0 2024-01-20T11:23:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.669+0000 7fd64affd640 10 client.5994.objecter ms_dispatch 0x7fd654000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.669+0000 7fd64affd640 3 client.5994.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.669+0000 7fd64affd640 3 client.5994.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.669+0000 7fd64affd640 20 client.5994.objecter dump_active .. 0 homeless 2024-01-20T11:23:15.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.900+0000 7fd64affd640 1 -- 172.21.15.110:0/3172881307 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd64c0557c0 con 0x7fd6541371e0 2024-01-20T11:23:15.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.900+0000 7fd64affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:15.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:15.900+0000 7fd64affd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:16.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.074+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd65414c7c0 con 0x7fd63806c8e0 2024-01-20T11:23:16.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.077+0000 7fd64affd640 1 -- 172.21.15.110:0/3172881307 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63780 (secure 0 0 0) 0x7fd650031ab0 con 0x7fd63806c8e0 2024-01-20T11:23:16.078 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:16.079 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:16.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 10 librados: watch_flush enter 2024-01-20T11:23:16.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 10 librados: watch_flush exit 2024-01-20T11:23:16.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 20 client.5994.objecter shutdown clearing up homeless session... 2024-01-20T11:23:16.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 10 client.5994.objecter successfully canceled tick 2024-01-20T11:23:16.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd63806c8e0 msgr2=0x7fd63806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd63806c8e0 0x7fd63806eda0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd650036000 tx=0x7fd65003d040 comp rx=0 tx=0).stop 2024-01-20T11:23:16.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 10 monclient: shutdown 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6541371e0 msgr2=0x7fd65413be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.079+0000 7fd65c489640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6541371e0 0x7fd65413be80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00efc0 tx=0x7fd64c00c490 comp rx=0 tx=0).stop 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.080+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 shutdown_connections 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.080+0000 7fd65c489640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd63806c8e0 0x7fd63806eda0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.080+0000 7fd65c489640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6541371e0 0x7fd65413be80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.080+0000 7fd65c489640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65413f780 0x7fd65413b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.080+0000 7fd65c489640 1 --2- 172.21.15.110:0/3172881307 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd654066620 0x7fd654136a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.080+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 >> 172.21.15.110:0/3172881307 conn(0x7fd6540b09e0 msgr2=0x7fd654073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.080+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 shutdown_connections 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.080+0000 7fd65c489640 1 -- 172.21.15.110:0/3172881307 wait complete. 2024-01-20T11:23:16.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.080+0000 7fd65c489640 1 librados: shutdown 2024-01-20T11:23:16.091 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-01-20T11:23:14.808623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002739"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":6.0599999999999996},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":6.6120000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":8.3000000000000007},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":7.8410000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":11.509},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":9.984}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":9.0990000000000002},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":13.554}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":15.861000000000001},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":10.657}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":15.214},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":16.599}]}]},{"osd":7,"up_from":21,"seq":90194313316,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.5229999999999997},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":11.542999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":6.7750000000000004},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.515000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":14.023},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.425000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":8.1379999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":16.192}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":9.3079999999999998},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":15.253}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":17.239000000000001},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":10.196}]}]},{"osd":3,"up_from":21,"seq":90194313317,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.3840000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":4.7300000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":8.1329999999999991},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":11.378}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":9.1300000000000008},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":9.9510000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":14.657999999999999},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.746}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":12.166},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":10.615}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":16.945},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.676}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444131,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":10.401},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":8.0690000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.8780000000000001},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.212}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.965},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":7.7110000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.563000000000001},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.019}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":11.597},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.173}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":9.7650000000000006},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":14.779999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444132,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":4.9039999999999999},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":8.234}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.3760000000000003},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.407999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":12.459},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":14.41}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.794},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":13.682}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.7230000000000008},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":15.414}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":9.0839999999999996},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":16.216000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444132,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":7.1589999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":4.2320000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.2829999999999995},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":10.047000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":13.125},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":14.638}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":14.053000000000001},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":8.6180000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":15.407},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":17.568999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":18.556999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.972}]}]},{"osd":6,"up_from":15,"seq":64424509540,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":9.3339999999999996},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":9.6969999999999992}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":6.7190000000000003},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":10.385}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":15.52},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":14.124000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":11.148999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":16.373000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":17.288},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":9.0099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":12.692},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":13.380000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:16.092 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:16.093 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:16.093 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:16.093 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:16.093 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:16.093 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:16.093 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:16.093 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 130, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:19:18.003350+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:19:18.003350+0000', 'last_peered': '2024-01-20T11:19:18.003350+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:19:18.003350+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:16.093 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:16.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.190+0000 7fa5e4e10640 10 monclient: get_monmap_and_config 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.190+0000 7fa5e4e10640 10 monclient: build_initial_monmap 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.190+0000 7fa5e4e10640 10 monclient: monmap: 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:16.191665+0000 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:16.191665+0000 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:16.191 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:16.192 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.190+0000 7fa5e4e10640 1 Processor -- start 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.190+0000 7fa5e4e10640 1 -- start start 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.190+0000 7fa5e4e10640 10 monclient: init 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fa5e00a96f0 0x7fa5e013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 10 monclient(hunting): picked mon.noname-c con 0x7fa5e00a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5e013fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e00a9ad0 0x7fa5e0142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 10 monclient(hunting): picked mon.noname-b con 0x7fa5e00a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e00b25c0 con 0x7fa5e013fa10 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e00aaea0 con 0x7fa5e00a96f0 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00ab0a0 con 0x7fa5e00a9ad0 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5e4e10640 10 monclient(hunting): authenticate will time out at 1839.542631s 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5ddd9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa5e013fa10 0x7fa5e01423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48040/0 (socket says 172.21.15.110:48040) 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.191+0000 7fa5ddd9b640 1 -- 172.21.15.110:0/620322743 learned_addr learned my addr 172.21.15.110:0/620322743 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5ded9d640 1 --2- 172.21.15.110:0/620322743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e00a9ad0 0x7fa5e0142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5de59c640 1 --2- 172.21.15.110:0/620322743 >> v2:172.21.15.134:3300/0 conn(0x7fa5e00a96f0 0x7fa5e013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5ded9d640 10 monclient(hunting): get_auth_request con 0x7fa5e00a9ad0 auth_method 0 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5ded9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5ded9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5de59c640 10 monclient(hunting): get_auth_request con 0x7fa5e00a96f0 auth_method 0 2024-01-20T11:23:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5de59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5de59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5ded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5ded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.192+0000 7fa5ded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.193+0000 7fa5de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.193+0000 7fa5de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:16.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.193+0000 7fa5de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:16.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5dd59a640 1 -- 172.21.15.110:0/620322743 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201919803 0 0) 0x7fa5e00b25c0 con 0x7fa5e013fa10 2024-01-20T11:23:16.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5dd59a640 10 monclient(hunting): _negotiate 2024-01-20T11:23:16.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5dd59a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5dd59a640 10 monclient(hunting): my global_id is 5997 2024-01-20T11:23:16.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5dd59a640 1 -- 172.21.15.110:0/620322743 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c8003520 con 0x7fa5e013fa10 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5ded9d640 10 monclient(hunting): handle_auth_done global_id 5639 payload 274 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5ded9d640 1 -- 172.21.15.110:0/620322743 >> v2:172.21.15.134:3300/0 conn(0x7fa5e00a96f0 msgr2=0x7fa5e013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5ded9d640 1 --2- 172.21.15.110:0/620322743 >> v2:172.21.15.134:3300/0 conn(0x7fa5e00a96f0 0x7fa5e013f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5ded9d640 1 -- 172.21.15.110:0/620322743 >> v1:172.21.15.110:6789/0 conn(0x7fa5e013fa10 legacy=0x7fa5e01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5ded9d640 10 monclient: _finish_hunting 0 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5ded9d640 1 monclient: found mon.noname-b 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5ded9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5ded9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.194+0000 7fa5ded9d640 1 -- 172.21.15.110:0/620322743 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e00b3fb0 con 0x7fa5e00a9ad0 2024-01-20T11:23:16.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.196+0000 7fa5ded9d640 1 --2- 172.21.15.110:0/620322743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e00a9ad0 0x7fa5e0142af0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00097a0 tx=0x7fa5d002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5d40f169308ae4c server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.198+0000 7fa5dd59a640 1 -- 172.21.15.110:0/620322743 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d002e070 con 0x7fa5e00a9ad0 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.198+0000 7fa5dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.198+0000 7fa5dd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.198+0000 7fa5dd59a640 10 monclient: dump: 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:16.199 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:16.200 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.198+0000 7fa5dd59a640 10 monclient: _finish_auth 0 2024-01-20T11:23:16.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.198+0000 7fa5dd59a640 10 monclient: _check_auth_tickets 2024-01-20T11:23:16.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.198+0000 7fa5dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:16.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.198+0000 7fa5e4e10640 5 monclient: authenticate success, global_id 5639 2024-01-20T11:23:16.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.198+0000 7fa5e4e10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5dd59a640 1 -- 172.21.15.110:0/620322743 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d00315a0 con 0x7fa5e00a9ad0 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5dd59a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5dd59a640 1 -- 172.21.15.110:0/620322743 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0031910 con 0x7fa5e00a9ad0 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5dd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5dd59a640 10 monclient: dump: 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.201 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:16.202 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.202 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:16.202 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:16.202 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:16.202 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:16.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 4 set_mon_vals no callback set 2024-01-20T11:23:16.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 10 monclient: shutdown 2024-01-20T11:23:16.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:16.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 1 -- 172.21.15.110:0/620322743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e00a9ad0 msgr2=0x7fa5e0142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 1 --2- 172.21.15.110:0/620322743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e00a9ad0 0x7fa5e0142af0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00097a0 tx=0x7fa5d002b840 comp rx=0 tx=0).stop 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 1 -- 172.21.15.110:0/620322743 shutdown_connections 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 1 --2- 172.21.15.110:0/620322743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e00a9ad0 0x7fa5e0142af0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 1 --2- 172.21.15.110:0/620322743 >> v2:172.21.15.134:3300/0 conn(0x7fa5e00a96f0 0x7fa5e013f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 1 -- 172.21.15.110:0/620322743 >> 172.21.15.110:0/620322743 conn(0x7fa5e00b7bc0 msgr2=0x7fa5e00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 1 -- 172.21.15.110:0/620322743 shutdown_connections 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.200+0000 7fa5e4e10640 1 -- 172.21.15.110:0/620322743 wait complete. 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.201+0000 7fa5e4e10640 10 monclient: build_initial_monmap 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.201+0000 7fa5e4e10640 10 monclient: monmap: 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:16.206 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.201+0000 7fa5e4e10640 1 librados: starting msgr at 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.201+0000 7fa5e4e10640 1 librados: starting objecter 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 Processor -- start 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 -- start start 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 librados: setting wanted keys 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 librados: calling monclient init 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient: init 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:23:16.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5e00a96f0 0x7fa5e01349f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient(hunting): picked mon.noname-b con 0x7fa5e00a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e00a9ad0 0x7fa5e0137420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient(hunting): picked mon.noname-a con 0x7fa5e00a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e01d7180 0x7fa5e0134000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient(hunting): picked mon.noname-c con 0x7fa5e01d7180 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e0136860 con 0x7fa5e00a9ad0 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e0137960 con 0x7fa5e00a96f0 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e0136f00 con 0x7fa5e01d7180 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.202+0000 7fa5e4e10640 10 monclient(hunting): authenticate will time out at 1839.553873s 2024-01-20T11:23:16.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e01d7180 0x7fa5e0134000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ddd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e00a9ad0 0x7fa5e0137420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5de59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5e00a96f0 0x7fa5e01349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e01d7180 0x7fa5e0134000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:48798/0 (socket says 172.21.15.110:48798) 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 1 -- 172.21.15.110:0/1360017958 learned_addr learned my addr 172.21.15.110:0/1360017958 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 10 monclient(hunting): get_auth_request con 0x7fa5e01d7180 auth_method 0 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ddd9b640 10 monclient(hunting): get_auth_request con 0x7fa5e00a9ad0 auth_method 0 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ddd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5de59c640 10 monclient(hunting): get_auth_request con 0x7fa5e00a96f0 auth_method 0 2024-01-20T11:23:16.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5de59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5de59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.203+0000 7fa5ded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.204+0000 7fa5ddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.204+0000 7fa5ddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.204+0000 7fa5ddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.204+0000 7fa5de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.204+0000 7fa5de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:16.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.204+0000 7fa5de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 10 monclient(hunting): handle_auth_done global_id 5642 payload 824 2024-01-20T11:23:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 1 -- 172.21.15.110:0/1360017958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5e00a96f0 msgr2=0x7fa5e01349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5e00a96f0 0x7fa5e01349f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 1 -- 172.21.15.110:0/1360017958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e00a9ad0 msgr2=0x7fa5e0137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e00a9ad0 0x7fa5e0137420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 10 monclient: _finish_hunting 0 2024-01-20T11:23:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 1 monclient: found mon.noname-c 2024-01-20T11:23:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.206+0000 7fa5ded9d640 1 -- 172.21.15.110:0/1360017958 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d0009400 con 0x7fa5e01d7180 2024-01-20T11:23:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.209+0000 7fa5ded9d640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e01d7180 0x7fa5e0134000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0034040 tx=0x7fa5d00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.209+0000 7fa5ceffd640 10 client.?.objecter ms_handle_connect 0x7fa5e01d7180 2024-01-20T11:23:16.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.209+0000 7fa5ceffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 1 -- 172.21.15.110:0/1360017958 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0003810 con 0x7fa5e01d7180 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 10 monclient: dump: 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:16.214 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 10 monclient: _finish_auth 0 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 1 -- 172.21.15.110:0/1360017958 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d0040070 con 0x7fa5e01d7180 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5e4e10640 5 monclient: authenticate success, global_id 5642 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5e4e10640 10 monclient: _renew_subs 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5e4e10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e0136bf0 con 0x7fa5e01d7180 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 1 -- 172.21.15.110:0/1360017958 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0003b90 con 0x7fa5e01d7180 2024-01-20T11:23:16.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.212+0000 7fa5ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.213+0000 7fa5ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.213+0000 7fa5ceffd640 10 monclient: dump: 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.213+0000 7fa5dcd99640 4 set_mon_vals no callback set 2024-01-20T11:23:16.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.213+0000 7fa5e4e10640 10 client.5642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.213+0000 7fa5e4e10640 10 monclient: _renew_subs 2024-01-20T11:23:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.213+0000 7fa5e4e10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.213+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e01355d0 con 0x7fa5e01d7180 2024-01-20T11:23:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.213+0000 7fa5e4e10640 1 librados: init done 2024-01-20T11:23:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.214+0000 7fa5e4e10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.214+0000 7fa5e4e10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:16.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.214+0000 7fa5e4e10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.214+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e01357c0 con 0x7fa5e01d7180 2024-01-20T11:23:16.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.455+0000 7fa5ceffd640 1 -- 172.21.15.110:0/1360017958 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa5d0003cf0 con 0x7fa5e01d7180 2024-01-20T11:23:16.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.455+0000 7fa5ceffd640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa5c006c860 0x7fa5c006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.455+0000 7fa5de59c640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa5c006c860 0x7fa5c006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.455+0000 7fa5de59c640 10 monclient: get_auth_request con 0x7fa5c006c860 auth_method 0 2024-01-20T11:23:16.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.456+0000 7fa5ceffd640 10 client.5642.objecter ms_handle_connect 0x7fa5c006c860 2024-01-20T11:23:16.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.456+0000 7fa5de59c640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa5c006c860 0x7fa5c006ed20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009b40 tx=0x7fa5d8009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:16.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.470+0000 7fa5ceffd640 1 -- 172.21.15.110:0/1360017958 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa5d0042070 con 0x7fa5e01d7180 2024-01-20T11:23:16.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.470+0000 7fa5ceffd640 10 client.5642.objecter ms_dispatch 0x7fa5e0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:16.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.470+0000 7fa5ceffd640 3 client.5642.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:16.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.470+0000 7fa5ceffd640 3 client.5642.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:16.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.470+0000 7fa5ceffd640 20 client.5642.objecter dump_active .. 0 homeless 2024-01-20T11:23:16.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.470+0000 7fa5ceffd640 1 -- 172.21.15.110:0/1360017958 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa5d0076c10 con 0x7fa5e01d7180 2024-01-20T11:23:16.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.470+0000 7fa5ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:16.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.470+0000 7fa5ceffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:16.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.707+0000 7fa5e4e10640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:16.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.707+0000 7fa5e4e10640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:16.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.707+0000 7fa5e4e10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.707+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa5e0130990 con 0x7fa5e01d7180 2024-01-20T11:23:16.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.712+0000 7fa5ceffd640 1 -- 172.21.15.110:0/1360017958 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fa5d007a230 con 0x7fa5e01d7180 2024-01-20T11:23:16.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.712+0000 7fa5ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:16.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.712+0000 7fa5ceffd640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:16.714 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:16.714 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":531,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:16.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.714+0000 7fa5e4e10640 10 librados: watch_flush enter 2024-01-20T11:23:16.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.714+0000 7fa5e4e10640 10 librados: watch_flush exit 2024-01-20T11:23:16.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.714+0000 7fa5e4e10640 20 client.5642.objecter shutdown clearing up homeless session... 2024-01-20T11:23:16.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.714+0000 7fa5e4e10640 10 client.5642.objecter successfully canceled tick 2024-01-20T11:23:16.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa5c006c860 msgr2=0x7fa5c006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa5c006c860 0x7fa5c006ed20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009b40 tx=0x7fa5d8009190 comp rx=0 tx=0).stop 2024-01-20T11:23:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 10 monclient: shutdown 2024-01-20T11:23:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e01d7180 msgr2=0x7fa5e0134000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e01d7180 0x7fa5e0134000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0034040 tx=0x7fa5d00046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 shutdown_connections 2024-01-20T11:23:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa5c006c860 0x7fa5c006ed20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5e01d7180 0x7fa5e0134000 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e00a9ad0 0x7fa5e0137420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 --2- 172.21.15.110:0/1360017958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5e00a96f0 0x7fa5e01349f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 >> 172.21.15.110:0/1360017958 conn(0x7fa5e00b7bc0 msgr2=0x7fa5e00b6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 shutdown_connections 2024-01-20T11:23:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 -- 172.21.15.110:0/1360017958 wait complete. 2024-01-20T11:23:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.715+0000 7fa5e4e10640 1 librados: shutdown 2024-01-20T11:23:16.727 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:16.728 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:16.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.826+0000 7fda8ab1d640 10 monclient: get_monmap_and_config 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.826+0000 7fda8ab1d640 10 monclient: build_initial_monmap 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.826+0000 7fda8ab1d640 10 monclient: monmap: 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:16.828320+0000 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:16.828320+0000 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:16.828 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.827+0000 7fda8ab1d640 1 Processor -- start 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.827+0000 7fda8ab1d640 1 -- start start 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.827+0000 7fda8ab1d640 10 monclient: init 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fda840a96f0 0x7fda8413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 10 monclient(hunting): picked mon.noname-c con 0x7fda840a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fda8413fa10 0x7fda841423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 10 monclient(hunting): picked mon.noname-a con 0x7fda8413fa10 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda840a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:16.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840b25c0 con 0x7fda840a9ad0 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda840aaea0 con 0x7fda8413fa10 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda840ab0a0 con 0x7fda840a96f0 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda8ab1d640 10 monclient(hunting): authenticate will time out at 1840.179630s 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.828+0000 7fda83fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fda8413fa10 0x7fda841423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda83fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fda8413fa10 0x7fda841423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47570/0 (socket says 172.21.15.110:47570) 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda83fff640 1 -- 172.21.15.110:0/2783086891 learned_addr learned my addr 172.21.15.110:0/2783086891 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda83fff640 10 monclient(hunting): get_auth_request con 0x7fda8413fa10 auth_method 0 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda83fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda83fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:16.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:16.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:16.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.829+0000 7fda888b9640 1 --2- 172.21.15.110:0/2783086891 >> v2:172.21.15.134:3300/0 conn(0x7fda840a96f0 0x7fda8413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.830+0000 7fda888b9640 10 monclient(hunting): get_auth_request con 0x7fda840a96f0 auth_method 0 2024-01-20T11:23:16.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.830+0000 7fda888b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.830+0000 7fda888b9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.830+0000 7fda888b9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.831+0000 7fda83fff640 10 monclient(hunting): handle_auth_done global_id 6003 payload 274 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.831+0000 7fda83fff640 1 -- 172.21.15.110:0/2783086891 >> v2:172.21.15.134:3300/0 conn(0x7fda840a96f0 msgr2=0x7fda8413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.831+0000 7fda83fff640 1 --2- 172.21.15.110:0/2783086891 >> v2:172.21.15.134:3300/0 conn(0x7fda840a96f0 0x7fda8413f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.831+0000 7fda83fff640 1 -- 172.21.15.110:0/2783086891 >> v1:172.21.15.134:6789/0 conn(0x7fda840a9ad0 legacy=0x7fda84142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.831+0000 7fda83fff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.831+0000 7fda83fff640 1 monclient: found mon.noname-a 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.831+0000 7fda83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.831+0000 7fda83fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.831+0000 7fda83fff640 1 -- 172.21.15.110:0/2783086891 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda840b3fb0 con 0x7fda8413fa10 2024-01-20T11:23:16.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.833+0000 7fda83fff640 1 --2- 172.21.15.110:0/2783086891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda8413fa10 0x7fda841423e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fda840b5720 tx=0x7fda7002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed51975f5a32c93 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:16.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 1 -- 172.21.15.110:0/2783086891 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70031870 con 0x7fda8413fa10 2024-01-20T11:23:16.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:16.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:16.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 10 monclient: dump: 2024-01-20T11:23:16.837 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:16.837 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.837 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.837 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda8ab1d640 5 monclient: authenticate success, global_id 6003 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 1 -- 172.21.15.110:0/2783086891 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda700319d0 con 0x7fda8413fa10 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda8ab1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:16.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda8ab1d640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda8ab1d640 4 set_mon_vals no callback set 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 1 -- 172.21.15.110:0/2783086891 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70031d60 con 0x7fda8413fa10 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.836+0000 7fda837fe640 10 monclient: dump: 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:16.839 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:16.840 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:16.840 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 10 monclient: shutdown 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 1 -- 172.21.15.110:0/2783086891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda8413fa10 msgr2=0x7fda841423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 1 --2- 172.21.15.110:0/2783086891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda8413fa10 0x7fda841423e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fda840b5720 tx=0x7fda7002b910 comp rx=0 tx=0).stop 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 1 -- 172.21.15.110:0/2783086891 shutdown_connections 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 1 --2- 172.21.15.110:0/2783086891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda8413fa10 0x7fda841423e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 1 --2- 172.21.15.110:0/2783086891 >> v2:172.21.15.134:3300/0 conn(0x7fda840a96f0 0x7fda8413f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 1 -- 172.21.15.110:0/2783086891 >> 172.21.15.110:0/2783086891 conn(0x7fda840b7bc0 msgr2=0x7fda840b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 1 -- 172.21.15.110:0/2783086891 shutdown_connections 2024-01-20T11:23:16.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 1 -- 172.21.15.110:0/2783086891 wait complete. 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 10 monclient: build_initial_monmap 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.837+0000 7fda8ab1d640 10 monclient: monmap: 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.838+0000 7fda8ab1d640 1 librados: starting msgr at 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.838+0000 7fda8ab1d640 1 librados: starting objecter 2024-01-20T11:23:16.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.838+0000 7fda8ab1d640 1 Processor -- start 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.838+0000 7fda8ab1d640 1 -- start start 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.838+0000 7fda8ab1d640 1 librados: setting wanted keys 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.838+0000 7fda8ab1d640 1 librados: calling monclient init 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.838+0000 7fda8ab1d640 10 monclient: init 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.838+0000 7fda8ab1d640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda840a96f0 0x7fda84133280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 10 monclient(hunting): picked mon.noname-a con 0x7fda840a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda840a9ad0 0x7fda8413e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 10 monclient(hunting): picked mon.noname-c con 0x7fda840a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda84132360 0x7fda841380a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 10 monclient(hunting): picked mon.noname-b con 0x7fda84132360 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:16.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8413e970 con 0x7fda840a96f0 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84132b50 con 0x7fda84132360 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda84134500 con 0x7fda840a9ad0 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda8ab1d640 10 monclient(hunting): authenticate will time out at 1840.190255s 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda888b9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda840a96f0 0x7fda84133280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda83fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda840a9ad0 0x7fda8413e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda890ba640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda84132360 0x7fda841380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda888b9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda840a96f0 0x7fda84133280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47572/0 (socket says 172.21.15.110:47572) 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda888b9640 1 -- 172.21.15.110:0/3860727234 learned_addr learned my addr 172.21.15.110:0/3860727234 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda888b9640 10 monclient(hunting): get_auth_request con 0x7fda840a96f0 auth_method 0 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda888b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda888b9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda888b9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda890ba640 10 monclient(hunting): get_auth_request con 0x7fda84132360 auth_method 0 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda890ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda890ba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda890ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda83fff640 10 monclient(hunting): get_auth_request con 0x7fda840a9ad0 auth_method 0 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda83fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.839+0000 7fda83fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.840+0000 7fda888b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.840+0000 7fda888b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.840+0000 7fda888b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.840+0000 7fda890ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.840+0000 7fda890ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:16.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.840+0000 7fda890ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:16.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.840+0000 7fda83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:16.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.840+0000 7fda83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:16.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.840+0000 7fda83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:16.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 10 monclient(hunting): handle_auth_done global_id 5645 payload 824 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 1 -- 172.21.15.110:0/3860727234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda84132360 msgr2=0x7fda841380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda84132360 0x7fda841380a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 1 -- 172.21.15.110:0/3860727234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda840a96f0 msgr2=0x7fda84133280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda840a96f0 0x7fda84133280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 1 monclient: found mon.noname-c 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.843+0000 7fda83fff640 1 -- 172.21.15.110:0/3860727234 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda700094d0 con 0x7fda840a9ad0 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.845+0000 7fda83fff640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda840a9ad0 0x7fda8413e430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fda84067a00 tx=0x7fda700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.845+0000 7fda817fa640 10 client.?.objecter ms_handle_connect 0x7fda840a9ad0 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.845+0000 7fda817fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 1 -- 172.21.15.110:0/3860727234 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7003aca0 con 0x7fda840a9ad0 2024-01-20T11:23:16.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 10 monclient: dump: 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 10 monclient: _finish_auth 0 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 10 monclient: _check_auth_tickets 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 1 -- 172.21.15.110:0/3860727234 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7003ae00 con 0x7fda840a9ad0 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda8ab1d640 5 monclient: authenticate success, global_id 5645 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda8ab1d640 10 monclient: _renew_subs 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda8ab1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda84132d60 con 0x7fda840a9ad0 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 1 -- 172.21.15.110:0/3860727234 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70044630 con 0x7fda840a9ad0 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda817fa640 10 monclient: dump: 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:16.852 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda82ffd640 4 set_mon_vals no callback set 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda8ab1d640 10 client.5645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda8ab1d640 10 monclient: _renew_subs 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda8ab1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda841385e0 con 0x7fda840a9ad0 2024-01-20T11:23:16.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.848+0000 7fda8ab1d640 1 librados: init done 2024-01-20T11:23:16.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.849+0000 7fda8ab1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:16.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.849+0000 7fda8ab1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:16.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.849+0000 7fda8ab1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:16.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.849+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda84132fa0 con 0x7fda840a9ad0 2024-01-20T11:23:16.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.868+0000 7fda817fa640 1 -- 172.21.15.110:0/3860727234 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fda7004d450 con 0x7fda840a9ad0 2024-01-20T11:23:16.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.868+0000 7fda817fa640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda6806c8e0 0x7fda6806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:16.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.869+0000 7fda888b9640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda6806c8e0 0x7fda6806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:16.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.869+0000 7fda888b9640 10 monclient: get_auth_request con 0x7fda6806c8e0 auth_method 0 2024-01-20T11:23:16.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.869+0000 7fda888b9640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda6806c8e0 0x7fda6806eda0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fda78006fd0 tx=0x7fda78008040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:16.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.869+0000 7fda817fa640 10 client.5645.objecter ms_handle_connect 0x7fda6806c8e0 2024-01-20T11:23:16.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.870+0000 7fda817fa640 1 -- 172.21.15.110:0/3860727234 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fda7002d040 con 0x7fda840a9ad0 2024-01-20T11:23:16.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.870+0000 7fda817fa640 10 client.5645.objecter ms_dispatch 0x7fda84000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:16.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.870+0000 7fda817fa640 3 client.5645.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:16.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.870+0000 7fda817fa640 3 client.5645.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:16.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:16.871+0000 7fda817fa640 20 client.5645.objecter dump_active .. 0 homeless 2024-01-20T11:23:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.103+0000 7fda817fa640 1 -- 172.21.15.110:0/3860727234 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fda70076c00 con 0x7fda840a9ad0 2024-01-20T11:23:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.103+0000 7fda817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:17.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.103+0000 7fda817fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:17.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.295+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fda84132fa0 con 0x7fda6806c8e0 2024-01-20T11:23:17.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.298+0000 7fda817fa640 1 -- 172.21.15.110:0/3860727234 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63767 (secure 0 0 0) 0x7fda84132fa0 con 0x7fda6806c8e0 2024-01-20T11:23:17.299 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:17.299 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.299+0000 7fda8ab1d640 10 librados: watch_flush enter 2024-01-20T11:23:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 10 librados: watch_flush exit 2024-01-20T11:23:17.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 20 client.5645.objecter shutdown clearing up homeless session... 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 10 client.5645.objecter successfully canceled tick 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda6806c8e0 msgr2=0x7fda6806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda6806c8e0 0x7fda6806eda0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fda78006fd0 tx=0x7fda78008040 comp rx=0 tx=0).stop 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 10 monclient: shutdown 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda840a9ad0 msgr2=0x7fda8413e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda840a9ad0 0x7fda8413e430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fda84067a00 tx=0x7fda700046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 shutdown_connections 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fda6806c8e0 0x7fda6806eda0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda84132360 0x7fda841380a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fda840a9ad0 0x7fda8413e430 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 --2- 172.21.15.110:0/3860727234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda840a96f0 0x7fda84133280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 >> 172.21.15.110:0/3860727234 conn(0x7fda840b7bc0 msgr2=0x7fda84136a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 shutdown_connections 2024-01-20T11:23:17.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 -- 172.21.15.110:0/3860727234 wait complete. 2024-01-20T11:23:17.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:17.300+0000 7fda8ab1d640 1 librados: shutdown 2024-01-20T11:23:17.311 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-01-20T11:23:16.809086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002756"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":130,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:19:18.003350+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:19:18.003350+0000","last_peered":"2024-01-20T11:19:18.003350+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:19:18.003350+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":6.0599999999999996},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":6.6120000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":8.3000000000000007},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":7.8410000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":11.509},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":9.984}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":9.0990000000000002},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":13.554}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":15.861000000000001},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":10.657}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":15.214},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":16.599}]}]},{"osd":7,"up_from":21,"seq":90194313317,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":11.593999999999999},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.3209999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":7.218},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.427}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.901},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.65}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":11.256},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":17.222000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.468999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":9.0709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":9.8710000000000004},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.138999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313317,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.3840000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":4.7300000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":8.1329999999999991},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":11.378}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":9.1300000000000008},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":9.9510000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":14.657999999999999},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.746}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":12.166},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":10.615}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":16.945},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.676}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444131,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":10.401},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":8.0690000000000008}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.8780000000000001},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.212}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":13.965},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":7.7110000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.563000000000001},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.019}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":11.597},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.173}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":9.7650000000000006},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":14.779999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444132,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":4.9039999999999999},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":8.234}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.3760000000000003},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.407999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":12.459},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":14.41}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.794},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":13.682}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.7230000000000008},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":15.414}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":9.0839999999999996},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":16.216000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444132,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":7.1589999999999998},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":4.2320000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.2829999999999995},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":10.047000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":13.125},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":14.638}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":14.053000000000001},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":8.6180000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":15.407},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":17.568999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":18.556999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.972}]}]},{"osd":6,"up_from":15,"seq":64424509541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":6.1689999999999996},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.1080000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.4589999999999996},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.067}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":10.823},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.5170000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.5259999999999998},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.282}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":8.8079999999999998},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.198}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":15.077999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":11.441000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:18.859 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~261s 2024-01-20T11:23:20.313 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:20.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.413+0000 7f69a2215640 10 monclient: get_monmap_and_config 2024-01-20T11:23:20.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.413+0000 7f69a2215640 10 monclient: build_initial_monmap 2024-01-20T11:23:20.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.413+0000 7f69a2215640 10 monclient: monmap: 2024-01-20T11:23:20.414 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:20.414 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.414 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:20.414760+0000 2024-01-20T11:23:20.414 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:20.414760+0000 2024-01-20T11:23:20.414 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:20.414 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.415 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:20.415 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:20.415 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:20.415 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:20.415 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:20.415 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.413+0000 7f69a2215640 1 Processor -- start 2024-01-20T11:23:20.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.413+0000 7f69a2215640 1 -- start start 2024-01-20T11:23:20.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.413+0000 7f69a2215640 10 monclient: init 2024-01-20T11:23:20.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:20.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-01-20T11:23:20.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f699c0a96f0 0x7f699c13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 10 monclient(hunting): picked mon.noname-a con 0x7f699c0a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:20.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f699c13fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 0x7f699c142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 10 monclient(hunting): picked mon.noname-b con 0x7f699c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699c0b25c0 con 0x7f699c13fa10 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699c0aaea0 con 0x7f699c0a96f0 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c0ab0a0 con 0x7f699c0a9ad0 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.414+0000 7f69a2215640 10 monclient(hunting): authenticate will time out at 1843.765693s 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f699c0a96f0 0x7f699c13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 0x7f699c142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699affd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f699c13fa10 0x7f699c1423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:54968/0 (socket says 172.21.15.110:54968) 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699affd640 1 -- 172.21.15.110:0/728908188 learned_addr learned my addr 172.21.15.110:0/728908188 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699b7fe640 10 monclient(hunting): get_auth_request con 0x7f699c0a96f0 auth_method 0 2024-01-20T11:23:20.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699bfff640 10 monclient(hunting): get_auth_request con 0x7f699c0a9ad0 auth_method 0 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.415+0000 7f699b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699b7fe640 10 monclient(hunting): handle_auth_done global_id 6009 payload 274 2024-01-20T11:23:20.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699b7fe640 1 -- 172.21.15.110:0/728908188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 msgr2=0x7f699c142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:20.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699b7fe640 1 --2- 172.21.15.110:0/728908188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 0x7f699c142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699b7fe640 1 -- 172.21.15.110:0/728908188 >> v1:172.21.15.134:6789/0 conn(0x7f699c13fa10 legacy=0x7f699c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:20.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699b7fe640 1 monclient: found mon.noname-a 2024-01-20T11:23:20.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:20.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:20.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699b7fe640 1 -- 172.21.15.110:0/728908188 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c0b3fb0 con 0x7f699c0a96f0 2024-01-20T11:23:20.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.417+0000 7f699bfff640 1 --2- 172.21.15.110:0/728908188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 0x7f699c142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:20.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.419+0000 7f699b7fe640 1 --2- 172.21.15.110:0/728908188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 0x7f699c13f4d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6994009d90 tx=0x7f699402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511160525351691a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:20.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 1 -- 172.21.15.110:0/728908188 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699402cbc0 con 0x7f699c0a96f0 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 10 monclient: dump: 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:20.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 1 -- 172.21.15.110:0/728908188 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699402cd20 con 0x7f699c0a96f0 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 1 -- 172.21.15.110:0/728908188 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6994034660 con 0x7f699c0a96f0 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f699a7fc640 10 monclient: dump: 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:20.426 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f69a2215640 5 monclient: authenticate success, global_id 6009 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f69a2215640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.422+0000 7f69a2215640 4 set_mon_vals no callback set 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 10 monclient: shutdown 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 1 -- 172.21.15.110:0/728908188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 msgr2=0x7f699c13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 1 --2- 172.21.15.110:0/728908188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 0x7f699c13f4d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6994009d90 tx=0x7f699402be10 comp rx=0 tx=0).stop 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 1 -- 172.21.15.110:0/728908188 shutdown_connections 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 1 --2- 172.21.15.110:0/728908188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 0x7f699c142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 1 --2- 172.21.15.110:0/728908188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 0x7f699c13f4d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 1 -- 172.21.15.110:0/728908188 >> 172.21.15.110:0/728908188 conn(0x7f699c0b7bc0 msgr2=0x7f699c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 1 -- 172.21.15.110:0/728908188 shutdown_connections 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.423+0000 7f69a2215640 1 -- 172.21.15.110:0/728908188 wait complete. 2024-01-20T11:23:20.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.424+0000 7f69a2215640 10 monclient: build_initial_monmap 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.424+0000 7f69a2215640 10 monclient: monmap: 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:20.428 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.424+0000 7f69a2215640 1 librados: starting msgr at 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.424+0000 7f69a2215640 1 librados: starting objecter 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.424+0000 7f69a2215640 1 Processor -- start 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.424+0000 7f69a2215640 1 -- start start 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.424+0000 7f69a2215640 1 librados: setting wanted keys 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.424+0000 7f69a2215640 1 librados: calling monclient init 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.424+0000 7f69a2215640 10 monclient: init 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 0x7f699c136360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient(hunting): picked mon.noname-a con 0x7f699c0a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 0x7f699c140290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient(hunting): picked mon.noname-c con 0x7f699c0a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c13d760 0x7f699c12ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient(hunting): picked mon.noname-b con 0x7f699c13d760 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c13fe00 con 0x7f699c0a96f0 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c1407d0 con 0x7f699c13d760 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c13d340 con 0x7f699c0a9ad0 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f69a2215640 10 monclient(hunting): authenticate will time out at 1843.776666s 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.425+0000 7f699b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 0x7f699c136360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 0x7f699c140290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c13d760 0x7f699c12ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 0x7f699c136360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50392/0 (socket says 172.21.15.110:50392) 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699b7fe640 1 -- 172.21.15.110:0/3721862348 learned_addr learned my addr 172.21.15.110:0/3721862348 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699b7fe640 10 monclient(hunting): get_auth_request con 0x7f699c0a96f0 auth_method 0 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699bfff640 10 monclient(hunting): get_auth_request con 0x7f699c13d760 auth_method 0 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699affd640 10 monclient(hunting): get_auth_request con 0x7f699c0a9ad0 auth_method 0 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.426+0000 7f699affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 10 monclient(hunting): handle_auth_done global_id 6112 payload 824 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 1 -- 172.21.15.110:0/3721862348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 msgr2=0x7f699c140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 0x7f699c140290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 1 -- 172.21.15.110:0/3721862348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 msgr2=0x7f699c136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 0x7f699c136360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 1 monclient: found mon.noname-b 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699bfff640 1 -- 172.21.15.110:0/3721862348 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c009400 con 0x7f699c13d760 2024-01-20T11:23:20.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.429+0000 7f699b7fe640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 0x7f699c136360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.432+0000 7f699bfff640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c13d760 0x7f699c12ee70 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f698c00e880 tx=0x7f698c00ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.432+0000 7f6983fff640 10 client.?.objecter ms_handle_connect 0x7f699c13d760 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.432+0000 7f6983fff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f6983fff640 1 -- 172.21.15.110:0/3721862348 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c004020 con 0x7f699c13d760 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f6983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f6983fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f6983fff640 10 monclient: dump: 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.437 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f6983fff640 10 monclient: _finish_auth 0 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f6983fff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f6983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f69a2215640 5 monclient: authenticate success, global_id 6112 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f69a2215640 10 monclient: _renew_subs 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f69a2215640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6994009a70 con 0x7f699c13d760 2024-01-20T11:23:20.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f69a2215640 10 client.6112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f69a2215640 10 monclient: _renew_subs 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f69a2215640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c13cf60 con 0x7f699c13d760 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.434+0000 7f69a2215640 1 librados: init done 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f69a2215640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f69a2215640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f69a2215640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c13eef0 con 0x7f699c13d760 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f6983fff640 1 -- 172.21.15.110:0/3721862348 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c004180 con 0x7f699c13d760 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f6983fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f6983fff640 1 -- 172.21.15.110:0/3721862348 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c0044f0 con 0x7f699c13d760 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f6983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f6983fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f6983fff640 10 monclient: dump: 2024-01-20T11:23:20.439 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.435+0000 7f6999ffb640 4 set_mon_vals no callback set 2024-01-20T11:23:20.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.674+0000 7f6983fff640 1 -- 172.21.15.110:0/3721862348 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f698c00c940 con 0x7f699c13d760 2024-01-20T11:23:20.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.674+0000 7f6983fff640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f698806c860 0x7f698806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.674+0000 7f699b7fe640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f698806c860 0x7f698806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.674+0000 7f699b7fe640 10 monclient: get_auth_request con 0x7f698806c860 auth_method 0 2024-01-20T11:23:20.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.675+0000 7f699b7fe640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f698806c860 0x7f698806ed20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6994000f80 tx=0x7f6994030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:20.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.675+0000 7f6983fff640 10 client.6112.objecter ms_handle_connect 0x7f698806c860 2024-01-20T11:23:20.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.677+0000 7f6983fff640 1 -- 172.21.15.110:0/3721862348 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f698c0588c0 con 0x7f699c13d760 2024-01-20T11:23:20.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.677+0000 7f6983fff640 10 client.6112.objecter ms_dispatch 0x7f699c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:20.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.677+0000 7f6983fff640 3 client.6112.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:20.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.677+0000 7f6983fff640 3 client.6112.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:20.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.677+0000 7f6983fff640 20 client.6112.objecter dump_active .. 0 homeless 2024-01-20T11:23:20.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.689+0000 7f6983fff640 1 -- 172.21.15.110:0/3721862348 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f698c054c00 con 0x7f699c13d760 2024-01-20T11:23:20.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.689+0000 7f6983fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:20.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.689+0000 7f6983fff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:20.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.856+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f699c13ff70 con 0x7f698806c860 2024-01-20T11:23:20.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.859+0000 7f6983fff640 1 -- 172.21.15.110:0/3721862348 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63775 (secure 0 0 0) 0x7f699c13ff70 con 0x7f698806c860 2024-01-20T11:23:20.860 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:20.860 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 10 librados: watch_flush enter 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 10 librados: watch_flush exit 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 20 client.6112.objecter shutdown clearing up homeless session... 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 10 client.6112.objecter successfully canceled tick 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f698806c860 msgr2=0x7f698806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f698806c860 0x7f698806ed20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6994000f80 tx=0x7f6994030000 comp rx=0 tx=0).stop 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 10 monclient: shutdown 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c13d760 msgr2=0x7f699c12ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.860+0000 7f69a2215640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c13d760 0x7f699c12ee70 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f698c00e880 tx=0x7f698c00ed40 comp rx=0 tx=0).stop 2024-01-20T11:23:20.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.861+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 shutdown_connections 2024-01-20T11:23:20.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.861+0000 7f69a2215640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f698806c860 0x7f698806ed20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.861+0000 7f69a2215640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c13d760 0x7f699c12ee70 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.861+0000 7f69a2215640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f699c0a9ad0 0x7f699c140290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.861+0000 7f69a2215640 1 --2- 172.21.15.110:0/3721862348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f699c0a96f0 0x7f699c136360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.861+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 >> 172.21.15.110:0/3721862348 conn(0x7f699c0b7bc0 msgr2=0x7f699c137440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:20.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.861+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 shutdown_connections 2024-01-20T11:23:20.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.861+0000 7f69a2215640 1 -- 172.21.15.110:0/3721862348 wait complete. 2024-01-20T11:23:20.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.861+0000 7f69a2215640 1 librados: shutdown 2024-01-20T11:23:20.872 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-01-20T11:23:20.810212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002993"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":12.379},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.5529999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":6.6890000000000001},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":14.098000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":9.8780000000000001},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":10.829000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.512},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":8.6280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":7.3840000000000003},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.768000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":11.66},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":7.9509999999999996}]}]},{"osd":7,"up_from":21,"seq":90194313317,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":11.593999999999999},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.3209999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":7.218},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.427}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.901},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.65}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":11.256},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":17.222000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.468999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":9.0709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":9.8710000000000004},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.138999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313318,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0010000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":6.625}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":9.016},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":16.451000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.231},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":8.0960000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":13.948},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.77}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":11.194000000000001},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":12.135}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.363},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.653}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444132,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.8029999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":11.372}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.7350000000000003},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":13.167}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.141},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.5269999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.5},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.518000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":12.375999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.8959999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":9.1379999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":14.957000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444133,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.806},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.7670000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":8.9399999999999995},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.932}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":9.5269999999999992},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":11.561}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.3889999999999993},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.406000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":15.465},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":12.247}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":16.297999999999998},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":12.887}]}]},{"osd":0,"up_from":17,"seq":73014444133,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.7110000000000003},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.5800000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":10.916},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":14.475}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":11.577999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.4440000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":10.852},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":12.337}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.763999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.175000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.32},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.705}]}]},{"osd":6,"up_from":15,"seq":64424509541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":6.1689999999999996},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.1080000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.4589999999999996},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.067}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":10.823},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.5170000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.5259999999999998},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.282}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":8.8079999999999998},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.198}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":15.077999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":11.441000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:20.873 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.971+0000 7f273eaa9640 10 monclient: get_monmap_and_config 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.971+0000 7f273eaa9640 10 monclient: build_initial_monmap 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.971+0000 7f273eaa9640 10 monclient: monmap: 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:20.972785+0000 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:20.972785+0000 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:20.972 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:20.973 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:20.973 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:20.973 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.971+0000 7f273eaa9640 1 Processor -- start 2024-01-20T11:23:20.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.971+0000 7f273eaa9640 1 -- start start 2024-01-20T11:23:20.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.971+0000 7f273eaa9640 10 monclient: init 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f27380a96f0 0x7f273813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 10 monclient(hunting): picked mon.noname-a con 0x7f27380a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f273813fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a9ad0 0x7f2738142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 10 monclient(hunting): picked mon.noname-b con 0x7f27380a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27380b25c0 con 0x7f273813fa10 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27380aaea0 con 0x7f27380a96f0 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27380ab0a0 con 0x7f27380a9ad0 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.972+0000 7f273eaa9640 10 monclient(hunting): authenticate will time out at 1844.323765s 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273c845640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f27380a96f0 0x7f273813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273d046640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a9ad0 0x7f2738142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f2737fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f273813fa10 0x7f27381423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:54972/0 (socket says 172.21.15.110:54972) 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f2737fff640 1 -- 172.21.15.110:0/1559976797 learned_addr learned my addr 172.21.15.110:0/1559976797 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273c845640 10 monclient(hunting): get_auth_request con 0x7f27380a96f0 auth_method 0 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273c845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273c845640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273c845640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273d046640 10 monclient(hunting): get_auth_request con 0x7f27380a9ad0 auth_method 0 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273d046640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273d046640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273d046640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273c845640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273c845640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.973+0000 7f273c845640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.974+0000 7f273d046640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.974+0000 7f273d046640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.974+0000 7f273d046640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f27377fe640 1 -- 172.21.15.110:0/1559976797 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391328973 0 0) 0x7f27380b25c0 con 0x7f273813fa10 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f27377fe640 10 monclient(hunting): _negotiate 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f27377fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f27377fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f27377fe640 10 monclient(hunting): my global_id is 6115 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f27377fe640 1 -- 172.21.15.110:0/1559976797 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2728003520 con 0x7f273813fa10 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f273d046640 10 monclient(hunting): handle_auth_done global_id 5654 payload 274 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f273d046640 1 -- 172.21.15.110:0/1559976797 >> v2:172.21.15.110:3300/0 conn(0x7f27380a96f0 msgr2=0x7f273813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f273d046640 1 --2- 172.21.15.110:0/1559976797 >> v2:172.21.15.110:3300/0 conn(0x7f27380a96f0 0x7f273813f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f273d046640 1 -- 172.21.15.110:0/1559976797 >> v1:172.21.15.134:6789/0 conn(0x7f273813fa10 legacy=0x7f27381423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f273d046640 10 monclient: _finish_hunting 0 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f273d046640 1 monclient: found mon.noname-b 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f273d046640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f273d046640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:20.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.975+0000 7f273d046640 1 -- 172.21.15.110:0/1559976797 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27380b3fb0 con 0x7f27380a9ad0 2024-01-20T11:23:20.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.978+0000 7f273d046640 1 --2- 172.21.15.110:0/1559976797 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a9ad0 0x7f2738142af0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2730009d90 tx=0x7f273002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff0ad2ae42f409b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:20.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 1 -- 172.21.15.110:0/1559976797 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273002cbc0 con 0x7f27380a9ad0 2024-01-20T11:23:20.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:20.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:20.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 10 monclient: dump: 2024-01-20T11:23:20.982 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 1 -- 172.21.15.110:0/1559976797 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273002cd20 con 0x7f27380a9ad0 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f273eaa9640 5 monclient: authenticate success, global_id 5654 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f273eaa9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f273eaa9640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f273eaa9640 4 set_mon_vals no callback set 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 1 -- 172.21.15.110:0/1559976797 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730034660 con 0x7f27380a9ad0 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.981+0000 7f27377fe640 10 monclient: dump: 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:20.984 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:20.985 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:20.985 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 10 monclient: shutdown 2024-01-20T11:23:20.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 1 -- 172.21.15.110:0/1559976797 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a9ad0 msgr2=0x7f2738142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 1 --2- 172.21.15.110:0/1559976797 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a9ad0 0x7f2738142af0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2730009d90 tx=0x7f273002be10 comp rx=0 tx=0).stop 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 1 -- 172.21.15.110:0/1559976797 shutdown_connections 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 1 --2- 172.21.15.110:0/1559976797 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a9ad0 0x7f2738142af0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 1 --2- 172.21.15.110:0/1559976797 >> v2:172.21.15.110:3300/0 conn(0x7f27380a96f0 0x7f273813f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 1 -- 172.21.15.110:0/1559976797 >> 172.21.15.110:0/1559976797 conn(0x7f27380b7bc0 msgr2=0x7f27380b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 1 -- 172.21.15.110:0/1559976797 shutdown_connections 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.982+0000 7f273eaa9640 1 -- 172.21.15.110:0/1559976797 wait complete. 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.983+0000 7f273eaa9640 10 monclient: build_initial_monmap 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.983+0000 7f273eaa9640 10 monclient: monmap: 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:20.988 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.983+0000 7f273eaa9640 1 librados: starting msgr at 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.983+0000 7f273eaa9640 1 librados: starting objecter 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.983+0000 7f273eaa9640 1 Processor -- start 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.983+0000 7f273eaa9640 1 -- start start 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.983+0000 7f273eaa9640 1 librados: setting wanted keys 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.983+0000 7f273eaa9640 1 librados: calling monclient init 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.983+0000 7f273eaa9640 10 monclient: init 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:20.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a96f0 0x7f273813a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient(hunting): picked mon.noname-c con 0x7f27380a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27380a9ad0 0x7f27381d7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient(hunting): picked mon.noname-b con 0x7f27380a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27380ab830 0x7f27380abcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient(hunting): picked mon.noname-a con 0x7f27380ab830 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27381d7f20 con 0x7f27380ab830 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380ac760 con 0x7f27380a9ad0 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27380ac920 con 0x7f27380a96f0 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:20.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273eaa9640 10 monclient(hunting): authenticate will time out at 1844.335479s 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273c845640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a96f0 0x7f273813a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.984+0000 7f273d046640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27380ab830 0x7f27380abcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273c845640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a96f0 0x7f273813a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57044/0 (socket says 172.21.15.110:57044) 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273c845640 1 -- 172.21.15.110:0/3250244521 learned_addr learned my addr 172.21.15.110:0/3250244521 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273d046640 10 monclient(hunting): get_auth_request con 0x7f27380ab830 auth_method 0 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273d046640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273d046640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273d046640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273c845640 10 monclient(hunting): get_auth_request con 0x7f27380a96f0 auth_method 0 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273c845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273c845640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273c845640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f2737fff640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27380a9ad0 0x7f27381d7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f2737fff640 10 monclient(hunting): get_auth_request con 0x7f27380a9ad0 auth_method 0 2024-01-20T11:23:20.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f2737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f2737fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f2737fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273c845640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273c845640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273c845640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273d046640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273d046640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f273d046640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f2737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f2737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:20.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.985+0000 7f2737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 10 monclient(hunting): handle_auth_done global_id 5657 payload 824 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 1 -- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27380a9ad0 msgr2=0x7f27381d7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27380a9ad0 0x7f27381d7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 1 -- 172.21.15.110:0/3250244521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27380ab830 msgr2=0x7f27380abcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27380ab830 0x7f27380abcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 10 monclient: _finish_hunting 0 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 1 monclient: found mon.noname-c 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:23:20.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.988+0000 7f273c845640 1 -- 172.21.15.110:0/3250244521 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2730009a70 con 0x7f27380a96f0 2024-01-20T11:23:20.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.990+0000 7f273c845640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a96f0 0x7f273813a720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f272c00e9b0 tx=0x7f272c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:20.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.990+0000 7f27357fa640 10 client.?.objecter ms_handle_connect 0x7f27380a96f0 2024-01-20T11:23:20.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.990+0000 7f27357fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:20.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 1 -- 172.21.15.110:0/3250244521 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c009670 con 0x7f27380a96f0 2024-01-20T11:23:20.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:20.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:20.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 10 monclient: dump: 2024-01-20T11:23:20.995 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:20.995 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.995 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 10 monclient: _finish_auth 0 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 10 monclient: _check_auth_tickets 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 1 -- 172.21.15.110:0/3250244521 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272c00c9a0 con 0x7f27380a96f0 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 1 -- 172.21.15.110:0/3250244521 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c00cd50 con 0x7f27380a96f0 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:20.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f27357fa640 10 monclient: dump: 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f273eaa9640 5 monclient: authenticate success, global_id 5657 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f273eaa9640 10 monclient: _renew_subs 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f273eaa9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:20.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27381d7a70 con 0x7f27380a96f0 2024-01-20T11:23:20.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f2736ffd640 4 set_mon_vals no callback set 2024-01-20T11:23:20.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f273eaa9640 10 client.5657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:20.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f273eaa9640 10 monclient: _renew_subs 2024-01-20T11:23:20.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f273eaa9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:20.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27380ac3d0 con 0x7f27380a96f0 2024-01-20T11:23:20.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.994+0000 7f273eaa9640 1 librados: init done 2024-01-20T11:23:20.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.995+0000 7f273eaa9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:20.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.995+0000 7f273eaa9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:20.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.995+0000 7f273eaa9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:20.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:20.995+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27380b3da0 con 0x7f27380a96f0 2024-01-20T11:23:21.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.238+0000 7f27357fa640 1 -- 172.21.15.110:0/3250244521 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f272c013470 con 0x7f27380a96f0 2024-01-20T11:23:21.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.238+0000 7f27357fa640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f271c06c8b0 0x7f271c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:21.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.238+0000 7f2737fff640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f271c06c8b0 0x7f271c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:21.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.238+0000 7f2737fff640 10 monclient: get_auth_request con 0x7f271c06c8b0 auth_method 0 2024-01-20T11:23:21.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.239+0000 7f27357fa640 10 client.5657.objecter ms_handle_connect 0x7f271c06c8b0 2024-01-20T11:23:21.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.239+0000 7f2737fff640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f271c06c8b0 0x7f271c06ed70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f27380b5c70 tx=0x7f2724009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:21.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.251+0000 7f27357fa640 1 -- 172.21.15.110:0/3250244521 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f272c058bd0 con 0x7f27380a96f0 2024-01-20T11:23:21.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.251+0000 7f27357fa640 10 client.5657.objecter ms_dispatch 0x7f2738000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:21.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.251+0000 7f27357fa640 3 client.5657.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:21.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.251+0000 7f27357fa640 3 client.5657.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:21.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.251+0000 7f27357fa640 20 client.5657.objecter dump_active .. 0 homeless 2024-01-20T11:23:21.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.251+0000 7f27357fa640 1 -- 172.21.15.110:0/3250244521 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f272c054f10 con 0x7f27380a96f0 2024-01-20T11:23:21.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.251+0000 7f27357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:21.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.251+0000 7f27357fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:21.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.445+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2738075490 con 0x7f271c06c8b0 2024-01-20T11:23:21.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.448+0000 7f27357fa640 1 -- 172.21.15.110:0/3250244521 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63775 (secure 0 0 0) 0x7f2738075490 con 0x7f271c06c8b0 2024-01-20T11:23:21.449 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 10 librados: watch_flush enter 2024-01-20T11:23:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 10 librados: watch_flush exit 2024-01-20T11:23:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 20 client.5657.objecter shutdown clearing up homeless session... 2024-01-20T11:23:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 10 client.5657.objecter successfully canceled tick 2024-01-20T11:23:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f271c06c8b0 msgr2=0x7f271c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f271c06c8b0 0x7f271c06ed70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f27380b5c70 tx=0x7f2724009040 comp rx=0 tx=0).stop 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 10 monclient: shutdown 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a96f0 msgr2=0x7f273813a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a96f0 0x7f273813a720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f272c00e9b0 tx=0x7f272c00ee70 comp rx=0 tx=0).stop 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 shutdown_connections 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f271c06c8b0 0x7f271c06ed70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27380ab830 0x7f27380abcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27380a9ad0 0x7f27381d7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 --2- 172.21.15.110:0/3250244521 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27380a96f0 0x7f273813a720 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 >> 172.21.15.110:0/3250244521 conn(0x7f27380b7bc0 msgr2=0x7f2738141460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 shutdown_connections 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 -- 172.21.15.110:0/3250244521 wait complete. 2024-01-20T11:23:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.450+0000 7f273eaa9640 1 librados: shutdown 2024-01-20T11:23:21.462 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-01-20T11:23:20.810212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002993"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":12.379},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.5529999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":6.6890000000000001},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":14.098000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":9.8780000000000001},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":10.829000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.512},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":8.6280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":7.3840000000000003},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.768000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":11.66},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":7.9509999999999996}]}]},{"osd":7,"up_from":21,"seq":90194313317,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":11.593999999999999},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.3209999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":7.218},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.427}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.901},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.65}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":11.256},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":17.222000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.468999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":9.0709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":9.8710000000000004},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.138999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313318,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0010000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":6.625}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":9.016},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":16.451000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.231},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":8.0960000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":13.948},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.77}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":11.194000000000001},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":12.135}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.363},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.653}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444132,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.8029999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":11.372}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.7350000000000003},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":13.167}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.141},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.5269999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.5},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.518000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":12.375999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.8959999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":9.1379999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":14.957000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444133,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.806},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.7670000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":8.9399999999999995},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.932}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":9.5269999999999992},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":11.561}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.3889999999999993},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.406000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":15.465},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":12.247}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":16.297999999999998},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":12.887}]}]},{"osd":0,"up_from":17,"seq":73014444133,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.7110000000000003},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.5800000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":10.916},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":14.475}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":11.577999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.4440000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":10.852},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":12.337}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.763999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.175000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.32},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.705}]}]},{"osd":6,"up_from":15,"seq":64424509541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":6.1689999999999996},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.1080000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.4589999999999996},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.067}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":10.823},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.5170000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.5259999999999998},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.282}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":8.8079999999999998},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.198}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":15.077999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":11.441000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:21.463 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:21.463 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:21.464 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:21.464 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:21.464 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:21.464 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:21.464 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:21.464 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 131, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:21:18.077638+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:21:18.077638+0000', 'last_peered': '2024-01-20T11:21:18.077638+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:21:18.077638+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:21.464 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:21.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.561+0000 7f8c678cf640 10 monclient: get_monmap_and_config 2024-01-20T11:23:21.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.561+0000 7f8c678cf640 10 monclient: build_initial_monmap 2024-01-20T11:23:21.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.561+0000 7f8c678cf640 10 monclient: monmap: 2024-01-20T11:23:21.562 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:21.562 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:21.562 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:21.563035+0000 2024-01-20T11:23:21.562 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:21.563035+0000 2024-01-20T11:23:21.562 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:21.563 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:21.563 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:21.563 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:21.563 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:21.563 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:21.563 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:21.563 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:21.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.561+0000 7f8c678cf640 1 Processor -- start 2024-01-20T11:23:21.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 1 -- start start 2024-01-20T11:23:21.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 10 monclient: init 2024-01-20T11:23:21.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:21.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-01-20T11:23:21.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c600a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:21.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f8c600a9ad0 0x7f8c601423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:21.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 10 monclient(hunting): picked mon.noname-c con 0x7f8c600a9ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c60142920 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c600b25c0 con 0x7f8c60142920 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6013fbe0 con 0x7f8c600a96f0 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c600aaea0 con 0x7f8c600a9ad0 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.562+0000 7f8c678cf640 10 monclient(hunting): authenticate will time out at 1844.914031s 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.563+0000 7f8c65e6c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8c60142920 0x7f8c6014c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55718/0 (socket says 172.21.15.110:55718) 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.563+0000 7f8c65e6c640 1 -- 172.21.15.110:0/765901934 learned_addr learned my addr 172.21.15.110:0/765901934 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:21.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.563+0000 7f8c64e6a640 1 --2- 172.21.15.110:0/765901934 >> v2:172.21.15.134:3300/0 conn(0x7f8c600a9ad0 0x7f8c601423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:21.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.564+0000 7f8c64e6a640 10 monclient(hunting): get_auth_request con 0x7f8c600a9ad0 auth_method 0 2024-01-20T11:23:21.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.564+0000 7f8c64e6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:21.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.564+0000 7f8c64e6a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:21.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.564+0000 7f8c64e6a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.565+0000 7f8c64e6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.565+0000 7f8c64e6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.565+0000 7f8c64e6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.565+0000 7f8c57fff640 1 -- 172.21.15.110:0/765901934 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561891434 0 0) 0x7f8c600b25c0 con 0x7f8c60142920 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.565+0000 7f8c57fff640 10 monclient(hunting): _negotiate 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.565+0000 7f8c57fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.565+0000 7f8c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.565+0000 7f8c57fff640 10 monclient(hunting): my global_id is 6021 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.565+0000 7f8c57fff640 1 -- 172.21.15.110:0/765901934 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c4c003520 con 0x7f8c60142920 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.566+0000 7f8c64e6a640 10 monclient(hunting): handle_auth_done global_id 6121 payload 274 2024-01-20T11:23:21.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.566+0000 7f8c64e6a640 1 -- 172.21.15.110:0/765901934 >> v1:172.21.15.134:6789/0 conn(0x7f8c600a96f0 legacy=0x7f8c6013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:21.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.566+0000 7f8c64e6a640 1 -- 172.21.15.110:0/765901934 >> v1:172.21.15.110:6789/0 conn(0x7f8c60142920 legacy=0x7f8c6014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:21.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.566+0000 7f8c64e6a640 10 monclient: _finish_hunting 0 2024-01-20T11:23:21.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.566+0000 7f8c64e6a640 1 monclient: found mon.noname-c 2024-01-20T11:23:21.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.566+0000 7f8c64e6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:21.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.566+0000 7f8c64e6a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:23:21.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.566+0000 7f8c64e6a640 1 -- 172.21.15.110:0/765901934 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c600ab0a0 con 0x7f8c600a9ad0 2024-01-20T11:23:21.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.571+0000 7f8c64e6a640 1 --2- 172.21.15.110:0/765901934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c600a9ad0 0x7f8c601423e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8c500097a0 tx=0x7f8c5002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6007ffd25a3a1373 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:21.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.573+0000 7f8c57fff640 1 -- 172.21.15.110:0/765901934 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c50031830 con 0x7f8c600a9ad0 2024-01-20T11:23:21.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.573+0000 7f8c57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:21.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.573+0000 7f8c57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:21.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.573+0000 7f8c57fff640 10 monclient: dump: 2024-01-20T11:23:21.574 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:21.574 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:21.574 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.573+0000 7f8c57fff640 10 monclient: _finish_auth 0 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.573+0000 7f8c57fff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.573+0000 7f8c57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.573+0000 7f8c678cf640 5 monclient: authenticate success, global_id 6121 2024-01-20T11:23:21.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.573+0000 7f8c678cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:21.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.575+0000 7f8c57fff640 1 -- 172.21.15.110:0/765901934 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c50031990 con 0x7f8c600a9ad0 2024-01-20T11:23:21.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.575+0000 7f8c57fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:21.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.575+0000 7f8c57fff640 1 -- 172.21.15.110:0/765901934 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c50031d00 con 0x7f8c600a9ad0 2024-01-20T11:23:21.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.575+0000 7f8c57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:21.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.575+0000 7f8c57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:21.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.575+0000 7f8c57fff640 10 monclient: dump: 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.575+0000 7f8c678cf640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.575+0000 7f8c678cf640 4 set_mon_vals no callback set 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 10 monclient: shutdown 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 1 -- 172.21.15.110:0/765901934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c600a9ad0 msgr2=0x7f8c601423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 1 --2- 172.21.15.110:0/765901934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c600a9ad0 0x7f8c601423e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8c500097a0 tx=0x7f8c5002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 1 -- 172.21.15.110:0/765901934 shutdown_connections 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 1 --2- 172.21.15.110:0/765901934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c600a9ad0 0x7f8c601423e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 1 -- 172.21.15.110:0/765901934 >> 172.21.15.110:0/765901934 conn(0x7f8c600b7bc0 msgr2=0x7f8c600b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 1 -- 172.21.15.110:0/765901934 shutdown_connections 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 1 -- 172.21.15.110:0/765901934 wait complete. 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 10 monclient: build_initial_monmap 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.576+0000 7f8c678cf640 10 monclient: monmap: 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:21.581 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 librados: starting msgr at 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 librados: starting objecter 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 Processor -- start 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 -- start start 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 librados: setting wanted keys 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 librados: calling monclient init 2024-01-20T11:23:21.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient: init 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c600a96f0 0x7f8c601381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient(hunting): picked mon.noname-a con 0x7f8c600a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c600a9ad0 0x7f8c60138720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient(hunting): picked mon.noname-c con 0x7f8c600a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c60142920 0x7f8c601dab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient(hunting): picked mon.noname-b con 0x7f8c60142920 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60138c60 con 0x7f8c600a96f0 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c601db810 con 0x7f8c60142920 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60138e10 con 0x7f8c600a9ad0 2024-01-20T11:23:21.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.577+0000 7f8c678cf640 10 monclient(hunting): authenticate will time out at 1844.929051s 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c65e6c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c60142920 0x7f8c601dab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c64e6a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c600a9ad0 0x7f8c60138720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c65e6c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c60142920 0x7f8c601dab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:35020/0 (socket says 172.21.15.110:35020) 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c65e6c640 1 -- 172.21.15.110:0/2884184595 learned_addr learned my addr 172.21.15.110:0/2884184595 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c65e6c640 10 monclient(hunting): get_auth_request con 0x7f8c60142920 auth_method 0 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c65e6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c65e6c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c65e6c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c64e6a640 10 monclient(hunting): get_auth_request con 0x7f8c600a9ad0 auth_method 0 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c64e6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c64e6a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.578+0000 7f8c64e6a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:21.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.579+0000 7f8c65e6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:21.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.579+0000 7f8c65e6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:21.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.579+0000 7f8c65e6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:21.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.579+0000 7f8c64e6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:21.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.579+0000 7f8c64e6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:21.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.579+0000 7f8c64e6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:21.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 10 monclient(hunting): handle_auth_done global_id 6127 payload 824 2024-01-20T11:23:21.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 1 -- 172.21.15.110:0/2884184595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c600a9ad0 msgr2=0x7f8c60138720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:21.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c600a9ad0 0x7f8c60138720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:21.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 1 -- 172.21.15.110:0/2884184595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c600a96f0 msgr2=0x7f8c601381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:21.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c600a96f0 0x7f8c601381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:21.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 10 monclient: _finish_hunting 0 2024-01-20T11:23:21.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 1 monclient: found mon.noname-b 2024-01-20T11:23:21.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:21.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:21.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.581+0000 7f8c65e6c640 1 -- 172.21.15.110:0/2884184595 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c50009400 con 0x7f8c60142920 2024-01-20T11:23:21.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.584+0000 7f8c65e6c640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c60142920 0x7f8c601dab50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8c5800cb40 tx=0x7f8c5800a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:21.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.584+0000 7f8c55ffb640 10 client.?.objecter ms_handle_connect 0x7f8c60142920 2024-01-20T11:23:21.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.584+0000 7f8c55ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:21.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 1 -- 172.21.15.110:0/2884184595 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5800aca0 con 0x7f8c60142920 2024-01-20T11:23:21.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:21.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:21.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 10 monclient: dump: 2024-01-20T11:23:21.589 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:21.589 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:21.589 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:21.589 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:21.589 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 10 monclient: _finish_auth 0 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 1 -- 172.21.15.110:0/2884184595 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5800ae00 con 0x7f8c60142920 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c678cf640 5 monclient: authenticate success, global_id 6127 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c678cf640 10 monclient: _renew_subs 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c678cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c60139090 con 0x7f8c60142920 2024-01-20T11:23:21.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 1 -- 172.21.15.110:0/2884184595 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c58012620 con 0x7f8c60142920 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c55ffb640 10 monclient: dump: 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:21.591 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c56ffd640 4 set_mon_vals no callback set 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c678cf640 10 client.6127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c678cf640 10 monclient: _renew_subs 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c678cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c601db270 con 0x7f8c60142920 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.587+0000 7f8c678cf640 1 librados: init done 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.588+0000 7f8c678cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.588+0000 7f8c678cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.588+0000 7f8c678cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:21.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.588+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c600af0a0 con 0x7f8c60142920 2024-01-20T11:23:21.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.607+0000 7f8c55ffb640 1 -- 172.21.15.110:0/2884184595 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8c58007070 con 0x7f8c60142920 2024-01-20T11:23:21.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.608+0000 7f8c55ffb640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8c4406c860 0x7f8c4406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:21.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.608+0000 7f8c6566b640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8c4406c860 0x7f8c4406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:21.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.608+0000 7f8c6566b640 10 monclient: get_auth_request con 0x7f8c4406c860 auth_method 0 2024-01-20T11:23:21.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.608+0000 7f8c55ffb640 10 client.6127.objecter ms_handle_connect 0x7f8c4406c860 2024-01-20T11:23:21.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.608+0000 7f8c6566b640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8c4406c860 0x7f8c4406ed20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c000f60 tx=0x7f8c5c0079b0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:21.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.611+0000 7f8c55ffb640 1 -- 172.21.15.110:0/2884184595 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8c58058520 con 0x7f8c60142920 2024-01-20T11:23:21.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.611+0000 7f8c55ffb640 10 client.6127.objecter ms_dispatch 0x7f8c60000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:21.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.611+0000 7f8c55ffb640 3 client.6127.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:21.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.611+0000 7f8c55ffb640 3 client.6127.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:21.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.611+0000 7f8c55ffb640 20 client.6127.objecter dump_active .. 0 homeless 2024-01-20T11:23:21.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.843+0000 7f8c55ffb640 1 -- 172.21.15.110:0/2884184595 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8c58054860 con 0x7f8c60142920 2024-01-20T11:23:21.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.843+0000 7f8c55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:21.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:21.843+0000 7f8c55ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:22.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.079+0000 7f8c678cf640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:22.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.079+0000 7f8c678cf640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:22.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.079+0000 7f8c678cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:22.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.079+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f8c600af0a0 con 0x7f8c60142920 2024-01-20T11:23:22.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.085+0000 7f8c55ffb640 1 -- 172.21.15.110:0/2884184595 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f8c600af0a0 con 0x7f8c60142920 2024-01-20T11:23:22.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.085+0000 7f8c55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:22.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.085+0000 7f8c55ffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:22.086 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:22.086 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":536,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:22.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.086+0000 7f8c678cf640 10 librados: watch_flush enter 2024-01-20T11:23:22.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 10 librados: watch_flush exit 2024-01-20T11:23:22.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 20 client.6127.objecter shutdown clearing up homeless session... 2024-01-20T11:23:22.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 10 client.6127.objecter successfully canceled tick 2024-01-20T11:23:22.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8c4406c860 msgr2=0x7f8c4406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:22.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8c4406c860 0x7f8c4406ed20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c000f60 tx=0x7f8c5c0079b0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 10 monclient: shutdown 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c60142920 msgr2=0x7f8c601dab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c60142920 0x7f8c601dab50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8c5800cb40 tx=0x7f8c5800a060 comp rx=0 tx=0).stop 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 shutdown_connections 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f8c4406c860 0x7f8c4406ed20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c60142920 0x7f8c601dab50 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c600a9ad0 0x7f8c60138720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 --2- 172.21.15.110:0/2884184595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c600a96f0 0x7f8c601381e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 >> 172.21.15.110:0/2884184595 conn(0x7f8c600b7bc0 msgr2=0x7f8c6013c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 shutdown_connections 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 -- 172.21.15.110:0/2884184595 wait complete. 2024-01-20T11:23:22.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.087+0000 7f8c678cf640 1 librados: shutdown 2024-01-20T11:23:22.098 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:22.099 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:22.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.196+0000 7f137c1a9640 10 monclient: get_monmap_and_config 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.196+0000 7f137c1a9640 10 monclient: build_initial_monmap 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.196+0000 7f137c1a9640 10 monclient: monmap: 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:22.198492+0000 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:22.198492+0000 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:22.198 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:22.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.197+0000 7f137c1a9640 1 Processor -- start 2024-01-20T11:23:22.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.197+0000 7f137c1a9640 1 -- start start 2024-01-20T11:23:22.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.197+0000 7f137c1a9640 10 monclient: init 2024-01-20T11:23:22.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:22.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-01-20T11:23:22.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740b0ba0 0x7f13740ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:22.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient(hunting): picked mon.noname-b con 0x7f13740b0ba0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:22.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13740adc30 0x7f13740af130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient(hunting): picked mon.noname-a con 0x7f13740adc30 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f13740ac4d0 0x7f137413bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient(hunting): picked mon.noname-c con 0x7f13740ac4d0 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13740b0f80 con 0x7f13740adc30 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13740ad530 con 0x7f13740ac4d0 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13740ad810 con 0x7f13740b0ba0 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f137c1a9640 10 monclient(hunting): authenticate will time out at 1845.549462s 2024-01-20T11:23:22.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f1379744640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13740adc30 0x7f13740af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.198+0000 7f1379f45640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740b0ba0 0x7f13740ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f137a746640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f13740ac4d0 0x7f137413bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379744640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13740adc30 0x7f13740af130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50438/0 (socket says 172.21.15.110:50438) 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379744640 1 -- 172.21.15.110:0/1760210391 learned_addr learned my addr 172.21.15.110:0/1760210391 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379744640 10 monclient(hunting): get_auth_request con 0x7f13740adc30 auth_method 0 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379744640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379744640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379744640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379f45640 10 monclient(hunting): get_auth_request con 0x7f13740b0ba0 auth_method 0 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379f45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379f45640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379f45640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f137a746640 10 monclient(hunting): get_auth_request con 0x7f13740ac4d0 auth_method 0 2024-01-20T11:23:22.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f137a746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f137a746640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f137a746640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379744640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379744640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f1379744640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f137a746640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f137a746640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.199+0000 7f137a746640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.200+0000 7f1379f45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.200+0000 7f1379f45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:22.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.200+0000 7f1379f45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:22.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 10 monclient(hunting): handle_auth_done global_id 6130 payload 274 2024-01-20T11:23:22.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 1 -- 172.21.15.110:0/1760210391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740b0ba0 msgr2=0x7f13740ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:22.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 1 --2- 172.21.15.110:0/1760210391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740b0ba0 0x7f13740ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 1 -- 172.21.15.110:0/1760210391 >> v2:172.21.15.110:3300/0 conn(0x7f13740adc30 msgr2=0x7f13740af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:22.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 1 --2- 172.21.15.110:0/1760210391 >> v2:172.21.15.110:3300/0 conn(0x7f13740adc30 0x7f13740af130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 10 monclient: _finish_hunting 0 2024-01-20T11:23:22.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 1 monclient: found mon.noname-c 2024-01-20T11:23:22.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:22.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:23:22.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.201+0000 7f137a746640 1 -- 172.21.15.110:0/1760210391 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13740ab110 con 0x7f13740ac4d0 2024-01-20T11:23:22.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.203+0000 7f137a746640 1 --2- 172.21.15.110:0/1760210391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740ac4d0 0x7f137413bf30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f137413fc80 tx=0x7f136c02bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0958f7993a967f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:22.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 1 -- 172.21.15.110:0/1760210391 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c02cb40 con 0x7f13740ac4d0 2024-01-20T11:23:22.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:22.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:22.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 10 monclient: dump: 2024-01-20T11:23:22.208 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:22.208 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 10 monclient: _finish_auth 0 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 10 monclient: _check_auth_tickets 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 1 -- 172.21.15.110:0/1760210391 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136c02cca0 con 0x7f13740ac4d0 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:22.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 1 -- 172.21.15.110:0/1760210391 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c034660 con 0x7f13740ac4d0 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f137c1a9640 5 monclient: authenticate success, global_id 6130 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f137c1a9640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f137c1a9640 4 set_mon_vals no callback set 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.207+0000 7f1378f43640 10 monclient: dump: 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:22.210 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:22.211 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:22.211 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:22.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 10 monclient: shutdown 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 1 -- 172.21.15.110:0/1760210391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740ac4d0 msgr2=0x7f137413bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 1 --2- 172.21.15.110:0/1760210391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740ac4d0 0x7f137413bf30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f137413fc80 tx=0x7f136c02bd90 comp rx=0 tx=0).stop 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 1 -- 172.21.15.110:0/1760210391 shutdown_connections 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 1 --2- 172.21.15.110:0/1760210391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740ac4d0 0x7f137413bf30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 1 --2- 172.21.15.110:0/1760210391 >> v2:172.21.15.110:3300/0 conn(0x7f13740adc30 0x7f13740af130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 1 --2- 172.21.15.110:0/1760210391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740b0ba0 0x7f13740ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 1 -- 172.21.15.110:0/1760210391 >> 172.21.15.110:0/1760210391 conn(0x7f13740b3e60 msgr2=0x7f13740b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 1 -- 172.21.15.110:0/1760210391 shutdown_connections 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 1 -- 172.21.15.110:0/1760210391 wait complete. 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 10 monclient: build_initial_monmap 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.208+0000 7f137c1a9640 10 monclient: monmap: 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:22.215 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 librados: starting msgr at 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 librados: starting objecter 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 Processor -- start 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 -- start start 2024-01-20T11:23:22.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 librados: setting wanted keys 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 librados: calling monclient init 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient: init 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740ac4d0 0x7f137413cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient(hunting): picked mon.noname-c con 0x7f13740ac4d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740b0ba0 0x7f1374132850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient(hunting): picked mon.noname-b con 0x7f13740b0ba0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f137413d4c0 0x7f1374132d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient(hunting): picked mon.noname-a con 0x7f137413d4c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13740b31c0 con 0x7f137413d4c0 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:22.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137413db80 con 0x7f13740b0ba0 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13741332d0 con 0x7f13740ac4d0 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.209+0000 7f137c1a9640 10 monclient(hunting): authenticate will time out at 1845.561093s 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379f45640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740ac4d0 0x7f137413cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379744640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740b0ba0 0x7f1374132850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f137a746640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f137413d4c0 0x7f1374132d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379744640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740b0ba0 0x7f1374132850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:35042/0 (socket says 172.21.15.110:35042) 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379744640 1 -- 172.21.15.110:0/2602169802 learned_addr learned my addr 172.21.15.110:0/2602169802 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379744640 10 monclient(hunting): get_auth_request con 0x7f13740b0ba0 auth_method 0 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379744640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379744640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379744640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:22.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f137a746640 10 monclient(hunting): get_auth_request con 0x7f137413d4c0 auth_method 0 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f137a746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f137a746640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f137a746640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379f45640 10 monclient(hunting): get_auth_request con 0x7f13740ac4d0 auth_method 0 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379f45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379f45640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.210+0000 7f1379f45640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.211+0000 7f1379744640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.211+0000 7f1379744640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.211+0000 7f1379744640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.211+0000 7f1379f45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.211+0000 7f1379f45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.211+0000 7f1379f45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.211+0000 7f137a746640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:22.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.211+0000 7f137a746640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.211+0000 7f137a746640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 10 monclient(hunting): handle_auth_done global_id 6133 payload 824 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 1 -- 172.21.15.110:0/2602169802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740ac4d0 msgr2=0x7f137413cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740ac4d0 0x7f137413cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 1 -- 172.21.15.110:0/2602169802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f137413d4c0 msgr2=0x7f1374132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f137413d4c0 0x7f1374132d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 10 monclient: _finish_hunting 0 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 1 monclient: found mon.noname-b 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379744640 1 -- 172.21.15.110:0/2602169802 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c009a70 con 0x7f13740b0ba0 2024-01-20T11:23:22.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.213+0000 7f1379f45640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740ac4d0 0x7f137413cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.216+0000 7f1379744640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740b0ba0 0x7f1374132850 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1364009fd0 tx=0x7f136400ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.216+0000 7f136a7fc640 10 client.?.objecter ms_handle_connect 0x7f13740b0ba0 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.216+0000 7f136a7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 1 -- 172.21.15.110:0/2602169802 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136400c940 con 0x7f13740b0ba0 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 10 monclient: dump: 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 1 -- 172.21.15.110:0/2602169802 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136400caa0 con 0x7f13740b0ba0 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 1 -- 172.21.15.110:0/2602169802 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136400ce30 con 0x7f13740b0ba0 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f137c1a9640 5 monclient: authenticate success, global_id 6133 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f137c1a9640 10 monclient: _renew_subs 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f137c1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137413cab0 con 0x7f13740b0ba0 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:22.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136bfff640 4 set_mon_vals no callback set 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.219+0000 7f136a7fc640 10 monclient: dump: 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.220+0000 7f137c1a9640 10 client.6133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.220+0000 7f137c1a9640 10 monclient: _renew_subs 2024-01-20T11:23:22.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.220+0000 7f137c1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:22.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.220+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13741419e0 con 0x7f13740b0ba0 2024-01-20T11:23:22.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.220+0000 7f137c1a9640 1 librados: init done 2024-01-20T11:23:22.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.220+0000 7f137c1a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:22.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.220+0000 7f137c1a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:22.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.220+0000 7f137c1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:22.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.220+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1374075490 con 0x7f13740b0ba0 2024-01-20T11:23:22.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.239+0000 7f136a7fc640 1 -- 172.21.15.110:0/2602169802 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f1364013470 con 0x7f13740b0ba0 2024-01-20T11:23:22.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.239+0000 7f136a7fc640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f135806c8b0 0x7f135806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:22.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.240+0000 7f1379f45640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f135806c8b0 0x7f135806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:22.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.240+0000 7f1379f45640 10 monclient: get_auth_request con 0x7f135806c8b0 auth_method 0 2024-01-20T11:23:22.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.240+0000 7f1379f45640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f135806c8b0 0x7f135806ed70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1370009e10 tx=0x7f1370009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:22.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.240+0000 7f136a7fc640 10 client.6133.objecter ms_handle_connect 0x7f135806c8b0 2024-01-20T11:23:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.468+0000 7f136a7fc640 1 -- 172.21.15.110:0/2602169802 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1364058aa0 con 0x7f13740b0ba0 2024-01-20T11:23:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.468+0000 7f136a7fc640 10 client.6133.objecter ms_dispatch 0x7f1374000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.468+0000 7f136a7fc640 3 client.6133.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.468+0000 7f136a7fc640 3 client.6133.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:22.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.468+0000 7f136a7fc640 20 client.6133.objecter dump_active .. 0 homeless 2024-01-20T11:23:22.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.480+0000 7f136a7fc640 1 -- 172.21.15.110:0/2602169802 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f1364054de0 con 0x7f13740b0ba0 2024-01-20T11:23:22.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.480+0000 7f136a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:22.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.480+0000 7f136a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:22.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.651+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1374069970 con 0x7f135806c8b0 2024-01-20T11:23:22.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.654+0000 7f136a7fc640 1 -- 172.21.15.110:0/2602169802 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63775 (secure 0 0 0) 0x7f1374069970 con 0x7f135806c8b0 2024-01-20T11:23:22.810 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:22.810 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:22.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 10 librados: watch_flush enter 2024-01-20T11:23:22.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 10 librados: watch_flush exit 2024-01-20T11:23:22.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 20 client.6133.objecter shutdown clearing up homeless session... 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 10 client.6133.objecter successfully canceled tick 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f135806c8b0 msgr2=0x7f135806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f135806c8b0 0x7f135806ed70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1370009e10 tx=0x7f1370009190 comp rx=0 tx=0).stop 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 10 monclient: shutdown 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740b0ba0 msgr2=0x7f1374132850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740b0ba0 0x7f1374132850 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1364009fd0 tx=0x7f136400ed40 comp rx=0 tx=0).stop 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 shutdown_connections 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f135806c8b0 0x7f135806ed70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f137413d4c0 0x7f1374132d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13740b0ba0 0x7f1374132850 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 --2- 172.21.15.110:0/2602169802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13740ac4d0 0x7f137413cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 >> 172.21.15.110:0/2602169802 conn(0x7f13740b3e60 msgr2=0x7f13740aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 shutdown_connections 2024-01-20T11:23:22.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 -- 172.21.15.110:0/2602169802 wait complete. 2024-01-20T11:23:22.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:22.656+0000 7f137c1a9640 1 librados: shutdown 2024-01-20T11:23:22.814 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-01-20T11:23:20.810212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002993"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":12.379},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":5.5529999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":6.6890000000000001},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":14.098000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":9.8780000000000001},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":10.829000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.512},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":8.6280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":7.3840000000000003},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":14.768000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":11.66},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":7.9509999999999996}]}]},{"osd":7,"up_from":21,"seq":90194313317,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":11.593999999999999},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":8.3209999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":7.218},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.427}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.901},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":13.65}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":11.256},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":17.222000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.468999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":9.0709999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":9.8710000000000004},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":15.138999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313318,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.0010000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":6.625}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":9.016},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":16.451000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.231},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":8.0960000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":13.948},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.77}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":11.194000000000001},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":12.135}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.363},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.653}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444132,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.8029999999999999},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":11.372}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":6.7350000000000003},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":13.167}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.141},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.5269999999999992}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.5},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.518000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":12.375999999999999},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.8959999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":9.1379999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":14.957000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444133,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.806},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":7.7670000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":8.9399999999999995},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.932}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":9.5269999999999992},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":11.561}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.3889999999999993},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.406000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":15.465},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":12.247}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":16.297999999999998},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":12.887}]}]},{"osd":0,"up_from":17,"seq":73014444133,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":5.7110000000000003},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":8.5800000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":10.916},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":14.475}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":11.577999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.4440000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":10.852},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":12.337}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":13.763999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":12.175000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.32},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.705}]}]},{"osd":6,"up_from":15,"seq":64424509541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":6.1689999999999996},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":8.1080000000000005}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.4589999999999996},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.067}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":10.823},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":7.5170000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.5259999999999998},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.282}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":8.8079999999999998},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.198}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":15.077999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":11.441000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:24.806 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:15.830 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:24.808 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:15.832 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:24.825 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:15.849 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:24.855 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:15.879 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:24.856 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:09:15.862 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:24.858 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:15.882 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:24.859 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:09:15.865 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:24.874 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:15.898 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:24.874 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:09:15.880 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:25.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~267s 2024-01-20T11:23:25.816 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:25.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.917+0000 7f3e3e234640 10 monclient: get_monmap_and_config 2024-01-20T11:23:25.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.917+0000 7f3e3e234640 10 monclient: build_initial_monmap 2024-01-20T11:23:25.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.917+0000 7f3e3e234640 10 monclient: monmap: 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:25.919300+0000 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:25.919300+0000 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:25.919 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.918+0000 7f3e3e234640 1 Processor -- start 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.918+0000 7f3e3e234640 1 -- start start 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.918+0000 7f3e3e234640 10 monclient: init 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e380b4d40 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e380a9b80 0x7f3e38142470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 10 monclient(hunting): picked mon.noname-a con 0x7f3e380a9b80 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 0x7f3e381429b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 10 monclient(hunting): picked mon.noname-b con 0x7f3e380b40f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e38142ef0 con 0x7f3e380b4d40 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3813fbd0 con 0x7f3e380a9b80 2024-01-20T11:23:25.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:25.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380b5120 con 0x7f3e380b40f0 2024-01-20T11:23:25.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:25.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.919+0000 7f3e3e234640 10 monclient(hunting): authenticate will time out at 1849.270772s 2024-01-20T11:23:25.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.949+0000 7f3e36ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e380a9b80 0x7f3e38142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:25.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.952+0000 7f3e36ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e380a9b80 0x7f3e38142470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50446/0 (socket says 172.21.15.110:50446) 2024-01-20T11:23:25.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.952+0000 7f3e36ffd640 1 -- 172.21.15.110:0/4028330867 learned_addr learned my addr 172.21.15.110:0/4028330867 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:25.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.952+0000 7f3e36ffd640 10 monclient(hunting): get_auth_request con 0x7f3e380a9b80 auth_method 0 2024-01-20T11:23:25.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.952+0000 7f3e36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:25.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.952+0000 7f3e36ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:25.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.952+0000 7f3e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:25.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.952+0000 7f3e37fff640 1 --2- 172.21.15.110:0/4028330867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 0x7f3e381429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:25.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.953+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:25.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.953+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:25.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.953+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:25.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.953+0000 7f3e37fff640 10 monclient(hunting): get_auth_request con 0x7f3e380b40f0 auth_method 0 2024-01-20T11:23:25.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.953+0000 7f3e37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:25.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.953+0000 7f3e37fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:25.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.953+0000 7f3e37fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:25.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.955+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_done global_id 6030 payload 274 2024-01-20T11:23:25.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.955+0000 7f3e36ffd640 1 -- 172.21.15.110:0/4028330867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 msgr2=0x7f3e381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:25.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.955+0000 7f3e36ffd640 1 --2- 172.21.15.110:0/4028330867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 0x7f3e381429b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:25.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.955+0000 7f3e36ffd640 1 -- 172.21.15.110:0/4028330867 >> v1:172.21.15.134:6789/0 conn(0x7f3e380b4d40 legacy=0x7f3e380b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:25.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.955+0000 7f3e36ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:23:25.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.955+0000 7f3e36ffd640 1 monclient: found mon.noname-a 2024-01-20T11:23:25.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.955+0000 7f3e36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:25.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.955+0000 7f3e36ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:25.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.955+0000 7f3e36ffd640 1 -- 172.21.15.110:0/4028330867 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e38141fe0 con 0x7f3e380a9b80 2024-01-20T11:23:25.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.957+0000 7f3e36ffd640 1 --2- 172.21.15.110:0/4028330867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e380a9b80 0x7f3e38142470 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3e200098a0 tx=0x7f3e2002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d6d302bc160099 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:25.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 1 -- 172.21.15.110:0/4028330867 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20031520 con 0x7f3e380a9b80 2024-01-20T11:23:25.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:25.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:25.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 10 monclient: dump: 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:25.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 1 -- 172.21.15.110:0/4028330867 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e20031680 con 0x7f3e380a9b80 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 1 -- 172.21.15.110:0/4028330867 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20031a10 con 0x7f3e380a9b80 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e3e234640 5 monclient: authenticate success, global_id 6030 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e3e234640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e3e234640 4 set_mon_vals no callback set 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.963+0000 7f3e367fc640 10 monclient: dump: 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:25.966 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:25.967 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:25.967 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:25.967 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:25.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 10 monclient: shutdown 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 1 -- 172.21.15.110:0/4028330867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e380a9b80 msgr2=0x7f3e38142470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 1 --2- 172.21.15.110:0/4028330867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e380a9b80 0x7f3e38142470 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3e200098a0 tx=0x7f3e2002b7c0 comp rx=0 tx=0).stop 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 1 -- 172.21.15.110:0/4028330867 shutdown_connections 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 1 --2- 172.21.15.110:0/4028330867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 0x7f3e381429b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 1 --2- 172.21.15.110:0/4028330867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e380a9b80 0x7f3e38142470 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 1 -- 172.21.15.110:0/4028330867 >> 172.21.15.110:0/4028330867 conn(0x7f3e38066590 msgr2=0x7f3e380669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 1 -- 172.21.15.110:0/4028330867 shutdown_connections 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.964+0000 7f3e3e234640 1 -- 172.21.15.110:0/4028330867 wait complete. 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.965+0000 7f3e3e234640 10 monclient: build_initial_monmap 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.965+0000 7f3e3e234640 10 monclient: monmap: 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.965+0000 7f3e3e234640 1 librados: starting msgr at 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.965+0000 7f3e3e234640 1 librados: starting objecter 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.965+0000 7f3e3e234640 1 Processor -- start 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.965+0000 7f3e3e234640 1 -- start start 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.965+0000 7f3e3e234640 1 librados: setting wanted keys 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.965+0000 7f3e3e234640 1 librados: calling monclient init 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.965+0000 7f3e3e234640 10 monclient: init 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:23:25.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e380a9b80 0x7f3e381397c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient(hunting): picked mon.noname-a con 0x7f3e380a9b80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 0x7f3e38136560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient(hunting): picked mon.noname-c con 0x7f3e380b40f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e380b4d40 0x7f3e38137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient(hunting): picked mon.noname-b con 0x7f3e380b4d40 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e38134ae0 con 0x7f3e380a9b80 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e38137880 con 0x7f3e380b4d40 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e38137b60 con 0x7f3e380b40f0 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e3e234640 10 monclient(hunting): authenticate will time out at 1849.317568s 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.966+0000 7f3e36ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 0x7f3e38136560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e380b4d40 0x7f3e38137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e380b4d40 0x7f3e38137f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:35058/0 (socket says 172.21.15.110:35058) 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 1 -- 172.21.15.110:0/764663092 learned_addr learned my addr 172.21.15.110:0/764663092 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 10 monclient(hunting): get_auth_request con 0x7f3e380b4d40 auth_method 0 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e36ffd640 10 monclient(hunting): get_auth_request con 0x7f3e380b40f0 auth_method 0 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e36ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:25.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:25.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.967+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:25.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.969+0000 7f3e377fe640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e380a9b80 0x7f3e381397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:25.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_done global_id 5672 payload 824 2024-01-20T11:23:25.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 1 -- 172.21.15.110:0/764663092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e380b4d40 msgr2=0x7f3e38137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e380b4d40 0x7f3e38137f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 1 -- 172.21.15.110:0/764663092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e380a9b80 msgr2=0x7f3e381397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e380a9b80 0x7f3e381397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 1 monclient: found mon.noname-c 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.970+0000 7f3e36ffd640 1 -- 172.21.15.110:0/764663092 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e200094d0 con 0x7f3e380b40f0 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.973+0000 7f3e36ffd640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 0x7f3e38136560 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3e38072750 tx=0x7f3e200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.973+0000 7f3e2ffff640 10 client.?.objecter ms_handle_connect 0x7f3e380b40f0 2024-01-20T11:23:25.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.973+0000 7f3e2ffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 1 -- 172.21.15.110:0/764663092 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20031520 con 0x7f3e380b40f0 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 10 monclient: dump: 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:25.978 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 10 monclient: _finish_auth 0 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 1 -- 172.21.15.110:0/764663092 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e20033c50 con 0x7f3e380b40f0 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 1 -- 172.21.15.110:0/764663092 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2003c400 con 0x7f3e380b40f0 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e3e234640 5 monclient: authenticate success, global_id 5672 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e3e234640 10 monclient: _renew_subs 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e3e234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e381340f0 con 0x7f3e380b40f0 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e35ffb640 4 set_mon_vals no callback set 2024-01-20T11:23:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.976+0000 7f3e2ffff640 10 monclient: dump: 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.977+0000 7f3e3e234640 10 client.5672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.977+0000 7f3e3e234640 10 monclient: _renew_subs 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.977+0000 7f3e3e234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.977+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e381386a0 con 0x7f3e380b40f0 2024-01-20T11:23:25.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.977+0000 7f3e3e234640 1 librados: init done 2024-01-20T11:23:25.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.978+0000 7f3e3e234640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:25.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.978+0000 7f3e3e234640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:25.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.978+0000 7f3e3e234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:25.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:25.978+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e381379c0 con 0x7f3e380b40f0 2024-01-20T11:23:26.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.267+0000 7f3e2ffff640 1 -- 172.21.15.110:0/764663092 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3e2003a040 con 0x7f3e380b40f0 2024-01-20T11:23:26.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.267+0000 7f3e2ffff640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3e1806c860 0x7f3e1806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:26.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.268+0000 7f3e377fe640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3e1806c860 0x7f3e1806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:26.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.268+0000 7f3e377fe640 10 monclient: get_auth_request con 0x7f3e1806c860 auth_method 0 2024-01-20T11:23:26.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.268+0000 7f3e377fe640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3e1806c860 0x7f3e1806ed20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3e28007dc0 tx=0x7f3e280072f0 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:26.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.268+0000 7f3e2ffff640 10 client.5672.objecter ms_handle_connect 0x7f3e1806c860 2024-01-20T11:23:26.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.284+0000 7f3e2ffff640 1 -- 172.21.15.110:0/764663092 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3e20079d50 con 0x7f3e380b40f0 2024-01-20T11:23:26.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.284+0000 7f3e2ffff640 10 client.5672.objecter ms_dispatch 0x7f3e38000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:26.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.284+0000 7f3e2ffff640 3 client.5672.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:26.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.284+0000 7f3e2ffff640 3 client.5672.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:26.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.284+0000 7f3e2ffff640 20 client.5672.objecter dump_active .. 0 homeless 2024-01-20T11:23:26.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.284+0000 7f3e2ffff640 1 -- 172.21.15.110:0/764663092 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f3e20076090 con 0x7f3e380b40f0 2024-01-20T11:23:26.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.284+0000 7f3e2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:26.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.284+0000 7f3e2ffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:26.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.450+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e3814c860 con 0x7f3e1806c860 2024-01-20T11:23:26.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.454+0000 7f3e2ffff640 1 -- 172.21.15.110:0/764663092 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63818 (secure 0 0 0) 0x7f3e280079b0 con 0x7f3e1806c860 2024-01-20T11:23:26.455 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:26.455 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:26.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 10 librados: watch_flush enter 2024-01-20T11:23:26.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 10 librados: watch_flush exit 2024-01-20T11:23:26.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 20 client.5672.objecter shutdown clearing up homeless session... 2024-01-20T11:23:26.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 10 client.5672.objecter successfully canceled tick 2024-01-20T11:23:26.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3e1806c860 msgr2=0x7f3e1806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:26.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3e1806c860 0x7f3e1806ed20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3e28007dc0 tx=0x7f3e280072f0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 10 monclient: shutdown 2024-01-20T11:23:26.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:26.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 msgr2=0x7f3e38136560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 0x7f3e38136560 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3e38072750 tx=0x7f3e200046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 shutdown_connections 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f3e1806c860 0x7f3e1806ed20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e380b4d40 0x7f3e38137f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e380b40f0 0x7f3e38136560 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 --2- 172.21.15.110:0/764663092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e380a9b80 0x7f3e381397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.455+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 >> 172.21.15.110:0/764663092 conn(0x7f3e38066590 msgr2=0x7f3e380b28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.456+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 shutdown_connections 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.456+0000 7f3e3e234640 1 -- 172.21.15.110:0/764663092 wait complete. 2024-01-20T11:23:26.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.456+0000 7f3e3e234640 1 librados: shutdown 2024-01-20T11:23:26.468 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-01-20T11:23:24.810906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002758"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":4.5650000000000004},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":6.4219999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.1699999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":13.715}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":10.186999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":13.044}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":14.352},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":7.5419999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.754},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":15.468999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":11.762},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":16.172999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313318,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":9.8339999999999996},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":6.4969999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":6.1449999999999996},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.087}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":11.833},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":12.459}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.8719999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":15.231}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":8.6579999999999995},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":11.122}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":13.710000000000001},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":9.1809999999999992}]}]},{"osd":3,"up_from":21,"seq":90194313319,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":8.0129999999999999},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.2370000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":9.3960000000000008},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":11.398}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":8.8030000000000008},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.032}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.074},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.513999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":16.422999999999998},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.279}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":12.712},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.954000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444133,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.343},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.5819999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":5.8879999999999999},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.291}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.881},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.1300000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":13.446},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.709}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":16.029},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.8100000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.031000000000001},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.32}]}]},{"osd":1,"up_from":17,"seq":73014444134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":13.811999999999999},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":5.4610000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.4169999999999998},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":6.7949999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":15.196999999999999},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":9.8190000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.5220000000000002},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.2360000000000007},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":10.494999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":12.997},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":11.092000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444134,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":8.0050000000000008},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":6.4100000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.7889999999999997},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":8.5969999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":10.343999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":5.8259999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":7.327},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":14.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":11.034000000000001},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":9.1850000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":11.712999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":14.815}]}]},{"osd":6,"up_from":15,"seq":64424509542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":11.452},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":19.834}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":13.291},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":12.491}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":10.379},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":15.016999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":13.625999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":9.8260000000000005}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":21.539000000000001},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":10.932}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":19.141999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":22.689}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:26.469 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.568+0000 7ffb483a9640 10 monclient: get_monmap_and_config 2024-01-20T11:23:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.568+0000 7ffb483a9640 10 monclient: build_initial_monmap 2024-01-20T11:23:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.568+0000 7ffb483a9640 10 monclient: monmap: 2024-01-20T11:23:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:26.570100+0000 2024-01-20T11:23:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:26.570100+0000 2024-01-20T11:23:26.569 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:26.570 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:26.570 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 1 Processor -- start 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 1 -- start start 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient: init 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffb40071c60 0x7ffb40142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient(hunting): picked mon.noname-a con 0x7ffb40071c60 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb400ac540 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 0x7ffb40149510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient(hunting): picked mon.noname-b con 0x7ffb400b0930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb400ad610 con 0x7ffb400ac540 2024-01-20T11:23:26.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb40072040 con 0x7ffb40071c60 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb401426c0 con 0x7ffb400b0930 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.569+0000 7ffb483a9640 10 monclient(hunting): authenticate will time out at 1849.921042s 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46145640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffb40071c60 0x7ffb40142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46946640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 0x7ffb40149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb45944640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ffb400ac540 0x7ffb400ab4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:55002/0 (socket says 172.21.15.110:55002) 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb45944640 1 -- 172.21.15.110:0/160443299 learned_addr learned my addr 172.21.15.110:0/160443299 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46145640 10 monclient(hunting): get_auth_request con 0x7ffb40071c60 auth_method 0 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46145640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46145640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46145640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46946640 10 monclient(hunting): get_auth_request con 0x7ffb400b0930 auth_method 0 2024-01-20T11:23:26.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46946640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.570+0000 7ffb46946640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.571+0000 7ffb46145640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.571+0000 7ffb46145640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.571+0000 7ffb46145640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.571+0000 7ffb46946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.571+0000 7ffb46946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:26.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.571+0000 7ffb46946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.573+0000 7ffb46145640 10 monclient(hunting): handle_auth_done global_id 6033 payload 274 2024-01-20T11:23:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.573+0000 7ffb46145640 1 -- 172.21.15.110:0/160443299 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 msgr2=0x7ffb40149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.573+0000 7ffb46145640 1 --2- 172.21.15.110:0/160443299 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 0x7ffb40149510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.573+0000 7ffb46145640 1 -- 172.21.15.110:0/160443299 >> v1:172.21.15.134:6789/0 conn(0x7ffb400ac540 legacy=0x7ffb400ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.573+0000 7ffb46145640 10 monclient: _finish_hunting 0 2024-01-20T11:23:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.573+0000 7ffb46145640 1 monclient: found mon.noname-a 2024-01-20T11:23:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.573+0000 7ffb46145640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.573+0000 7ffb46145640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:26.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.573+0000 7ffb46145640 1 -- 172.21.15.110:0/160443299 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb400721b0 con 0x7ffb40071c60 2024-01-20T11:23:26.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.575+0000 7ffb46145640 1 --2- 172.21.15.110:0/160443299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 0x7ffb40142180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ffb300097a0 tx=0x7ffb3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f5f5d375e71179 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 1 -- 172.21.15.110:0/160443299 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30031830 con 0x7ffb40071c60 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 10 monclient: dump: 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:26.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 10 monclient: _finish_auth 0 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 10 monclient: _check_auth_tickets 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 1 -- 172.21.15.110:0/160443299 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30031990 con 0x7ffb40071c60 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb483a9640 5 monclient: authenticate success, global_id 6033 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb483a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb483a9640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 1 -- 172.21.15.110:0/160443299 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30031d20 con 0x7ffb40071c60 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb483a9640 4 set_mon_vals no callback set 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.578+0000 7ffb45143640 10 monclient: dump: 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:26.581 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:26.582 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:26.582 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:26.582 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:26.582 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:26.582 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:26.582 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:26.582 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 10 monclient: shutdown 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 1 -- 172.21.15.110:0/160443299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 msgr2=0x7ffb40142180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 1 --2- 172.21.15.110:0/160443299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 0x7ffb40142180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ffb300097a0 tx=0x7ffb3002b8d0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 1 -- 172.21.15.110:0/160443299 shutdown_connections 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 1 --2- 172.21.15.110:0/160443299 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 0x7ffb40149510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 1 --2- 172.21.15.110:0/160443299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 0x7ffb40142180 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 1 -- 172.21.15.110:0/160443299 >> 172.21.15.110:0/160443299 conn(0x7ffb400b3e60 msgr2=0x7ffb400b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 1 -- 172.21.15.110:0/160443299 shutdown_connections 2024-01-20T11:23:26.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.579+0000 7ffb483a9640 1 -- 172.21.15.110:0/160443299 wait complete. 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.580+0000 7ffb483a9640 10 monclient: build_initial_monmap 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.580+0000 7ffb483a9640 10 monclient: monmap: 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.580+0000 7ffb483a9640 1 librados: starting msgr at 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.580+0000 7ffb483a9640 1 librados: starting objecter 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.580+0000 7ffb483a9640 1 Processor -- start 2024-01-20T11:23:26.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.580+0000 7ffb483a9640 1 -- start start 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.580+0000 7ffb483a9640 1 librados: setting wanted keys 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.580+0000 7ffb483a9640 1 librados: calling monclient init 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.580+0000 7ffb483a9640 10 monclient: init 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 0x7ffb4014b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient(hunting): picked mon.noname-a con 0x7ffb40071c60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb400ac540 0x7ffb4013bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient(hunting): picked mon.noname-b con 0x7ffb400ac540 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 0x7ffb4013c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient(hunting): picked mon.noname-c con 0x7ffb400b0930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40147fa0 con 0x7ffb40071c60 2024-01-20T11:23:26.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40148250 con 0x7ffb400ac540 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40148530 con 0x7ffb400b0930 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb483a9640 10 monclient(hunting): authenticate will time out at 1849.932578s 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.581+0000 7ffb46145640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 0x7ffb4014b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46946640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 0x7ffb4013c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46145640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 0x7ffb4014b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50470/0 (socket says 172.21.15.110:50470) 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46145640 1 -- 172.21.15.110:0/1743885279 learned_addr learned my addr 172.21.15.110:0/1743885279 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46145640 10 monclient(hunting): get_auth_request con 0x7ffb40071c60 auth_method 0 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46145640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46145640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46145640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb45944640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb400ac540 0x7ffb4013bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:26.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46946640 10 monclient(hunting): get_auth_request con 0x7ffb400b0930 auth_method 0 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46946640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46946640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb45944640 10 monclient(hunting): get_auth_request con 0x7ffb400ac540 auth_method 0 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb45944640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb45944640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb45944640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46145640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46145640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46145640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb46946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.582+0000 7ffb45944640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:26.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.583+0000 7ffb45944640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.583+0000 7ffb45944640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 10 monclient(hunting): handle_auth_done global_id 6036 payload 824 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 1 -- 172.21.15.110:0/1743885279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 msgr2=0x7ffb4013c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 0x7ffb4013c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 1 -- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb400ac540 msgr2=0x7ffb4013bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb400ac540 0x7ffb4013bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 10 monclient: _finish_hunting 0 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 1 monclient: found mon.noname-a 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:26.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.585+0000 7ffb46145640 1 -- 172.21.15.110:0/1743885279 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb30009400 con 0x7ffb40071c60 2024-01-20T11:23:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.588+0000 7ffb46145640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 0x7ffb4014b1f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ffb300098c0 tx=0x7ffb300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.588+0000 7ffb36ffd640 10 client.?.objecter ms_handle_connect 0x7ffb40071c60 2024-01-20T11:23:26.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.588+0000 7ffb36ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.591+0000 7ffb36ffd640 1 -- 172.21.15.110:0/1743885279 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3003a6f0 con 0x7ffb40071c60 2024-01-20T11:23:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.591+0000 7ffb36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.591+0000 7ffb36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.591+0000 7ffb36ffd640 10 monclient: dump: 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.591+0000 7ffb36ffd640 10 monclient: _finish_auth 0 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.591+0000 7ffb36ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:26.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.591+0000 7ffb36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.591+0000 7ffb483a9640 5 monclient: authenticate success, global_id 6036 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 10 monclient: _renew_subs 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb40148a50 con 0x7ffb40071c60 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 10 client.6036.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 10 monclient: _renew_subs 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4014b730 con 0x7ffb40071c60 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 1 librados: init done 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.592+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb40075490 con 0x7ffb40071c60 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.593+0000 7ffb36ffd640 1 -- 172.21.15.110:0/1743885279 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3003a850 con 0x7ffb40071c60 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.593+0000 7ffb36ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.594+0000 7ffb36ffd640 1 -- 172.21.15.110:0/1743885279 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3003abe0 con 0x7ffb40071c60 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.594+0000 7ffb36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.594+0000 7ffb36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.594+0000 7ffb36ffd640 10 monclient: dump: 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:26.597 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.594+0000 7ffb44942640 4 set_mon_vals no callback set 2024-01-20T11:23:26.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.615+0000 7ffb36ffd640 1 -- 172.21.15.110:0/1743885279 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ffb3003ad40 con 0x7ffb40071c60 2024-01-20T11:23:26.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.615+0000 7ffb36ffd640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb2406c860 0x7ffb2406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:26.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.616+0000 7ffb45944640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb2406c860 0x7ffb2406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:26.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.616+0000 7ffb45944640 10 monclient: get_auth_request con 0x7ffb2406c860 auth_method 0 2024-01-20T11:23:26.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.616+0000 7ffb36ffd640 10 client.6036.objecter ms_handle_connect 0x7ffb2406c860 2024-01-20T11:23:26.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.616+0000 7ffb45944640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb2406c860 0x7ffb2406ed20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ffb4013fb10 tx=0x7ffb3c007e80 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:26.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.846+0000 7ffb36ffd640 1 -- 172.21.15.110:0/1743885279 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffb3007a870 con 0x7ffb40071c60 2024-01-20T11:23:26.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.846+0000 7ffb36ffd640 10 client.6036.objecter ms_dispatch 0x7ffb40000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:26.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.846+0000 7ffb36ffd640 3 client.6036.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:26.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.846+0000 7ffb36ffd640 3 client.6036.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:26.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.847+0000 7ffb36ffd640 20 client.6036.objecter dump_active .. 0 homeless 2024-01-20T11:23:26.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.847+0000 7ffb36ffd640 1 -- 172.21.15.110:0/1743885279 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ffb30076bb0 con 0x7ffb40071c60 2024-01-20T11:23:26.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.847+0000 7ffb36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:26.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:26.847+0000 7ffb36ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:27.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.020+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb40149bf0 con 0x7ffb2406c860 2024-01-20T11:23:27.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.023+0000 7ffb36ffd640 1 -- 172.21.15.110:0/1743885279 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63902 (secure 0 0 0) 0x7ffb3c001460 con 0x7ffb2406c860 2024-01-20T11:23:27.024 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:27.025 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:27.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.024+0000 7ffb483a9640 10 librados: watch_flush enter 2024-01-20T11:23:27.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.024+0000 7ffb483a9640 10 librados: watch_flush exit 2024-01-20T11:23:27.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.024+0000 7ffb483a9640 20 client.6036.objecter shutdown clearing up homeless session... 2024-01-20T11:23:27.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.024+0000 7ffb483a9640 10 client.6036.objecter successfully canceled tick 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.024+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb2406c860 msgr2=0x7ffb2406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.024+0000 7ffb483a9640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb2406c860 0x7ffb2406ed20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ffb4013fb10 tx=0x7ffb3c007e80 comp rx=0 tx=0).stop 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 10 monclient: shutdown 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 msgr2=0x7ffb4014b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 0x7ffb4014b1f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ffb300098c0 tx=0x7ffb300046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 shutdown_connections 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7ffb2406c860 0x7ffb2406ed20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffb400b0930 0x7ffb4013c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb400ac540 0x7ffb4013bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 --2- 172.21.15.110:0/1743885279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb40071c60 0x7ffb4014b1f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 >> 172.21.15.110:0/1743885279 conn(0x7ffb400b3e60 msgr2=0x7ffb401414c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 shutdown_connections 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 -- 172.21.15.110:0/1743885279 wait complete. 2024-01-20T11:23:27.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.025+0000 7ffb483a9640 1 librados: shutdown 2024-01-20T11:23:27.036 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-01-20T11:23:26.811414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":4.5650000000000004},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":6.4219999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.1699999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":13.715}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":10.186999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":13.044}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":14.352},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":7.5419999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.754},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":15.468999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":11.762},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":16.172999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313319,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.9879999999999995},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.1520000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":14.186999999999999},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":6.6609999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":11.111000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":9.7370000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":8.2949999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":10.497999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":12.874000000000001},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":13.551}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":16.228000000000002},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":14.848000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313319,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":8.0129999999999999},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.2370000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":9.3960000000000008},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":11.398}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":8.8030000000000008},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.032}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.074},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.513999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":16.422999999999998},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.279}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":12.712},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.954000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444133,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.343},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.5819999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":5.8879999999999999},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.291}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.881},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.1300000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":13.446},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.709}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":16.029},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.8100000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.031000000000001},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.32}]}]},{"osd":1,"up_from":17,"seq":73014444134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":13.811999999999999},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":5.4610000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.4169999999999998},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":6.7949999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":15.196999999999999},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":9.8190000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.5220000000000002},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.2360000000000007},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":10.494999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":12.997},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":11.092000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444134,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":8.0050000000000008},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":6.4100000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.7889999999999997},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":8.5969999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":10.343999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":5.8259999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":7.327},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":14.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":11.034000000000001},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":9.1850000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":11.712999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":14.815}]}]},{"osd":6,"up_from":15,"seq":64424509543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.984},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":10.144}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.9260000000000002},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":11.919}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.505000000000001},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":6.6369999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.141999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":12.885999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":15.753},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":8.7509999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":7.2549999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.609000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:27.038 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:27.038 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:27.038 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:27.038 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:27.038 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:27.038 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:27.038 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:27.038 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 131, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:21:18.077638+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:21:18.077638+0000', 'last_peered': '2024-01-20T11:21:18.077638+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:21:18.077638+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:27.039 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:27.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.165+0000 7f150f89b640 10 monclient: get_monmap_and_config 2024-01-20T11:23:27.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.165+0000 7f150f89b640 10 monclient: build_initial_monmap 2024-01-20T11:23:27.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.165+0000 7f150f89b640 10 monclient: monmap: 2024-01-20T11:23:27.166 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:27.166 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.166 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:27.166986+0000 2024-01-20T11:23:27.166 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:27.166986+0000 2024-01-20T11:23:27.166 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:27.167 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.167 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:27.167 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:27.167 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:27.167 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:27.167 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:27.167 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.165+0000 7f150f89b640 1 Processor -- start 2024-01-20T11:23:27.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.166+0000 7f150f89b640 1 -- start start 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.166+0000 7f150f89b640 10 monclient: init 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f15080b4d10 0x7f15080b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient(hunting): picked mon.noname-c con 0x7f15080b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b17c0 0x7f15080b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient(hunting): picked mon.noname-b con 0x7f15080b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1508145980 0x7f1508145e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient(hunting): picked mon.noname-a con 0x7f1508145980 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15080b50f0 con 0x7f1508145980 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15080b2120 con 0x7f15080b4d10 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15080aade0 con 0x7f15080b17c0 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150f89b640 10 monclient(hunting): authenticate will time out at 1850.518436s 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.167+0000 7f150ce36640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b17c0 0x7f15080b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150d637640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f15080b4d10 0x7f15080b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150ce36640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b17c0 0x7f15080b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57098/0 (socket says 172.21.15.110:57098) 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150d637640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f15080b4d10 0x7f15080b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:35080/0 (socket says 172.21.15.110:35080) 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150ce36640 1 -- 172.21.15.110:0/71334823 learned_addr learned my addr 172.21.15.110:0/71334823 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150ce36640 10 monclient(hunting): get_auth_request con 0x7f15080b17c0 auth_method 0 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150ce36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150ce36640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150ce36640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150d637640 10 monclient(hunting): get_auth_request con 0x7f15080b4d10 auth_method 0 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150d637640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150d637640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150d637640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150d637640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150d637640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150d637640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150ce36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150ce36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.168+0000 7f150ce36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:27.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 10 monclient(hunting): handle_auth_done global_id 6145 payload 274 2024-01-20T11:23:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 1 -- 172.21.15.110:0/71334823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b17c0 msgr2=0x7f15080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 1 --2- 172.21.15.110:0/71334823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b17c0 0x7f15080b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 1 -- 172.21.15.110:0/71334823 >> v2:172.21.15.110:3300/0 conn(0x7f1508145980 msgr2=0x7f1508145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 1 --2- 172.21.15.110:0/71334823 >> v2:172.21.15.110:3300/0 conn(0x7f1508145980 0x7f1508145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 10 monclient: _finish_hunting 0 2024-01-20T11:23:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 1 monclient: found mon.noname-c 2024-01-20T11:23:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.170+0000 7f150d637640 1 -- 172.21.15.110:0/71334823 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1508145460 con 0x7f15080b4d10 2024-01-20T11:23:27.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.174+0000 7f150d637640 1 --2- 172.21.15.110:0/71334823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b4d10 0x7f15080b3da0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f15080b8cd0 tx=0x7f14f802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b832e09547aeb97 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:27.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 1 -- 172.21.15.110:0/71334823 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f802f050 con 0x7f15080b4d10 2024-01-20T11:23:27.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:27.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:27.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 10 monclient: dump: 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 10 monclient: _finish_auth 0 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:27.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 1 -- 172.21.15.110:0/71334823 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8031700 con 0x7f15080b4d10 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 1 -- 172.21.15.110:0/71334823 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8031a90 con 0x7f15080b4d10 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f14fffff640 10 monclient: dump: 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:27.181 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:27.182 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f150f89b640 5 monclient: authenticate success, global_id 6145 2024-01-20T11:23:27.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f150f89b640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:27.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.178+0000 7f150f89b640 4 set_mon_vals no callback set 2024-01-20T11:23:27.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 10 monclient: shutdown 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 1 -- 172.21.15.110:0/71334823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b4d10 msgr2=0x7f15080b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 1 --2- 172.21.15.110:0/71334823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b4d10 0x7f15080b3da0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f15080b8cd0 tx=0x7f14f802b7a0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 1 -- 172.21.15.110:0/71334823 shutdown_connections 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 1 --2- 172.21.15.110:0/71334823 >> v2:172.21.15.110:3300/0 conn(0x7f1508145980 0x7f1508145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 1 --2- 172.21.15.110:0/71334823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b17c0 0x7f15080b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 1 --2- 172.21.15.110:0/71334823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b4d10 0x7f15080b3da0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 1 -- 172.21.15.110:0/71334823 >> 172.21.15.110:0/71334823 conn(0x7f1508066590 msgr2=0x7f15080669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 1 -- 172.21.15.110:0/71334823 shutdown_connections 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.179+0000 7f150f89b640 1 -- 172.21.15.110:0/71334823 wait complete. 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.180+0000 7f150f89b640 10 monclient: build_initial_monmap 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.180+0000 7f150f89b640 10 monclient: monmap: 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:27.186 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.180+0000 7f150f89b640 1 librados: starting msgr at 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.180+0000 7f150f89b640 1 librados: starting objecter 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.180+0000 7f150f89b640 1 Processor -- start 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.180+0000 7f150f89b640 1 -- start start 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.180+0000 7f150f89b640 1 librados: setting wanted keys 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.180+0000 7f150f89b640 1 librados: calling monclient init 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.180+0000 7f150f89b640 10 monclient: init 2024-01-20T11:23:27.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b17c0 0x7f15081db140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient(hunting): picked mon.noname-b con 0x7f15080b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b4d10 0x7f15081d79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient(hunting): picked mon.noname-c con 0x7f15080b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1508145980 0x7f15081d7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient(hunting): picked mon.noname-a con 0x7f1508145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081384f0 con 0x7f1508145980 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081dba60 con 0x7f15080b17c0 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15081db680 con 0x7f15080b4d10 2024-01-20T11:23:27.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150f89b640 10 monclient(hunting): authenticate will time out at 1850.532390s 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150de38640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1508145980 0x7f15081d7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150ce36640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b4d10 0x7f15081d79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150d637640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b17c0 0x7f15081db140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150de38640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1508145980 0x7f15081d7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50494/0 (socket says 172.21.15.110:50494) 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150de38640 1 -- 172.21.15.110:0/2493304423 learned_addr learned my addr 172.21.15.110:0/2493304423 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150de38640 10 monclient(hunting): get_auth_request con 0x7f1508145980 auth_method 0 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150de38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150de38640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150de38640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150d637640 10 monclient(hunting): get_auth_request con 0x7f15080b17c0 auth_method 0 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150d637640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150d637640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.181+0000 7f150d637640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150ce36640 10 monclient(hunting): get_auth_request con 0x7f15080b4d10 auth_method 0 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150ce36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150ce36640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150ce36640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150d637640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150d637640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150d637640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150de38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150de38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150de38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150ce36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150ce36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.182+0000 7f150ce36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 10 monclient(hunting): handle_auth_done global_id 6148 payload 824 2024-01-20T11:23:27.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 1 -- 172.21.15.110:0/2493304423 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b4d10 msgr2=0x7f15081d79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b4d10 0x7f15081d79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 1 -- 172.21.15.110:0/2493304423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1508145980 msgr2=0x7f15081d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1508145980 0x7f15081d7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 10 monclient: _finish_hunting 0 2024-01-20T11:23:27.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 1 monclient: found mon.noname-b 2024-01-20T11:23:27.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:27.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.185+0000 7f150d637640 1 -- 172.21.15.110:0/2493304423 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f8009400 con 0x7f15080b17c0 2024-01-20T11:23:27.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.187+0000 7f150d637640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b17c0 0x7f15081db140 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f14f8033000 tx=0x7f14f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.187+0000 7f14fdffb640 10 client.?.objecter ms_handle_connect 0x7f15080b17c0 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.187+0000 7f14fdffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 1 -- 172.21.15.110:0/2493304423 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8033a30 con 0x7f15080b17c0 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 10 monclient: dump: 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:27.193 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 10 monclient: _finish_auth 0 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 10 monclient: _check_auth_tickets 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 1 -- 172.21.15.110:0/2493304423 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8033b90 con 0x7f15080b17c0 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f150f89b640 5 monclient: authenticate success, global_id 6148 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f150f89b640 10 monclient: _renew_subs 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f150f89b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15081d8450 con 0x7f15080b17c0 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 1 -- 172.21.15.110:0/2493304423 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f803c400 con 0x7f15080b17c0 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:27.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14fdffb640 10 monclient: dump: 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f14ff7fe640 4 set_mon_vals no callback set 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f150f89b640 10 client.6148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f150f89b640 10 monclient: _renew_subs 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f150f89b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1508134990 con 0x7f15080b17c0 2024-01-20T11:23:27.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.191+0000 7f150f89b640 1 librados: init done 2024-01-20T11:23:27.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.192+0000 7f150f89b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:27.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.192+0000 7f150f89b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:27.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.192+0000 7f150f89b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.192+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15081db8a0 con 0x7f15080b17c0 2024-01-20T11:23:27.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.435+0000 7f14fdffb640 1 -- 172.21.15.110:0/2493304423 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f14f803a040 con 0x7f15080b17c0 2024-01-20T11:23:27.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.436+0000 7f14fdffb640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f14ec06c8b0 0x7f14ec06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.436+0000 7f150ce36640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f14ec06c8b0 0x7f14ec06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.436+0000 7f150ce36640 10 monclient: get_auth_request con 0x7f14ec06c8b0 auth_method 0 2024-01-20T11:23:27.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.437+0000 7f150ce36640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f14ec06c8b0 0x7f14ec06ed70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1508141ed0 tx=0x7f1504009240 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:27.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.437+0000 7f14fdffb640 10 client.6148.objecter ms_handle_connect 0x7f14ec06c8b0 2024-01-20T11:23:27.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.439+0000 7f14fdffb640 1 -- 172.21.15.110:0/2493304423 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f14f807a3c0 con 0x7f15080b17c0 2024-01-20T11:23:27.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.439+0000 7f14fdffb640 10 client.6148.objecter ms_dispatch 0x7f1508000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:27.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.439+0000 7f14fdffb640 3 client.6148.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:27.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.439+0000 7f14fdffb640 3 client.6148.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:27.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.439+0000 7f14fdffb640 20 client.6148.objecter dump_active .. 0 homeless 2024-01-20T11:23:27.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.449+0000 7f14fdffb640 1 -- 172.21.15.110:0/2493304423 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f14f8076700 con 0x7f15080b17c0 2024-01-20T11:23:27.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.449+0000 7f14fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:27.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.449+0000 7f14fdffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.690+0000 7f150f89b640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.690+0000 7f150f89b640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.690+0000 7f150f89b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.690+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1508075490 con 0x7f15080b17c0 2024-01-20T11:23:27.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.697+0000 7f14fdffb640 1 -- 172.21.15.110:0/2493304423 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f14f8043520 con 0x7f15080b17c0 2024-01-20T11:23:27.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.697+0000 7f14fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:27.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.697+0000 7f14fdffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:27.705 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:27.705 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":542,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:27.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 10 librados: watch_flush enter 2024-01-20T11:23:27.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 10 librados: watch_flush exit 2024-01-20T11:23:27.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 20 client.6148.objecter shutdown clearing up homeless session... 2024-01-20T11:23:27.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 10 client.6148.objecter successfully canceled tick 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f14ec06c8b0 msgr2=0x7f14ec06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f14ec06c8b0 0x7f14ec06ed70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1508141ed0 tx=0x7f1504009240 comp rx=0 tx=0).stop 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 10 monclient: shutdown 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b17c0 msgr2=0x7f15081db140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b17c0 0x7f15081db140 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f14f8033000 tx=0x7f14f80046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 shutdown_connections 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f14ec06c8b0 0x7f14ec06ed70 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1508145980 0x7f15081d7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15080b4d10 0x7f15081d79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 --2- 172.21.15.110:0/2493304423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15080b17c0 0x7f15081db140 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 >> 172.21.15.110:0/2493304423 conn(0x7f1508066590 msgr2=0x7f15080b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:27.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 shutdown_connections 2024-01-20T11:23:27.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 -- 172.21.15.110:0/2493304423 wait complete. 2024-01-20T11:23:27.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.699+0000 7f150f89b640 1 librados: shutdown 2024-01-20T11:23:27.717 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:27.717 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:27.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.815+0000 7f2d09f3e640 10 monclient: get_monmap_and_config 2024-01-20T11:23:27.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.815+0000 7f2d09f3e640 10 monclient: build_initial_monmap 2024-01-20T11:23:27.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.815+0000 7f2d09f3e640 10 monclient: monmap: 2024-01-20T11:23:27.817 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:27.817 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.817 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:27.817047+0000 2024-01-20T11:23:27.817 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:27.817047+0000 2024-01-20T11:23:27.817 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:27.817 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.818 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:27.818 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:27.818 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:27.818 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:27.818 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:27.818 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 1 Processor -- start 2024-01-20T11:23:27.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 1 -- start start 2024-01-20T11:23:27.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 10 monclient: init 2024-01-20T11:23:27.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:27.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-01-20T11:23:27.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d0406bb00 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:27.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d040af790 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:27.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f2d040aeb20 0x7f2d040aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d040aeb20 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:27.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0406bee0 con 0x7f2d0406bb00 2024-01-20T11:23:27.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d040ada30 con 0x7f2d040af790 2024-01-20T11:23:27.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d040add10 con 0x7f2d040aeb20 2024-01-20T11:23:27.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:27.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.816+0000 7f2d09f3e640 10 monclient(hunting): authenticate will time out at 1851.168035s 2024-01-20T11:23:27.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.817+0000 7f2d037fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2d0406bb00 0x7f2d040b03c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55724/0 (socket says 172.21.15.110:55724) 2024-01-20T11:23:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.817+0000 7f2d037fe640 1 -- 172.21.15.110:0/2780624060 learned_addr learned my addr 172.21.15.110:0/2780624060 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.817+0000 7f2d03fff640 1 --2- 172.21.15.110:0/2780624060 >> v2:172.21.15.134:3300/0 conn(0x7f2d040aeb20 0x7f2d040aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.818+0000 7f2d03fff640 10 monclient(hunting): get_auth_request con 0x7f2d040aeb20 auth_method 0 2024-01-20T11:23:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.818+0000 7f2d03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.818+0000 7f2d03fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.818+0000 7f2d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.819+0000 7f2d03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.819+0000 7f2d03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.819+0000 7f2d03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.819+0000 7f2d027fc640 1 -- 172.21.15.110:0/2780624060 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334726875 0 0) 0x7f2d0406bee0 con 0x7f2d0406bb00 2024-01-20T11:23:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.819+0000 7f2d027fc640 10 monclient(hunting): _negotiate 2024-01-20T11:23:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.819+0000 7f2d027fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.819+0000 7f2d027fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.819+0000 7f2d027fc640 10 monclient(hunting): my global_id is 6042 2024-01-20T11:23:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.819+0000 7f2d027fc640 1 -- 172.21.15.110:0/2780624060 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cec003520 con 0x7f2d0406bb00 2024-01-20T11:23:27.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.822+0000 7f2d03fff640 10 monclient(hunting): handle_auth_done global_id 6151 payload 274 2024-01-20T11:23:27.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.822+0000 7f2d03fff640 1 -- 172.21.15.110:0/2780624060 >> v1:172.21.15.134:6789/0 conn(0x7f2d040af790 legacy=0x7f2d040ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.822+0000 7f2d03fff640 1 -- 172.21.15.110:0/2780624060 >> v1:172.21.15.110:6789/0 conn(0x7f2d0406bb00 legacy=0x7f2d040b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.822+0000 7f2d03fff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:27.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.822+0000 7f2d03fff640 1 monclient: found mon.noname-c 2024-01-20T11:23:27.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.822+0000 7f2d03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:27.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.822+0000 7f2d03fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.822+0000 7f2d03fff640 1 -- 172.21.15.110:0/2780624060 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d040adff0 con 0x7f2d040aeb20 2024-01-20T11:23:27.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.825+0000 7f2d03fff640 1 --2- 172.21.15.110:0/2780624060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 0x7f2d040aef90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009d90 tx=0x7f2cfc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf7e80ad166ccc45 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:27.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 1 -- 172.21.15.110:0/2780624060 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc02cc60 con 0x7f2d040aeb20 2024-01-20T11:23:27.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:27.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:27.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 10 monclient: dump: 2024-01-20T11:23:27.831 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:27.831 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.831 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.831 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.832 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:27.832 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.832 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:27.832 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:27.832 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:27.832 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:27.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:27.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:27.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 1 -- 172.21.15.110:0/2780624060 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc02cdc0 con 0x7f2d040aeb20 2024-01-20T11:23:27.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 5 monclient: authenticate success, global_id 6151 2024-01-20T11:23:27.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 4 set_mon_vals no callback set 2024-01-20T11:23:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 1 -- 172.21.15.110:0/2780624060 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc034660 con 0x7f2d040aeb20 2024-01-20T11:23:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:27.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d027fc640 10 monclient: dump: 2024-01-20T11:23:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.835 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:27.836 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.836 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:27.836 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:27.836 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:27.836 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 10 monclient: shutdown 2024-01-20T11:23:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 1 -- 172.21.15.110:0/2780624060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 msgr2=0x7f2d040aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 1 --2- 172.21.15.110:0/2780624060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 0x7f2d040aef90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009d90 tx=0x7f2cfc02beb0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 1 -- 172.21.15.110:0/2780624060 shutdown_connections 2024-01-20T11:23:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 1 --2- 172.21.15.110:0/2780624060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 0x7f2d040aef90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 1 -- 172.21.15.110:0/2780624060 >> 172.21.15.110:0/2780624060 conn(0x7f2d04066d60 msgr2=0x7f2d040b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:27.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 1 -- 172.21.15.110:0/2780624060 shutdown_connections 2024-01-20T11:23:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.830+0000 7f2d09f3e640 1 -- 172.21.15.110:0/2780624060 wait complete. 2024-01-20T11:23:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.831+0000 7f2d09f3e640 10 monclient: build_initial_monmap 2024-01-20T11:23:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.831+0000 7f2d09f3e640 10 monclient: monmap: 2024-01-20T11:23:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:27.848 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:27.849 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:27.849 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.831+0000 7f2d09f3e640 1 librados: starting msgr at 2024-01-20T11:23:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.831+0000 7f2d09f3e640 1 librados: starting objecter 2024-01-20T11:23:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.831+0000 7f2d09f3e640 1 Processor -- start 2024-01-20T11:23:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.831+0000 7f2d09f3e640 1 -- start start 2024-01-20T11:23:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.831+0000 7f2d09f3e640 1 librados: setting wanted keys 2024-01-20T11:23:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.831+0000 7f2d09f3e640 1 librados: calling monclient init 2024-01-20T11:23:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.831+0000 7f2d09f3e640 10 monclient: init 2024-01-20T11:23:27.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d0406bb00 0x7f2d04120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d0406bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 0x7f2d04121770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient(hunting): picked mon.noname-b con 0x7f2d040aeb20 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d040af790 0x7f2d041dac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient(hunting): picked mon.noname-a con 0x7f2d040af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:27.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d04122f50 con 0x7f2d040af790 2024-01-20T11:23:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d04121010 con 0x7f2d040aeb20 2024-01-20T11:23:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0411e5f0 con 0x7f2d0406bb00 2024-01-20T11:23:27.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d09f3e640 10 monclient(hunting): authenticate will time out at 1851.183688s 2024-01-20T11:23:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.832+0000 7f2d03fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d040af790 0x7f2d041dac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d037fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d0406bb00 0x7f2d04120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d02ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 0x7f2d04121770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d03fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d040af790 0x7f2d041dac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50496/0 (socket says 172.21.15.110:50496) 2024-01-20T11:23:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d03fff640 1 -- 172.21.15.110:0/3586031948 learned_addr learned my addr 172.21.15.110:0/3586031948 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:27.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d03fff640 10 monclient(hunting): get_auth_request con 0x7f2d040af790 auth_method 0 2024-01-20T11:23:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d03fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d02ffd640 10 monclient(hunting): get_auth_request con 0x7f2d040aeb20 auth_method 0 2024-01-20T11:23:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d02ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d037fe640 10 monclient(hunting): get_auth_request con 0x7f2d0406bb00 auth_method 0 2024-01-20T11:23:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d037fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d037fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:27.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.833+0000 7f2d02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.834+0000 7f2d037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.834+0000 7f2d037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.834+0000 7f2d037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 10 monclient(hunting): handle_auth_done global_id 6157 payload 824 2024-01-20T11:23:27.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 1 -- 172.21.15.110:0/3586031948 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d0406bb00 msgr2=0x7f2d04120ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d0406bb00 0x7f2d04120ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 1 -- 172.21.15.110:0/3586031948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d040af790 msgr2=0x7f2d041dac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d040af790 0x7f2d041dac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:23:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 1 monclient: found mon.noname-b 2024-01-20T11:23:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.836+0000 7f2d02ffd640 1 -- 172.21.15.110:0/3586031948 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc009a70 con 0x7f2d040aeb20 2024-01-20T11:23:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.838+0000 7f2d02ffd640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 0x7f2d04121770 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2d040afbc0 tx=0x7f2cf800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.838+0000 7f2cebfff640 10 client.?.objecter ms_handle_connect 0x7f2d040aeb20 2024-01-20T11:23:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.838+0000 7f2cebfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 1 -- 172.21.15.110:0/3586031948 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf80052d0 con 0x7f2d040aeb20 2024-01-20T11:23:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:27.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 10 monclient: dump: 2024-01-20T11:23:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:27.859 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:27.860 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 10 monclient: _finish_auth 0 2024-01-20T11:23:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 1 -- 172.21.15.110:0/3586031948 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf8005430 con 0x7f2d040aeb20 2024-01-20T11:23:27.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 1 -- 172.21.15.110:0/3586031948 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8006a60 con 0x7f2d040aeb20 2024-01-20T11:23:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d09f3e640 5 monclient: authenticate success, global_id 6157 2024-01-20T11:23:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d09f3e640 10 monclient: _renew_subs 2024-01-20T11:23:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d09f3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d04121cb0 con 0x7f2d040aeb20 2024-01-20T11:23:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:27.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2cebfff640 10 monclient: dump: 2024-01-20T11:23:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:27.862 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:27.863 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d01ffb640 4 set_mon_vals no callback set 2024-01-20T11:23:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d09f3e640 10 client.6157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d09f3e640 10 monclient: _renew_subs 2024-01-20T11:23:27.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d09f3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d041db340 con 0x7f2d040aeb20 2024-01-20T11:23:27.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.842+0000 7f2d09f3e640 1 librados: init done 2024-01-20T11:23:27.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.843+0000 7f2d09f3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:27.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.843+0000 7f2d09f3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:27.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.843+0000 7f2d09f3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:27.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.843+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d041db560 con 0x7f2d040aeb20 2024-01-20T11:23:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.861+0000 7f2cebfff640 1 -- 172.21.15.110:0/3586031948 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2cf8003640 con 0x7f2d040aeb20 2024-01-20T11:23:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.862+0000 7f2cebfff640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2cf006c8b0 0x7f2cf006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:27.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.862+0000 7f2d037fe640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2cf006c8b0 0x7f2cf006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.862+0000 7f2d037fe640 10 monclient: get_auth_request con 0x7f2cf006c8b0 auth_method 0 2024-01-20T11:23:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.862+0000 7f2d037fe640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2cf006c8b0 0x7f2cf006ed70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4005880 tx=0x7f2cf4009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.862+0000 7f2cebfff640 10 client.6157.objecter ms_handle_connect 0x7f2cf006c8b0 2024-01-20T11:23:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.864+0000 7f2cebfff640 1 -- 172.21.15.110:0/3586031948 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2cf8057990 con 0x7f2d040aeb20 2024-01-20T11:23:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.864+0000 7f2cebfff640 10 client.6157.objecter ms_dispatch 0x7f2d04000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.864+0000 7f2cebfff640 3 client.6157.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:27.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.864+0000 7f2cebfff640 3 client.6157.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:27.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:27.864+0000 7f2cebfff640 20 client.6157.objecter dump_active .. 0 homeless 2024-01-20T11:23:28.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.091+0000 7f2cebfff640 1 -- 172.21.15.110:0/3586031948 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2cf8053cd0 con 0x7f2d040aeb20 2024-01-20T11:23:28.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.091+0000 7f2cebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:28.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.091+0000 7f2cebfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:28.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.269+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d0411e7b0 con 0x7f2cf006c8b0 2024-01-20T11:23:28.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.272+0000 7f2cebfff640 1 -- 172.21.15.110:0/3586031948 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63902 (secure 0 0 0) 0x7f2d0411e7b0 con 0x7f2cf006c8b0 2024-01-20T11:23:28.273 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:28.274 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:28.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 10 librados: watch_flush enter 2024-01-20T11:23:28.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 10 librados: watch_flush exit 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 20 client.6157.objecter shutdown clearing up homeless session... 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 10 client.6157.objecter successfully canceled tick 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2cf006c8b0 msgr2=0x7f2cf006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2cf006c8b0 0x7f2cf006ed70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4005880 tx=0x7f2cf4009040 comp rx=0 tx=0).stop 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 10 monclient: shutdown 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 msgr2=0x7f2d04121770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 0x7f2d04121770 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2d040afbc0 tx=0x7f2cf800c820 comp rx=0 tx=0).stop 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 shutdown_connections 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f2cf006c8b0 0x7f2cf006ed70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d040af790 0x7f2d041dac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d040aeb20 0x7f2d04121770 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 --2- 172.21.15.110:0/3586031948 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d0406bb00 0x7f2d04120ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:28.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 >> 172.21.15.110:0/3586031948 conn(0x7f2d04066d60 msgr2=0x7f2d040b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:28.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 shutdown_connections 2024-01-20T11:23:28.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 -- 172.21.15.110:0/3586031948 wait complete. 2024-01-20T11:23:28.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:28.274+0000 7f2d09f3e640 1 librados: shutdown 2024-01-20T11:23:28.286 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-01-20T11:23:26.811414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":4.5650000000000004},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":6.4219999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":9.1699999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":13.715}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":10.186999999999999},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":13.044}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":14.352},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":7.5419999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.754},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":15.468999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":11.762},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":16.172999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313319,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.9879999999999995},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.1520000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":14.186999999999999},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":6.6609999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":11.111000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":9.7370000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":8.2949999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":10.497999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":12.874000000000001},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":13.551}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":16.228000000000002},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":14.848000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313319,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":8.0129999999999999},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.2370000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":9.3960000000000008},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":11.398}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":8.8030000000000008},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":12.032}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.074},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":15.513999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":16.422999999999998},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":13.279}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":12.712},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.954000000000001}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444133,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":8.2729999999999997,"15min":9.6010000000000009},"min":{"1min":3.911,"5min":3.911,"15min":3.911},"max":{"1min":20.539999999999999,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.343},{"interface":"front","average":{"1min":9.0039999999999996,"5min":8.1969999999999992,"15min":8.5340000000000007},"min":{"1min":4.2839999999999998,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":15.808999999999999,"5min":17.154,"15min":17.154},"last":7.5819999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":8.7940000000000005,"5min":8.6129999999999995,"15min":12.054},"min":{"1min":5.1959999999999997,"5min":4.5890000000000004,"15min":4.1660000000000004},"max":{"1min":16.329999999999998,"5min":17.024000000000001,"15min":65.064999999999998},"last":5.8879999999999999},{"interface":"front","average":{"1min":11.132999999999999,"5min":11.146000000000001,"15min":14.635},"min":{"1min":5.508,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":17.391999999999999,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.291}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.881},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":9.1300000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":13.446},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.709}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:22:23 2024","interfaces":[{"interface":"back","average":{"1min":14.833,"5min":13.718,"15min":15.182},"min":{"1min":9.7959999999999994,"5min":8.1170000000000009,"15min":7.6050000000000004},"max":{"1min":22.48,"5min":22.48,"15min":35.186999999999998},"last":16.029},{"interface":"front","average":{"1min":10.081,"5min":10.657,"15min":10.800000000000001},"min":{"1min":6.7110000000000003,"5min":5.6260000000000003,"15min":4.7850000000000001},"max":{"1min":16.463000000000001,"5min":16.632000000000001,"15min":32.984000000000002},"last":9.8100000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":11.031000000000001},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.32}]}]},{"osd":1,"up_from":17,"seq":73014444134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":13.811999999999999},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":5.4610000000000003}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":7.4169999999999998},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":6.7949999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":15.196999999999999},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":9.8190000000000008}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":8.5220000000000002},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":14.611000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":9.2360000000000007},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":10.494999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":12.997},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":11.092000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444134,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":8.0050000000000008},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":6.4100000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":9.7889999999999997},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":8.5969999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":10.343999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":5.8259999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":7.327},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":14.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":11.034000000000001},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":9.1850000000000005}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":11.712999999999999},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":14.815}]}]},{"osd":6,"up_from":15,"seq":64424509543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.984},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":10.144}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.9260000000000002},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":11.919}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.505000000000001},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":6.6369999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.141999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":12.885999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":15.753},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":8.7509999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":7.2549999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.609000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:31.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~273s 2024-01-20T11:23:31.288 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.386+0000 7fb803fb0640 10 monclient: get_monmap_and_config 2024-01-20T11:23:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.386+0000 7fb803fb0640 10 monclient: build_initial_monmap 2024-01-20T11:23:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.386+0000 7fb803fb0640 10 monclient: monmap: 2024-01-20T11:23:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:31.387 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.388 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:31.388055+0000 2024-01-20T11:23:31.388 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:31.388055+0000 2024-01-20T11:23:31.388 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:31.388 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.388 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:31.391 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 1 Processor -- start 2024-01-20T11:23:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 1 -- start start 2024-01-20T11:23:31.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 10 monclient: init 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7fc071c60 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7fc0ac540 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc0b0930 0x7fb7fc14a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 10 monclient(hunting): picked mon.noname-b con 0x7fb7fc0b0930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0ad610 con 0x7fb7fc071c60 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0721b0 con 0x7fb7fc0ac540 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc072040 con 0x7fb7fc0b0930 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.387+0000 7fb803fb0640 10 monclient(hunting): authenticate will time out at 1854.738989s 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.388+0000 7fb801d4c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb7fc071c60 0x7fb7fc142180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46982/0 (socket says 172.21.15.110:46982) 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.388+0000 7fb801d4c640 1 -- 172.21.15.110:0/2440913061 learned_addr learned my addr 172.21.15.110:0/2440913061 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:31.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.388+0000 7fb80254d640 1 --2- 172.21.15.110:0/2440913061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc0b0930 0x7fb7fc14a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.388+0000 7fb80254d640 10 monclient(hunting): get_auth_request con 0x7fb7fc0b0930 auth_method 0 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.388+0000 7fb80254d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.388+0000 7fb80254d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.388+0000 7fb80254d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.389+0000 7fb80254d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.389+0000 7fb80254d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.389+0000 7fb80254d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 1 -- 172.21.15.110:0/2440913061 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495589126 0 0) 0x7fb7fc0721b0 con 0x7fb7fc0ac540 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 10 monclient(hunting): _negotiate 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 10 monclient(hunting): my global_id is 6160 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 1 -- 172.21.15.110:0/2440913061 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e8005730 con 0x7fb7fc0ac540 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 1 -- 172.21.15.110:0/2440913061 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448704677 0 0) 0x7fb7fc0ad610 con 0x7fb7fc071c60 2024-01-20T11:23:31.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 10 monclient(hunting): _negotiate 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 10 monclient(hunting): my global_id is 6048 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb800d4a640 1 -- 172.21.15.110:0/2440913061 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc0721b0 con 0x7fb7fc071c60 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb80254d640 10 monclient(hunting): handle_auth_done global_id 5690 payload 274 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb80254d640 1 -- 172.21.15.110:0/2440913061 >> v1:172.21.15.134:6789/0 conn(0x7fb7fc0ac540 legacy=0x7fb7fc0ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb80254d640 1 -- 172.21.15.110:0/2440913061 >> v1:172.21.15.110:6789/0 conn(0x7fb7fc071c60 legacy=0x7fb7fc142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb80254d640 10 monclient: _finish_hunting 0 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb80254d640 1 monclient: found mon.noname-b 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb80254d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb80254d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.390+0000 7fb80254d640 1 -- 172.21.15.110:0/2440913061 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc072340 con 0x7fb7fc0b0930 2024-01-20T11:23:31.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.392+0000 7fb80254d640 1 --2- 172.21.15.110:0/2440913061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc0b0930 0x7fb7fc14a520 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4009dc0 tx=0x7fb7f402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c704a814e95eb1b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.394+0000 7fb800d4a640 1 -- 172.21.15.110:0/2440913061 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f402ccd0 con 0x7fb7fc0b0930 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.394+0000 7fb800d4a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.394+0000 7fb800d4a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.394+0000 7fb800d4a640 10 monclient: dump: 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:31.398 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.394+0000 7fb800d4a640 10 monclient: _finish_auth 0 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.394+0000 7fb800d4a640 10 monclient: _check_auth_tickets 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.394+0000 7fb800d4a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.394+0000 7fb803fb0640 5 monclient: authenticate success, global_id 5690 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.394+0000 7fb803fb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb800d4a640 1 -- 172.21.15.110:0/2440913061 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f402ce30 con 0x7fb7fc0b0930 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb800d4a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb800d4a640 1 -- 172.21.15.110:0/2440913061 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4034640 con 0x7fb7fc0b0930 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb800d4a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb800d4a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb800d4a640 10 monclient: dump: 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.399 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb803fb0640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb803fb0640 4 set_mon_vals no callback set 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb803fb0640 10 monclient: shutdown 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb803fb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb803fb0640 1 -- 172.21.15.110:0/2440913061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc0b0930 msgr2=0x7fb7fc14a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb803fb0640 1 --2- 172.21.15.110:0/2440913061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc0b0930 0x7fb7fc14a520 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4009dc0 tx=0x7fb7f402bf20 comp rx=0 tx=0).stop 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb803fb0640 1 -- 172.21.15.110:0/2440913061 shutdown_connections 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb803fb0640 1 --2- 172.21.15.110:0/2440913061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc0b0930 0x7fb7fc14a520 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.396+0000 7fb803fb0640 1 -- 172.21.15.110:0/2440913061 >> 172.21.15.110:0/2440913061 conn(0x7fb7fc0b3e60 msgr2=0x7fb7fc0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:31.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.397+0000 7fb803fb0640 1 -- 172.21.15.110:0/2440913061 shutdown_connections 2024-01-20T11:23:31.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.397+0000 7fb803fb0640 1 -- 172.21.15.110:0/2440913061 wait complete. 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.397+0000 7fb803fb0640 10 monclient: build_initial_monmap 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.397+0000 7fb803fb0640 10 monclient: monmap: 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:31.404 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.397+0000 7fb803fb0640 1 librados: starting msgr at 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.397+0000 7fb803fb0640 1 librados: starting objecter 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 Processor -- start 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 -- start start 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 librados: setting wanted keys 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 librados: calling monclient init 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient: init 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc071c60 0x7fb7fc141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient(hunting): picked mon.noname-c con 0x7fb7fc071c60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7fc0ac540 0x7fb7fc1d6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient(hunting): picked mon.noname-a con 0x7fb7fc0ac540 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7fc0b0930 0x7fb7fc1d74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient(hunting): picked mon.noname-b con 0x7fb7fc0b0930 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc1353c0 con 0x7fb7fc0ac540 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc06ebb0 con 0x7fb7fc0b0930 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc06ee90 con 0x7fb7fc071c60 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.398+0000 7fb803fb0640 10 monclient(hunting): authenticate will time out at 1854.749887s 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb801d4c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc071c60 0x7fb7fc141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7fc0ac540 0x7fb7fc1d6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80254d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7fc0b0930 0x7fb7fc1d74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7fc0ac540 0x7fb7fc1d6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58256/0 (socket says 172.21.15.110:58256) 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 1 -- 172.21.15.110:0/3313311787 learned_addr learned my addr 172.21.15.110:0/3313311787 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:31.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 10 monclient(hunting): get_auth_request con 0x7fb7fc0ac540 auth_method 0 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb801d4c640 10 monclient(hunting): get_auth_request con 0x7fb7fc071c60 auth_method 0 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb801d4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb801d4c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb801d4c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80254d640 10 monclient(hunting): get_auth_request con 0x7fb7fc0b0930 auth_method 0 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80254d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80254d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80254d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb80154b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:31.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb801d4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb801d4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.399+0000 7fb801d4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.400+0000 7fb80254d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.400+0000 7fb80254d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.400+0000 7fb80254d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 10 monclient(hunting): handle_auth_done global_id 6051 payload 824 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 1 -- 172.21.15.110:0/3313311787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc071c60 msgr2=0x7fb7fc141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc071c60 0x7fb7fc141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 1 -- 172.21.15.110:0/3313311787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7fc0b0930 msgr2=0x7fb7fc1d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7fc0b0930 0x7fb7fc1d74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 10 monclient: _finish_hunting 0 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 1 monclient: found mon.noname-a 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:31.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb80154b640 1 -- 172.21.15.110:0/3313311787 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4009a70 con 0x7fb7fc0ac540 2024-01-20T11:23:31.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.402+0000 7fb801d4c640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc071c60 0x7fb7fc141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:31.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.404+0000 7fb80154b640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7fc0ac540 0x7fb7fc1d6f70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00c430 tx=0x7fb7ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:31.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.404+0000 7fb7f27fc640 10 client.?.objecter ms_handle_connect 0x7fb7fc0ac540 2024-01-20T11:23:31.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.404+0000 7fb7f27fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 1 -- 172.21.15.110:0/3313311787 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec0052d0 con 0x7fb7fc0ac540 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 10 monclient: dump: 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:31.412 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 1 -- 172.21.15.110:0/3313311787 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec005430 con 0x7fb7fc0ac540 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb803fb0640 5 monclient: authenticate success, global_id 6051 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb803fb0640 10 monclient: _renew_subs 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 1 -- 172.21.15.110:0/3313311787 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec003810 con 0x7fb7fc0ac540 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb803fb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc135630 con 0x7fb7fc0ac540 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f3fff640 4 set_mon_vals no callback set 2024-01-20T11:23:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb7f27fc640 10 monclient: dump: 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb803fb0640 10 client.6051.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb803fb0640 10 monclient: _renew_subs 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb803fb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc1d7bd0 con 0x7fb7fc0ac540 2024-01-20T11:23:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.408+0000 7fb803fb0640 1 librados: init done 2024-01-20T11:23:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.409+0000 7fb803fb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.409+0000 7fb803fb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.409+0000 7fb803fb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.409+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc075490 con 0x7fb7fc0ac540 2024-01-20T11:23:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.427+0000 7fb7f27fc640 1 -- 172.21.15.110:0/3313311787 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb7ec003970 con 0x7fb7fc0ac540 2024-01-20T11:23:31.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.428+0000 7fb7f27fc640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7e006c890 0x7fb7e006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.428+0000 7fb801d4c640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7e006c890 0x7fb7e006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.428+0000 7fb801d4c640 10 monclient: get_auth_request con 0x7fb7e006c890 auth_method 0 2024-01-20T11:23:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.429+0000 7fb7f27fc640 10 client.6051.objecter ms_handle_connect 0x7fb7e006c890 2024-01-20T11:23:31.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.429+0000 7fb801d4c640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7e006c890 0x7fb7e006ed50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80011e0 tx=0x7fb7f8007d30 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.664+0000 7fb7f27fc640 1 -- 172.21.15.110:0/3313311787 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb7ec057cf0 con 0x7fb7fc0ac540 2024-01-20T11:23:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.664+0000 7fb7f27fc640 10 client.6051.objecter ms_dispatch 0x7fb7fc000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:31.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.664+0000 7fb7f27fc640 3 client.6051.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.664+0000 7fb7f27fc640 3 client.6051.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.664+0000 7fb7f27fc640 20 client.6051.objecter dump_active .. 0 homeless 2024-01-20T11:23:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.664+0000 7fb7f27fc640 1 -- 172.21.15.110:0/3313311787 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fb7ec054030 con 0x7fb7fc0ac540 2024-01-20T11:23:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.664+0000 7fb7f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:31.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.664+0000 7fb7f27fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:31.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.844+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7fc0a9480 con 0x7fb7e006c890 2024-01-20T11:23:31.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.847+0000 7fb7f27fc640 1 -- 172.21.15.110:0/3313311787 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63855 (secure 0 0 0) 0x7fb7f80072c0 con 0x7fb7e006c890 2024-01-20T11:23:31.848 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:31.849 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 10 librados: watch_flush enter 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 10 librados: watch_flush exit 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 20 client.6051.objecter shutdown clearing up homeless session... 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 10 client.6051.objecter successfully canceled tick 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7e006c890 msgr2=0x7fb7e006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7e006c890 0x7fb7e006ed50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80011e0 tx=0x7fb7f8007d30 comp rx=0 tx=0).stop 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 10 monclient: shutdown 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7fc0ac540 msgr2=0x7fb7fc1d6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7fc0ac540 0x7fb7fc1d6f70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00c430 tx=0x7fb7ec00c8f0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 shutdown_connections 2024-01-20T11:23:31.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fb7e006c890 0x7fb7e006ed50 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7fc0b0930 0x7fb7fc1d74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7fc0ac540 0x7fb7fc1d6f70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 --2- 172.21.15.110:0/3313311787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7fc071c60 0x7fb7fc141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 >> 172.21.15.110:0/3313311787 conn(0x7fb7fc0b3e60 msgr2=0x7fb7fc0a9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:31.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 shutdown_connections 2024-01-20T11:23:31.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 -- 172.21.15.110:0/3313311787 wait complete. 2024-01-20T11:23:31.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.849+0000 7fb803fb0640 1 librados: shutdown 2024-01-20T11:23:31.860 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-01-20T11:23:30.812364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002749"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":3.734},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":9.1159999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":10.385999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":6.6299999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":12.605},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.6959999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":8.5009999999999994},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":16.061}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":13.548999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":11.167999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":16.849},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":14.516}]}]},{"osd":7,"up_from":21,"seq":90194313319,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.9879999999999995},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.1520000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":14.186999999999999},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":6.6609999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":11.111000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":9.7370000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":8.2949999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":10.497999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":12.874000000000001},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":13.551}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":16.228000000000002},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":14.848000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313320,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":5.8460000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":7.4569999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":6.4489999999999998},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.577}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":8.6560000000000006},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":13.853}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":14.666},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.050000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":11.644},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":15.436}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":16.227},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.013}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":6.21},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":8.8399999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.1470000000000002},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":11.888}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":15.773},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":12.302}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":17.895},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":14.273999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":16.486000000000001},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":13.635999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":12.913},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":18.689}]}]},{"osd":1,"up_from":17,"seq":73014444135,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.6529999999999996},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":6.9420000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":13.709},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":8.6129999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":15.156000000000001},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":10.353999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.000999999999999},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":10.670999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":13.063000000000001},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":9.3559999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":14.449},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":11.682}]}]},{"osd":0,"up_from":17,"seq":73014444135,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.2169999999999996},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":10.978999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.2670000000000003},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":10.167999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":8.4239999999999995},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":13.949}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":15.423999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":13.359}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":12.391999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":18.672000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":17.515000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":14.686999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.984},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":10.144}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.9260000000000002},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":11.919}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.505000000000001},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":6.6369999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.141999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":12.885999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":15.753},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":8.7509999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":7.2549999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.609000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:31.862 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:31.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.963+0000 7f312f0c2640 10 monclient: get_monmap_and_config 2024-01-20T11:23:31.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.963+0000 7f312f0c2640 10 monclient: build_initial_monmap 2024-01-20T11:23:31.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.963+0000 7f312f0c2640 10 monclient: monmap: 2024-01-20T11:23:31.964 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:31.964 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.965 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:31.964638+0000 2024-01-20T11:23:31.965 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:31.964638+0000 2024-01-20T11:23:31.965 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:31.965 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.965 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:31.965 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:31.966 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:31.966 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:31.966 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:31.966 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.963+0000 7f312f0c2640 1 Processor -- start 2024-01-20T11:23:31.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.963+0000 7f312f0c2640 1 -- start start 2024-01-20T11:23:31.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.963+0000 7f312f0c2640 10 monclient: init 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f31280a96f0 0x7f312813f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 10 monclient(hunting): picked mon.noname-c con 0x7f31280a96f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f312813fa10 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f31280a9ad0 0x7f3128142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 10 monclient(hunting): picked mon.noname-a con 0x7f31280a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31280b25c0 con 0x7f312813fa10 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31280aaea0 con 0x7f31280a9ad0 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31280ab0a0 con 0x7f31280a96f0 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312f0c2640 10 monclient(hunting): authenticate will time out at 1855.315626s 2024-01-20T11:23:31.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.964+0000 7f312d65f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f31280a9ad0 0x7f3128142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312d65f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f31280a9ad0 0x7f3128142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58270/0 (socket says 172.21.15.110:58270) 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312d65f640 1 -- 172.21.15.110:0/412859874 learned_addr learned my addr 172.21.15.110:0/412859874 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312d65f640 10 monclient(hunting): get_auth_request con 0x7f31280a9ad0 auth_method 0 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312d65f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312d65f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312d65f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312ce5e640 1 --2- 172.21.15.110:0/412859874 >> v2:172.21.15.134:3300/0 conn(0x7f31280a96f0 0x7f312813f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312d65f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312d65f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.965+0000 7f312d65f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.966+0000 7f312ce5e640 10 monclient(hunting): get_auth_request con 0x7f31280a96f0 auth_method 0 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.966+0000 7f312ce5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:31.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.966+0000 7f312ce5e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.966+0000 7f312ce5e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312ce5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312ce5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312ce5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312d65f640 10 monclient(hunting): handle_auth_done global_id 6054 payload 274 2024-01-20T11:23:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312d65f640 1 -- 172.21.15.110:0/412859874 >> v2:172.21.15.134:3300/0 conn(0x7f31280a96f0 msgr2=0x7f312813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312d65f640 1 --2- 172.21.15.110:0/412859874 >> v2:172.21.15.134:3300/0 conn(0x7f31280a96f0 0x7f312813f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312d65f640 1 -- 172.21.15.110:0/412859874 >> v1:172.21.15.134:6789/0 conn(0x7f312813fa10 legacy=0x7f31281423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312d65f640 10 monclient: _finish_hunting 0 2024-01-20T11:23:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312d65f640 1 monclient: found mon.noname-a 2024-01-20T11:23:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312d65f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312d65f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.967+0000 7f312d65f640 1 -- 172.21.15.110:0/412859874 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31280b3fb0 con 0x7f31280a9ad0 2024-01-20T11:23:31.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.969+0000 7f312d65f640 1 --2- 172.21.15.110:0/412859874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a9ad0 0x7f3128142af0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f31280b5720 tx=0x7f312002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2230fb4972bfa478 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:31.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 1 -- 172.21.15.110:0/412859874 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312002ca80 con 0x7f31280a9ad0 2024-01-20T11:23:31.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:31.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:31.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 10 monclient: dump: 2024-01-20T11:23:31.974 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 1 -- 172.21.15.110:0/412859874 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312002cbe0 con 0x7f31280a9ad0 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:31.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f312f0c2640 5 monclient: authenticate success, global_id 6054 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 1 -- 172.21.15.110:0/412859874 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31200343f0 con 0x7f31280a9ad0 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f312f0c2640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f312f0c2640 4 set_mon_vals no callback set 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.972+0000 7f31277fe640 10 monclient: dump: 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 10 monclient: shutdown 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 1 -- 172.21.15.110:0/412859874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a9ad0 msgr2=0x7f3128142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 1 --2- 172.21.15.110:0/412859874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a9ad0 0x7f3128142af0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f31280b5720 tx=0x7f312002bee0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 1 -- 172.21.15.110:0/412859874 shutdown_connections 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 1 --2- 172.21.15.110:0/412859874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a9ad0 0x7f3128142af0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 1 --2- 172.21.15.110:0/412859874 >> v2:172.21.15.134:3300/0 conn(0x7f31280a96f0 0x7f312813f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 1 -- 172.21.15.110:0/412859874 >> 172.21.15.110:0/412859874 conn(0x7f31280b7bc0 msgr2=0x7f31280b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 1 -- 172.21.15.110:0/412859874 shutdown_connections 2024-01-20T11:23:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.973+0000 7f312f0c2640 1 -- 172.21.15.110:0/412859874 wait complete. 2024-01-20T11:23:31.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.974+0000 7f312f0c2640 10 monclient: build_initial_monmap 2024-01-20T11:23:31.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.974+0000 7f312f0c2640 10 monclient: monmap: 2024-01-20T11:23:31.981 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.974+0000 7f312f0c2640 1 librados: starting msgr at 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.974+0000 7f312f0c2640 1 librados: starting objecter 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.974+0000 7f312f0c2640 1 Processor -- start 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.974+0000 7f312f0c2640 1 -- start start 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.974+0000 7f312f0c2640 1 librados: setting wanted keys 2024-01-20T11:23:31.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.974+0000 7f312f0c2640 1 librados: calling monclient init 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.974+0000 7f312f0c2640 10 monclient: init 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a96f0 0x7f312813a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient(hunting): picked mon.noname-a con 0x7f31280a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31280a9ad0 0x7f312813b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient(hunting): picked mon.noname-c con 0x7f31280a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31280ac4d0 0x7f312813b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient(hunting): picked mon.noname-b con 0x7f31280ac4d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312813acd0 con 0x7f31280a96f0 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3128138f00 con 0x7f31280ac4d0 2024-01-20T11:23:31.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31280aba80 con 0x7f31280a9ad0 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312f0c2640 10 monclient(hunting): authenticate will time out at 1855.326601s 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f312ce5e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a96f0 0x7f312813a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.975+0000 7f3127fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31280a9ad0 0x7f312813b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312ce5e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a96f0 0x7f312813a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58276/0 (socket says 172.21.15.110:58276) 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312ce5e640 1 -- 172.21.15.110:0/3921162481 learned_addr learned my addr 172.21.15.110:0/3921162481 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312ce5e640 10 monclient(hunting): get_auth_request con 0x7f31280a96f0 auth_method 0 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312ce5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312ce5e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312ce5e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312d65f640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31280ac4d0 0x7f312813b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f3127fff640 10 monclient(hunting): get_auth_request con 0x7f31280a9ad0 auth_method 0 2024-01-20T11:23:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f3127fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f3127fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f3127fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312d65f640 10 monclient(hunting): get_auth_request con 0x7f31280ac4d0 auth_method 0 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312d65f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312d65f640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312d65f640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312ce5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312ce5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f312ce5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f3127fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f3127fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.976+0000 7f3127fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.977+0000 7f312d65f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.977+0000 7f312d65f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.977+0000 7f312d65f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 10 monclient(hunting): handle_auth_done global_id 5696 payload 824 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 1 -- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31280ac4d0 msgr2=0x7f312813b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31280ac4d0 0x7f312813b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 1 -- 172.21.15.110:0/3921162481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a96f0 msgr2=0x7f312813a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a96f0 0x7f312813a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 1 monclient: found mon.noname-c 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:23:31.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.979+0000 7f3127fff640 1 -- 172.21.15.110:0/3921162481 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3120009b30 con 0x7f31280a9ad0 2024-01-20T11:23:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.982+0000 7f3127fff640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31280a9ad0 0x7f312813b100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f31280aac70 tx=0x7f311c00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.982+0000 7f31257fa640 10 client.?.objecter ms_handle_connect 0x7f31280a9ad0 2024-01-20T11:23:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.982+0000 7f31257fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 1 -- 172.21.15.110:0/3921162481 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c0052d0 con 0x7f31280a9ad0 2024-01-20T11:23:31.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 10 monclient: dump: 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 10 monclient: _finish_auth 0 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 10 monclient: _check_auth_tickets 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 1 -- 172.21.15.110:0/3921162481 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311c005450 con 0x7f31280a9ad0 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 1 -- 172.21.15.110:0/3921162481 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c007260 con 0x7f31280a9ad0 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f31257fa640 10 monclient: dump: 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:31.990 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f312f0c2640 5 monclient: authenticate success, global_id 5696 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f312f0c2640 10 monclient: _renew_subs 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f312f0c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31280abd60 con 0x7f31280a9ad0 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f3126ffd640 4 set_mon_vals no callback set 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f312f0c2640 10 client.5696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f312f0c2640 10 monclient: _renew_subs 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f312f0c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312813bb80 con 0x7f31280a9ad0 2024-01-20T11:23:31.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.985+0000 7f312f0c2640 1 librados: init done 2024-01-20T11:23:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.986+0000 7f312f0c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.986+0000 7f312f0c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.986+0000 7f312f0c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:31.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:31.986+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312813be00 con 0x7f31280a9ad0 2024-01-20T11:23:32.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.004+0000 7f31257fa640 1 -- 172.21.15.110:0/3921162481 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f311c0073c0 con 0x7f31280a9ad0 2024-01-20T11:23:32.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.005+0000 7f31257fa640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f311406c8b0 0x7f311406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:32.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.005+0000 7f312ce5e640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f311406c8b0 0x7f311406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:32.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.005+0000 7f312ce5e640 10 monclient: get_auth_request con 0x7f311406c8b0 auth_method 0 2024-01-20T11:23:32.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.005+0000 7f312ce5e640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f311406c8b0 0x7f311406ed70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3118009be0 tx=0x7f3118009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:32.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.005+0000 7f31257fa640 10 client.5696.objecter ms_handle_connect 0x7f311406c8b0 2024-01-20T11:23:32.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.006+0000 7f31257fa640 1 -- 172.21.15.110:0/3921162481 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f311c0578e0 con 0x7f31280a9ad0 2024-01-20T11:23:32.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.006+0000 7f31257fa640 10 client.5696.objecter ms_dispatch 0x7f3128000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:32.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.006+0000 7f31257fa640 3 client.5696.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:32.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.006+0000 7f31257fa640 3 client.5696.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:32.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.006+0000 7f31257fa640 20 client.5696.objecter dump_active .. 0 homeless 2024-01-20T11:23:32.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.234+0000 7f31257fa640 1 -- 172.21.15.110:0/3921162481 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f311c053c20 con 0x7f31280a9ad0 2024-01-20T11:23:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.234+0000 7f31257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.234+0000 7f31257fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:32.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.409+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3128075490 con 0x7f311406c8b0 2024-01-20T11:23:32.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.413+0000 7f31257fa640 1 -- 172.21.15.110:0/3921162481 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63855 (secure 0 0 0) 0x7f3128075490 con 0x7f311406c8b0 2024-01-20T11:23:32.416 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 10 librados: watch_flush enter 2024-01-20T11:23:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 10 librados: watch_flush exit 2024-01-20T11:23:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 20 client.5696.objecter shutdown clearing up homeless session... 2024-01-20T11:23:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 10 client.5696.objecter successfully canceled tick 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f311406c8b0 msgr2=0x7f311406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f311406c8b0 0x7f311406ed70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3118009be0 tx=0x7f3118009190 comp rx=0 tx=0).stop 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 10 monclient: shutdown 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31280a9ad0 msgr2=0x7f312813b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31280a9ad0 0x7f312813b100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f31280aac70 tx=0x7f311c00c820 comp rx=0 tx=0).stop 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 shutdown_connections 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f311406c8b0 0x7f311406ed70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.414+0000 7f312f0c2640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31280ac4d0 0x7f312813b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.415+0000 7f312f0c2640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31280a9ad0 0x7f312813b100 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.415+0000 7f312f0c2640 1 --2- 172.21.15.110:0/3921162481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31280a96f0 0x7f312813a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.415+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 >> 172.21.15.110:0/3921162481 conn(0x7f31280b7bc0 msgr2=0x7f31280b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:32.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.415+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 shutdown_connections 2024-01-20T11:23:32.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.415+0000 7f312f0c2640 1 -- 172.21.15.110:0/3921162481 wait complete. 2024-01-20T11:23:32.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.415+0000 7f312f0c2640 1 librados: shutdown 2024-01-20T11:23:32.426 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-01-20T11:23:30.812364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002749"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:31.739289+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:19:31.739289+0000","last_peered":"2024-01-20T11:19:31.739289+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:19:31.739289+0000","last_undegraded":"2024-01-20T11:19:31.739289+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":3.734},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":9.1159999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":10.385999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":6.6299999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":12.605},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.6959999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":8.5009999999999994},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":16.061}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":13.548999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":11.167999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":16.849},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":14.516}]}]},{"osd":7,"up_from":21,"seq":90194313319,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":8.9879999999999995},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":7.1520000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":14.186999999999999},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":6.6609999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":11.111000000000001},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":9.7370000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":8.2949999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":10.497999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":12.874000000000001},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":13.551}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":16.228000000000002},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":14.848000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313320,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":5.8460000000000001},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":7.4569999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":6.4489999999999998},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.577}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":8.6560000000000006},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":13.853}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":14.666},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":12.050000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":11.644},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":15.436}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":16.227},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":13.013}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":6.21},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":8.8399999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.1470000000000002},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":11.888}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":15.773},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":12.302}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":17.895},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":14.273999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":16.486000000000001},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":13.635999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":12.913},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":18.689}]}]},{"osd":1,"up_from":17,"seq":73014444135,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.6529999999999996},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":6.9420000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":13.709},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":8.6129999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":15.156000000000001},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":10.353999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.000999999999999},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":10.670999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":13.063000000000001},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":9.3559999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":14.449},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":11.682}]}]},{"osd":0,"up_from":17,"seq":73014444135,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.2169999999999996},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":10.978999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.2670000000000003},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":10.167999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":8.4239999999999995},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":13.949}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":15.423999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":13.359}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":12.391999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":18.672000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":17.515000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":14.686999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.0229999999999997,"5min":7.4089999999999998,"15min":10.131},"min":{"1min":4.0170000000000003,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.696,"5min":14.696,"15min":106.059},"last":10.984},{"interface":"front","average":{"1min":8.7699999999999996,"5min":8.3230000000000004,"15min":12.25},"min":{"1min":4.2560000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":14.217000000000001,"5min":14.807,"15min":157.67699999999999},"last":10.144}]},{"osd":1,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":8.1980000000000004,"5min":9.2379999999999995,"15min":22.050999999999998},"min":{"1min":5.2770000000000001,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":12.175000000000001,"5min":16.399000000000001,"15min":189.37700000000001},"last":5.9260000000000002},{"interface":"front","average":{"1min":10.986000000000001,"5min":11.058,"15min":23.510999999999999},"min":{"1min":5.6619999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":16.228000000000002,"5min":17.751000000000001,"15min":190.304},"last":11.919}]},{"osd":2,"last update":"Sat Jan 20 11:22:27 2024","interfaces":[{"interface":"back","average":{"1min":12.018000000000001,"5min":10.882999999999999,"15min":13.366},"min":{"1min":7.4219999999999997,"5min":6.2549999999999999,"15min":5.3300000000000001},"max":{"1min":16.905000000000001,"5min":17.491,"15min":128.67099999999999},"last":14.505000000000001},{"interface":"front","average":{"1min":9.8789999999999996,"5min":10.548999999999999,"15min":14.054},"min":{"1min":5.8760000000000003,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":14.379,"5min":17.050000000000001,"15min":101.75700000000001},"last":6.6369999999999996}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":15.141999999999999},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":12.885999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":15.753},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":8.7509999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":7.2549999999999999},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":16.609000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:32.428 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:32.428 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:32.428 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:32.428 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:32.428 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:32.429 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:31.739289+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:19:31.739289+0000', 'last_peered': '2024-01-20T11:19:31.739289+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:19:31.739289+0000', 'last_undegraded': '2024-01-20T11:19:31.739289+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:32.429 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:32.429 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 131, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:21:18.077638+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:21:18.077638+0000', 'last_peered': '2024-01-20T11:21:18.077638+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:21:18.077638+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:32.429 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.530+0000 7fe92be56640 10 monclient: get_monmap_and_config 2024-01-20T11:23:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.530+0000 7fe92be56640 10 monclient: build_initial_monmap 2024-01-20T11:23:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.530+0000 7fe92be56640 10 monclient: monmap: 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:32.531820+0000 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:32.531820+0000 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:32.532 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.530+0000 7fe92be56640 1 Processor -- start 2024-01-20T11:23:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 1 -- start start 2024-01-20T11:23:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient: init 2024-01-20T11:23:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-01-20T11:23:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9240a96f0 0x7fe92413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient(hunting): picked mon.noname-a con 0x7fe9240a96f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe92413fa10 0x7fe9241423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient(hunting): picked mon.noname-c con 0x7fe92413fa10 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9240a9ad0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9240b25c0 con 0x7fe9240a9ad0 2024-01-20T11:23:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9240aaea0 con 0x7fe9240a96f0 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9240ab0a0 con 0x7fe92413fa10 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.531+0000 7fe92be56640 10 monclient(hunting): authenticate will time out at 1855.882794s 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9240a96f0 0x7fe92413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9240a96f0 0x7fe92413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58286/0 (socket says 172.21.15.110:58286) 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 1 -- 172.21.15.110:0/1614757121 learned_addr learned my addr 172.21.15.110:0/1614757121 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 10 monclient(hunting): get_auth_request con 0x7fe9240a96f0 auth_method 0 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe9293f1640 1 --2- 172.21.15.110:0/1614757121 >> v2:172.21.15.134:3300/0 conn(0x7fe92413fa10 0x7fe9241423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.532+0000 7fe929bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.533+0000 7fe9293f1640 10 monclient(hunting): get_auth_request con 0x7fe92413fa10 auth_method 0 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.533+0000 7fe9293f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.533+0000 7fe9293f1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.533+0000 7fe9293f1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe9293f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe9293f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe9293f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe929bf2640 10 monclient(hunting): handle_auth_done global_id 6060 payload 274 2024-01-20T11:23:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe929bf2640 1 -- 172.21.15.110:0/1614757121 >> v2:172.21.15.134:3300/0 conn(0x7fe92413fa10 msgr2=0x7fe9241423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe929bf2640 1 --2- 172.21.15.110:0/1614757121 >> v2:172.21.15.134:3300/0 conn(0x7fe92413fa10 0x7fe9241423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe929bf2640 1 -- 172.21.15.110:0/1614757121 >> v1:172.21.15.134:6789/0 conn(0x7fe9240a9ad0 legacy=0x7fe924142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe929bf2640 10 monclient: _finish_hunting 0 2024-01-20T11:23:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe929bf2640 1 monclient: found mon.noname-a 2024-01-20T11:23:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe929bf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe929bf2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.534+0000 7fe929bf2640 1 -- 172.21.15.110:0/1614757121 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240b3fb0 con 0x7fe9240a96f0 2024-01-20T11:23:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.536+0000 7fe929bf2640 1 --2- 172.21.15.110:0/1614757121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a96f0 0x7fe92413f4d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe92406d0f0 tx=0x7fe92002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc6fe5ec5b2a13b server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 1 -- 172.21.15.110:0/1614757121 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920031870 con 0x7fe9240a96f0 2024-01-20T11:23:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 10 monclient: dump: 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:32.541 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 10 monclient: _finish_auth 0 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 10 monclient: _check_auth_tickets 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 1 -- 172.21.15.110:0/1614757121 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9200319d0 con 0x7fe9240a96f0 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe92be56640 5 monclient: authenticate success, global_id 6060 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe92be56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe92be56640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 1 -- 172.21.15.110:0/1614757121 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920031d60 con 0x7fe9240a96f0 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe92be56640 4 set_mon_vals no callback set 2024-01-20T11:23:32.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.539+0000 7fe928bf0640 10 monclient: dump: 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:32.543 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 10 monclient: shutdown 2024-01-20T11:23:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 1 -- 172.21.15.110:0/1614757121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a96f0 msgr2=0x7fe92413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 1 --2- 172.21.15.110:0/1614757121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a96f0 0x7fe92413f4d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe92406d0f0 tx=0x7fe92002b910 comp rx=0 tx=0).stop 2024-01-20T11:23:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 1 -- 172.21.15.110:0/1614757121 shutdown_connections 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 1 --2- 172.21.15.110:0/1614757121 >> v2:172.21.15.134:3300/0 conn(0x7fe92413fa10 0x7fe9241423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 1 --2- 172.21.15.110:0/1614757121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a96f0 0x7fe92413f4d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 1 -- 172.21.15.110:0/1614757121 >> 172.21.15.110:0/1614757121 conn(0x7fe9240b7bc0 msgr2=0x7fe9240b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 1 -- 172.21.15.110:0/1614757121 shutdown_connections 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.540+0000 7fe92be56640 1 -- 172.21.15.110:0/1614757121 wait complete. 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.541+0000 7fe92be56640 10 monclient: build_initial_monmap 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.541+0000 7fe92be56640 10 monclient: monmap: 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.541+0000 7fe92be56640 1 librados: starting msgr at 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.541+0000 7fe92be56640 1 librados: starting objecter 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 Processor -- start 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 -- start start 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 librados: setting wanted keys 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 librados: calling monclient init 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient: init 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9240a96f0 0x7fe92413a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient(hunting): picked mon.noname-c con 0x7fe9240a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a9ad0 0x7fe92413b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient(hunting): picked mon.noname-a con 0x7fe9240a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9241dab70 0x7fe9241db020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient(hunting): picked mon.noname-b con 0x7fe9241dab70 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92413aca0 con 0x7fe9240a9ad0 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9241da6c0 con 0x7fe9241dab70 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9241db560 con 0x7fe9240a96f0 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.542+0000 7fe92be56640 10 monclient(hunting): authenticate will time out at 1855.893808s 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a9ad0 0x7fe92413b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe929bf2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9240a96f0 0x7fe92413a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe92a3f3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9241dab70 0x7fe9241db020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a9ad0 0x7fe92413b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58288/0 (socket says 172.21.15.110:58288) 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 1 -- 172.21.15.110:0/2986488333 learned_addr learned my addr 172.21.15.110:0/2986488333 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 10 monclient(hunting): get_auth_request con 0x7fe9240a9ad0 auth_method 0 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe92a3f3640 10 monclient(hunting): get_auth_request con 0x7fe9241dab70 auth_method 0 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe92a3f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe92a3f3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe92a3f3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe929bf2640 10 monclient(hunting): get_auth_request con 0x7fe9240a96f0 auth_method 0 2024-01-20T11:23:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe929bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe929bf2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe929bf2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.543+0000 7fe9293f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.544+0000 7fe92a3f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.544+0000 7fe92a3f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.544+0000 7fe92a3f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.544+0000 7fe929bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.544+0000 7fe929bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.544+0000 7fe929bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 10 monclient(hunting): handle_auth_done global_id 6063 payload 824 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 1 -- 172.21.15.110:0/2986488333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9240a96f0 msgr2=0x7fe92413a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9240a96f0 0x7fe92413a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 1 -- 172.21.15.110:0/2986488333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9241dab70 msgr2=0x7fe9241db020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9241dab70 0x7fe9241db020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 10 monclient: _finish_hunting 0 2024-01-20T11:23:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 1 monclient: found mon.noname-a 2024-01-20T11:23:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.546+0000 7fe9293f1640 1 -- 172.21.15.110:0/2986488333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9200094d0 con 0x7fe9240a9ad0 2024-01-20T11:23:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.548+0000 7fe9293f1640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a9ad0 0x7fe92413b330 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe91400e870 tx=0x7fe91400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:32.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.548+0000 7fe91a7fc640 10 client.?.objecter ms_handle_connect 0x7fe9240a9ad0 2024-01-20T11:23:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.548+0000 7fe91a7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe91a7fc640 1 -- 172.21.15.110:0/2986488333 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91400ee60 con 0x7fe9240a9ad0 2024-01-20T11:23:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe91a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe91a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe91a7fc640 10 monclient: dump: 2024-01-20T11:23:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:32.558 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe91a7fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe91a7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe91a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe91a7fc640 1 -- 172.21.15.110:0/2986488333 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91400c940 con 0x7fe9240a9ad0 2024-01-20T11:23:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe92be56640 5 monclient: authenticate success, global_id 6063 2024-01-20T11:23:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.551+0000 7fe92be56640 10 monclient: _renew_subs 2024-01-20T11:23:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92413b8d0 con 0x7fe9240a9ad0 2024-01-20T11:23:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe91a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe91a7fc640 1 -- 172.21.15.110:0/2986488333 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91400ccf0 con 0x7fe9240a9ad0 2024-01-20T11:23:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe91a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe91a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe91a7fc640 10 monclient: dump: 2024-01-20T11:23:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe91bfff640 4 set_mon_vals no callback set 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 10 client.6063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 10 monclient: _renew_subs 2024-01-20T11:23:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe924139850 con 0x7fe9240a9ad0 2024-01-20T11:23:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 1 librados: init done 2024-01-20T11:23:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.552+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe924075490 con 0x7fe9240a9ad0 2024-01-20T11:23:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.571+0000 7fe91a7fc640 1 -- 172.21.15.110:0/2986488333 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe914013470 con 0x7fe9240a9ad0 2024-01-20T11:23:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.571+0000 7fe91a7fc640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe90806c890 0x7fe90806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:32.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.572+0000 7fe929bf2640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe90806c890 0x7fe90806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:32.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.572+0000 7fe929bf2640 10 monclient: get_auth_request con 0x7fe90806c890 auth_method 0 2024-01-20T11:23:32.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.572+0000 7fe929bf2640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe90806c890 0x7fe90806ed50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe920036000 tx=0x7fe92003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:32.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.572+0000 7fe91a7fc640 10 client.6063.objecter ms_handle_connect 0x7fe90806c890 2024-01-20T11:23:32.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.802+0000 7fe91a7fc640 1 -- 172.21.15.110:0/2986488333 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe9140589d0 con 0x7fe9240a9ad0 2024-01-20T11:23:32.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.802+0000 7fe91a7fc640 10 client.6063.objecter ms_dispatch 0x7fe924000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.802+0000 7fe91a7fc640 3 client.6063.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.802+0000 7fe91a7fc640 3 client.6063.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.802+0000 7fe91a7fc640 20 client.6063.objecter dump_active .. 0 homeless 2024-01-20T11:23:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.802+0000 7fe91a7fc640 1 -- 172.21.15.110:0/2986488333 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe914054d10 con 0x7fe9240a9ad0 2024-01-20T11:23:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.802+0000 7fe91a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:32.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:32.802+0000 7fe91a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.046+0000 7fe92be56640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.046+0000 7fe92be56640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.046+0000 7fe92be56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:33.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.046+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fe924069850 con 0x7fe9240a9ad0 2024-01-20T11:23:33.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.052+0000 7fe91a7fc640 1 -- 172.21.15.110:0/2986488333 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fe914021520 con 0x7fe9240a9ad0 2024-01-20T11:23:33.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.052+0000 7fe91a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:33.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.052+0000 7fe91a7fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:33.053 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:33.053 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":547,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:33.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 10 librados: watch_flush enter 2024-01-20T11:23:33.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 10 librados: watch_flush exit 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 20 client.6063.objecter shutdown clearing up homeless session... 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 10 client.6063.objecter successfully canceled tick 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe90806c890 msgr2=0x7fe90806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe90806c890 0x7fe90806ed50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe920036000 tx=0x7fe92003c040 comp rx=0 tx=0).stop 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 10 monclient: shutdown 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a9ad0 msgr2=0x7fe92413b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a9ad0 0x7fe92413b330 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe91400e870 tx=0x7fe91400ed30 comp rx=0 tx=0).stop 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 shutdown_connections 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe90806c890 0x7fe90806ed50 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe9241dab70 0x7fe9241db020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9240a9ad0 0x7fe92413b330 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 --2- 172.21.15.110:0/2986488333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9240a96f0 0x7fe92413a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 >> 172.21.15.110:0/2986488333 conn(0x7fe9240b7bc0 msgr2=0x7fe9240b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:33.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 shutdown_connections 2024-01-20T11:23:33.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 -- 172.21.15.110:0/2986488333 wait complete. 2024-01-20T11:23:33.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.054+0000 7fe92be56640 1 librados: shutdown 2024-01-20T11:23:33.065 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:33.066 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.166+0000 7f4899fb0640 10 monclient: get_monmap_and_config 2024-01-20T11:23:33.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.166+0000 7f4899fb0640 10 monclient: build_initial_monmap 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.166+0000 7f4899fb0640 10 monclient: monmap: 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:33.168343+0000 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:33.168343+0000 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:33.168 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:33.169 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.167+0000 7f4899fb0640 1 Processor -- start 2024-01-20T11:23:33.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.167+0000 7f4899fb0640 1 -- start start 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.167+0000 7f4899fb0640 10 monclient: init 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f48940accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 10 monclient(hunting): picked mon.noname-b con 0x7f48940b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f489413d0e0 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f489413d520 0x7f4894143910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 10 monclient(hunting): picked mon.noname-a con 0x7f489413d520 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489406dcc0 con 0x7f489413d0e0 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489413d9b0 con 0x7f489413d520 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48940ad450 con 0x7f48940b0c00 2024-01-20T11:23:33.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4899fb0640 10 monclient(hunting): authenticate will time out at 1856.519401s 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f48937fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f48940accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.168+0000 7f4893fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f489413d520 0x7f4894143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f4893fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f489413d520 0x7f4894143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58302/0 (socket says 172.21.15.110:58302) 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f4893fff640 1 -- 172.21.15.110:0/2727746342 learned_addr learned my addr 172.21.15.110:0/2727746342 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f4893fff640 10 monclient(hunting): get_auth_request con 0x7f489413d520 auth_method 0 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f4893fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f4893fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f4893fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f48937fe640 10 monclient(hunting): get_auth_request con 0x7f48940b0c00 auth_method 0 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f48937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f48937fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f48937fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:33.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f48937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:33.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f48937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:33.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f48937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:33.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f4893fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:33.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f4893fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:33.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.169+0000 7f4893fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:33.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.172+0000 7f48937fe640 10 monclient(hunting): handle_auth_done global_id 5702 payload 274 2024-01-20T11:23:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.172+0000 7f48937fe640 1 -- 172.21.15.110:0/2727746342 >> v2:172.21.15.110:3300/0 conn(0x7f489413d520 msgr2=0x7f4894143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.172+0000 7f48937fe640 1 --2- 172.21.15.110:0/2727746342 >> v2:172.21.15.110:3300/0 conn(0x7f489413d520 0x7f4894143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.172+0000 7f48937fe640 1 -- 172.21.15.110:0/2727746342 >> v1:172.21.15.110:6789/0 conn(0x7f489413d0e0 legacy=0x7f4894143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.172+0000 7f48937fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.172+0000 7f48937fe640 1 monclient: found mon.noname-b 2024-01-20T11:23:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.172+0000 7f48937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.172+0000 7f48937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.172+0000 7f48937fe640 1 -- 172.21.15.110:0/2727746342 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48940ad8e0 con 0x7f48940b0c00 2024-01-20T11:23:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.174+0000 7f48937fe640 1 --2- 172.21.15.110:0/2727746342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f48940accf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f48840097a0 tx=0x7f488402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a167500f85d7c5bb server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 1 -- 172.21.15.110:0/2727746342 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884030710 con 0x7f48940b0c00 2024-01-20T11:23:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 10 monclient: dump: 2024-01-20T11:23:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:33.179 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 1 -- 172.21.15.110:0/2727746342 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488402e050 con 0x7f48940b0c00 2024-01-20T11:23:33.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f4899fb0640 5 monclient: authenticate success, global_id 5702 2024-01-20T11:23:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f4899fb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f4899fb0640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f4899fb0640 4 set_mon_vals no callback set 2024-01-20T11:23:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 1 -- 172.21.15.110:0/2727746342 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884030a90 con 0x7f48940b0c00 2024-01-20T11:23:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:33.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.178+0000 7f48927fc640 10 monclient: dump: 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:33.182 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 10 monclient: shutdown 2024-01-20T11:23:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 1 -- 172.21.15.110:0/2727746342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 msgr2=0x7f48940accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 1 --2- 172.21.15.110:0/2727746342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f48940accf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f48840097a0 tx=0x7f488402b710 comp rx=0 tx=0).stop 2024-01-20T11:23:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 1 -- 172.21.15.110:0/2727746342 shutdown_connections 2024-01-20T11:23:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 1 --2- 172.21.15.110:0/2727746342 >> v2:172.21.15.110:3300/0 conn(0x7f489413d520 0x7f4894143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 1 --2- 172.21.15.110:0/2727746342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f48940accf0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 1 -- 172.21.15.110:0/2727746342 >> 172.21.15.110:0/2727746342 conn(0x7f48940b3e60 msgr2=0x7f48940b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 1 -- 172.21.15.110:0/2727746342 shutdown_connections 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 1 -- 172.21.15.110:0/2727746342 wait complete. 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 10 monclient: build_initial_monmap 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.179+0000 7f4899fb0640 10 monclient: monmap: 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:33.187 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 librados: starting msgr at 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 librados: starting objecter 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 Processor -- start 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 -- start start 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 librados: setting wanted keys 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 librados: calling monclient init 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient: init 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f489413b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient(hunting): picked mon.noname-c con 0x7f48940b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f489413d0e0 0x7f4894137420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:33.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient(hunting): picked mon.noname-a con 0x7f489413d0e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f489413d520 0x7f4894138d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient(hunting): picked mon.noname-b con 0x7f489413d520 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4894135850 con 0x7f489413d0e0 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4894138650 con 0x7f489413d520 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4894138930 con 0x7f48940b0c00 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.180+0000 7f4899fb0640 10 monclient(hunting): authenticate will time out at 1856.532095s 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f48937fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f489413b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:33.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f4892ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f489413d0e0 0x7f4894137420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f4893fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f489413d520 0x7f4894138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f48937fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f489413b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:60636/0 (socket says 172.21.15.110:60636) 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f48937fe640 1 -- 172.21.15.110:0/2270276931 learned_addr learned my addr 172.21.15.110:0/2270276931 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f48937fe640 10 monclient(hunting): get_auth_request con 0x7f48940b0c00 auth_method 0 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f48937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f48937fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f48937fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f4892ffd640 10 monclient(hunting): get_auth_request con 0x7f489413d0e0 auth_method 0 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f4892ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f4892ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.181+0000 7f4892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4893fff640 10 monclient(hunting): get_auth_request con 0x7f489413d520 auth_method 0 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4893fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4893fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:33.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4893fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f48937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f48937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f48937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4892ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4892ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4892ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4893fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4893fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.182+0000 7f4893fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 10 monclient(hunting): handle_auth_done global_id 5705 payload 824 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 1 -- 172.21.15.110:0/2270276931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f489413d520 msgr2=0x7f4894138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f489413d520 0x7f4894138d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 1 -- 172.21.15.110:0/2270276931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f489413d0e0 msgr2=0x7f4894137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:33.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f489413d0e0 0x7f4894137420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:33.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 1 monclient: found mon.noname-c 2024-01-20T11:23:33.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:33.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-20T11:23:33.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.185+0000 7f48937fe640 1 -- 172.21.15.110:0/2270276931 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4884009400 con 0x7f48940b0c00 2024-01-20T11:23:33.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.187+0000 7f48937fe640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f489413b7e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f48840098c0 tx=0x7f48840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.187+0000 7f487bfff640 10 client.?.objecter ms_handle_connect 0x7f48940b0c00 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.187+0000 7f487bfff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 1 -- 172.21.15.110:0/2270276931 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884032950 con 0x7f48940b0c00 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 10 monclient: dump: 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:33.194 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 10 monclient: _finish_auth 0 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 1 -- 172.21.15.110:0/2270276931 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4884032ad0 con 0x7f48940b0c00 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 1 -- 172.21.15.110:0/2270276931 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884032e80 con 0x7f48940b0c00 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f487bfff640 10 monclient: dump: 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:33.195 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4899fb0640 5 monclient: authenticate success, global_id 5705 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4899fb0640 10 monclient: _renew_subs 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4899fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4894135470 con 0x7f48940b0c00 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4891ffb640 4 set_mon_vals no callback set 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4899fb0640 10 client.5705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:33.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4899fb0640 10 monclient: _renew_subs 2024-01-20T11:23:33.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4899fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:33.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4894139290 con 0x7f48940b0c00 2024-01-20T11:23:33.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.191+0000 7f4899fb0640 1 librados: init done 2024-01-20T11:23:33.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.192+0000 7f4899fb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:33.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.192+0000 7f4899fb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:33.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.192+0000 7f4899fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-20T11:23:33.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.192+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4894075490 con 0x7f48940b0c00 2024-01-20T11:23:33.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.430+0000 7f487bfff640 1 -- 172.21.15.110:0/2270276931 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4884039070 con 0x7f48940b0c00 2024-01-20T11:23:33.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.431+0000 7f487bfff640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f488006c8b0 0x7f488006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:33.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.431+0000 7f4892ffd640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f488006c8b0 0x7f488006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:33.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.431+0000 7f4892ffd640 10 monclient: get_auth_request con 0x7f488006c8b0 auth_method 0 2024-01-20T11:23:33.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.432+0000 7f4892ffd640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f488006c8b0 0x7f488006ed70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4894143a80 tx=0x7f4888009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:33.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.432+0000 7f487bfff640 10 client.5705.objecter ms_handle_connect 0x7f488006c8b0 2024-01-20T11:23:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.443+0000 7f487bfff640 1 -- 172.21.15.110:0/2270276931 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f488407a320 con 0x7f48940b0c00 2024-01-20T11:23:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.443+0000 7f487bfff640 10 client.5705.objecter ms_dispatch 0x7f4894000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.443+0000 7f487bfff640 3 client.5705.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.443+0000 7f487bfff640 3 client.5705.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.443+0000 7f487bfff640 20 client.5705.objecter dump_active .. 0 homeless 2024-01-20T11:23:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.443+0000 7f487bfff640 1 -- 172.21.15.110:0/2270276931 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4884076660 con 0x7f48940b0c00 2024-01-20T11:23:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.443+0000 7f487bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:33.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.443+0000 7f487bfff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:33.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.627+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4894135c40 con 0x7f488006c8b0 2024-01-20T11:23:33.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.630+0000 7f487bfff640 1 -- 172.21.15.110:0/2270276931 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63793 (secure 0 0 0) 0x7f4894135c40 con 0x7f488006c8b0 2024-01-20T11:23:33.631 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:33.632 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 10 librados: watch_flush enter 2024-01-20T11:23:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 10 librados: watch_flush exit 2024-01-20T11:23:33.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 20 client.5705.objecter shutdown clearing up homeless session... 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 10 client.5705.objecter successfully canceled tick 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f488006c8b0 msgr2=0x7f488006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f488006c8b0 0x7f488006ed70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4894143a80 tx=0x7f4888009040 comp rx=0 tx=0).stop 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 10 monclient: shutdown 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 msgr2=0x7f489413b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f489413b7e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f48840098c0 tx=0x7f48840046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 shutdown_connections 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f488006c8b0 0x7f488006ed70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f489413d520 0x7f4894138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f489413d0e0 0x7f4894137420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 --2- 172.21.15.110:0/2270276931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48940b0c00 0x7f489413b7e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:33.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 >> 172.21.15.110:0/2270276931 conn(0x7f48940b3e60 msgr2=0x7f48940ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:33.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 shutdown_connections 2024-01-20T11:23:33.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 -- 172.21.15.110:0/2270276931 wait complete. 2024-01-20T11:23:33.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:33.633+0000 7f4899fb0640 1 librados: shutdown 2024-01-20T11:23:33.647 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-01-20T11:23:32.812957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002744"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":3.734},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":9.1159999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":10.385999999999999},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":6.6299999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":12.605},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.6959999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":8.5009999999999994},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":16.061}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":13.548999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":11.167999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":16.849},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":14.516}]}]},{"osd":7,"up_from":21,"seq":90194313320,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":4.9480000000000004},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":6.8390000000000004}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":8.7720000000000002},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":12.859}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":14.491},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":7.8460000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":10.875999999999999},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":15.124000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":11.484999999999999},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":9.5730000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":10.279999999999999},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":12.179}]}]},{"osd":3,"up_from":21,"seq":90194313321,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":4.6920000000000002},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":7.3419999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":9.4580000000000002},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":14.504}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.037000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":7.7999999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.714},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":9.7639999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":12.657999999999999},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":11.976000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":11.426},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.135999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444134,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":6.21},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":8.8399999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":8.1470000000000002},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":11.888}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":15.773},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":12.302}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":17.895},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":14.273999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":16.486000000000001},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":13.635999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":12.913},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":18.689}]}]},{"osd":1,"up_from":17,"seq":73014444135,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":7.8769999999999998,"5min":8.0289999999999999,"15min":9.0860000000000003},"min":{"1min":4.766,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":15.151999999999999,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.6529999999999996},{"interface":"front","average":{"1min":8.2449999999999992,"5min":8.6630000000000003,"15min":9.8529999999999998},"min":{"1min":4.8419999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":16.481999999999999,"5min":16.992999999999999,"15min":30.439},"last":6.9420000000000002}]},{"osd":2,"last update":"Sat Jan 20 11:22:28 2024","interfaces":[{"interface":"back","average":{"1min":10.384,"5min":10.766999999999999,"15min":12.516},"min":{"1min":5.8620000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":14.872,"5min":17.009,"15min":24.725000000000001},"last":13.709},{"interface":"front","average":{"1min":8.9830000000000005,"5min":9.9459999999999997,"15min":12.311999999999999},"min":{"1min":6.1879999999999997,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":13.305999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":8.6129999999999995}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":15.156000000000001},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":10.353999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.000999999999999},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":10.670999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":13.063000000000001},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":9.3559999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":14.449},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":11.682}]}]},{"osd":0,"up_from":17,"seq":73014444135,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":7.7160000000000002,"15min":12.423999999999999},"min":{"1min":4.5890000000000004,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":22.428999999999998,"5min":22.428999999999998,"15min":114.843},"last":6.2169999999999996},{"interface":"front","average":{"1min":8.5,"5min":8.7949999999999999,"15min":13.323},"min":{"1min":4.2949999999999999,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":15.388,"5min":18.504999999999999,"15min":121.044},"last":10.978999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":9.1240000000000006,"5min":8.2260000000000009,"15min":22.457000000000001},"min":{"1min":5.7240000000000002,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":19.783000000000001,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.2670000000000003},{"interface":"front","average":{"1min":11.162000000000001,"5min":10.853,"15min":24.469000000000001},"min":{"1min":6.1150000000000002,"5min":4.819,"15min":4.819},"max":{"1min":23.260999999999999,"5min":23.260999999999999,"15min":373.88299999999998},"last":10.167999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":8.4239999999999995},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":13.949}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":15.423999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":13.359}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:26 2024","interfaces":[{"interface":"back","average":{"1min":14.763999999999999,"5min":13.332000000000001,"15min":27.045999999999999},"min":{"1min":9.093,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":28.087,"5min":28.087,"15min":375.733},"last":12.391999999999999},{"interface":"front","average":{"1min":13.304,"5min":13.881,"15min":25.611000000000001},"min":{"1min":9.2560000000000002,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":26.254999999999999,"5min":26.254999999999999,"15min":374.654},"last":18.672000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":17.515000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":14.686999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":6.8369999999999997},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":5.8739999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":10.797000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":12.131}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":7.9989999999999997},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":15.69}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.458,"5min":12.198,"15min":14.772},"min":{"1min":6.5549999999999997,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":16.440000000000001,"5min":18.550999999999998,"15min":81.373999999999995},"last":8.8670000000000009},{"interface":"front","average":{"1min":12.842000000000001,"5min":12.593999999999999,"15min":15.023999999999999},"min":{"1min":6.2290000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":17.616,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.486000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":9.8350000000000009},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.901}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.351000000000001,"5min":13.529,"15min":15.577},"min":{"1min":8.4039999999999999,"5min":8.1419999999999995,"15min":8.1419999999999995},"max":{"1min":16.850000000000001,"5min":18.468,"15min":25.303999999999998},"last":13.472},{"interface":"front","average":{"1min":12.115,"5min":12.904,"15min":15.042},"min":{"1min":7.718,"5min":7.149,"15min":7.149},"max":{"1min":17.273,"5min":19.515000000000001,"15min":22.838999999999999},"last":12.743}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:36.650 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:36.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.750+0000 7f09323a9640 10 monclient: get_monmap_and_config 2024-01-20T11:23:36.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.750+0000 7f09323a9640 10 monclient: build_initial_monmap 2024-01-20T11:23:36.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.750+0000 7f09323a9640 10 monclient: monmap: 2024-01-20T11:23:36.751 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:36.752134+0000 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:36.752134+0000 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:36.752 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 1 Processor -- start 2024-01-20T11:23:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 1 -- start start 2024-01-20T11:23:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 10 monclient: init 2024-01-20T11:23:36.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f092c145760 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 10 monclient(hunting): picked mon.noname-b con 0x7f092c0a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f092c13f8b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c146250 con 0x7f092c145760 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c0b20c0 con 0x7f092c13f8b0 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092c0ab050 con 0x7f092c0a9d80 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:36.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.751+0000 7f09323a9640 10 monclient(hunting): authenticate will time out at 1860.103175s 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.752+0000 7f092bfff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f092c145760 0x7f092c13f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46988/0 (socket says 172.21.15.110:46988) 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.752+0000 7f092bfff640 1 -- 172.21.15.110:0/1394651328 learned_addr learned my addr 172.21.15.110:0/1394651328 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.752+0000 7f092b7fe640 1 --2- 172.21.15.110:0/1394651328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.753+0000 7f092b7fe640 10 monclient(hunting): get_auth_request con 0x7f092c0a9d80 auth_method 0 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.753+0000 7f092b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.753+0000 7f092b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.753+0000 7f092b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.753+0000 7f092b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.753+0000 7f092b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:36.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.753+0000 7f092b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.754+0000 7f092affd640 1 -- 172.21.15.110:0/1394651328 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251669533 0 0) 0x7f092c146250 con 0x7f092c145760 2024-01-20T11:23:36.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.754+0000 7f092affd640 10 monclient(hunting): _negotiate 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.754+0000 7f092affd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.754+0000 7f092affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.754+0000 7f092affd640 10 monclient(hunting): my global_id is 6072 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.754+0000 7f092affd640 1 -- 172.21.15.110:0/1394651328 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0914003520 con 0x7f092c145760 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.755+0000 7f092b7fe640 10 monclient(hunting): handle_auth_done global_id 5708 payload 274 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.755+0000 7f092b7fe640 1 -- 172.21.15.110:0/1394651328 >> v1:172.21.15.134:6789/0 conn(0x7f092c13f8b0 legacy=0x7f092c0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.755+0000 7f092b7fe640 1 -- 172.21.15.110:0/1394651328 >> v1:172.21.15.110:6789/0 conn(0x7f092c145760 legacy=0x7f092c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.755+0000 7f092b7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.755+0000 7f092b7fe640 1 monclient: found mon.noname-b 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.755+0000 7f092b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.755+0000 7f092b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.755+0000 7f092b7fe640 1 -- 172.21.15.110:0/1394651328 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c0b2d80 con 0x7f092c0a9d80 2024-01-20T11:23:36.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.757+0000 7f092b7fe640 1 --2- 172.21.15.110:0/1394651328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c145b40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f09200097d0 tx=0x7f092002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13c2fab27ca8b59 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 1 -- 172.21.15.110:0/1394651328 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920031890 con 0x7f092c0a9d80 2024-01-20T11:23:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:36.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 10 monclient: dump: 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:36.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 10 monclient: _finish_auth 0 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 1 -- 172.21.15.110:0/1394651328 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09200319f0 con 0x7f092c0a9d80 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 1 -- 172.21.15.110:0/1394651328 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920031d80 con 0x7f092c0a9d80 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f09323a9640 5 monclient: authenticate success, global_id 5708 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f09323a9640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f09323a9640 4 set_mon_vals no callback set 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.760+0000 7f092affd640 10 monclient: dump: 2024-01-20T11:23:36.763 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:36.764 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:36.764 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 10 monclient: shutdown 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 1 -- 172.21.15.110:0/1394651328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 msgr2=0x7f092c145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 1 --2- 172.21.15.110:0/1394651328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c145b40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f09200097d0 tx=0x7f092002b930 comp rx=0 tx=0).stop 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 1 -- 172.21.15.110:0/1394651328 shutdown_connections 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 1 --2- 172.21.15.110:0/1394651328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c145b40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 1 -- 172.21.15.110:0/1394651328 >> 172.21.15.110:0/1394651328 conn(0x7f092c0663b0 msgr2=0x7f092c0667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 1 -- 172.21.15.110:0/1394651328 shutdown_connections 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 1 -- 172.21.15.110:0/1394651328 wait complete. 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 10 monclient: build_initial_monmap 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.761+0000 7f09323a9640 10 monclient: monmap: 2024-01-20T11:23:36.768 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.762+0000 7f09323a9640 1 librados: starting msgr at 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.762+0000 7f09323a9640 1 librados: starting objecter 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.762+0000 7f09323a9640 1 Processor -- start 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.762+0000 7f09323a9640 1 -- start start 2024-01-20T11:23:36.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.762+0000 7f09323a9640 1 librados: setting wanted keys 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.762+0000 7f09323a9640 1 librados: calling monclient init 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.762+0000 7f09323a9640 10 monclient: init 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.762+0000 7f09323a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 10 monclient(hunting): picked mon.noname-c con 0x7f092c0a9d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f092c13f8b0 0x7f092c0b7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 10 monclient(hunting): picked mon.noname-b con 0x7f092c13f8b0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f092c145760 0x7f092c0b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 10 monclient(hunting): picked mon.noname-a con 0x7f092c145760 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c0b7770 con 0x7f092c145760 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c137710 con 0x7f092c13f8b0 2024-01-20T11:23:36.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092c1379f0 con 0x7f092c0a9d80 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f09323a9640 10 monclient(hunting): authenticate will time out at 1860.114284s 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f092bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f0930946640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f092c145760 0x7f092c0b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f0930946640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f092c145760 0x7f092c0b80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58318/0 (socket says 172.21.15.110:58318) 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f0930946640 1 -- 172.21.15.110:0/3766543967 learned_addr learned my addr 172.21.15.110:0/3766543967 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f0930946640 10 monclient(hunting): get_auth_request con 0x7f092c145760 auth_method 0 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f0930946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f0930946640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f0930946640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f092b7fe640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f092c13f8b0 0x7f092c0b7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f092bfff640 10 monclient(hunting): get_auth_request con 0x7f092c0a9d80 auth_method 0 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f092bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:36.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f092bfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.763+0000 7f092bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092b7fe640 10 monclient(hunting): get_auth_request con 0x7f092c13f8b0 auth_method 0 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f0930946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f0930946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f0930946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.764+0000 7f092b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:36.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 10 monclient(hunting): handle_auth_done global_id 6075 payload 824 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 1 -- 172.21.15.110:0/3766543967 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 msgr2=0x7f092c142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c142250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 1 -- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f092c13f8b0 msgr2=0x7f092c0b7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f092c13f8b0 0x7f092c0b7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 10 monclient: _finish_hunting 0 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 1 monclient: found mon.noname-a 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f0930946640 1 -- 172.21.15.110:0/3766543967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920009400 con 0x7f092c145760 2024-01-20T11:23:36.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.767+0000 7f092bfff640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.769+0000 7f0930946640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f092c145760 0x7f092c0b80e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f092400c730 tx=0x7f092400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.769+0000 7f0928ff9640 10 client.?.objecter ms_handle_connect 0x7f092c145760 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.769+0000 7f0928ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f0928ff9640 1 -- 172.21.15.110:0/3766543967 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924003b70 con 0x7f092c145760 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f0928ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f0928ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f0928ff9640 10 monclient: dump: 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:36.775 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f0928ff9640 10 monclient: _finish_auth 0 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f0928ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f0928ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f0928ff9640 1 -- 172.21.15.110:0/3766543967 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0924003cd0 con 0x7f092c145760 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f09323a9640 5 monclient: authenticate success, global_id 6075 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f09323a9640 10 monclient: _renew_subs 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.772+0000 7f09323a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c137cd0 con 0x7f092c145760 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f0928ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f0928ff9640 1 -- 172.21.15.110:0/3766543967 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092400aa90 con 0x7f092c145760 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f0928ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f0928ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f0928ff9640 10 monclient: dump: 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:36.777 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f092a7fc640 4 set_mon_vals no callback set 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 10 client.6075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 10 monclient: _renew_subs 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092c1daea0 con 0x7f092c145760 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 1 librados: init done 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:36.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.773+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092c075490 con 0x7f092c145760 2024-01-20T11:23:36.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.791+0000 7f0928ff9640 1 -- 172.21.15.110:0/3766543967 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f092400abf0 con 0x7f092c145760 2024-01-20T11:23:36.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.791+0000 7f0928ff9640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f091806c8b0 0x7f091806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:36.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.791+0000 7f092bfff640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f091806c8b0 0x7f091806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:36.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.791+0000 7f092bfff640 10 monclient: get_auth_request con 0x7f091806c8b0 auth_method 0 2024-01-20T11:23:36.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.792+0000 7f092bfff640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f091806c8b0 0x7f091806ed70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f091c005880 tx=0x7f091c009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:36.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.792+0000 7f0928ff9640 10 client.6075.objecter ms_handle_connect 0x7f091806c8b0 2024-01-20T11:23:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.795+0000 7f0928ff9640 1 -- 172.21.15.110:0/3766543967 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f092400ce40 con 0x7f092c145760 2024-01-20T11:23:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.795+0000 7f0928ff9640 10 client.6075.objecter ms_dispatch 0x7f092c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.795+0000 7f0928ff9640 3 client.6075.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.795+0000 7f0928ff9640 3 client.6075.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:36.795+0000 7f0928ff9640 20 client.6075.objecter dump_active .. 0 homeless 2024-01-20T11:23:37.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.029+0000 7f0928ff9640 1 -- 172.21.15.110:0/3766543967 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0924054cb0 con 0x7f092c145760 2024-01-20T11:23:37.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.029+0000 7f0928ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:37.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.029+0000 7f0928ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:37.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.202+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f092c0b3380 con 0x7f091806c8b0 2024-01-20T11:23:37.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.204+0000 7f0928ff9640 1 -- 172.21.15.110:0/3766543967 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63722 (secure 0 0 0) 0x7f091c001800 con 0x7f091806c8b0 2024-01-20T11:23:37.206 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:37.206 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:37.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 10 librados: watch_flush enter 2024-01-20T11:23:37.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 10 librados: watch_flush exit 2024-01-20T11:23:37.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 20 client.6075.objecter shutdown clearing up homeless session... 2024-01-20T11:23:37.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 10 client.6075.objecter successfully canceled tick 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f091806c8b0 msgr2=0x7f091806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f091806c8b0 0x7f091806ed70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f091c005880 tx=0x7f091c009040 comp rx=0 tx=0).stop 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 10 monclient: shutdown 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f092c145760 msgr2=0x7f092c0b80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.206+0000 7f09323a9640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f092c145760 0x7f092c0b80e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f092400c730 tx=0x7f092400cbf0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.207+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 shutdown_connections 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.207+0000 7f09323a9640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f091806c8b0 0x7f091806ed70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.207+0000 7f09323a9640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f092c145760 0x7f092c0b80e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.207+0000 7f09323a9640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f092c13f8b0 0x7f092c0b7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.207+0000 7f09323a9640 1 --2- 172.21.15.110:0/3766543967 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f092c0a9d80 0x7f092c142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.207+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 >> 172.21.15.110:0/3766543967 conn(0x7f092c0663b0 msgr2=0x7f092c13d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:37.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.207+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 shutdown_connections 2024-01-20T11:23:37.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.207+0000 7f09323a9640 1 -- 172.21.15.110:0/3766543967 wait complete. 2024-01-20T11:23:37.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.207+0000 7f09323a9640 1 librados: shutdown 2024-01-20T11:23:37.218 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-01-20T11:23:36.813975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003068"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":11.645},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":6.327}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":7.117},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":15.217000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":9.4149999999999991},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":14.378}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":16.016999999999999},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":7.7590000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.148999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":16.826000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":13.689},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":12.375999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.532},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":6.2400000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":8.4049999999999994},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":14.367000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.161},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":11.523999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.1790000000000003},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":15.901999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":9.4819999999999993},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":12.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":13.407},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":10.436}]}]},{"osd":3,"up_from":21,"seq":90194313321,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":4.6920000000000002},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":7.3419999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":9.4580000000000002},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":14.504}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.037000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":7.7999999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.714},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":9.7639999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":12.657999999999999},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":11.976000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":11.426},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.135999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444135,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.5330000000000004},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":10.045}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":11.974},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.574}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":10.688000000000001},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":6.883}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":11.291},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":8.1460000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":8.8979999999999997},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":14.837}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":13.170999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.755000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444136,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":5.9909999999999997},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":9.0679999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":8.3290000000000006},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":7.6959999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":9.7840000000000007},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":10.456}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.112},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":7.0490000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":12.965999999999999},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":13.603999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":14.618},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":15.428000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444136,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":11.07},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":10.477}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":5.6280000000000001},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":11.762}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":15.644},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":7.9379999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":9.7219999999999995},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":7.0620000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":13.055999999999999},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":13.685}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.611000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":8.7189999999999994}]}]},{"osd":6,"up_from":15,"seq":64424509545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":4.4219999999999997},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":10.842000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.377000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":8.0370000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":13.741},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":9.9629999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":17.295000000000002},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.625999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":16.423999999999999},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.426}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":13.045},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":9.3399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:37.220 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.317+0000 7f9e757a9640 10 monclient: get_monmap_and_config 2024-01-20T11:23:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.317+0000 7f9e757a9640 10 monclient: build_initial_monmap 2024-01-20T11:23:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.317+0000 7f9e757a9640 10 monclient: monmap: 2024-01-20T11:23:37.318 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:37.319476+0000 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:37.319476+0000 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:37.319 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.318+0000 7f9e757a9640 1 Processor -- start 2024-01-20T11:23:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.318+0000 7f9e757a9640 1 -- start start 2024-01-20T11:23:37.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.318+0000 7f9e757a9640 10 monclient: init 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9e70071c60 0x7f9e70142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient(hunting): picked mon.noname-c con 0x7f9e70071c60 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e700ac540 0x7f9e700ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient(hunting): picked mon.noname-b con 0x7f9e700ac540 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e700b0930 0x7f9e700af130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient(hunting): picked mon.noname-a con 0x7f9e700b0930 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e70072040 con 0x7f9e700b0930 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e700ad610 con 0x7f9e70071c60 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e701426c0 con 0x7f9e700ac540 2024-01-20T11:23:37.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e757a9640 10 monclient(hunting): authenticate will time out at 1860.670286s 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e6f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e700b0930 0x7f9e700af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e6e7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e700ac540 0x7f9e700ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e6effd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f9e70071c60 0x7f9e70142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e6f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e700b0930 0x7f9e700af130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58324/0 (socket says 172.21.15.110:58324) 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e6f7fe640 1 -- 172.21.15.110:0/1335757766 learned_addr learned my addr 172.21.15.110:0/1335757766 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e6f7fe640 10 monclient(hunting): get_auth_request con 0x7f9e700b0930 auth_method 0 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.319+0000 7f9e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6e7fc640 10 monclient(hunting): get_auth_request con 0x7f9e700ac540 auth_method 0 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6effd640 10 monclient(hunting): get_auth_request con 0x7f9e70071c60 auth_method 0 2024-01-20T11:23:37.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.320+0000 7f9e6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_done global_id 6078 payload 274 2024-01-20T11:23:37.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 1 -- 172.21.15.110:0/1335757766 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e700ac540 msgr2=0x7f9e700ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:37.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 1 --2- 172.21.15.110:0/1335757766 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e700ac540 0x7f9e700ab4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 1 -- 172.21.15.110:0/1335757766 >> v2:172.21.15.134:3300/0 conn(0x7f9e70071c60 msgr2=0x7f9e70142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:37.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 1 --2- 172.21.15.110:0/1335757766 >> v2:172.21.15.134:3300/0 conn(0x7f9e70071c60 0x7f9e70142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:37.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 1 monclient: found mon.noname-a 2024-01-20T11:23:37.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.322+0000 7f9e6f7fe640 1 -- 172.21.15.110:0/1335757766 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700aba00 con 0x7f9e700b0930 2024-01-20T11:23:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.324+0000 7f9e6f7fe640 1 --2- 172.21.15.110:0/1335757766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 0x7f9e700af130 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9e700ada90 tx=0x7f9e6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ad078578decc2f server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 1 -- 172.21.15.110:0/1335757766 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68031870 con 0x7f9e700b0930 2024-01-20T11:23:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 10 monclient: dump: 2024-01-20T11:23:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:37.328 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.329 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.329 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.329 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:37.330 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.330 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 10 monclient: _finish_auth 0 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 10 monclient: _check_auth_tickets 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 1 -- 172.21.15.110:0/1335757766 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e680319d0 con 0x7f9e700b0930 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 1 -- 172.21.15.110:0/1335757766 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68031d60 con 0x7f9e700b0930 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e757a9640 5 monclient: authenticate success, global_id 6078 2024-01-20T11:23:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e757a9640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e757a9640 4 set_mon_vals no callback set 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.327+0000 7f9e6dffb640 10 monclient: dump: 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:37.332 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:37.333 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:37.333 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 10 monclient: shutdown 2024-01-20T11:23:37.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:37.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 1 -- 172.21.15.110:0/1335757766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 msgr2=0x7f9e700af130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 1 --2- 172.21.15.110:0/1335757766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 0x7f9e700af130 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9e700ada90 tx=0x7f9e6802b910 comp rx=0 tx=0).stop 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 1 -- 172.21.15.110:0/1335757766 shutdown_connections 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 1 --2- 172.21.15.110:0/1335757766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 0x7f9e700af130 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 1 --2- 172.21.15.110:0/1335757766 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e700ac540 0x7f9e700ab4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 1 --2- 172.21.15.110:0/1335757766 >> v2:172.21.15.134:3300/0 conn(0x7f9e70071c60 0x7f9e70142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 1 -- 172.21.15.110:0/1335757766 >> 172.21.15.110:0/1335757766 conn(0x7f9e700b3e60 msgr2=0x7f9e700b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 1 -- 172.21.15.110:0/1335757766 shutdown_connections 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.328+0000 7f9e757a9640 1 -- 172.21.15.110:0/1335757766 wait complete. 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.329+0000 7f9e757a9640 10 monclient: build_initial_monmap 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.329+0000 7f9e757a9640 10 monclient: monmap: 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:37.338 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.329+0000 7f9e757a9640 1 librados: starting msgr at 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.329+0000 7f9e757a9640 1 librados: starting objecter 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 Processor -- start 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 -- start start 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 librados: setting wanted keys 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 librados: calling monclient init 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient: init 2024-01-20T11:23:37.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e70071c60 0x7f9e700704a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient(hunting): picked mon.noname-c con 0x7f9e70071c60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e700ac540 0x7f9e7013e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient(hunting): picked mon.noname-b con 0x7f9e700ac540 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 0x7f9e701d6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient(hunting): picked mon.noname-a con 0x7f9e700b0930 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7013e210 con 0x7f9e700b0930 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70120f30 con 0x7f9e700ac540 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7013de50 con 0x7f9e70071c60 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e757a9640 10 monclient(hunting): authenticate will time out at 1860.681765s 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.330+0000 7f9e6f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 0x7f9e701d6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6effd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e70071c60 0x7f9e700704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 0x7f9e701d6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58334/0 (socket says 172.21.15.110:58334) 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6f7fe640 1 -- 172.21.15.110:0/2952620780 learned_addr learned my addr 172.21.15.110:0/2952620780 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6f7fe640 10 monclient(hunting): get_auth_request con 0x7f9e700b0930 auth_method 0 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6e7fc640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e700ac540 0x7f9e7013e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6effd640 10 monclient(hunting): get_auth_request con 0x7f9e70071c60 auth_method 0 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6effd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6e7fc640 10 monclient(hunting): get_auth_request con 0x7f9e700ac540 auth_method 0 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.331+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.332+0000 7f9e6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.332+0000 7f9e6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.332+0000 7f9e6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.332+0000 7f9e6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.332+0000 7f9e6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.332+0000 7f9e6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 10 monclient(hunting): handle_auth_done global_id 6081 payload 824 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 1 -- 172.21.15.110:0/2952620780 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e70071c60 msgr2=0x7f9e700704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e70071c60 0x7f9e700704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 1 -- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e700ac540 msgr2=0x7f9e7013e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e700ac540 0x7f9e7013e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 1 monclient: found mon.noname-a 2024-01-20T11:23:37.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:37.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:37.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.334+0000 7f9e6f7fe640 1 -- 172.21.15.110:0/2952620780 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e680094d0 con 0x7f9e700b0930 2024-01-20T11:23:37.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.336+0000 7f9e6f7fe640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 0x7f9e701d6f20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9e680090b0 tx=0x7f9e680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:37.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.336+0000 7f9e577fe640 10 client.?.objecter ms_handle_connect 0x7f9e700b0930 2024-01-20T11:23:37.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.336+0000 7f9e577fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:37.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 1 -- 172.21.15.110:0/2952620780 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6803ac60 con 0x7f9e700b0930 2024-01-20T11:23:37.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:37.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:37.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 10 monclient: dump: 2024-01-20T11:23:37.345 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 1 -- 172.21.15.110:0/2952620780 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6803ade0 con 0x7f9e700b0930 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:37.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 1 -- 172.21.15.110:0/2952620780 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68042630 con 0x7f9e700b0930 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e577fe640 10 monclient: dump: 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e757a9640 5 monclient: authenticate success, global_id 6081 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e757a9640 10 monclient: _renew_subs 2024-01-20T11:23:37.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e757a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7011fbd0 con 0x7f9e700b0930 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e6cff9640 4 set_mon_vals no callback set 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e757a9640 10 client.6081.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e757a9640 10 monclient: _renew_subs 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e757a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e701d7460 con 0x7f9e700b0930 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.340+0000 7f9e757a9640 1 librados: init done 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.341+0000 7f9e757a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.341+0000 7f9e757a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.341+0000 7f9e757a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:37.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.341+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7013e380 con 0x7f9e700b0930 2024-01-20T11:23:37.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.361+0000 7f9e577fe640 1 -- 172.21.15.110:0/2952620780 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9e680303c0 con 0x7f9e700b0930 2024-01-20T11:23:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.361+0000 7f9e577fe640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9e5c06c8b0 0x7f9e5c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.361+0000 7f9e6effd640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9e5c06c8b0 0x7f9e5c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.361+0000 7f9e6effd640 10 monclient: get_auth_request con 0x7f9e5c06c8b0 auth_method 0 2024-01-20T11:23:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.362+0000 7f9e577fe640 10 client.6081.objecter ms_handle_connect 0x7f9e5c06c8b0 2024-01-20T11:23:37.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.362+0000 7f9e6effd640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9e5c06c8b0 0x7f9e5c06ed70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9e600113e0 tx=0x7f9e6000b040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:37.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.363+0000 7f9e577fe640 1 -- 172.21.15.110:0/2952620780 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9e6802d040 con 0x7f9e700b0930 2024-01-20T11:23:37.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.363+0000 7f9e577fe640 10 client.6081.objecter ms_dispatch 0x7f9e70000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:37.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.363+0000 7f9e577fe640 3 client.6081.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:37.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.363+0000 7f9e577fe640 3 client.6081.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:37.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.363+0000 7f9e577fe640 20 client.6081.objecter dump_active .. 0 homeless 2024-01-20T11:23:37.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~279s 2024-01-20T11:23:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.604+0000 7f9e577fe640 1 -- 172.21.15.110:0/2952620780 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9e68076f10 con 0x7f9e700b0930 2024-01-20T11:23:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.604+0000 7f9e577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:37.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.604+0000 7f9e577fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:37.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.777+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e7013e010 con 0x7f9e5c06c8b0 2024-01-20T11:23:37.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.781+0000 7f9e577fe640 1 -- 172.21.15.110:0/2952620780 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63722 (secure 0 0 0) 0x7f9e7013e010 con 0x7f9e5c06c8b0 2024-01-20T11:23:37.781 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:37.782 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:37.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 10 librados: watch_flush enter 2024-01-20T11:23:37.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 10 librados: watch_flush exit 2024-01-20T11:23:37.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 20 client.6081.objecter shutdown clearing up homeless session... 2024-01-20T11:23:37.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 10 client.6081.objecter successfully canceled tick 2024-01-20T11:23:37.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9e5c06c8b0 msgr2=0x7f9e5c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9e5c06c8b0 0x7f9e5c06ed70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9e600113e0 tx=0x7f9e6000b040 comp rx=0 tx=0).stop 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 10 monclient: shutdown 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 msgr2=0x7f9e701d6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 0x7f9e701d6f20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9e680090b0 tx=0x7f9e680046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 shutdown_connections 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f9e5c06c8b0 0x7f9e5c06ed70 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e700b0930 0x7f9e701d6f20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e700ac540 0x7f9e7013e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 --2- 172.21.15.110:0/2952620780 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e70071c60 0x7f9e700704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 >> 172.21.15.110:0/2952620780 conn(0x7f9e700b3e60 msgr2=0x7f9e7013bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:37.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 shutdown_connections 2024-01-20T11:23:37.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 -- 172.21.15.110:0/2952620780 wait complete. 2024-01-20T11:23:37.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.782+0000 7f9e757a9640 1 librados: shutdown 2024-01-20T11:23:37.794 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-01-20T11:23:36.813975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003068"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.8309999999999995,"5min":8.0760000000000005,"15min":12.488},"min":{"1min":4.6459999999999999,"5min":3.444,"15min":3.444},"max":{"1min":24.600000000000001,"5min":24.600000000000001,"15min":100.05800000000001},"last":11.645},{"interface":"front","average":{"1min":7.75,"5min":8.0570000000000004,"15min":13.321},"min":{"1min":4.9219999999999997,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":16.242999999999999,"5min":16.242999999999999,"15min":91.789000000000001},"last":6.327}]},{"osd":1,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":9.7249999999999996,"5min":10.018000000000001,"15min":15.039},"min":{"1min":5.1189999999999998,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":22.195,"5min":22.195,"15min":98.700000000000003},"last":7.117},{"interface":"front","average":{"1min":10.832000000000001,"5min":10.269,"15min":14.444000000000001},"min":{"1min":5.9189999999999996,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":22.798999999999999,"5min":22.798999999999999,"15min":95.603999999999999},"last":15.217000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":11.244,"5min":11.446999999999999,"15min":16.242000000000001},"min":{"1min":5.8789999999999996,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":19.283999999999999,"5min":19.283999999999999,"15min":131.09700000000001},"last":9.4149999999999991},{"interface":"front","average":{"1min":12.183,"5min":12.103999999999999,"15min":15.958},"min":{"1min":7.827,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":20.315000000000001,"5min":20.315000000000001,"15min":105.43899999999999},"last":14.378}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.896000000000001,"5min":11.053000000000001,"15min":17.751999999999999},"min":{"1min":8.3539999999999992,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":27.010999999999999,"5min":27.010999999999999,"15min":94.911000000000001},"last":16.016999999999999},{"interface":"front","average":{"1min":11.119,"5min":12.666,"15min":19.504999999999999},"min":{"1min":6.1070000000000002,"5min":6.1070000000000002,"15min":6.1070000000000002},"max":{"1min":18.577000000000002,"5min":18.768999999999998,"15min":100.633},"last":7.7590000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.797000000000001,"5min":13.731,"15min":19.533999999999999},"min":{"1min":6.282,"5min":6.282,"15min":6.282},"max":{"1min":23.946999999999999,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.148999999999999},{"interface":"front","average":{"1min":14.269,"5min":12.705,"15min":18.329000000000001},"min":{"1min":9.5,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":25.286000000000001,"5min":25.286000000000001,"15min":107.962},"last":16.826000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.355,"5min":11.606,"15min":16.963000000000001},"min":{"1min":7.6980000000000004,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":20.991,"5min":20.991,"15min":96.911000000000001},"last":13.689},{"interface":"front","average":{"1min":14.157999999999999,"5min":14.840999999999999,"15min":20.463000000000001},"min":{"1min":9.5800000000000001,"5min":9.157,"15min":9.157},"max":{"1min":27.356000000000002,"5min":27.356000000000002,"15min":129.828},"last":12.375999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.532},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":6.2400000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":8.4049999999999994},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":14.367000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.161},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":11.523999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.1790000000000003},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":15.901999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":9.4819999999999993},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":12.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":13.407},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":10.436}]}]},{"osd":3,"up_from":21,"seq":90194313321,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":4.6920000000000002},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":7.3419999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":9.4580000000000002},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":14.504}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":10.037000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":7.7999999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":10.714},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":9.7639999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":12.657999999999999},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":11.976000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":11.426},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":15.135999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444135,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.5330000000000004},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":10.045}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":11.974},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":12.574}]},{"osd":3,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":13.522,"5min":12.704000000000001,"15min":15.756},"min":{"1min":8.218,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":19.655000000000001,"5min":20.829000000000001,"15min":49.237000000000002},"last":10.688000000000001},{"interface":"front","average":{"1min":10.372999999999999,"5min":10.352,"15min":13.218},"min":{"1min":6.351,"5min":5.6769999999999996,"15min":5.6769999999999996},"max":{"1min":14.926,"5min":17.103999999999999,"15min":52.905999999999999},"last":6.883}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":11.291},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":8.1460000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":8.8979999999999997},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":14.837}]},{"osd":7,"last update":"Sat Jan 20 11:22:33 2024","interfaces":[{"interface":"back","average":{"1min":12.589,"5min":12.407999999999999,"15min":13.272},"min":{"1min":7.8849999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":18.527000000000001,"5min":18.609000000000002,"15min":24.266999999999999},"last":13.170999999999999},{"interface":"front","average":{"1min":15.347,"5min":14.818,"15min":16.847000000000001},"min":{"1min":9.5760000000000005,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":21.297999999999998,"5min":21.884,"15min":28.148},"last":15.755000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444136,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":5.9909999999999997},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":9.0679999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":8.3290000000000006},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":7.6959999999999997}]},{"osd":3,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":12.996,"5min":12.728,"15min":21.442},"min":{"1min":8.0030000000000001,"5min":6.6479999999999997,"15min":6.6479999999999997},"max":{"1min":16.638000000000002,"5min":21.178999999999998,"15min":259.483},"last":9.7840000000000007},{"interface":"front","average":{"1min":11.009,"5min":9.7490000000000006,"15min":16.971},"min":{"1min":5.8220000000000001,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":19.297999999999998,"5min":19.297999999999998,"15min":243.24199999999999},"last":10.456}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.112},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":7.0490000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":10.81,"15min":11.802},"min":{"1min":7.9370000000000003,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.792999999999999,"5min":16.792999999999999,"15min":27.233000000000001},"last":12.965999999999999},{"interface":"front","average":{"1min":12.414,"5min":13.391,"15min":14.898},"min":{"1min":8.4299999999999997,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.02,"5min":21.553000000000001,"15min":31.634},"last":13.603999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:31 2024","interfaces":[{"interface":"back","average":{"1min":14.214,"5min":12.733000000000001,"15min":24.533999999999999},"min":{"1min":9.6370000000000005,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":19.882000000000001,"5min":20.875,"15min":344.98099999999999},"last":14.618},{"interface":"front","average":{"1min":14.396000000000001,"5min":14.727,"15min":25.501000000000001},"min":{"1min":9.1669999999999998,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":21.021999999999998,"5min":22.863,"15min":313.45999999999998},"last":15.428000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444136,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":11.07},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":10.477}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":5.6280000000000001},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":11.762}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":15.644},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":7.9379999999999997}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":9.7219999999999995},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":7.0620000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":13.055999999999999},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":13.685}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.611000000000001},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":8.7189999999999994}]}]},{"osd":6,"up_from":15,"seq":64424509545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":4.4219999999999997},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":10.842000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.377000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":8.0370000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":13.741},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":9.9629999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":17.295000000000002},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.625999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":16.423999999999999},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.426}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":13.045},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":9.3399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:37.795 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:37.795 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:37.795 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:37.795 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:37.796 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:37.796 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 16, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:31.815524+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:21:31.815524+0000', 'last_peered': '2024-01-20T11:21:31.815524+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:21:31.815524+0000', 'last_undegraded': '2024-01-20T11:21:31.815524+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:37.796 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:37.796 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 131, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:21:18.077638+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:21:18.077638+0000', 'last_peered': '2024-01-20T11:21:18.077638+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:21:18.077638+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:37.796 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:37.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.893+0000 7f69f9389640 10 monclient: get_monmap_and_config 2024-01-20T11:23:37.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.893+0000 7f69f9389640 10 monclient: build_initial_monmap 2024-01-20T11:23:37.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.893+0000 7f69f9389640 10 monclient: monmap: 2024-01-20T11:23:37.894 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:37.894 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.894 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:37.894844+0000 2024-01-20T11:23:37.894 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:37.894844+0000 2024-01-20T11:23:37.894 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:37.895 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.895 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:37.895 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:37.895 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:37.895 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:37.895 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:37.895 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.893+0000 7f69f9389640 1 Processor -- start 2024-01-20T11:23:37.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.893+0000 7f69f9389640 1 -- start start 2024-01-20T11:23:37.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.893+0000 7f69f9389640 10 monclient: init 2024-01-20T11:23:37.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:37.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-01-20T11:23:37.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69f40b0c00 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69f40accf0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f69f4143910 0x7f69f414b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 10 monclient(hunting): picked mon.noname-c con 0x7f69f4143910 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40ad450 con 0x7f69f40b0c00 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40ad8e0 con 0x7f69f40accf0 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f40ad130 con 0x7f69f4143910 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.894+0000 7f69f9389640 10 monclient(hunting): authenticate will time out at 1861.245807s 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.895+0000 7f69f2ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f69f40b0c00 0x7f69f413d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46990/0 (socket says 172.21.15.110:46990) 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.895+0000 7f69f2ffd640 1 -- 172.21.15.110:0/1042990811 learned_addr learned my addr 172.21.15.110:0/1042990811 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:37.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.895+0000 7f69f37fe640 1 --2- 172.21.15.110:0/1042990811 >> v2:172.21.15.134:3300/0 conn(0x7f69f4143910 0x7f69f414b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.896+0000 7f69f37fe640 10 monclient(hunting): get_auth_request con 0x7f69f4143910 auth_method 0 2024-01-20T11:23:37.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.896+0000 7f69f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.896+0000 7f69f37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.896+0000 7f69f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.897+0000 7f69f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.897+0000 7f69f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.897+0000 7f69f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.897+0000 7f69f1ffb640 1 -- 172.21.15.110:0/1042990811 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352598582 0 0) 0x7f69f40ad450 con 0x7f69f40b0c00 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.897+0000 7f69f1ffb640 10 monclient(hunting): _negotiate 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.897+0000 7f69f1ffb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.897+0000 7f69f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.897+0000 7f69f1ffb640 10 monclient(hunting): my global_id is 6084 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.897+0000 7f69f1ffb640 1 -- 172.21.15.110:0/1042990811 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69dc003520 con 0x7f69f40b0c00 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.899+0000 7f69f37fe640 10 monclient(hunting): handle_auth_done global_id 6196 payload 274 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.899+0000 7f69f37fe640 1 -- 172.21.15.110:0/1042990811 >> v1:172.21.15.134:6789/0 conn(0x7f69f40accf0 legacy=0x7f69f4143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.899+0000 7f69f37fe640 1 -- 172.21.15.110:0/1042990811 >> v1:172.21.15.110:6789/0 conn(0x7f69f40b0c00 legacy=0x7f69f413d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.899+0000 7f69f37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.899+0000 7f69f37fe640 1 monclient: found mon.noname-c 2024-01-20T11:23:37.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.899+0000 7f69f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:37.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.899+0000 7f69f37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.134:3300/0 2024-01-20T11:23:37.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.899+0000 7f69f37fe640 1 -- 172.21.15.110:0/1042990811 --> v2:172.21.15.134:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f4143d70 con 0x7f69f4143910 2024-01-20T11:23:37.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.901+0000 7f69f37fe640 1 --2- 172.21.15.110:0/1042990811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f4143910 0x7f69f414b560 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009d90 tx=0x7f69ec02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf0a1ef666c7e7e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:37.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 1 -- 172.21.15.110:0/1042990811 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec02cbd0 con 0x7f69f4143910 2024-01-20T11:23:37.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:37.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-20T11:23:37.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 10 monclient: dump: 2024-01-20T11:23:37.908 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:37.908 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.908 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.908 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.908 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 10 monclient: _finish_auth 0 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 5 monclient: authenticate success, global_id 6196 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 1 -- 172.21.15.110:0/1042990811 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec02cd30 con 0x7f69f4143910 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:37.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 4 set_mon_vals no callback set 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 1 -- 172.21.15.110:0/1042990811 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec02cea0 con 0x7f69f4143910 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f1ffb640 10 monclient: dump: 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:37.910 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:37.911 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 10 monclient: shutdown 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 1 -- 172.21.15.110:0/1042990811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f4143910 msgr2=0x7f69f414b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 1 --2- 172.21.15.110:0/1042990811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f4143910 0x7f69f414b560 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009d90 tx=0x7f69ec02be20 comp rx=0 tx=0).stop 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 1 -- 172.21.15.110:0/1042990811 shutdown_connections 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.907+0000 7f69f9389640 1 --2- 172.21.15.110:0/1042990811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f4143910 0x7f69f414b560 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.908+0000 7f69f9389640 1 -- 172.21.15.110:0/1042990811 >> 172.21.15.110:0/1042990811 conn(0x7f69f40b3e60 msgr2=0x7f69f40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.908+0000 7f69f9389640 1 -- 172.21.15.110:0/1042990811 shutdown_connections 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.908+0000 7f69f9389640 1 -- 172.21.15.110:0/1042990811 wait complete. 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.908+0000 7f69f9389640 10 monclient: build_initial_monmap 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.908+0000 7f69f9389640 10 monclient: monmap: 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:37.915 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.908+0000 7f69f9389640 1 librados: starting msgr at 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.908+0000 7f69f9389640 1 librados: starting objecter 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 Processor -- start 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 -- start start 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 librados: setting wanted keys 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 librados: calling monclient init 2024-01-20T11:23:37.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient: init 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69f40accf0 0x7f69f40704d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient(hunting): picked mon.noname-c con 0x7f69f40accf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f40b0c00 0x7f69f41d4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient(hunting): picked mon.noname-b con 0x7f69f40b0c00 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69f4143910 0x7f69f41d5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient(hunting): picked mon.noname-a con 0x7f69f4143910 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4144890 con 0x7f69f4143910 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40713e0 con 0x7f69f40b0c00 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4071900 con 0x7f69f40accf0 2024-01-20T11:23:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.909+0000 7f69f9389640 10 monclient(hunting): authenticate will time out at 1861.261002s 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69f4143910 0x7f69f41d5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f40b0c00 0x7f69f41d4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69f40accf0 0x7f69f40704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69f4143910 0x7f69f41d5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58342/0 (socket says 172.21.15.110:58342) 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f37fe640 1 -- 172.21.15.110:0/3179253901 learned_addr learned my addr 172.21.15.110:0/3179253901 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f37fe640 10 monclient(hunting): get_auth_request con 0x7f69f4143910 auth_method 0 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f27fc640 10 monclient(hunting): get_auth_request con 0x7f69f40b0c00 auth_method 0 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f2ffd640 10 monclient(hunting): get_auth_request con 0x7f69f40accf0 auth_method 0 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.910+0000 7f69f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.911+0000 7f69f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.911+0000 7f69f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.911+0000 7f69f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.911+0000 7f69f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.911+0000 7f69f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.911+0000 7f69f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.911+0000 7f69f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.911+0000 7f69f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.911+0000 7f69f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 10 monclient(hunting): handle_auth_done global_id 6087 payload 824 2024-01-20T11:23:37.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 1 -- 172.21.15.110:0/3179253901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69f40accf0 msgr2=0x7f69f40704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69f40accf0 0x7f69f40704d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 1 -- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f40b0c00 msgr2=0x7f69f41d4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f40b0c00 0x7f69f41d4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 1 monclient: found mon.noname-a 2024-01-20T11:23:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f37fe640 1 -- 172.21.15.110:0/3179253901 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec009a70 con 0x7f69f4143910 2024-01-20T11:23:37.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.913+0000 7f69f27fc640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f40b0c00 0x7f69f41d4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:37.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.915+0000 7f69f37fe640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69f4143910 0x7f69f41d5300 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009d90 tx=0x7f69ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:37.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.915+0000 7f69db7fe640 10 client.?.objecter ms_handle_connect 0x7f69f4143910 2024-01-20T11:23:37.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.915+0000 7f69db7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:37.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69db7fe640 1 -- 172.21.15.110:0/3179253901 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec02cea0 con 0x7f69f4143910 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69db7fe640 10 monclient: dump: 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:37.923 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69db7fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69db7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69db7fe640 1 -- 172.21.15.110:0/3179253901 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec040650 con 0x7f69f4143910 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69f9389640 5 monclient: authenticate success, global_id 6087 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69f9389640 10 monclient: _renew_subs 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69f9389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f413d970 con 0x7f69f4143910 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69db7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69f0ff9640 4 set_mon_vals no callback set 2024-01-20T11:23:37.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.919+0000 7f69f9389640 10 client.6087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.920+0000 7f69f9389640 10 monclient: _renew_subs 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.920+0000 7f69db7fe640 1 -- 172.21.15.110:0/3179253901 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec040a00 con 0x7f69f4143910 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.920+0000 7f69f9389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.920+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f41d8530 con 0x7f69f4143910 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.920+0000 7f69f9389640 1 librados: init done 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.920+0000 7f69db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.920+0000 7f69db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.920+0000 7f69db7fe640 10 monclient: dump: 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:37.925 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.921+0000 7f69f9389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.921+0000 7f69f9389640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.921+0000 7f69f9389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:37.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:37.921+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f4072cc0 con 0x7f69f4143910 2024-01-20T11:23:38.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.163+0000 7f69db7fe640 1 -- 172.21.15.110:0/3179253901 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f69ec040b60 con 0x7f69f4143910 2024-01-20T11:23:38.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.163+0000 7f69db7fe640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f69e006c860 0x7f69e006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.164+0000 7f69f2ffd640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f69e006c860 0x7f69e006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.164+0000 7f69f2ffd640 10 monclient: get_auth_request con 0x7f69e006c860 auth_method 0 2024-01-20T11:23:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.164+0000 7f69f2ffd640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f69e006c860 0x7f69e006ed20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f69e40011e0 tx=0x7f69e4007d30 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.164+0000 7f69db7fe640 10 client.6087.objecter ms_handle_connect 0x7f69e006c860 2024-01-20T11:23:38.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.176+0000 7f69db7fe640 1 -- 172.21.15.110:0/3179253901 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f69ec07b4b0 con 0x7f69f4143910 2024-01-20T11:23:38.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.176+0000 7f69db7fe640 10 client.6087.objecter ms_dispatch 0x7f69f4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:38.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.176+0000 7f69db7fe640 3 client.6087.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:38.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.176+0000 7f69db7fe640 3 client.6087.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:38.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.176+0000 7f69db7fe640 20 client.6087.objecter dump_active .. 0 homeless 2024-01-20T11:23:38.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.177+0000 7f69db7fe640 1 -- 172.21.15.110:0/3179253901 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f69ec0777f0 con 0x7f69f4143910 2024-01-20T11:23:38.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.177+0000 7f69db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:38.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.177+0000 7f69db7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:38.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.423+0000 7f69f9389640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:38.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.423+0000 7f69f9389640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:38.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.423+0000 7f69f9389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:38.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.423+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f69f41449d0 con 0x7f69f4143910 2024-01-20T11:23:38.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.428+0000 7f69db7fe640 1 -- 172.21.15.110:0/3179253901 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f69ec039520 con 0x7f69f4143910 2024-01-20T11:23:38.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.428+0000 7f69db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:38.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.428+0000 7f69db7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:38.429 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:38.430 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":552,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 10 librados: watch_flush enter 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 10 librados: watch_flush exit 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 20 client.6087.objecter shutdown clearing up homeless session... 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 10 client.6087.objecter successfully canceled tick 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f69e006c860 msgr2=0x7f69e006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f69e006c860 0x7f69e006ed20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f69e40011e0 tx=0x7f69e4007d30 comp rx=0 tx=0).stop 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 10 monclient: shutdown 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69f4143910 msgr2=0x7f69f41d5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:38.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69f4143910 0x7f69f41d5300 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009d90 tx=0x7f69ec004a40 comp rx=0 tx=0).stop 2024-01-20T11:23:38.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 shutdown_connections 2024-01-20T11:23:38.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f69e006c860 0x7f69e006ed20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:38.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69f4143910 0x7f69f41d5300 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:38.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69f40b0c00 0x7f69f41d4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:38.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 --2- 172.21.15.110:0/3179253901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69f40accf0 0x7f69f40704d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:38.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 >> 172.21.15.110:0/3179253901 conn(0x7f69f40b3e60 msgr2=0x7f69f40afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:38.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 shutdown_connections 2024-01-20T11:23:38.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 -- 172.21.15.110:0/3179253901 wait complete. 2024-01-20T11:23:38.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.430+0000 7f69f9389640 1 librados: shutdown 2024-01-20T11:23:38.442 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:38.442 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:38.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.543+0000 7fe31e0da640 10 monclient: get_monmap_and_config 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.543+0000 7fe31e0da640 10 monclient: build_initial_monmap 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.544+0000 7fe31e0da640 10 monclient: monmap: 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:38.545587+0000 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:38.545587+0000 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:38.545 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:38.546 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:38.546 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:38.546 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:38.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.544+0000 7fe31e0da640 1 Processor -- start 2024-01-20T11:23:38.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.544+0000 7fe31e0da640 1 -- start start 2024-01-20T11:23:38.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.544+0000 7fe31e0da640 10 monclient: init 2024-01-20T11:23:38.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:38.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-01-20T11:23:38.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3180ad890 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:38.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe3180adc70 0x7fe318142110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:38.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 10 monclient(hunting): picked mon.noname-a con 0x7fe3180adc70 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fe31806b5d0 0x7fe318142650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 10 monclient(hunting): picked mon.noname-c con 0x7fe31806b5d0 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31806bab0 con 0x7fe3180ad890 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe318072530 con 0x7fe3180adc70 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31813fa10 con 0x7fe31806b5d0 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe31e0da640 10 monclient(hunting): authenticate will time out at 1861.896566s 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.545+0000 7fe316ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe3180adc70 0x7fe318142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe316ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe3180adc70 0x7fe318142110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58346/0 (socket says 172.21.15.110:58346) 2024-01-20T11:23:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe316ffd640 1 -- 172.21.15.110:0/2459794979 learned_addr learned my addr 172.21.15.110:0/2459794979 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe316ffd640 10 monclient(hunting): get_auth_request con 0x7fe3180adc70 auth_method 0 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe316ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe316ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe316ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe316ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe317fff640 1 --2- 172.21.15.110:0/2459794979 >> v2:172.21.15.134:3300/0 conn(0x7fe31806b5d0 0x7fe318142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.546+0000 7fe316ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.547+0000 7fe317fff640 10 monclient(hunting): get_auth_request con 0x7fe31806b5d0 auth_method 0 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.547+0000 7fe317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.547+0000 7fe317fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:38.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.547+0000 7fe317fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:38.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.548+0000 7fe316ffd640 10 monclient(hunting): handle_auth_done global_id 6090 payload 274 2024-01-20T11:23:38.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.548+0000 7fe316ffd640 1 -- 172.21.15.110:0/2459794979 >> v2:172.21.15.134:3300/0 conn(0x7fe31806b5d0 msgr2=0x7fe318142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:38.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.548+0000 7fe316ffd640 1 --2- 172.21.15.110:0/2459794979 >> v2:172.21.15.134:3300/0 conn(0x7fe31806b5d0 0x7fe318142650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:38.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.548+0000 7fe316ffd640 1 -- 172.21.15.110:0/2459794979 >> v1:172.21.15.134:6789/0 conn(0x7fe3180ad890 legacy=0x7fe318071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:38.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.548+0000 7fe316ffd640 10 monclient: _finish_hunting 0 2024-01-20T11:23:38.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.548+0000 7fe316ffd640 1 monclient: found mon.noname-a 2024-01-20T11:23:38.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.548+0000 7fe316ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:38.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.548+0000 7fe316ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:38.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.548+0000 7fe316ffd640 1 -- 172.21.15.110:0/2459794979 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318142b90 con 0x7fe3180adc70 2024-01-20T11:23:38.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.550+0000 7fe316ffd640 1 --2- 172.21.15.110:0/2459794979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3180adc70 0x7fe318142110 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe3180af500 tx=0x7fe30c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2235e30276be4af7 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:38.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 1 -- 172.21.15.110:0/2459794979 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c031870 con 0x7fe3180adc70 2024-01-20T11:23:38.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:38.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:38.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 10 monclient: dump: 2024-01-20T11:23:38.555 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:38.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 1 -- 172.21.15.110:0/2459794979 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c0319d0 con 0x7fe3180adc70 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 1 -- 172.21.15.110:0/2459794979 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c031d60 con 0x7fe3180adc70 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe31e0da640 5 monclient: authenticate success, global_id 6090 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe31e0da640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe31e0da640 4 set_mon_vals no callback set 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.553+0000 7fe3167fc640 10 monclient: dump: 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:38.557 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 10 monclient: shutdown 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 1 -- 172.21.15.110:0/2459794979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3180adc70 msgr2=0x7fe318142110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:38.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 1 --2- 172.21.15.110:0/2459794979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3180adc70 0x7fe318142110 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe3180af500 tx=0x7fe30c02b910 comp rx=0 tx=0).stop 2024-01-20T11:23:38.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 1 -- 172.21.15.110:0/2459794979 shutdown_connections 2024-01-20T11:23:38.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 1 --2- 172.21.15.110:0/2459794979 >> v2:172.21.15.134:3300/0 conn(0x7fe31806b5d0 0x7fe318142650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:38.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 1 --2- 172.21.15.110:0/2459794979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3180adc70 0x7fe318142110 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:38.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 1 -- 172.21.15.110:0/2459794979 >> 172.21.15.110:0/2459794979 conn(0x7fe31806a5f0 msgr2=0x7fe31806aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:38.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 1 -- 172.21.15.110:0/2459794979 shutdown_connections 2024-01-20T11:23:38.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.554+0000 7fe31e0da640 1 -- 172.21.15.110:0/2459794979 wait complete. 2024-01-20T11:23:38.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.555+0000 7fe31e0da640 10 monclient: build_initial_monmap 2024-01-20T11:23:38.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.555+0000 7fe31e0da640 10 monclient: monmap: 2024-01-20T11:23:38.562 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:38.562 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:38.562 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.555+0000 7fe31e0da640 1 librados: starting msgr at 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.555+0000 7fe31e0da640 1 librados: starting objecter 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.555+0000 7fe31e0da640 1 Processor -- start 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.555+0000 7fe31e0da640 1 -- start start 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.555+0000 7fe31e0da640 1 librados: setting wanted keys 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.555+0000 7fe31e0da640 1 librados: calling monclient init 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.555+0000 7fe31e0da640 10 monclient: init 2024-01-20T11:23:38.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe31806b5d0 0x7fe318139160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient(hunting): picked mon.noname-b con 0x7fe31806b5d0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3180ad890 0x7fe31813b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient(hunting): picked mon.noname-c con 0x7fe3180ad890 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3181396a0 0x7fe3181daa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient(hunting): picked mon.noname-a con 0x7fe3181396a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31813a6a0 con 0x7fe3181396a0 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31813aed0 con 0x7fe31806b5d0 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe318137170 con 0x7fe3180ad890 2024-01-20T11:23:38.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe31e0da640 10 monclient(hunting): authenticate will time out at 1861.907592s 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe317fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3181396a0 0x7fe3181daa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.556+0000 7fe316ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3180ad890 0x7fe31813b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe3177fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe31806b5d0 0x7fe318139160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe317fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3181396a0 0x7fe3181daa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58356/0 (socket says 172.21.15.110:58356) 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe317fff640 1 -- 172.21.15.110:0/2902931331 learned_addr learned my addr 172.21.15.110:0/2902931331 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe317fff640 10 monclient(hunting): get_auth_request con 0x7fe3181396a0 auth_method 0 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe317fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe317fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe3177fe640 10 monclient(hunting): get_auth_request con 0x7fe31806b5d0 auth_method 0 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe3177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe3177fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:38.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe3177fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe316ffd640 10 monclient(hunting): get_auth_request con 0x7fe3180ad890 auth_method 0 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe316ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe316ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe317fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe317fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe317fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe3177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe3177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe3177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe316ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe316ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.557+0000 7fe316ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 10 monclient(hunting): handle_auth_done global_id 6093 payload 824 2024-01-20T11:23:38.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 1 -- 172.21.15.110:0/2902931331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3180ad890 msgr2=0x7fe31813b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:38.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3180ad890 0x7fe31813b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:38.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 1 -- 172.21.15.110:0/2902931331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe31806b5d0 msgr2=0x7fe318139160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:38.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe31806b5d0 0x7fe318139160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:38.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:38.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 1 monclient: found mon.noname-a 2024-01-20T11:23:38.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:38.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:38.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.560+0000 7fe317fff640 1 -- 172.21.15.110:0/2902931331 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c0094d0 con 0x7fe3181396a0 2024-01-20T11:23:38.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.563+0000 7fe317fff640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3181396a0 0x7fe3181daa00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe310007e60 tx=0x7fe31000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:38.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.563+0000 7fe2fffff640 10 client.?.objecter ms_handle_connect 0x7fe3181396a0 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.563+0000 7fe2fffff640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.566+0000 7fe2fffff640 1 -- 172.21.15.110:0/2902931331 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310014420 con 0x7fe3181396a0 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.566+0000 7fe2fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.566+0000 7fe2fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.566+0000 7fe2fffff640 10 monclient: dump: 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:38.569 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.566+0000 7fe2fffff640 10 monclient: _finish_auth 0 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.566+0000 7fe2fffff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.566+0000 7fe2fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.566+0000 7fe2fffff640 1 -- 172.21.15.110:0/2902931331 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3100145a0 con 0x7fe3181396a0 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.566+0000 7fe2fffff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe2fffff640 1 -- 172.21.15.110:0/2902931331 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310014950 con 0x7fe3181396a0 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe31e0da640 5 monclient: authenticate success, global_id 6093 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe31e0da640 10 monclient: _renew_subs 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe31e0da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe318137a10 con 0x7fe3181396a0 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe315ffb640 4 set_mon_vals no callback set 2024-01-20T11:23:38.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe2fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe2fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe2fffff640 10 monclient: dump: 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe31e0da640 10 client.6093.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe31e0da640 10 monclient: _renew_subs 2024-01-20T11:23:38.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe31e0da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:38.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3181da560 con 0x7fe3181396a0 2024-01-20T11:23:38.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.567+0000 7fe31e0da640 1 librados: init done 2024-01-20T11:23:38.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.568+0000 7fe31e0da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:38.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.568+0000 7fe31e0da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:38.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.568+0000 7fe31e0da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:38.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.568+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe318075490 con 0x7fe3181396a0 2024-01-20T11:23:38.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.808+0000 7fe2fffff640 1 -- 172.21.15.110:0/2902931331 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe310014ab0 con 0x7fe3181396a0 2024-01-20T11:23:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.809+0000 7fe2fffff640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe30406c860 0x7fe30406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.809+0000 7fe3177fe640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe30406c860 0x7fe30406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.809+0000 7fe3177fe640 10 monclient: get_auth_request con 0x7fe30406c860 auth_method 0 2024-01-20T11:23:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.810+0000 7fe3177fe640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe30406c860 0x7fe30406ed20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe308001780 tx=0x7fe308009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:38.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.810+0000 7fe2fffff640 10 client.6093.objecter ms_handle_connect 0x7fe30406c860 2024-01-20T11:23:38.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.822+0000 7fe2fffff640 1 -- 172.21.15.110:0/2902931331 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe310059990 con 0x7fe3181396a0 2024-01-20T11:23:38.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.822+0000 7fe2fffff640 10 client.6093.objecter ms_dispatch 0x7fe318000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:38.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.822+0000 7fe2fffff640 3 client.6093.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:38.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.822+0000 7fe2fffff640 3 client.6093.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:38.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.822+0000 7fe2fffff640 20 client.6093.objecter dump_active .. 0 homeless 2024-01-20T11:23:38.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.822+0000 7fe2fffff640 1 -- 172.21.15.110:0/2902931331 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe310055cd0 con 0x7fe3181396a0 2024-01-20T11:23:38.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.822+0000 7fe2fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:38.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:38.822+0000 7fe2fffff640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:39.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.000+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe318137330 con 0x7fe30406c860 2024-01-20T11:23:39.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.004+0000 7fe2fffff640 1 -- 172.21.15.110:0/2902931331 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63782 (secure 0 0 0) 0x7fe318137330 con 0x7fe30406c860 2024-01-20T11:23:39.004 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:39.005 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 10 librados: watch_flush enter 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 10 librados: watch_flush exit 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 20 client.6093.objecter shutdown clearing up homeless session... 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 10 client.6093.objecter successfully canceled tick 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe30406c860 msgr2=0x7fe30406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe30406c860 0x7fe30406ed20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe308001780 tx=0x7fe308009040 comp rx=0 tx=0).stop 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 10 monclient: shutdown 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3181396a0 msgr2=0x7fe3181daa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3181396a0 0x7fe3181daa00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe310007e60 tx=0x7fe31000e4b0 comp rx=0 tx=0).stop 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 shutdown_connections 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fe30406c860 0x7fe30406ed20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:39.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3181396a0 0x7fe3181daa00 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:39.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3180ad890 0x7fe31813b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:39.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 --2- 172.21.15.110:0/2902931331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe31806b5d0 0x7fe318139160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:39.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 >> 172.21.15.110:0/2902931331 conn(0x7fe31806a5f0 msgr2=0x7fe3180af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:39.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 shutdown_connections 2024-01-20T11:23:39.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 -- 172.21.15.110:0/2902931331 wait complete. 2024-01-20T11:23:39.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:39.005+0000 7fe31e0da640 1 librados: shutdown 2024-01-20T11:23:39.017 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-01-20T11:23:38.814332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280619,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.2789999999999999},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":5.9829999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":9.2400000000000002},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":12.157999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":8.1649999999999991},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.473000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.895},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":14.669}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.215},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":15.413}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":16.109000000000002},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":13.547000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.532},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":6.2400000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":8.4049999999999994},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":14.367000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.161},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":11.523999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.1790000000000003},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":15.901999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":9.4819999999999993},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":12.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":13.407},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":10.436}]}]},{"osd":3,"up_from":21,"seq":90194313322,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":9.0540000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.7560000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":6.8230000000000004},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.7189999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":12.428000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":7.4820000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":8.4819999999999993},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":14.978999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":14.358000000000001},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":10.433}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.140000000000001},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":11.673999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444136,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.1020000000000003},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":6.9379999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":7.7699999999999996},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":13.567}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.488},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":9.0640000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":9.7469999999999999},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.067}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":11.627000000000001},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":8.4580000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":10.336},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":15.311}]}]},{"osd":1,"up_from":17,"seq":73014444137,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.3449999999999998},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":8.0649999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":10.32},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":11.032}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":5.3570000000000002},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":12.382999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.669},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":8.7300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":13.198},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":9.7149999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":15.653},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":13.784000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444137,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":10.670999999999999},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":5.8949999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":8.3699999999999992},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":13.314}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":16.190999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":11.49}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":12.087999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":9.077}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":13.925000000000001},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":12.718999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":16.952000000000002},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":9.9719999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":4.4219999999999997},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":10.842000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.377000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":8.0370000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":13.741},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":9.9629999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":17.295000000000002},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.625999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":16.423999999999999},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.426}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":13.045},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":9.3399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:42.019 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.118+0000 7fa383e75640 10 monclient: get_monmap_and_config 2024-01-20T11:23:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.118+0000 7fa383e75640 10 monclient: build_initial_monmap 2024-01-20T11:23:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.118+0000 7fa383e75640 10 monclient: monmap: 2024-01-20T11:23:42.118 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:42.118905+0000 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:42.118905+0000 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.118+0000 7fa383e75640 1 Processor -- start 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 1 -- start start 2024-01-20T11:23:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient: init 2024-01-20T11:23:42.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:42.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-01-20T11:23:42.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c0b0c00 0x7fa37c13f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient(hunting): picked mon.noname-b con 0x7fa37c0b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:42.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa37c0adbc0 0x7fa37c13f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient(hunting): picked mon.noname-a con 0x7fa37c0adbc0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa37c1420b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0aaa40 con 0x7fa37c1420b0 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37c072680 con 0x7fa37c0adbc0 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c0aac90 con 0x7fa37c0b0c00 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.119+0000 7fa383e75640 10 monclient(hunting): authenticate will time out at 1865.469832s 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381c11640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c0b0c00 0x7fa37c13f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381410640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa37c0adbc0 0x7fa37c13f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa382412640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa37c1420b0 0x7fa37c142520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:51034/0 (socket says 172.21.15.110:51034) 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa382412640 1 -- 172.21.15.110:0/1463878677 learned_addr learned my addr 172.21.15.110:0/1463878677 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381410640 10 monclient(hunting): get_auth_request con 0x7fa37c0adbc0 auth_method 0 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381410640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381410640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381410640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381c11640 10 monclient(hunting): get_auth_request con 0x7fa37c0b0c00 auth_method 0 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381c11640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.120+0000 7fa381c11640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.121+0000 7fa381c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.121+0000 7fa381c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.121+0000 7fa381c11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.121+0000 7fa381410640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.121+0000 7fa381410640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.121+0000 7fa381410640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa381c11640 10 monclient(hunting): handle_auth_done global_id 5726 payload 274 2024-01-20T11:23:42.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa381c11640 1 -- 172.21.15.110:0/1463878677 >> v2:172.21.15.110:3300/0 conn(0x7fa37c0adbc0 msgr2=0x7fa37c13f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:42.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa381c11640 1 --2- 172.21.15.110:0/1463878677 >> v2:172.21.15.110:3300/0 conn(0x7fa37c0adbc0 0x7fa37c13f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa381c11640 1 -- 172.21.15.110:0/1463878677 >> v1:172.21.15.134:6789/0 conn(0x7fa37c1420b0 legacy=0x7fa37c142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:42.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa381c11640 10 monclient: _finish_hunting 0 2024-01-20T11:23:42.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa381c11640 1 monclient: found mon.noname-b 2024-01-20T11:23:42.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa380c0f640 1 -- 172.21.15.110:0/1463878677 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802873908 0 0) 0x7fa37c0aaa40 con 0x7fa37c1420b0 2024-01-20T11:23:42.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa381c11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:42.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa381c11640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:42.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa381c11640 1 -- 172.21.15.110:0/1463878677 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0aaf70 con 0x7fa37c0b0c00 2024-01-20T11:23:42.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.122+0000 7fa380c0f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:23:42.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.124+0000 7fa381c11640 1 --2- 172.21.15.110:0/1463878677 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c0b0c00 0x7fa37c13f190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa374009d90 tx=0x7fa37402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35b8fcf830ed03bd server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:42.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 1 -- 172.21.15.110:0/1463878677 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37402cbc0 con 0x7fa37c0b0c00 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 10 monclient: dump: 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:42.129 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 10 monclient: _finish_auth 0 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 10 monclient: _check_auth_tickets 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 1 -- 172.21.15.110:0/1463878677 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37402cd20 con 0x7fa37c0b0c00 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa383e75640 5 monclient: authenticate success, global_id 5726 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa383e75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa383e75640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa383e75640 4 set_mon_vals no callback set 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 1 -- 172.21.15.110:0/1463878677 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374034660 con 0x7fa37c0b0c00 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:42.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa380c0f640 10 monclient: dump: 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa383e75640 10 monclient: shutdown 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa383e75640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa383e75640 1 -- 172.21.15.110:0/1463878677 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c0b0c00 msgr2=0x7fa37c13f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.128+0000 7fa383e75640 1 --2- 172.21.15.110:0/1463878677 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c0b0c00 0x7fa37c13f190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa374009d90 tx=0x7fa37402be10 comp rx=0 tx=0).stop 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.129+0000 7fa383e75640 1 -- 172.21.15.110:0/1463878677 shutdown_connections 2024-01-20T11:23:42.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.129+0000 7fa383e75640 1 --2- 172.21.15.110:0/1463878677 >> v2:172.21.15.110:3300/0 conn(0x7fa37c0adbc0 0x7fa37c13f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.129+0000 7fa383e75640 1 --2- 172.21.15.110:0/1463878677 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c0b0c00 0x7fa37c13f190 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.129+0000 7fa383e75640 1 -- 172.21.15.110:0/1463878677 >> 172.21.15.110:0/1463878677 conn(0x7fa37c0b3e60 msgr2=0x7fa37c0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:42.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.129+0000 7fa383e75640 1 -- 172.21.15.110:0/1463878677 shutdown_connections 2024-01-20T11:23:42.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.129+0000 7fa383e75640 1 -- 172.21.15.110:0/1463878677 wait complete. 2024-01-20T11:23:42.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.129+0000 7fa383e75640 10 monclient: build_initial_monmap 2024-01-20T11:23:42.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.129+0000 7fa383e75640 10 monclient: monmap: 2024-01-20T11:23:42.135 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:42.135 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:42.135 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:42.135 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:42.135 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 librados: starting msgr at 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 librados: starting objecter 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 Processor -- start 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 -- start start 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 librados: setting wanted keys 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 librados: calling monclient init 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient: init 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:42.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa37c0adbc0 0x7fa37c1338b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient(hunting): picked mon.noname-b con 0x7fa37c0adbc0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa37c0b0c00 0x7fa37c1317b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient(hunting): picked mon.noname-a con 0x7fa37c0b0c00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c1420b0 0x7fa37c131cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient(hunting): picked mon.noname-c con 0x7fa37c1420b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c134c80 con 0x7fa37c0b0c00 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c0b12b0 con 0x7fa37c0adbc0 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c133df0 con 0x7fa37c1420b0 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:42.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.130+0000 7fa383e75640 10 monclient(hunting): authenticate will time out at 1865.480892s 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa382412640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c1420b0 0x7fa37c131cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa37c0b0c00 0x7fa37c1317b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381c11640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa37c0adbc0 0x7fa37c1338b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa37c0b0c00 0x7fa37c1317b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43638/0 (socket says 172.21.15.110:43638) 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 1 -- 172.21.15.110:0/3888125381 learned_addr learned my addr 172.21.15.110:0/3888125381 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 10 monclient(hunting): get_auth_request con 0x7fa37c0b0c00 auth_method 0 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa382412640 10 monclient(hunting): get_auth_request con 0x7fa37c1420b0 auth_method 0 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa382412640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa382412640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa382412640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381c11640 10 monclient(hunting): get_auth_request con 0x7fa37c0adbc0 auth_method 0 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381c11640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381c11640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.131+0000 7fa381410640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.132+0000 7fa382412640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.132+0000 7fa382412640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.132+0000 7fa382412640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.132+0000 7fa381c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.132+0000 7fa381c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.132+0000 7fa381c11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 10 monclient(hunting): handle_auth_done global_id 6099 payload 824 2024-01-20T11:23:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 1 -- 172.21.15.110:0/3888125381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c1420b0 msgr2=0x7fa37c131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c1420b0 0x7fa37c131cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 1 -- 172.21.15.110:0/3888125381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa37c0adbc0 msgr2=0x7fa37c1338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa37c0adbc0 0x7fa37c1338b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 10 monclient: _finish_hunting 0 2024-01-20T11:23:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 1 monclient: found mon.noname-a 2024-01-20T11:23:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.134+0000 7fa381410640 1 -- 172.21.15.110:0/3888125381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa374009a70 con 0x7fa37c0b0c00 2024-01-20T11:23:42.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.136+0000 7fa381410640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa37c0b0c00 0x7fa37c1317b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00e9b0 tx=0x7fa36c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.136+0000 7fa3727fc640 10 client.?.objecter ms_handle_connect 0x7fa37c0b0c00 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.136+0000 7fa3727fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 1 -- 172.21.15.110:0/3888125381 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c009670 con 0x7fa37c0b0c00 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 10 monclient: dump: 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.142 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 1 -- 172.21.15.110:0/3888125381 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c00c9a0 con 0x7fa37c0b0c00 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa383e75640 5 monclient: authenticate success, global_id 6099 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa383e75640 10 monclient: _renew_subs 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa383e75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37c134ee0 con 0x7fa37c0b0c00 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 1 -- 172.21.15.110:0/3888125381 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c00cd50 con 0x7fa37c0b0c00 2024-01-20T11:23:42.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa3727fc640 10 monclient: dump: 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa373fff640 4 set_mon_vals no callback set 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa383e75640 10 client.6099.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:42.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa383e75640 10 monclient: _renew_subs 2024-01-20T11:23:42.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa383e75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:42.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37c1340b0 con 0x7fa37c0b0c00 2024-01-20T11:23:42.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.140+0000 7fa383e75640 1 librados: init done 2024-01-20T11:23:42.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.141+0000 7fa383e75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:42.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.141+0000 7fa383e75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:42.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.141+0000 7fa383e75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:42.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.141+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37c075490 con 0x7fa37c0b0c00 2024-01-20T11:23:42.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.160+0000 7fa3727fc640 1 -- 172.21.15.110:0/3888125381 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa36c013470 con 0x7fa37c0b0c00 2024-01-20T11:23:42.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.161+0000 7fa3727fc640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa36006c8b0 0x7fa36006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.161+0000 7fa381c11640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa36006c8b0 0x7fa36006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.161+0000 7fa381c11640 10 monclient: get_auth_request con 0x7fa36006c8b0 auth_method 0 2024-01-20T11:23:42.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.162+0000 7fa3727fc640 10 client.6099.objecter ms_handle_connect 0x7fa36006c8b0 2024-01-20T11:23:42.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.162+0000 7fa381c11640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa36006c8b0 0x7fa36006ed70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa374036000 tx=0x7fa374030000 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.391+0000 7fa3727fc640 1 -- 172.21.15.110:0/3888125381 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa36c058b90 con 0x7fa37c0b0c00 2024-01-20T11:23:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.391+0000 7fa3727fc640 10 client.6099.objecter ms_dispatch 0x7fa37c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.391+0000 7fa3727fc640 3 client.6099.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.391+0000 7fa3727fc640 3 client.6099.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:42.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.391+0000 7fa3727fc640 20 client.6099.objecter dump_active .. 0 homeless 2024-01-20T11:23:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.391+0000 7fa3727fc640 1 -- 172.21.15.110:0/3888125381 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa36c054ed0 con 0x7fa37c0b0c00 2024-01-20T11:23:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.391+0000 7fa3727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:42.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.391+0000 7fa3727fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:42.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.564+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa37c1351c0 con 0x7fa36006c8b0 2024-01-20T11:23:42.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.567+0000 7fa3727fc640 1 -- 172.21.15.110:0/3888125381 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63782 (secure 0 0 0) 0x7fa37c1351c0 con 0x7fa36006c8b0 2024-01-20T11:23:42.567 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:42.568 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:42.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 10 librados: watch_flush enter 2024-01-20T11:23:42.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 10 librados: watch_flush exit 2024-01-20T11:23:42.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 20 client.6099.objecter shutdown clearing up homeless session... 2024-01-20T11:23:42.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 10 client.6099.objecter successfully canceled tick 2024-01-20T11:23:42.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa36006c8b0 msgr2=0x7fa36006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:42.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa36006c8b0 0x7fa36006ed70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa374036000 tx=0x7fa374030000 comp rx=0 tx=0).stop 2024-01-20T11:23:42.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 10 monclient: shutdown 2024-01-20T11:23:42.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa37c0b0c00 msgr2=0x7fa37c1317b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa37c0b0c00 0x7fa37c1317b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00e9b0 tx=0x7fa36c00ee70 comp rx=0 tx=0).stop 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 shutdown_connections 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa36006c8b0 0x7fa36006ed70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa37c1420b0 0x7fa37c131cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa37c0b0c00 0x7fa37c1317b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 --2- 172.21.15.110:0/3888125381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa37c0adbc0 0x7fa37c1338b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 >> 172.21.15.110:0/3888125381 conn(0x7fa37c0b3e60 msgr2=0x7fa37c138f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 shutdown_connections 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 -- 172.21.15.110:0/3888125381 wait complete. 2024-01-20T11:23:42.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.569+0000 7fa383e75640 1 librados: shutdown 2024-01-20T11:23:42.580 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-01-20T11:23:40.814793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002936"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280619,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.2789999999999999},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":5.9829999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":9.2400000000000002},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":12.157999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":8.1649999999999991},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.473000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.895},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":14.669}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.215},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":15.413}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":16.109000000000002},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":13.547000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":6.9790000000000001,"5min":7.4329999999999998,"15min":9.2490000000000006},"min":{"1min":4.298,"5min":3.9820000000000002,"15min":3.9820000000000002},"max":{"1min":14.342000000000001,"5min":15.970000000000001,"15min":38.005000000000003},"last":5.532},{"interface":"front","average":{"1min":8.3219999999999992,"5min":8.5600000000000005,"15min":11.965},"min":{"1min":4.4240000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":13.082000000000001,"5min":15.512,"15min":76.718999999999994},"last":6.2400000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.6519999999999992,"5min":8.9299999999999997,"15min":9.9800000000000004},"min":{"1min":6.1289999999999996,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":16.486000000000001,"5min":16.486000000000001,"15min":18.859000000000002},"last":8.4049999999999994},{"interface":"front","average":{"1min":11.1,"5min":10.843,"15min":12.472},"min":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":16.009,"5min":16.920000000000002,"15min":43.774000000000001},"last":14.367000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.811999999999999,"5min":11.74,"15min":15.311999999999999},"min":{"1min":7.218,"5min":6.9370000000000003,"15min":6.9370000000000003},"max":{"1min":17.186,"5min":17.428000000000001,"15min":49.210000000000001},"last":15.161},{"interface":"front","average":{"1min":10.872,"5min":10.885999999999999,"15min":14.731999999999999},"min":{"1min":7.0039999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":16.440000000000001,"5min":18.869,"15min":78.548000000000002},"last":11.523999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":11.657,"5min":11.106,"15min":13.834},"min":{"1min":7.1189999999999998,"5min":5.8129999999999997,"15min":5.8129999999999997},"max":{"1min":16.427,"5min":18.474,"15min":55.771000000000001},"last":7.1790000000000003},{"interface":"front","average":{"1min":13.611000000000001,"5min":13.460000000000001,"15min":16.024000000000001},"min":{"1min":8.7929999999999993,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":18.042999999999999,"5min":20.396999999999998,"15min":49.829999999999998},"last":15.901999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":12.442,"15min":16.058},"min":{"1min":9.5890000000000004,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":18.501999999999999,"5min":18.501999999999999,"15min":91.667000000000002},"last":9.4819999999999993},{"interface":"front","average":{"1min":12.861000000000001,"5min":11.949,"15min":15.943},"min":{"1min":8.3729999999999993,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":16.303999999999998,"5min":18.236999999999998,"15min":92.613},"last":12.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":14.513999999999999,"5min":13.499000000000001,"15min":17.146000000000001},"min":{"1min":10.103,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":17.32,"5min":19.93,"15min":80.102000000000004},"last":13.407},{"interface":"front","average":{"1min":15.208,"5min":14.167999999999999,"15min":17.568000000000001},"min":{"1min":10.394,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.707000000000001,"5min":19.315999999999999,"15min":53.398000000000003},"last":10.436}]}]},{"osd":3,"up_from":21,"seq":90194313322,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":7.4710000000000001,"5min":7.5910000000000002,"15min":13.557},"min":{"1min":4.8929999999999998,"5min":3.9260000000000002,"15min":3.8700000000000001},"max":{"1min":15.608000000000001,"5min":18.129000000000001,"15min":155.27799999999999},"last":9.0540000000000003},{"interface":"front","average":{"1min":8.7889999999999997,"5min":8.0670000000000002,"15min":12.368},"min":{"1min":3.923,"5min":3.923,"15min":3.923},"max":{"1min":14.705,"5min":17.398,"15min":144.16499999999999},"last":5.7560000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":9.8610000000000007,"5min":9.8490000000000002,"15min":13.972},"min":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":15.157999999999999,"5min":15.708,"15min":149.55000000000001},"last":6.8230000000000004},{"interface":"front","average":{"1min":10.596,"5min":11.003,"15min":16.84},"min":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":15.039999999999999,"5min":17.704000000000001,"15min":155.91200000000001},"last":9.7189999999999994}]},{"osd":2,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.241,"5min":10.624000000000001,"15min":15.744999999999999},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":5.1719999999999997},"max":{"1min":15.119,"5min":18.369,"15min":145.553},"last":12.428000000000001},{"interface":"front","average":{"1min":11.109,"5min":11.368,"15min":16.009},"min":{"1min":7.4279999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":16.981000000000002,"5min":19.343,"15min":147.66300000000001},"last":7.4820000000000002}]},{"osd":4,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":12.401999999999999,"5min":12.861000000000001,"15min":21.573},"min":{"1min":7.9530000000000003,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":19.559999999999999,"5min":19.559999999999999,"15min":284.00700000000001},"last":8.4819999999999993},{"interface":"front","average":{"1min":11.872,"5min":12.544,"15min":22.021000000000001},"min":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":16.622,"5min":19.489000000000001,"15min":284.62799999999999},"last":14.978999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":10.427,"5min":11.039999999999999,"15min":17.466999999999999},"min":{"1min":7.4539999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":16.536999999999999,"5min":18.204000000000001,"15min":146.744},"last":14.358000000000001},{"interface":"front","average":{"1min":12.42,"5min":13.433,"15min":18.681999999999999},"min":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":15.185,"5min":18.326000000000001,"15min":150.542},"last":10.433}]},{"osd":7,"last update":"Sat Jan 20 11:22:37 2024","interfaces":[{"interface":"back","average":{"1min":14.673999999999999,"5min":15.286,"15min":20.666},"min":{"1min":9.9540000000000006,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":17.949999999999999,"5min":20.244,"15min":159.179},"last":13.140000000000001},{"interface":"front","average":{"1min":13.414999999999999,"5min":13.688000000000001,"15min":20.210000000000001},"min":{"1min":9.7739999999999991,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":16.693999999999999,"5min":17.818999999999999,"15min":157.35599999999999},"last":11.673999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444136,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.1020000000000003},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":6.9379999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":7.7699999999999996},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":13.567}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.488},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":9.0640000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":9.7469999999999999},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.067}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":11.627000000000001},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":8.4580000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":10.336},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":15.311}]}]},{"osd":1,"up_from":17,"seq":73014444137,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.3449999999999998},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":8.0649999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":10.32},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":11.032}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":5.3570000000000002},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":12.382999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.669},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":8.7300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":13.198},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":9.7149999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":15.653},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":13.784000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444137,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":10.670999999999999},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":5.8949999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":8.3699999999999992},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":13.314}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":16.190999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":11.49}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":12.087999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":9.077}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":13.925000000000001},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":12.718999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":16.952000000000002},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":9.9719999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":4.4219999999999997},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":10.842000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.377000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":8.0370000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":13.741},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":9.9629999999999992}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":17.295000000000002},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.625999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":16.423999999999999},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.426}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":13.045},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":9.3399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:42.581 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.680+0000 7fa884d25640 10 monclient: get_monmap_and_config 2024-01-20T11:23:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.680+0000 7fa884d25640 10 monclient: build_initial_monmap 2024-01-20T11:23:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.680+0000 7fa884d25640 10 monclient: monmap: 2024-01-20T11:23:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.680 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:42.680729+0000 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:42.680729+0000 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.680+0000 7fa884d25640 1 Processor -- start 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.680+0000 7fa884d25640 1 -- start start 2024-01-20T11:23:42.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.680+0000 7fa884d25640 10 monclient: init 2024-01-20T11:23:42.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:42.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-01-20T11:23:42.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a96f0 0x7fa88013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 10 monclient(hunting): picked mon.noname-b con 0x7fa8800a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:42.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa88013fa10 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:42.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa8800a9ad0 0x7fa880142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 10 monclient(hunting): picked mon.noname-a con 0x7fa8800a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8800b25c0 con 0x7fa88013fa10 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8800aaea0 con 0x7fa8800a9ad0 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8800ab0a0 con 0x7fa8800a96f0 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.681+0000 7fa884d25640 10 monclient(hunting): authenticate will time out at 1866.031713s 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87ed9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa8800a9ad0 0x7fa880142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a96f0 0x7fa88013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87ed9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa8800a9ad0 0x7fa880142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43654/0 (socket says 172.21.15.110:43654) 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87ed9d640 1 -- 172.21.15.110:0/1125570898 learned_addr learned my addr 172.21.15.110:0/1125570898 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87ed9d640 10 monclient(hunting): get_auth_request con 0x7fa8800a9ad0 auth_method 0 2024-01-20T11:23:42.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87e59c640 10 monclient(hunting): get_auth_request con 0x7fa8800a96f0 auth_method 0 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.682+0000 7fa87ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.683+0000 7fa87ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.683+0000 7fa87e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.683+0000 7fa87e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.683+0000 7fa87e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.684+0000 7fa87ed9d640 10 monclient(hunting): handle_auth_done global_id 6102 payload 274 2024-01-20T11:23:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.684+0000 7fa87ed9d640 1 -- 172.21.15.110:0/1125570898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a96f0 msgr2=0x7fa88013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.684+0000 7fa87ed9d640 1 --2- 172.21.15.110:0/1125570898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a96f0 0x7fa88013f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.684+0000 7fa87ed9d640 1 -- 172.21.15.110:0/1125570898 >> v1:172.21.15.110:6789/0 conn(0x7fa88013fa10 legacy=0x7fa8801423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.684+0000 7fa87ed9d640 10 monclient: _finish_hunting 0 2024-01-20T11:23:42.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.684+0000 7fa87ed9d640 1 monclient: found mon.noname-a 2024-01-20T11:23:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.684+0000 7fa87ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.684+0000 7fa87ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.684+0000 7fa87ed9d640 1 -- 172.21.15.110:0/1125570898 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8800b3fb0 con 0x7fa8800a9ad0 2024-01-20T11:23:42.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.686+0000 7fa87ed9d640 1 --2- 172.21.15.110:0/1125570898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a9ad0 0x7fa880142af0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa878009870 tx=0x7fa87802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2b8cc80ba31626 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 1 -- 172.21.15.110:0/1125570898 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8780317c0 con 0x7fa8800a9ad0 2024-01-20T11:23:42.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 10 monclient: dump: 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 10 monclient: _finish_auth 0 2024-01-20T11:23:42.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 10 monclient: _check_auth_tickets 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 1 -- 172.21.15.110:0/1125570898 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa878031920 con 0x7fa8800a9ad0 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa884d25640 5 monclient: authenticate success, global_id 6102 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa884d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa884d25640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa884d25640 4 set_mon_vals no callback set 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 1 -- 172.21.15.110:0/1125570898 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa878031cb0 con 0x7fa8800a9ad0 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.690+0000 7fa87d59a640 10 monclient: dump: 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:42.693 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 10 monclient: shutdown 2024-01-20T11:23:42.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:42.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 1 -- 172.21.15.110:0/1125570898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a9ad0 msgr2=0x7fa880142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:42.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 1 --2- 172.21.15.110:0/1125570898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a9ad0 0x7fa880142af0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa878009870 tx=0x7fa87802b860 comp rx=0 tx=0).stop 2024-01-20T11:23:42.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 1 -- 172.21.15.110:0/1125570898 shutdown_connections 2024-01-20T11:23:42.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 1 --2- 172.21.15.110:0/1125570898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a9ad0 0x7fa880142af0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 1 --2- 172.21.15.110:0/1125570898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a96f0 0x7fa88013f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 1 -- 172.21.15.110:0/1125570898 >> 172.21.15.110:0/1125570898 conn(0x7fa8800b7bc0 msgr2=0x7fa8800b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 1 -- 172.21.15.110:0/1125570898 shutdown_connections 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.691+0000 7fa884d25640 1 -- 172.21.15.110:0/1125570898 wait complete. 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.692+0000 7fa884d25640 10 monclient: build_initial_monmap 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.692+0000 7fa884d25640 10 monclient: monmap: 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.692+0000 7fa884d25640 1 librados: starting msgr at 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.692+0000 7fa884d25640 1 librados: starting objecter 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 Processor -- start 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 -- start start 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 librados: setting wanted keys 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 librados: calling monclient init 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient: init 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a96f0 0x7fa880137270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient(hunting): picked mon.noname-a con 0x7fa8800a96f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a9ad0 0x7fa880132270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient(hunting): picked mon.noname-c con 0x7fa8800a9ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa880134c20 0x7fa8801327b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient(hunting): picked mon.noname-b con 0x7fa880134c20 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa880135980 con 0x7fa8800a96f0 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8801346c0 con 0x7fa880134c20 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa880132cf0 con 0x7fa8800a9ad0 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.693+0000 7fa884d25640 10 monclient(hunting): authenticate will time out at 1866.043892s 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a96f0 0x7fa880137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a96f0 0x7fa880137270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43656/0 (socket says 172.21.15.110:43656) 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 1 -- 172.21.15.110:0/3227691437 learned_addr learned my addr 172.21.15.110:0/3227691437 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 10 monclient(hunting): get_auth_request con 0x7fa8800a96f0 auth_method 0 2024-01-20T11:23:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87ed9d640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa880134c20 0x7fa8801327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87dd9b640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a9ad0 0x7fa880132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87ed9d640 10 monclient(hunting): get_auth_request con 0x7fa880134c20 auth_method 0 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.694+0000 7fa87e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87dd9b640 10 monclient(hunting): get_auth_request con 0x7fa8800a9ad0 auth_method 0 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.695+0000 7fa87dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 10 monclient(hunting): handle_auth_done global_id 6105 payload 824 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 1 -- 172.21.15.110:0/3227691437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a9ad0 msgr2=0x7fa880132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a9ad0 0x7fa880132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 1 -- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa880134c20 msgr2=0x7fa8801327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa880134c20 0x7fa8801327b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 10 monclient: _finish_hunting 0 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 1 monclient: found mon.noname-a 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.697+0000 7fa87e59c640 1 -- 172.21.15.110:0/3227691437 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8780094d0 con 0x7fa8800a96f0 2024-01-20T11:23:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.700+0000 7fa87e59c640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a96f0 0x7fa880137270 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa87000efc0 tx=0x7fa87000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:42.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.700+0000 7fa86effd640 10 client.?.objecter ms_handle_connect 0x7fa8800a96f0 2024-01-20T11:23:42.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.700+0000 7fa86effd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 1 -- 172.21.15.110:0/3227691437 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870014480 con 0x7fa8800a96f0 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 10 monclient: dump: 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:42.704 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 10 monclient: _finish_auth 0 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 1 -- 172.21.15.110:0/3227691437 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87000f040 con 0x7fa8800a96f0 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 1 -- 172.21.15.110:0/3227691437 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870014800 con 0x7fa8800a96f0 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa86effd640 10 monclient: dump: 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.705 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa884d25640 5 monclient: authenticate success, global_id 6105 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa884d25640 10 monclient: _renew_subs 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa87cd99640 4 set_mon_vals no callback set 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa884d25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.703+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa880132f70 con 0x7fa8800a96f0 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.704+0000 7fa884d25640 10 client.6105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.704+0000 7fa884d25640 10 monclient: _renew_subs 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.704+0000 7fa884d25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:42.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.704+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa880134330 con 0x7fa8800a96f0 2024-01-20T11:23:42.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.704+0000 7fa884d25640 1 librados: init done 2024-01-20T11:23:42.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.704+0000 7fa884d25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:42.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.704+0000 7fa884d25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:42.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.704+0000 7fa884d25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:42.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.704+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8800b3da0 con 0x7fa8800a96f0 2024-01-20T11:23:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.726+0000 7fa86effd640 1 -- 172.21.15.110:0/3227691437 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa870014960 con 0x7fa8800a96f0 2024-01-20T11:23:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.726+0000 7fa86effd640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa86006c860 0x7fa86006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.727+0000 7fa87dd9b640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa86006c860 0x7fa86006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.727+0000 7fa87dd9b640 10 monclient: get_auth_request con 0x7fa86006c860 auth_method 0 2024-01-20T11:23:42.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.727+0000 7fa86effd640 10 client.6105.objecter ms_handle_connect 0x7fa86006c860 2024-01-20T11:23:42.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.727+0000 7fa87dd9b640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa86006c860 0x7fa86006ed20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa8800b3a60 tx=0x7fa874006f90 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:42.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.729+0000 7fa86effd640 1 -- 172.21.15.110:0/3227691437 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa8700594c0 con 0x7fa8800a96f0 2024-01-20T11:23:42.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.729+0000 7fa86effd640 10 client.6105.objecter ms_dispatch 0x7fa880000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:42.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.729+0000 7fa86effd640 3 client.6105.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:42.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.729+0000 7fa86effd640 3 client.6105.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:42.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.729+0000 7fa86effd640 20 client.6105.objecter dump_active .. 0 homeless 2024-01-20T11:23:42.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.958+0000 7fa86effd640 1 -- 172.21.15.110:0/3227691437 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa870055800 con 0x7fa8800a96f0 2024-01-20T11:23:42.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.958+0000 7fa86effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:42.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:42.958+0000 7fa86effd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.125+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa880075490 con 0x7fa86006c860 2024-01-20T11:23:43.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.128+0000 7fa86effd640 1 -- 172.21.15.110:0/3227691437 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63919 (secure 0 0 0) 0x7fa880075490 con 0x7fa86006c860 2024-01-20T11:23:43.128 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:43.128 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.129+0000 7fa884d25640 10 librados: watch_flush enter 2024-01-20T11:23:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.129+0000 7fa884d25640 10 librados: watch_flush exit 2024-01-20T11:23:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.129+0000 7fa884d25640 20 client.6105.objecter shutdown clearing up homeless session... 2024-01-20T11:23:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.129+0000 7fa884d25640 10 client.6105.objecter successfully canceled tick 2024-01-20T11:23:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa86006c860 msgr2=0x7fa86006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa86006c860 0x7fa86006ed20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa8800b3a60 tx=0x7fa874006f90 comp rx=0 tx=0).stop 2024-01-20T11:23:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 10 monclient: shutdown 2024-01-20T11:23:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a96f0 msgr2=0x7fa880137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a96f0 0x7fa880137270 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa87000efc0 tx=0x7fa87000c490 comp rx=0 tx=0).stop 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 shutdown_connections 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fa86006c860 0x7fa86006ed20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa880134c20 0x7fa8801327b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8800a9ad0 0x7fa880132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 --2- 172.21.15.110:0/3227691437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8800a96f0 0x7fa880137270 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 >> 172.21.15.110:0/3227691437 conn(0x7fa8800b7bc0 msgr2=0x7fa8800b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 shutdown_connections 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 -- 172.21.15.110:0/3227691437 wait complete. 2024-01-20T11:23:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.130+0000 7fa884d25640 1 librados: shutdown 2024-01-20T11:23:43.142 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-01-20T11:23:42.815409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003045"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280619,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.2789999999999999},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":5.9829999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":9.2400000000000002},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":12.157999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":8.1649999999999991},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.473000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.895},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":14.669}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.215},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":15.413}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":16.109000000000002},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":13.547000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":5.1529999999999996},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":7.0019999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":9.0730000000000004},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.6940000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":10.349},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":8.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":7.8120000000000003},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":12.138}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.795999999999999},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":12.811}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":14},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":15.837}]}]},{"osd":3,"up_from":21,"seq":90194313323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.3890000000000002},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":9.8109999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":12.26},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":14.395}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":8.7200000000000006},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":11.548999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":16.527000000000001},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":10.835000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":13.380000000000001},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":18.132000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":17.324999999999999},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":15.308999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444136,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.1020000000000003},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":6.9379999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":7.7699999999999996},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":13.567}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.488},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":9.0640000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":9.7469999999999999},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.067}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":11.627000000000001},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":8.4580000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":10.336},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":15.311}]}]},{"osd":1,"up_from":17,"seq":73014444137,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.3449999999999998},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":8.0649999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":10.32},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":11.032}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":5.3570000000000002},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":12.382999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.669},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":8.7300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":13.198},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":9.7149999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":15.653},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":13.784000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444137,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":10.670999999999999},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":5.8949999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":8.3699999999999992},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":13.314}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":16.190999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":11.49}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":12.087999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":9.077}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":13.925000000000001},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":12.718999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":16.952000000000002},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":9.9719999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509546,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":8.0109999999999992},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":11.32}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":13.579000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":14.970000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":6.2530000000000001},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":10.645}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.9819999999999993},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":9.3559999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":12.081},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":7.3529999999999998},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":16.149000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:43.144 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:43.144 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:43.144 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:43.144 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:43.144 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:43.145 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 16, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:31.815524+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:21:31.815524+0000', 'last_peered': '2024-01-20T11:21:31.815524+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:21:31.815524+0000', 'last_undegraded': '2024-01-20T11:21:31.815524+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:43.145 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:43.145 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 131, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:21:18.077638+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:21:18.077638+0000', 'last_peered': '2024-01-20T11:21:18.077638+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:21:18.077638+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:43.145 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.242+0000 7f366b475640 10 monclient: get_monmap_and_config 2024-01-20T11:23:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.242+0000 7f366b475640 10 monclient: build_initial_monmap 2024-01-20T11:23:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.242+0000 7f366b475640 10 monclient: monmap: 2024-01-20T11:23:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:43.242894+0000 2024-01-20T11:23:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:43.242894+0000 2024-01-20T11:23:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:43.242 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:43.243 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:43.243 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.242+0000 7f366b475640 1 Processor -- start 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 1 -- start start 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient: init 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664066680 0x7f366413cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient(hunting): picked mon.noname-b con 0x7f3664066680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f366406dcf0 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3664143980 0x7f366413d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient(hunting): picked mon.noname-a con 0x7f3664143980 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3664143e10 con 0x7f366406dcf0 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366413d980 con 0x7f3664143980 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366406e2f0 con 0x7f3664066680 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.243+0000 7f366b475640 10 monclient(hunting): authenticate will time out at 1866.593959s 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669211640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664066680 0x7f366413cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669a12640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3664143980 0x7f366413d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669a12640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3664143980 0x7f366413d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43658/0 (socket says 172.21.15.110:43658) 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669a12640 1 -- 172.21.15.110:0/802084179 learned_addr learned my addr 172.21.15.110:0/802084179 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669a12640 10 monclient(hunting): get_auth_request con 0x7f3664143980 auth_method 0 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669a12640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669a12640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669211640 10 monclient(hunting): get_auth_request con 0x7f3664066680 auth_method 0 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669211640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.244+0000 7f3669211640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.245+0000 7f3669a12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.245+0000 7f3669a12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.245+0000 7f3669a12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.245+0000 7f3669211640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.245+0000 7f3669211640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.245+0000 7f3669211640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.247+0000 7f3669a12640 10 monclient(hunting): handle_auth_done global_id 6108 payload 274 2024-01-20T11:23:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.247+0000 7f3669a12640 1 -- 172.21.15.110:0/802084179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664066680 msgr2=0x7f366413cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.247+0000 7f3669a12640 1 --2- 172.21.15.110:0/802084179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664066680 0x7f366413cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.247+0000 7f3669a12640 1 -- 172.21.15.110:0/802084179 >> v1:172.21.15.110:6789/0 conn(0x7f366406dcf0 legacy=0x7f3664143270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.247+0000 7f3669a12640 10 monclient: _finish_hunting 0 2024-01-20T11:23:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.247+0000 7f3669a12640 1 monclient: found mon.noname-a 2024-01-20T11:23:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.247+0000 7f3669a12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.247+0000 7f3669a12640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.247+0000 7f3669a12640 1 -- 172.21.15.110:0/802084179 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3664072140 con 0x7f3664143980 2024-01-20T11:23:43.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.249+0000 7f3669a12640 1 --2- 172.21.15.110:0/802084179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3664143980 0x7f366413d440 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f365c009e50 tx=0x7f365c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39eb439f219ce951 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 1 -- 172.21.15.110:0/802084179 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c02ca70 con 0x7f3664143980 2024-01-20T11:23:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 10 monclient: dump: 2024-01-20T11:23:43.253 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 10 monclient: _finish_auth 0 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 1 -- 172.21.15.110:0/802084179 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365c02cbd0 con 0x7f3664143980 2024-01-20T11:23:43.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f366b475640 5 monclient: authenticate success, global_id 6108 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f366b475640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f366b475640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 1 -- 172.21.15.110:0/802084179 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c0343f0 con 0x7f3664143980 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f366b475640 4 set_mon_vals no callback set 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.253+0000 7f365bfff640 10 monclient: dump: 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.255 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:43.256 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:43.256 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:43.256 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 10 monclient: shutdown 2024-01-20T11:23:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 1 -- 172.21.15.110:0/802084179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3664143980 msgr2=0x7f366413d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 1 --2- 172.21.15.110:0/802084179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3664143980 0x7f366413d440 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f365c009e50 tx=0x7f365c02bed0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 1 -- 172.21.15.110:0/802084179 shutdown_connections 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 1 --2- 172.21.15.110:0/802084179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3664143980 0x7f366413d440 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 1 --2- 172.21.15.110:0/802084179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664066680 0x7f366413cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 1 -- 172.21.15.110:0/802084179 >> 172.21.15.110:0/802084179 conn(0x7f36640b09e0 msgr2=0x7f36640b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 1 -- 172.21.15.110:0/802084179 shutdown_connections 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.254+0000 7f366b475640 1 -- 172.21.15.110:0/802084179 wait complete. 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.255+0000 7f366b475640 10 monclient: build_initial_monmap 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.255+0000 7f366b475640 10 monclient: monmap: 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.255+0000 7f366b475640 1 librados: starting msgr at 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.255+0000 7f366b475640 1 librados: starting objecter 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.255+0000 7f366b475640 1 Processor -- start 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.255+0000 7f366b475640 1 -- start start 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.255+0000 7f366b475640 1 librados: setting wanted keys 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.255+0000 7f366b475640 1 librados: calling monclient init 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.255+0000 7f366b475640 10 monclient: init 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3664066680 0x7f36641d8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient(hunting): picked mon.noname-b con 0x7f3664066680 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366406dcf0 0x7f36641d9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient(hunting): picked mon.noname-a con 0x7f366406dcf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664143980 0x7f36640ac330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient(hunting): picked mon.noname-c con 0x7f3664143980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36641d8670 con 0x7f366406dcf0 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36641447b0 con 0x7f3664066680 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36640a9480 con 0x7f3664143980 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f366b475640 10 monclient(hunting): authenticate will time out at 1866.606325s 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3668a10640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366406dcf0 0x7f36641d9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3669a12640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664143980 0x7f36640ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3668a10640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366406dcf0 0x7f36641d9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43672/0 (socket says 172.21.15.110:43672) 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3668a10640 1 -- 172.21.15.110:0/1455917724 learned_addr learned my addr 172.21.15.110:0/1455917724 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3668a10640 10 monclient(hunting): get_auth_request con 0x7f366406dcf0 auth_method 0 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3668a10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3668a10640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3668a10640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3669a12640 10 monclient(hunting): get_auth_request con 0x7f3664143980 auth_method 0 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3669a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3669a12640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3669a12640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.256+0000 7f3669211640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3664066680 0x7f36641d8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3668a10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3668a10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3668a10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669211640 10 monclient(hunting): get_auth_request con 0x7f3664066680 auth_method 0 2024-01-20T11:23:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669211640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669211640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669a12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669a12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669a12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669211640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669211640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.257+0000 7f3669211640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 10 monclient(hunting): handle_auth_done global_id 6111 payload 824 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 1 -- 172.21.15.110:0/1455917724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664143980 msgr2=0x7f36640ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664143980 0x7f36640ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 1 -- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3664066680 msgr2=0x7f36641d8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3664066680 0x7f36641d8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 10 monclient: _finish_hunting 0 2024-01-20T11:23:43.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 1 monclient: found mon.noname-a 2024-01-20T11:23:43.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:43.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.260+0000 7f3668a10640 1 -- 172.21.15.110:0/1455917724 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c009b30 con 0x7f366406dcf0 2024-01-20T11:23:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.262+0000 7f3668a10640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366406dcf0 0x7f36641d9040 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f366000c410 tx=0x7f366000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.262+0000 7f3659ffb640 10 client.?.objecter ms_handle_connect 0x7f366406dcf0 2024-01-20T11:23:43.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.262+0000 7f3659ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 1 -- 172.21.15.110:0/1455917724 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36600035f0 con 0x7f366406dcf0 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 10 monclient: dump: 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:43.268 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 10 monclient: _finish_auth 0 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 1 -- 172.21.15.110:0/1455917724 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3660003750 con 0x7f366406dcf0 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 1 -- 172.21.15.110:0/1455917724 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660003b40 con 0x7f366406dcf0 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f366b475640 5 monclient: authenticate success, global_id 6111 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f366b475640 10 monclient: _renew_subs 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f366b475640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36640ac870 con 0x7f366406dcf0 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f365b7fe640 4 set_mon_vals no callback set 2024-01-20T11:23:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f3659ffb640 10 monclient: dump: 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f366b475640 10 client.6111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f366b475640 10 monclient: _renew_subs 2024-01-20T11:23:43.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f366b475640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36640acd50 con 0x7f366406dcf0 2024-01-20T11:23:43.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.266+0000 7f366b475640 1 librados: init done 2024-01-20T11:23:43.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.267+0000 7f366b475640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:43.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.267+0000 7f366b475640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:43.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.267+0000 7f366b475640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.267+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36641d87e0 con 0x7f366406dcf0 2024-01-20T11:23:43.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.510+0000 7f3659ffb640 1 -- 172.21.15.110:0/1455917724 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3660003ca0 con 0x7f366406dcf0 2024-01-20T11:23:43.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.510+0000 7f3659ffb640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f364806c8b0 0x7f364806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.511+0000 7f3669211640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f364806c8b0 0x7f364806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.511+0000 7f3669211640 10 monclient: get_auth_request con 0x7f364806c8b0 auth_method 0 2024-01-20T11:23:43.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.511+0000 7f3669211640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f364806c8b0 0x7f364806ed70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3654009b40 tx=0x7f3654009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:43.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.511+0000 7f3659ffb640 10 client.6111.objecter ms_handle_connect 0x7f364806c8b0 2024-01-20T11:23:43.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.525+0000 7f3659ffb640 1 -- 172.21.15.110:0/1455917724 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f36600010c0 con 0x7f366406dcf0 2024-01-20T11:23:43.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.525+0000 7f3659ffb640 10 client.6111.objecter ms_dispatch 0x7f3664000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:43.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.525+0000 7f3659ffb640 3 client.6111.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:43.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.525+0000 7f3659ffb640 3 client.6111.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:43.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.525+0000 7f3659ffb640 20 client.6111.objecter dump_active .. 0 homeless 2024-01-20T11:23:43.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.525+0000 7f3659ffb640 1 -- 172.21.15.110:0/1455917724 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f36600540b0 con 0x7f366406dcf0 2024-01-20T11:23:43.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.525+0000 7f3659ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:43.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.525+0000 7f3659ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:43.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~286s 2024-01-20T11:23:43.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.798+0000 7f366b475640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:43.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.798+0000 7f366b475640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:43.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.798+0000 7f366b475640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.798+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3664075490 con 0x7f366406dcf0 2024-01-20T11:23:43.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.804+0000 7f3659ffb640 1 -- 172.21.15.110:0/1455917724 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f3660017520 con 0x7f366406dcf0 2024-01-20T11:23:43.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.804+0000 7f3659ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:43.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.804+0000 7f3659ffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:43.804 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:43.804 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":558,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:43.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 10 librados: watch_flush enter 2024-01-20T11:23:43.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 10 librados: watch_flush exit 2024-01-20T11:23:43.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 20 client.6111.objecter shutdown clearing up homeless session... 2024-01-20T11:23:43.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 10 client.6111.objecter successfully canceled tick 2024-01-20T11:23:43.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f364806c8b0 msgr2=0x7f364806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f364806c8b0 0x7f364806ed70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3654009b40 tx=0x7f3654009190 comp rx=0 tx=0).stop 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 10 monclient: shutdown 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366406dcf0 msgr2=0x7f36641d9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366406dcf0 0x7f36641d9040 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f366000c410 tx=0x7f366000c8d0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 shutdown_connections 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f364806c8b0 0x7f364806ed70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3664143980 0x7f36640ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f366406dcf0 0x7f36641d9040 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 --2- 172.21.15.110:0/1455917724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3664066680 0x7f36641d8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 >> 172.21.15.110:0/1455917724 conn(0x7f36640b09e0 msgr2=0x7f366413fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 shutdown_connections 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 -- 172.21.15.110:0/1455917724 wait complete. 2024-01-20T11:23:43.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.806+0000 7f366b475640 1 librados: shutdown 2024-01-20T11:23:43.816 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:43.817 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.917+0000 7effcd007640 10 monclient: get_monmap_and_config 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.917+0000 7effcd007640 10 monclient: build_initial_monmap 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.917+0000 7effcd007640 10 monclient: monmap: 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:43.917766+0000 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:43.917766+0000 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:43.917 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:43.918 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:43.918 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:43.918 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.917+0000 7effcd007640 1 Processor -- start 2024-01-20T11:23:43.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.917+0000 7effcd007640 1 -- start start 2024-01-20T11:23:43.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.917+0000 7effcd007640 10 monclient: init 2024-01-20T11:23:43.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:43.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-01-20T11:23:43.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b4d10 0x7effc80b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 10 monclient(hunting): picked mon.noname-b con 0x7effc80b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:43.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7effc80b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7effc8145980 0x7effc8145e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 10 monclient(hunting): picked mon.noname-a con 0x7effc8145980 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc80aade0 con 0x7effc80b17c0 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc80b50f0 con 0x7effc8145980 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc8145460 con 0x7effc80b4d10 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effcd007640 10 monclient(hunting): authenticate will time out at 1867.268534s 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effc759e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7effc8145980 0x7effc8145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.918+0000 7effc6d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b4d10 0x7effc80b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc659c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7effc80b17c0 0x7effc80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:51036/0 (socket says 172.21.15.110:51036) 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc659c640 1 -- 172.21.15.110:0/979908524 learned_addr learned my addr 172.21.15.110:0/979908524 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:43.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc759e640 10 monclient(hunting): get_auth_request con 0x7effc8145980 auth_method 0 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc759e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc759e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc6d9d640 10 monclient(hunting): get_auth_request con 0x7effc80b4d10 auth_method 0 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc6d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.919+0000 7effc6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc759e640 10 monclient(hunting): handle_auth_done global_id 6114 payload 274 2024-01-20T11:23:43.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc759e640 1 -- 172.21.15.110:0/979908524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b4d10 msgr2=0x7effc80b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:43.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc759e640 1 --2- 172.21.15.110:0/979908524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b4d10 0x7effc80b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc759e640 1 -- 172.21.15.110:0/979908524 >> v1:172.21.15.134:6789/0 conn(0x7effc80b17c0 legacy=0x7effc80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc759e640 10 monclient: _finish_hunting 0 2024-01-20T11:23:43.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc759e640 1 monclient: found mon.noname-a 2024-01-20T11:23:43.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:43.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc759e640 1 -- 172.21.15.110:0/979908524 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8146350 con 0x7effc8145980 2024-01-20T11:23:43.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.921+0000 7effc6d9d640 1 --2- 172.21.15.110:0/979908524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b4d10 0x7effc80b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:43.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.923+0000 7effc759e640 1 --2- 172.21.15.110:0/979908524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc8145980 0x7effc8145e10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7effc0009d90 tx=0x7effc002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee944a38d8f8dd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:43.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 1 -- 172.21.15.110:0/979908524 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc002cbc0 con 0x7effc8145980 2024-01-20T11:23:43.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 10 monclient: dump: 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 10 monclient: _finish_auth 0 2024-01-20T11:23:43.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 1 -- 172.21.15.110:0/979908524 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc002cd20 con 0x7effc8145980 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 5 monclient: authenticate success, global_id 6114 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 1 -- 172.21.15.110:0/979908524 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc0034660 con 0x7effc8145980 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 4 set_mon_vals no callback set 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effc5d9b640 10 monclient: dump: 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.929 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 10 monclient: shutdown 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 1 -- 172.21.15.110:0/979908524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc8145980 msgr2=0x7effc8145e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 1 --2- 172.21.15.110:0/979908524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc8145980 0x7effc8145e10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7effc0009d90 tx=0x7effc002be10 comp rx=0 tx=0).stop 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 1 -- 172.21.15.110:0/979908524 shutdown_connections 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 1 --2- 172.21.15.110:0/979908524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc8145980 0x7effc8145e10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 1 --2- 172.21.15.110:0/979908524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b4d10 0x7effc80b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.927+0000 7effcd007640 1 -- 172.21.15.110:0/979908524 >> 172.21.15.110:0/979908524 conn(0x7effc8066590 msgr2=0x7effc80669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:43.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.928+0000 7effcd007640 1 -- 172.21.15.110:0/979908524 shutdown_connections 2024-01-20T11:23:43.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.928+0000 7effcd007640 1 -- 172.21.15.110:0/979908524 wait complete. 2024-01-20T11:23:43.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.928+0000 7effcd007640 10 monclient: build_initial_monmap 2024-01-20T11:23:43.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.928+0000 7effcd007640 10 monclient: monmap: 2024-01-20T11:23:43.934 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:43.934 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:43.934 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.929+0000 7effcd007640 1 librados: starting msgr at 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.929+0000 7effcd007640 1 librados: starting objecter 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.929+0000 7effcd007640 1 Processor -- start 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.929+0000 7effcd007640 1 -- start start 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.929+0000 7effcd007640 1 librados: setting wanted keys 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.929+0000 7effcd007640 1 librados: calling monclient init 2024-01-20T11:23:43.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.929+0000 7effcd007640 10 monclient: init 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b17c0 0x7effc8142af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient(hunting): picked mon.noname-c con 0x7effc80b17c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc80b4d10 0x7effc81d78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient(hunting): picked mon.noname-a con 0x7effc80b4d10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7effc8145980 0x7effc8146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient(hunting): picked mon.noname-b con 0x7effc8145980 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc81d7e20 con 0x7effc80b4d10 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc8146730 con 0x7effc8145980 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:43.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc8148100 con 0x7effc80b17c0 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effcd007640 10 monclient(hunting): authenticate will time out at 1867.280326s 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effc659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc80b4d10 0x7effc81d78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effc6d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b17c0 0x7effc8142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effc759e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7effc8145980 0x7effc8146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effc659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc80b4d10 0x7effc81d78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43680/0 (socket says 172.21.15.110:43680) 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effc659c640 1 -- 172.21.15.110:0/2573182665 learned_addr learned my addr 172.21.15.110:0/2573182665 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effc659c640 10 monclient(hunting): get_auth_request con 0x7effc80b4d10 auth_method 0 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effc659c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.930+0000 7effc659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc759e640 10 monclient(hunting): get_auth_request con 0x7effc8145980 auth_method 0 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc759e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc759e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc6d9d640 10 monclient(hunting): get_auth_request con 0x7effc80b17c0 auth_method 0 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc6d9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:43.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.931+0000 7effc6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 10 monclient(hunting): handle_auth_done global_id 6117 payload 824 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 1 -- 172.21.15.110:0/2573182665 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b17c0 msgr2=0x7effc8142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b17c0 0x7effc8142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 1 -- 172.21.15.110:0/2573182665 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7effc8145980 msgr2=0x7effc8146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7effc8145980 0x7effc8146c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 10 monclient: _finish_hunting 0 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 1 monclient: found mon.noname-a 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.934+0000 7effc659c640 1 -- 172.21.15.110:0/2573182665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb8009400 con 0x7effc80b4d10 2024-01-20T11:23:43.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.936+0000 7effc659c640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc80b4d10 0x7effc81d78e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7effbc00c3c0 tx=0x7effbc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.936+0000 7effaf7fe640 10 client.?.objecter ms_handle_connect 0x7effc80b4d10 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.936+0000 7effaf7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.939+0000 7effaf7fe640 1 -- 172.21.15.110:0/2573182665 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc005ce0 con 0x7effc80b4d10 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 10 monclient: dump: 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.942 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effcd007640 5 monclient: authenticate success, global_id 6117 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effcd007640 10 monclient: _renew_subs 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 1 -- 172.21.15.110:0/2573182665 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effbc005e40 con 0x7effc80b4d10 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effcd007640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc0009a70 con 0x7effc80b4d10 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 1 -- 172.21.15.110:0/2573182665 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc0072f0 con 0x7effc80b4d10 2024-01-20T11:23:43.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effaf7fe640 10 monclient: dump: 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effc559a640 4 set_mon_vals no callback set 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effcd007640 10 client.6117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:43.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effcd007640 10 monclient: _renew_subs 2024-01-20T11:23:43.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effcd007640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc814c390 con 0x7effc80b4d10 2024-01-20T11:23:43.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.940+0000 7effcd007640 1 librados: init done 2024-01-20T11:23:43.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.941+0000 7effcd007640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:43.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.941+0000 7effcd007640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:43.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.941+0000 7effcd007640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:43.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:43.941+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc8146590 con 0x7effc80b4d10 2024-01-20T11:23:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.181+0000 7effaf7fe640 1 -- 172.21.15.110:0/2573182665 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7effbc007450 con 0x7effc80b4d10 2024-01-20T11:23:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.181+0000 7effaf7fe640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7effb406c8e0 0x7effb406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:44.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.182+0000 7effc6d9d640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7effb406c8e0 0x7effb406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:44.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.182+0000 7effc6d9d640 10 monclient: get_auth_request con 0x7effb406c8e0 auth_method 0 2024-01-20T11:23:44.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.182+0000 7effc6d9d640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7effb406c8e0 0x7effb406eda0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7effb8009cd0 tx=0x7effb8009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:44.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.182+0000 7effaf7fe640 10 client.6117.objecter ms_handle_connect 0x7effb406c8e0 2024-01-20T11:23:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.194+0000 7effaf7fe640 1 -- 172.21.15.110:0/2573182665 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7effbc057a10 con 0x7effc80b4d10 2024-01-20T11:23:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.194+0000 7effaf7fe640 10 client.6117.objecter ms_dispatch 0x7effc8000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.194+0000 7effaf7fe640 3 client.6117.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.194+0000 7effaf7fe640 3 client.6117.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.194+0000 7effaf7fe640 20 client.6117.objecter dump_active .. 0 homeless 2024-01-20T11:23:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.194+0000 7effaf7fe640 1 -- 172.21.15.110:0/2573182665 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7effbc053d50 con 0x7effc80b4d10 2024-01-20T11:23:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.194+0000 7effaf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.194+0000 7effaf7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:44.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.360+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effc814c7f0 con 0x7effb406c8e0 2024-01-20T11:23:44.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.363+0000 7effaf7fe640 1 -- 172.21.15.110:0/2573182665 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63919 (secure 0 0 0) 0x7effb8009920 con 0x7effb406c8e0 2024-01-20T11:23:44.363 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:44.364 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 10 librados: watch_flush enter 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 10 librados: watch_flush exit 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 20 client.6117.objecter shutdown clearing up homeless session... 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 10 client.6117.objecter successfully canceled tick 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7effb406c8e0 msgr2=0x7effb406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7effb406c8e0 0x7effb406eda0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7effb8009cd0 tx=0x7effb8009190 comp rx=0 tx=0).stop 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 10 monclient: shutdown 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc80b4d10 msgr2=0x7effc81d78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:44.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc80b4d10 0x7effc81d78e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7effbc00c3c0 tx=0x7effbc00c880 comp rx=0 tx=0).stop 2024-01-20T11:23:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 shutdown_connections 2024-01-20T11:23:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7effb406c8e0 0x7effb406eda0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7effc8145980 0x7effc8146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effc80b4d10 0x7effc81d78e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 --2- 172.21.15.110:0/2573182665 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effc80b17c0 0x7effc8142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.365+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 >> 172.21.15.110:0/2573182665 conn(0x7effc8066590 msgr2=0x7effc8144460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.366+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 shutdown_connections 2024-01-20T11:23:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.366+0000 7effcd007640 1 -- 172.21.15.110:0/2573182665 wait complete. 2024-01-20T11:23:44.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:44.366+0000 7effcd007640 1 librados: shutdown 2024-01-20T11:23:44.378 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-01-20T11:23:42.815409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003045"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280619,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.2789999999999999},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":5.9829999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":9.2400000000000002},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":12.157999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":8.1649999999999991},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":11.473000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":10.895},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":14.669}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":10.215},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":15.413}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":16.109000000000002},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":13.547000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":5.1529999999999996},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":7.0019999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":9.0730000000000004},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.6940000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":10.349},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":8.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":7.8120000000000003},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":12.138}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.795999999999999},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":12.811}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":14},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":15.837}]}]},{"osd":3,"up_from":21,"seq":90194313323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.3890000000000002},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":9.8109999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":12.26},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":14.395}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":8.7200000000000006},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":11.548999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":16.527000000000001},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":10.835000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":13.380000000000001},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":18.132000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":17.324999999999999},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":15.308999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444136,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":4.1020000000000003},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":6.9379999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":7.7699999999999996},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":13.567}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.488},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":9.0640000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":13.981,"5min":13.76,"15min":16.388000000000002},"min":{"1min":7.8200000000000003,"5min":7.3869999999999996,"15min":7.3869999999999996},"max":{"1min":18.986000000000001,"5min":20.393000000000001,"15min":51.000999999999998},"last":9.7469999999999999},{"interface":"front","average":{"1min":12.003,"5min":12.244999999999999,"15min":13.826000000000001},"min":{"1min":7.431,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":17.805,"5min":21.629999999999999,"15min":48.299999999999997},"last":11.067}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":11.627000000000001},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":8.4580000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":10.336},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":15.311}]}]},{"osd":1,"up_from":17,"seq":73014444137,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":6.3449999999999998},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":8.0649999999999995}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":10.32},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":11.032}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":5.3570000000000002},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":12.382999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:36 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":11.598000000000001,"15min":13.307},"min":{"1min":5.9669999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":13.555,"5min":18.899999999999999,"15min":25.541},"last":11.669},{"interface":"front","average":{"1min":14.597,"5min":13.57,"15min":14.781000000000001},"min":{"1min":10.856999999999999,"5min":6.75,"15min":6.75},"max":{"1min":18.100999999999999,"5min":22.280000000000001,"15min":30.204999999999998},"last":8.7300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":13.198},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":9.7149999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":15.653},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":13.784000000000001}]}]},{"osd":0,"up_from":17,"seq":73014444137,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":10.670999999999999},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":5.8949999999999996}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":8.3699999999999992},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":13.314}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":16.190999999999999},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":11.49}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":12.087999999999999},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":9.077}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":13.925000000000001},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":12.718999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":16.952000000000002},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":9.9719999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509546,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":8.0109999999999992},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":11.32}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":13.579000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":14.970000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":6.2530000000000001},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":10.645}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.9819999999999993},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":9.3559999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:22:38 2024","interfaces":[{"interface":"back","average":{"1min":13.968,"5min":13.611000000000001,"15min":15.914999999999999},"min":{"1min":8.9339999999999993,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":18.495000000000001,"5min":18.495000000000001,"15min":70.254999999999995},"last":12.081},{"interface":"front","average":{"1min":11.864000000000001,"5min":12.443,"15min":15.521000000000001},"min":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":15.683,"5min":18.062000000000001,"15min":72.670000000000002},"last":14.199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":7.3529999999999998},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":16.149000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:44.846 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:35.871 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:44.849 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:35.873 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:44.864 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:35.889 34116== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:44.896 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:09:35.902 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:44.899 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:09:35.906 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:44.900 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:35.925 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:44.903 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:35.928 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:44.915 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:09:35.921 33847== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:44.919 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:35.943 34120== Warning: unimplemented fcntl command: 1036 2024-01-20T11:23:47.380 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.479+0000 7f86a47dc640 10 monclient: get_monmap_and_config 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.479+0000 7f86a47dc640 10 monclient: build_initial_monmap 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.479+0000 7f86a47dc640 10 monclient: monmap: 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:47.479696+0000 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:47.479696+0000 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:47.479 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:47.480 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:47.480 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.479+0000 7f86a47dc640 1 Processor -- start 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.479+0000 7f86a47dc640 1 -- start start 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.479+0000 7f86a47dc640 10 monclient: init 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f869c0a96f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f869c0a9ad0 0x7f869c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 10 monclient(hunting): picked mon.noname-a con 0x7f869c0a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f869c142920 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:47.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c0b25c0 con 0x7f869c0a96f0 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c13fbe0 con 0x7f869c142920 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c0aaea0 con 0x7f869c0a9ad0 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.480+0000 7f86a47dc640 10 monclient(hunting): authenticate will time out at 1870.830680s 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a1d77640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f869c0a9ad0 0x7f869c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a2d79640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f869c142920 0x7f869c14c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:51050/0 (socket says 172.21.15.110:51050) 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a2d79640 1 -- 172.21.15.110:0/2419386579 learned_addr learned my addr 172.21.15.110:0/2419386579 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a1d77640 10 monclient(hunting): get_auth_request con 0x7f869c0a9ad0 auth_method 0 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a1d77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a1d77640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:47.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a1d77640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:47.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a1d77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:47.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a1d77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:47.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.481+0000 7f86a1d77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:47.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.483+0000 7f86a1d77640 10 monclient(hunting): handle_auth_done global_id 6120 payload 274 2024-01-20T11:23:47.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.483+0000 7f86a1d77640 1 -- 172.21.15.110:0/2419386579 >> v1:172.21.15.134:6789/0 conn(0x7f869c142920 legacy=0x7f869c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:47.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.483+0000 7f86a1d77640 1 -- 172.21.15.110:0/2419386579 >> v1:172.21.15.110:6789/0 conn(0x7f869c0a96f0 legacy=0x7f869c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:47.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.483+0000 7f86a1d77640 10 monclient: _finish_hunting 0 2024-01-20T11:23:47.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.483+0000 7f86a1d77640 1 monclient: found mon.noname-a 2024-01-20T11:23:47.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.483+0000 7f86a1d77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:47.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.483+0000 7f86a1d77640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:47.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.483+0000 7f86a1d77640 1 -- 172.21.15.110:0/2419386579 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c0ab0a0 con 0x7f869c0a9ad0 2024-01-20T11:23:47.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.488+0000 7f86a1d77640 1 --2- 172.21.15.110:0/2419386579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 0x7f869c1423e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f868c0097a0 tx=0x7f868c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75ce3a7095c1324 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:47.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 1 -- 172.21.15.110:0/2419386579 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c02e070 con 0x7f869c0a9ad0 2024-01-20T11:23:47.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:47.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:47.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 10 monclient: dump: 2024-01-20T11:23:47.494 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:47.494 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 10 monclient: _finish_auth 0 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 10 monclient: _check_auth_tickets 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 1 -- 172.21.15.110:0/2419386579 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c0315a0 con 0x7f869c0a9ad0 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 1 -- 172.21.15.110:0/2419386579 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c031930 con 0x7f869c0a9ad0 2024-01-20T11:23:47.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a1576640 10 monclient: dump: 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a47dc640 5 monclient: authenticate success, global_id 6120 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a47dc640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:47.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.494+0000 7f86a47dc640 4 set_mon_vals no callback set 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.495+0000 7f86a47dc640 10 monclient: shutdown 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.495+0000 7f86a47dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.495+0000 7f86a47dc640 1 -- 172.21.15.110:0/2419386579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 msgr2=0x7f869c1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.495+0000 7f86a47dc640 1 --2- 172.21.15.110:0/2419386579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 0x7f869c1423e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f868c0097a0 tx=0x7f868c02b840 comp rx=0 tx=0).stop 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.495+0000 7f86a47dc640 1 -- 172.21.15.110:0/2419386579 shutdown_connections 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.495+0000 7f86a47dc640 1 --2- 172.21.15.110:0/2419386579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 0x7f869c1423e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.495+0000 7f86a47dc640 1 -- 172.21.15.110:0/2419386579 >> 172.21.15.110:0/2419386579 conn(0x7f869c0b7bc0 msgr2=0x7f869c0b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.495+0000 7f86a47dc640 1 -- 172.21.15.110:0/2419386579 shutdown_connections 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.495+0000 7f86a47dc640 1 -- 172.21.15.110:0/2419386579 wait complete. 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.496+0000 7f86a47dc640 10 monclient: build_initial_monmap 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.496+0000 7f86a47dc640 10 monclient: monmap: 2024-01-20T11:23:47.500 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.496+0000 7f86a47dc640 1 librados: starting msgr at 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.496+0000 7f86a47dc640 1 librados: starting objecter 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.496+0000 7f86a47dc640 1 Processor -- start 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.496+0000 7f86a47dc640 1 -- start start 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.496+0000 7f86a47dc640 1 librados: setting wanted keys 2024-01-20T11:23:47.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.496+0000 7f86a47dc640 1 librados: calling monclient init 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.496+0000 7f86a47dc640 10 monclient: init 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f869c0a96f0 0x7f869c0b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient(hunting): picked mon.noname-b con 0x7f869c0a96f0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 0x7f869c1d76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient(hunting): picked mon.noname-a con 0x7f869c0a9ad0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c142920 0x7f869c1d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient(hunting): picked mon.noname-c con 0x7f869c142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0afe20 con 0x7f869c0a9ad0 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0b0340 con 0x7f869c0a96f0 2024-01-20T11:23:47.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0b0500 con 0x7f869c142920 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a47dc640 10 monclient(hunting): authenticate will time out at 1870.847528s 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a1d77640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 0x7f869c1d76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.497+0000 7f86a2d79640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c142920 0x7f869c1d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2578640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f869c0a96f0 0x7f869c0b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a1d77640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 0x7f869c1d76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43696/0 (socket says 172.21.15.110:43696) 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a1d77640 1 -- 172.21.15.110:0/1003234748 learned_addr learned my addr 172.21.15.110:0/1003234748 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a1d77640 10 monclient(hunting): get_auth_request con 0x7f869c0a9ad0 auth_method 0 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a1d77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a1d77640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a1d77640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2d79640 10 monclient(hunting): get_auth_request con 0x7f869c142920 auth_method 0 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:47.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2d79640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2d79640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2578640 10 monclient(hunting): get_auth_request con 0x7f869c0a96f0 auth_method 0 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2578640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2578640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a1d77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a1d77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a1d77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2d79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2d79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2d79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2578640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2578640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:47.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.498+0000 7f86a2578640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:47.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 10 monclient(hunting): handle_auth_done global_id 6123 payload 824 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 1 -- 172.21.15.110:0/1003234748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c142920 msgr2=0x7f869c1d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c142920 0x7f869c1d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 1 -- 172.21.15.110:0/1003234748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f869c0a96f0 msgr2=0x7f869c0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f869c0a96f0 0x7f869c0b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 10 monclient: _finish_hunting 0 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 1 monclient: found mon.noname-a 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.501+0000 7f86a1d77640 1 -- 172.21.15.110:0/1003234748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c009400 con 0x7f869c0a9ad0 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.503+0000 7f86a1d77640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 0x7f869c1d76d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f868c0096f0 tx=0x7f868c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.503+0000 7f8692ffd640 10 client.?.objecter ms_handle_connect 0x7f869c0a9ad0 2024-01-20T11:23:47.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.503+0000 7f8692ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 1 -- 172.21.15.110:0/1003234748 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c032900 con 0x7f869c0a9ad0 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 10 monclient: dump: 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:47.508 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 10 monclient: _finish_auth 0 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 1 -- 172.21.15.110:0/1003234748 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c032aa0 con 0x7f869c0a9ad0 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a47dc640 5 monclient: authenticate success, global_id 6123 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a47dc640 10 monclient: _renew_subs 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 1 -- 172.21.15.110:0/1003234748 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c032e50 con 0x7f869c0a9ad0 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a47dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c0afa40 con 0x7f869c0a9ad0 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a0d75640 4 set_mon_vals no callback set 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:47.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f8692ffd640 10 monclient: dump: 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a47dc640 10 client.6123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a47dc640 10 monclient: _renew_subs 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a47dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c0b1080 con 0x7f869c0a9ad0 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.506+0000 7f86a47dc640 1 librados: init done 2024-01-20T11:23:47.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.507+0000 7f86a47dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:47.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.507+0000 7f86a47dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:47.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.507+0000 7f86a47dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:47.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.507+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c0b1ca0 con 0x7f869c0a9ad0 2024-01-20T11:23:47.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.748+0000 7f8692ffd640 1 -- 172.21.15.110:0/1003234748 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f868c04e430 con 0x7f869c0a9ad0 2024-01-20T11:23:47.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.748+0000 7f8692ffd640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f868006c8b0 0x7f868006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:47.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.749+0000 7f86a2578640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f868006c8b0 0x7f868006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:47.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.749+0000 7f86a2578640 10 monclient: get_auth_request con 0x7f868006c8b0 auth_method 0 2024-01-20T11:23:47.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.749+0000 7f86a2578640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f868006c8b0 0x7f868006ed70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8698009fd0 tx=0x7f869800d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:47.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.749+0000 7f8692ffd640 10 client.6123.objecter ms_handle_connect 0x7f868006c8b0 2024-01-20T11:23:47.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.766+0000 7f8692ffd640 1 -- 172.21.15.110:0/1003234748 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f868c07a0e0 con 0x7f869c0a9ad0 2024-01-20T11:23:47.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.766+0000 7f8692ffd640 10 client.6123.objecter ms_dispatch 0x7f869c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:47.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.766+0000 7f8692ffd640 3 client.6123.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:47.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.766+0000 7f8692ffd640 3 client.6123.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:47.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.766+0000 7f8692ffd640 20 client.6123.objecter dump_active .. 0 homeless 2024-01-20T11:23:47.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.766+0000 7f8692ffd640 1 -- 172.21.15.110:0/1003234748 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f868c076420 con 0x7f869c0a9ad0 2024-01-20T11:23:47.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.766+0000 7f8692ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:47.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.766+0000 7f8692ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:47.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.931+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f869c0b4400 con 0x7f868006c8b0 2024-01-20T11:23:47.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.935+0000 7f8692ffd640 1 -- 172.21.15.110:0/1003234748 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63929 (secure 0 0 0) 0x7f8698013340 con 0x7f868006c8b0 2024-01-20T11:23:47.935 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:47.935 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:47.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 10 librados: watch_flush enter 2024-01-20T11:23:47.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 10 librados: watch_flush exit 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 20 client.6123.objecter shutdown clearing up homeless session... 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 10 client.6123.objecter successfully canceled tick 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f868006c8b0 msgr2=0x7f868006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f868006c8b0 0x7f868006ed70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8698009fd0 tx=0x7f869800d040 comp rx=0 tx=0).stop 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 10 monclient: shutdown 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 msgr2=0x7f869c1d76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 0x7f869c1d76d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f868c0096f0 tx=0x7f868c0046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 shutdown_connections 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f868006c8b0 0x7f868006ed70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c142920 0x7f869c1d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c0a9ad0 0x7f869c1d76d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 --2- 172.21.15.110:0/1003234748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f869c0a96f0 0x7f869c0b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:47.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 >> 172.21.15.110:0/1003234748 conn(0x7f869c0b7bc0 msgr2=0x7f869c13c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:47.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 shutdown_connections 2024-01-20T11:23:47.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 -- 172.21.15.110:0/1003234748 wait complete. 2024-01-20T11:23:47.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:47.937+0000 7f86a47dc640 1 librados: shutdown 2024-01-20T11:23:47.947 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-01-20T11:23:46.816344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002881"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":8.9290000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":10.532},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":11.890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.052},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":15.555999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.692},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":7.4950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.335999999999999},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":17.390999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":9.7929999999999993},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":14.356999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":5.1529999999999996},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":7.0019999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":9.0730000000000004},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.6940000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":10.349},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":8.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":7.8120000000000003},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":12.138}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.795999999999999},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":12.811}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":14},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":15.837}]}]},{"osd":3,"up_from":21,"seq":90194313323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.3890000000000002},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":9.8109999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":12.26},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":14.395}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":8.7200000000000006},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":11.548999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":16.527000000000001},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":10.835000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":13.380000000000001},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":18.132000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":17.324999999999999},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":15.308999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444137,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.6479999999999997},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":11.099}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":7.3860000000000001},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":13.653}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.449},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":9.0380000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.913,"15min":16.091999999999999},"min":{"1min":9.5099999999999998,"5min":7.8200000000000003,"15min":7.3869999999999996},"max":{"1min":17.895,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.407},{"interface":"front","average":{"1min":12.028,"5min":12.231999999999999,"15min":13.638},"min":{"1min":8.1460000000000008,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.597999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":6.8090000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":10.394},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":11.728999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":8.4049999999999994},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":15.295999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":5.8250000000000002},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":10.055}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":8.7040000000000006},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":13.058999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":7.7480000000000002},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":13.808999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:23:40 2024","interfaces":[{"interface":"back","average":{"1min":9.6929999999999996,"5min":11.172000000000001,"15min":13},"min":{"1min":6.7290000000000001,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":14.914999999999999,"5min":18.899999999999999,"15min":25.541},"last":6.7290000000000001},{"interface":"front","average":{"1min":12.557,"5min":13.332000000000001,"15min":14.596},"min":{"1min":7.0490000000000004,"5min":6.75,"15min":6.75},"max":{"1min":16.565000000000001,"5min":22.280000000000001,"15min":30.204999999999998},"last":9.3030000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":11.58},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":14.577999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":12.44},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":15.435}]}]},{"osd":0,"up_from":17,"seq":73014444138,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":14.321},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":12.003}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.2060000000000004},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":8.4450000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":10.470000000000001},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":5.5839999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":6.4880000000000004},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":9.7289999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":15.117000000000001},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":12.75}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":13.269},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.083}]}]},{"osd":6,"up_from":15,"seq":64424509547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":16.888000000000002},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":8.2409999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.960000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":7.1559999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":9.2010000000000005},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":10.217000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":12.141},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":6.4610000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:23:41 2024","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":13.513999999999999,"15min":15.654},"min":{"1min":8.3879999999999999,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":21.539000000000001,"5min":21.539000000000001,"15min":70.254999999999995},"last":11.077999999999999},{"interface":"front","average":{"1min":12.259,"5min":12.333,"15min":15.17},"min":{"1min":7.3079999999999998,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":17.798999999999999,"5min":18.061,"15min":72.670000000000002},"last":15.42}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":14.750999999999999},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":14.093999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:47.949 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.056+0000 7fc257547640 10 monclient: get_monmap_and_config 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.056+0000 7fc257547640 10 monclient: build_initial_monmap 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.056+0000 7fc257547640 10 monclient: monmap: 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:48.056855+0000 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:48.056855+0000 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:48.056 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:48.057 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:48.057 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:48.057 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.056+0000 7fc257547640 1 Processor -- start 2024-01-20T11:23:48.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 1 -- start start 2024-01-20T11:23:48.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient: init 2024-01-20T11:23:48.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc2500b4d10 0x7fc2500b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient(hunting): picked mon.noname-c con 0x7fc2500b4d10 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc2500b17c0 0x7fc2500b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient(hunting): picked mon.noname-a con 0x7fc2500b17c0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc250145980 0x7fc250145e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient(hunting): picked mon.noname-b con 0x7fc250145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2500b50f0 con 0x7fc2500b17c0 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2500b2150 con 0x7fc2500b4d10 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500aade0 con 0x7fc250145980 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:48.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.057+0000 7fc257547640 10 monclient(hunting): authenticate will time out at 1871.407871s 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc254ae2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc2500b17c0 0x7fc2500b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc255ae4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc250145980 0x7fc250145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc2552e3640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7fc2500b4d10 0x7fc2500b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc254ae2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc2500b17c0 0x7fc2500b1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43712/0 (socket says 172.21.15.110:43712) 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc254ae2640 1 -- 172.21.15.110:0/1030568117 learned_addr learned my addr 172.21.15.110:0/1030568117 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc254ae2640 10 monclient(hunting): get_auth_request con 0x7fc2500b17c0 auth_method 0 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc254ae2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc254ae2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc254ae2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc2552e3640 10 monclient(hunting): get_auth_request con 0x7fc2500b4d10 auth_method 0 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc2552e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc2552e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc2552e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc255ae4640 10 monclient(hunting): get_auth_request con 0x7fc250145980 auth_method 0 2024-01-20T11:23:48.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc255ae4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc255ae4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.058+0000 7fc255ae4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.059+0000 7fc254ae2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.059+0000 7fc254ae2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.059+0000 7fc254ae2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.059+0000 7fc2552e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.059+0000 7fc2552e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.059+0000 7fc2552e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.059+0000 7fc255ae4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.059+0000 7fc255ae4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.059+0000 7fc255ae4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 10 monclient(hunting): handle_auth_done global_id 6126 payload 274 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 1 -- 172.21.15.110:0/1030568117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc250145980 msgr2=0x7fc250145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 1 --2- 172.21.15.110:0/1030568117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc250145980 0x7fc250145e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 1 -- 172.21.15.110:0/1030568117 >> v2:172.21.15.134:3300/0 conn(0x7fc2500b4d10 msgr2=0x7fc2500b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 1 --2- 172.21.15.110:0/1030568117 >> v2:172.21.15.134:3300/0 conn(0x7fc2500b4d10 0x7fc2500b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 10 monclient: _finish_hunting 0 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 1 monclient: found mon.noname-a 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:48.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.060+0000 7fc254ae2640 1 -- 172.21.15.110:0/1030568117 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc250145460 con 0x7fc2500b17c0 2024-01-20T11:23:48.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.062+0000 7fc254ae2640 1 --2- 172.21.15.110:0/1030568117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2500b17c0 0x7fc2500b1c10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc240009870 tx=0x7fc24002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a193a252c0bfd646 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 1 -- 172.21.15.110:0/1030568117 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2400317c0 con 0x7fc2500b17c0 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 10 monclient: dump: 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.066 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 10 monclient: _finish_auth 0 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 1 -- 172.21.15.110:0/1030568117 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc240031920 con 0x7fc2500b17c0 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc257547640 5 monclient: authenticate success, global_id 6126 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc257547640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 1 -- 172.21.15.110:0/1030568117 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240031cb0 con 0x7fc2500b17c0 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc257547640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc257547640 4 set_mon_vals no callback set 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:48.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc247fff640 10 monclient: dump: 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:48.068 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc257547640 10 monclient: shutdown 2024-01-20T11:23:48.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.066+0000 7fc257547640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:48.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 1 -- 172.21.15.110:0/1030568117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2500b17c0 msgr2=0x7fc2500b1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:48.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 1 --2- 172.21.15.110:0/1030568117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2500b17c0 0x7fc2500b1c10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc240009870 tx=0x7fc24002b860 comp rx=0 tx=0).stop 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 1 -- 172.21.15.110:0/1030568117 shutdown_connections 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 1 --2- 172.21.15.110:0/1030568117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc250145980 0x7fc250145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 1 --2- 172.21.15.110:0/1030568117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2500b17c0 0x7fc2500b1c10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 1 --2- 172.21.15.110:0/1030568117 >> v2:172.21.15.134:3300/0 conn(0x7fc2500b4d10 0x7fc2500b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 1 -- 172.21.15.110:0/1030568117 >> 172.21.15.110:0/1030568117 conn(0x7fc250066590 msgr2=0x7fc2500669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 1 -- 172.21.15.110:0/1030568117 shutdown_connections 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 1 -- 172.21.15.110:0/1030568117 wait complete. 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 10 monclient: build_initial_monmap 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.067+0000 7fc257547640 10 monclient: monmap: 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:48.072 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 librados: starting msgr at 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 librados: starting objecter 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 Processor -- start 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 -- start start 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 librados: setting wanted keys 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 librados: calling monclient init 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient: init 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2500b17c0 0x7fc250141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient(hunting): picked mon.noname-b con 0x7fc2500b17c0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2500b4d10 0x7fc2500abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient(hunting): picked mon.noname-c con 0x7fc2500b4d10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250145980 0x7fc250137fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient(hunting): picked mon.noname-a con 0x7fc250145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2501451b0 con 0x7fc250145980 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500ac250 con 0x7fc2500b17c0 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500ab780 con 0x7fc2500b4d10 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.068+0000 7fc257547640 10 monclient(hunting): authenticate will time out at 1871.419044s 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc255ae4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250145980 0x7fc250137fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc254ae2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2500b4d10 0x7fc2500abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc2552e3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2500b17c0 0x7fc250141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc255ae4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250145980 0x7fc250137fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43722/0 (socket says 172.21.15.110:43722) 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc255ae4640 1 -- 172.21.15.110:0/3140797404 learned_addr learned my addr 172.21.15.110:0/3140797404 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc255ae4640 10 monclient(hunting): get_auth_request con 0x7fc250145980 auth_method 0 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc255ae4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc255ae4640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc255ae4640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc254ae2640 10 monclient(hunting): get_auth_request con 0x7fc2500b4d10 auth_method 0 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc254ae2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc254ae2640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc254ae2640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc2552e3640 10 monclient(hunting): get_auth_request con 0x7fc2500b17c0 auth_method 0 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc2552e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc2552e3640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.069+0000 7fc2552e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.070+0000 7fc255ae4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.070+0000 7fc255ae4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.070+0000 7fc255ae4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.070+0000 7fc2552e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.070+0000 7fc2552e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.070+0000 7fc2552e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.070+0000 7fc254ae2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.070+0000 7fc254ae2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.070+0000 7fc254ae2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 10 monclient(hunting): handle_auth_done global_id 6129 payload 824 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 1 -- 172.21.15.110:0/3140797404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2500b4d10 msgr2=0x7fc2500abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2500b4d10 0x7fc2500abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 1 -- 172.21.15.110:0/3140797404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2500b17c0 msgr2=0x7fc250141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2500b17c0 0x7fc250141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 10 monclient: _finish_hunting 0 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 1 monclient: found mon.noname-a 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.072+0000 7fc255ae4640 1 -- 172.21.15.110:0/3140797404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2400094d0 con 0x7fc250145980 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.075+0000 7fc255ae4640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250145980 0x7fc250137fb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc24800efc0 tx=0x7fc24800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.075+0000 7fc245ffb640 10 client.?.objecter ms_handle_connect 0x7fc250145980 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.075+0000 7fc245ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:48.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc245ffb640 1 -- 172.21.15.110:0/3140797404 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc248009b10 con 0x7fc250145980 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc245ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc245ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc245ffb640 10 monclient: dump: 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc245ffb640 10 monclient: _finish_auth 0 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc245ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:23:48.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc245ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc257547640 5 monclient: authenticate success, global_id 6129 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc257547640 10 monclient: _renew_subs 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc257547640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2500ab9a0 con 0x7fc250145980 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc257547640 10 client.6129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc257547640 10 monclient: _renew_subs 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc257547640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2501386d0 con 0x7fc250145980 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.077+0000 7fc257547640 1 librados: init done 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc257547640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc257547640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc257547640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:48.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc250075490 con 0x7fc250145980 2024-01-20T11:23:48.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc245ffb640 1 -- 172.21.15.110:0/3140797404 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24800f040 con 0x7fc250145980 2024-01-20T11:23:48.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc245ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:48.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc245ffb640 1 -- 172.21.15.110:0/3140797404 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc248014680 con 0x7fc250145980 2024-01-20T11:23:48.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc245ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:48.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc245ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:48.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc245ffb640 10 monclient: dump: 2024-01-20T11:23:48.081 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:48.081 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.081 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.082 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.082 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:48.082 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.082 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:48.082 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:48.082 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:48.082 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.078+0000 7fc2477fe640 4 set_mon_vals no callback set 2024-01-20T11:23:48.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.320+0000 7fc245ffb640 1 -- 172.21.15.110:0/3140797404 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc2480147e0 con 0x7fc250145980 2024-01-20T11:23:48.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.320+0000 7fc245ffb640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc23406c860 0x7fc23406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.321+0000 7fc2552e3640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc23406c860 0x7fc23406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.321+0000 7fc2552e3640 10 monclient: get_auth_request con 0x7fc23406c860 auth_method 0 2024-01-20T11:23:48.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.321+0000 7fc2552e3640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc23406c860 0x7fc23406ed20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009b40 tx=0x7fc24c009190 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:48.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.321+0000 7fc245ffb640 10 client.6129.objecter ms_handle_connect 0x7fc23406c860 2024-01-20T11:23:48.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.334+0000 7fc245ffb640 1 -- 172.21.15.110:0/3140797404 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc248059310 con 0x7fc250145980 2024-01-20T11:23:48.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.334+0000 7fc245ffb640 10 client.6129.objecter ms_dispatch 0x7fc250000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:48.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.334+0000 7fc245ffb640 3 client.6129.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:48.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.334+0000 7fc245ffb640 3 client.6129.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:48.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.334+0000 7fc245ffb640 20 client.6129.objecter dump_active .. 0 homeless 2024-01-20T11:23:48.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.334+0000 7fc245ffb640 1 -- 172.21.15.110:0/3140797404 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc248055650 con 0x7fc250145980 2024-01-20T11:23:48.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.334+0000 7fc245ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:48.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.334+0000 7fc245ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:48.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.517+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2500b1ba0 con 0x7fc23406c860 2024-01-20T11:23:48.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.520+0000 7fc245ffb640 1 -- 172.21.15.110:0/3140797404 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63929 (secure 0 0 0) 0x7fc2500b1ba0 con 0x7fc23406c860 2024-01-20T11:23:48.520 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:48.521 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 10 librados: watch_flush enter 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 10 librados: watch_flush exit 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 20 client.6129.objecter shutdown clearing up homeless session... 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 10 client.6129.objecter successfully canceled tick 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc23406c860 msgr2=0x7fc23406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc23406c860 0x7fc23406ed20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009b40 tx=0x7fc24c009190 comp rx=0 tx=0).stop 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 10 monclient: shutdown 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250145980 msgr2=0x7fc250137fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250145980 0x7fc250137fb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc24800efc0 tx=0x7fc24800c490 comp rx=0 tx=0).stop 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 shutdown_connections 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc23406c860 0x7fc23406ed20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc250145980 0x7fc250137fb0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2500b4d10 0x7fc2500abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 --2- 172.21.15.110:0/3140797404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2500b17c0 0x7fc250141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 >> 172.21.15.110:0/3140797404 conn(0x7fc250066590 msgr2=0x7fc2500b83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:48.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 shutdown_connections 2024-01-20T11:23:48.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 -- 172.21.15.110:0/3140797404 wait complete. 2024-01-20T11:23:48.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.522+0000 7fc257547640 1 librados: shutdown 2024-01-20T11:23:48.534 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-01-20T11:23:46.816344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002881"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":8.9290000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":10.532},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":11.890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.052},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":15.555999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.692},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":7.4950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.335999999999999},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":17.390999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":9.7929999999999993},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":14.356999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":5.1529999999999996},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":7.0019999999999998}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":9.0730000000000004},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":9.6940000000000008}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":10.349},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":8.4489999999999998}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":7.8120000000000003},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":12.138}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.795999999999999},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":12.811}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":14},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":15.837}]}]},{"osd":3,"up_from":21,"seq":90194313323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.3890000000000002},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":9.8109999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":12.26},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":14.395}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":8.7200000000000006},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":11.548999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":16.527000000000001},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":10.835000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":13.380000000000001},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":18.132000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":17.324999999999999},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":15.308999999999999}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444137,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.6479999999999997},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":11.099}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":7.3860000000000001},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":13.653}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":12.449},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":9.0380000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.913,"15min":16.091999999999999},"min":{"1min":9.5099999999999998,"5min":7.8200000000000003,"15min":7.3869999999999996},"max":{"1min":17.895,"5min":20.393000000000001,"15min":51.000999999999998},"last":14.407},{"interface":"front","average":{"1min":12.028,"5min":12.231999999999999,"15min":13.638},"min":{"1min":8.1460000000000008,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.597999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":6.8090000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":10.394},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":11.728999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":8.4049999999999994},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":15.295999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":5.8250000000000002},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":10.055}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":8.7040000000000006},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":13.058999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":7.7480000000000002},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":13.808999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:23:40 2024","interfaces":[{"interface":"back","average":{"1min":9.6929999999999996,"5min":11.172000000000001,"15min":13},"min":{"1min":6.7290000000000001,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":14.914999999999999,"5min":18.899999999999999,"15min":25.541},"last":6.7290000000000001},{"interface":"front","average":{"1min":12.557,"5min":13.332000000000001,"15min":14.596},"min":{"1min":7.0490000000000004,"5min":6.75,"15min":6.75},"max":{"1min":16.565000000000001,"5min":22.280000000000001,"15min":30.204999999999998},"last":9.3030000000000008}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":11.58},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":14.577999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":12.44},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":15.435}]}]},{"osd":0,"up_from":17,"seq":73014444138,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":14.321},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":12.003}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.2060000000000004},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":8.4450000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.723000000000001,"5min":11.949,"15min":22.827000000000002},"min":{"1min":8.4269999999999996,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":27.292000000000002,"5min":27.292000000000002,"15min":358.738},"last":10.470000000000001},{"interface":"front","average":{"1min":10.308,"5min":10.736000000000001,"15min":12.256},"min":{"1min":5.3179999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":20.672000000000001,"5min":20.672000000000001,"15min":21.004999999999999},"last":5.5839999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":12.583,"15min":24.561},"min":{"1min":7.2709999999999999,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":23.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":6.4880000000000004},{"interface":"front","average":{"1min":11.798999999999999,"5min":10.345000000000001,"15min":21.959},"min":{"1min":6.2430000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":21.334,"5min":21.334,"15min":372.70800000000003},"last":9.7289999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":15.117000000000001},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":12.75}]},{"osd":7,"last update":"Sat Jan 20 11:22:39 2024","interfaces":[{"interface":"back","average":{"1min":14.972,"5min":14.936999999999999,"15min":16.509},"min":{"1min":8.625,"5min":8.625,"15min":8.625},"max":{"1min":25.673999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":13.269},{"interface":"front","average":{"1min":13.026,"5min":11.829000000000001,"15min":12.269},"min":{"1min":7.9909999999999997,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":24.547000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.083}]}]},{"osd":6,"up_from":15,"seq":64424509547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":16.888000000000002},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":8.2409999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.960000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":7.1559999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":9.2010000000000005},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":10.217000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":12.141},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":6.4610000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:23:41 2024","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":13.513999999999999,"15min":15.654},"min":{"1min":8.3879999999999999,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":21.539000000000001,"5min":21.539000000000001,"15min":70.254999999999995},"last":11.077999999999999},{"interface":"front","average":{"1min":12.259,"5min":12.333,"15min":15.17},"min":{"1min":7.3079999999999998,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":17.798999999999999,"5min":18.061,"15min":72.670000000000002},"last":15.42}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":14.750999999999999},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":14.093999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:48.535 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:48.535 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:48.535 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:48.535 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:19:41.675224+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:19:41.675224+0000', 'last_peered': '2024-01-20T11:19:41.675224+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:19:41.675224+0000', 'last_undegraded': '2024-01-20T11:19:41.675224+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:48.536 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:48.536 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 16, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:31.815524+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:21:31.815524+0000', 'last_peered': '2024-01-20T11:21:31.815524+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:21:31.815524+0000', 'last_undegraded': '2024-01-20T11:21:31.815524+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:48.536 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:48.536 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 131, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:21:18.077638+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:21:18.077638+0000', 'last_peered': '2024-01-20T11:21:18.077638+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:21:18.077638+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:48.536 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:48.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.637+0000 7f63a3a7c640 10 monclient: get_monmap_and_config 2024-01-20T11:23:48.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.637+0000 7f63a3a7c640 10 monclient: build_initial_monmap 2024-01-20T11:23:48.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.637+0000 7f63a3a7c640 10 monclient: monmap: 2024-01-20T11:23:48.637 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:48.637 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.637 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:48.638102+0000 2024-01-20T11:23:48.637 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:48.638102+0000 2024-01-20T11:23:48.637 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:48.638 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.638 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:48.638 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:48.638 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:48.638 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:48.638 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:48.638 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 1 Processor -- start 2024-01-20T11:23:48.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 1 -- start start 2024-01-20T11:23:48.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient: init 2024-01-20T11:23:48.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:48.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-01-20T11:23:48.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c06bb00 0x7f639c0b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient(hunting): picked mon.noname-b con 0x7f639c06bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f639c0af790 0x7f639c145770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient(hunting): picked mon.noname-c con 0x7f639c0af790 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f639c0b0900 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f639c06c220 con 0x7f639c0b0900 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f639c06bee0 con 0x7f639c0af790 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639c0ada30 con 0x7f639c06bb00 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.638+0000 7f63a3a7c640 10 monclient(hunting): authenticate will time out at 1871.989072s 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.639+0000 7f63a2019640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f639c0b0900 0x7f639c145cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56890/0 (socket says 172.21.15.110:56890) 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.639+0000 7f63a2019640 1 -- 172.21.15.110:0/1875927310 learned_addr learned my addr 172.21.15.110:0/1875927310 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.639+0000 7f63a1818640 1 --2- 172.21.15.110:0/1875927310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c06bb00 0x7f639c0b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.639+0000 7f63a1017640 1 --2- 172.21.15.110:0/1875927310 >> v2:172.21.15.134:3300/0 conn(0x7f639c0af790 0x7f639c145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.639+0000 7f63a1818640 10 monclient(hunting): get_auth_request con 0x7f639c06bb00 auth_method 0 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.639+0000 7f63a1818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.639+0000 7f63a1818640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.639+0000 7f63a1818640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1017640 10 monclient(hunting): get_auth_request con 0x7f639c0af790 auth_method 0 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1017640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1017640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1818640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1017640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.640+0000 7f63a1017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.641+0000 7f63a0816640 1 -- 172.21.15.110:0/1875927310 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592226003 0 0) 0x7f639c06c220 con 0x7f639c0b0900 2024-01-20T11:23:48.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.641+0000 7f63a0816640 10 monclient(hunting): _negotiate 2024-01-20T11:23:48.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.641+0000 7f63a0816640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.641+0000 7f63a0816640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.641+0000 7f63a0816640 10 monclient(hunting): my global_id is 6132 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.641+0000 7f63a0816640 1 -- 172.21.15.110:0/1875927310 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6388003520 con 0x7f639c0b0900 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.642+0000 7f63a1818640 10 monclient(hunting): handle_auth_done global_id 5759 payload 274 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.642+0000 7f63a1818640 1 -- 172.21.15.110:0/1875927310 >> v2:172.21.15.134:3300/0 conn(0x7f639c0af790 msgr2=0x7f639c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.642+0000 7f63a1818640 1 --2- 172.21.15.110:0/1875927310 >> v2:172.21.15.134:3300/0 conn(0x7f639c0af790 0x7f639c145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.642+0000 7f63a1818640 1 -- 172.21.15.110:0/1875927310 >> v1:172.21.15.110:6789/0 conn(0x7f639c0b0900 legacy=0x7f639c145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.642+0000 7f63a1818640 10 monclient: _finish_hunting 0 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.642+0000 7f63a1818640 1 monclient: found mon.noname-b 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.642+0000 7f63a1818640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.642+0000 7f63a1818640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:48.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.642+0000 7f63a1818640 1 -- 172.21.15.110:0/1875927310 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c0add10 con 0x7f639c06bb00 2024-01-20T11:23:48.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.644+0000 7f63a1818640 1 --2- 172.21.15.110:0/1875927310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c06bb00 0x7f639c0b03c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f63980097a0 tx=0x7f639802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b4203d1690b15bc server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:48.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.647+0000 7f63a0816640 1 -- 172.21.15.110:0/1875927310 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639802e070 con 0x7f639c06bb00 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.647+0000 7f63a0816640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.647+0000 7f63a0816640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.647+0000 7f63a0816640 10 monclient: dump: 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a0816640 10 monclient: _finish_auth 0 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a0816640 10 monclient: _check_auth_tickets 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a0816640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a0816640 1 -- 172.21.15.110:0/1875927310 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63980315a0 con 0x7f639c06bb00 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a0816640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a0816640 1 -- 172.21.15.110:0/1875927310 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398031930 con 0x7f639c06bb00 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 5 monclient: authenticate success, global_id 5759 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 4 set_mon_vals no callback set 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a0816640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a0816640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a0816640 10 monclient: dump: 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.649 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.650 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:48.650 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.650 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:48.650 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:48.650 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:48.650 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 10 monclient: shutdown 2024-01-20T11:23:48.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:48.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 1 -- 172.21.15.110:0/1875927310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c06bb00 msgr2=0x7f639c0b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:48.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 1 --2- 172.21.15.110:0/1875927310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c06bb00 0x7f639c0b03c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f63980097a0 tx=0x7f639802b840 comp rx=0 tx=0).stop 2024-01-20T11:23:48.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 1 -- 172.21.15.110:0/1875927310 shutdown_connections 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 1 --2- 172.21.15.110:0/1875927310 >> v2:172.21.15.134:3300/0 conn(0x7f639c0af790 0x7f639c145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 1 --2- 172.21.15.110:0/1875927310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c06bb00 0x7f639c0b03c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 1 -- 172.21.15.110:0/1875927310 >> 172.21.15.110:0/1875927310 conn(0x7f639c066d60 msgr2=0x7f639c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 1 -- 172.21.15.110:0/1875927310 shutdown_connections 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.648+0000 7f63a3a7c640 1 -- 172.21.15.110:0/1875927310 wait complete. 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.649+0000 7f63a3a7c640 10 monclient: build_initial_monmap 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.649+0000 7f63a3a7c640 10 monclient: monmap: 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:48.654 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 librados: starting msgr at 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 librados: starting objecter 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 Processor -- start 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 -- start start 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 librados: setting wanted keys 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 librados: calling monclient init 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient: init 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f639c06bb00 0x7f639c0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient(hunting): picked mon.noname-a con 0x7f639c06bb00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f639c0af790 0x7f639c11d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient(hunting): picked mon.noname-b con 0x7f639c0af790 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c0b0900 0x7f639c13bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient(hunting): picked mon.noname-c con 0x7f639c0b0900 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639c11de00 con 0x7f639c06bb00 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639c11dae0 con 0x7f639c0af790 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639c071300 con 0x7f639c0b0900 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.650+0000 7f63a3a7c640 10 monclient(hunting): authenticate will time out at 1872.000969s 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1818640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f639c06bb00 0x7f639c0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1017640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f639c0af790 0x7f639c11d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a2019640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c0b0900 0x7f639c13bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1818640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f639c06bb00 0x7f639c0b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43736/0 (socket says 172.21.15.110:43736) 2024-01-20T11:23:48.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1818640 1 -- 172.21.15.110:0/4243001055 learned_addr learned my addr 172.21.15.110:0/4243001055 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1818640 10 monclient(hunting): get_auth_request con 0x7f639c06bb00 auth_method 0 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1818640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1818640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1017640 10 monclient(hunting): get_auth_request con 0x7f639c0af790 auth_method 0 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1017640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a1017640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a2019640 10 monclient(hunting): get_auth_request con 0x7f639c0b0900 auth_method 0 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a2019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a2019640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.651+0000 7f63a2019640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.652+0000 7f63a1818640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.652+0000 7f63a1818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.652+0000 7f63a1818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.652+0000 7f63a1017640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.652+0000 7f63a1017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.652+0000 7f63a1017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.652+0000 7f63a2019640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:48.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.652+0000 7f63a2019640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:48.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.652+0000 7f63a2019640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 10 monclient(hunting): handle_auth_done global_id 6135 payload 824 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 1 -- 172.21.15.110:0/4243001055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c0b0900 msgr2=0x7f639c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c0b0900 0x7f639c13bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 1 -- 172.21.15.110:0/4243001055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f639c0af790 msgr2=0x7f639c11d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f639c0af790 0x7f639c11d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 10 monclient: _finish_hunting 0 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 1 monclient: found mon.noname-a 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.654+0000 7f63a1818640 1 -- 172.21.15.110:0/4243001055 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6398009400 con 0x7f639c06bb00 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.657+0000 7f63a1818640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f639c06bb00 0x7f639c0b4870 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6398034040 tx=0x7f63980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.657+0000 7f638e7fc640 10 client.?.objecter ms_handle_connect 0x7f639c06bb00 2024-01-20T11:23:48.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.657+0000 7f638e7fc640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 1 -- 172.21.15.110:0/4243001055 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398032630 con 0x7f639c06bb00 2024-01-20T11:23:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 10 monclient: dump: 2024-01-20T11:23:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:48.661 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 10 monclient: _finish_auth 0 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 10 monclient: _check_auth_tickets 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 1 -- 172.21.15.110:0/4243001055 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6398032790 con 0x7f639c06bb00 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f63a3a7c640 5 monclient: authenticate success, global_id 6135 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f63a3a7c640 10 monclient: _renew_subs 2024-01-20T11:23:48.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f63a3a7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639c071580 con 0x7f639c06bb00 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 1 -- 172.21.15.110:0/4243001055 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398032b40 con 0x7f639c06bb00 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638e7fc640 10 monclient: dump: 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:48.663 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f638ffff640 4 set_mon_vals no callback set 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f63a3a7c640 10 client.6135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f63a3a7c640 10 monclient: _renew_subs 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f63a3a7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639c13c430 con 0x7f639c06bb00 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.660+0000 7f63a3a7c640 1 librados: init done 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.661+0000 7f63a3a7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.661+0000 7f63a3a7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.661+0000 7f63a3a7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:48.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.661+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639c075490 con 0x7f639c06bb00 2024-01-20T11:23:48.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.682+0000 7f638e7fc640 1 -- 172.21.15.110:0/4243001055 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f639804d3f0 con 0x7f639c06bb00 2024-01-20T11:23:48.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.682+0000 7f638e7fc640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f638006c8b0 0x7f638006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:48.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.683+0000 7f63a1017640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f638006c8b0 0x7f638006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:48.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.683+0000 7f63a1017640 10 monclient: get_auth_request con 0x7f638006c8b0 auth_method 0 2024-01-20T11:23:48.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.683+0000 7f638e7fc640 10 client.6135.objecter ms_handle_connect 0x7f638006c8b0 2024-01-20T11:23:48.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.683+0000 7f63a1017640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f638006c8b0 0x7f638006ed70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f639c1db780 tx=0x7f6390009110 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:48.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.686+0000 7f638e7fc640 1 -- 172.21.15.110:0/4243001055 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f639807a020 con 0x7f639c06bb00 2024-01-20T11:23:48.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.686+0000 7f638e7fc640 10 client.6135.objecter ms_dispatch 0x7f639c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:48.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.686+0000 7f638e7fc640 3 client.6135.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:48.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.686+0000 7f638e7fc640 3 client.6135.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:48.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.686+0000 7f638e7fc640 20 client.6135.objecter dump_active .. 0 homeless 2024-01-20T11:23:48.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.914+0000 7f638e7fc640 1 -- 172.21.15.110:0/4243001055 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6398076360 con 0x7f639c06bb00 2024-01-20T11:23:48.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.914+0000 7f638e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:48.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:48.914+0000 7f638e7fc640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:49.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.163+0000 7f63a3a7c640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:49.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.163+0000 7f63a3a7c640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:49.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.163+0000 7f63a3a7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:49.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.163+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f639c143090 con 0x7f639c06bb00 2024-01-20T11:23:49.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.169+0000 7f638e7fc640 1 -- 172.21.15.110:0/4243001055 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7f6398079980 con 0x7f639c06bb00 2024-01-20T11:23:49.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.169+0000 7f638e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:49.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.169+0000 7f638e7fc640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:49.169 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:49.169 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":563,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:49.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.170+0000 7f63a3a7c640 10 librados: watch_flush enter 2024-01-20T11:23:49.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.170+0000 7f63a3a7c640 10 librados: watch_flush exit 2024-01-20T11:23:49.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.170+0000 7f63a3a7c640 20 client.6135.objecter shutdown clearing up homeless session... 2024-01-20T11:23:49.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.170+0000 7f63a3a7c640 10 client.6135.objecter successfully canceled tick 2024-01-20T11:23:49.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.170+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f638006c8b0 msgr2=0x7f638006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:49.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.170+0000 7f63a3a7c640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f638006c8b0 0x7f638006ed70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f639c1db780 tx=0x7f6390009110 comp rx=0 tx=0).stop 2024-01-20T11:23:49.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 10 monclient: shutdown 2024-01-20T11:23:49.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f639c06bb00 msgr2=0x7f639c0b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f639c06bb00 0x7f639c0b4870 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6398034040 tx=0x7f63980046c0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 shutdown_connections 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f638006c8b0 0x7f638006ed70 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f639c0b0900 0x7f639c13bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f639c0af790 0x7f639c11d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 --2- 172.21.15.110:0/4243001055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f639c06bb00 0x7f639c0b4870 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 >> 172.21.15.110:0/4243001055 conn(0x7f639c066d60 msgr2=0x7f639c144ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 shutdown_connections 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 -- 172.21.15.110:0/4243001055 wait complete. 2024-01-20T11:23:49.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.171+0000 7f63a3a7c640 1 librados: shutdown 2024-01-20T11:23:49.182 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:49.182 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:49.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.281+0000 7facb6aa9640 10 monclient: get_monmap_and_config 2024-01-20T11:23:49.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.281+0000 7facb6aa9640 10 monclient: build_initial_monmap 2024-01-20T11:23:49.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.281+0000 7facb6aa9640 10 monclient: monmap: 2024-01-20T11:23:49.282 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:49.282 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:49.282 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:49.281852+0000 2024-01-20T11:23:49.282 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:49.281852+0000 2024-01-20T11:23:49.282 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:49.282 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:49.282 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:49.282 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:49.283 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:49.283 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:49.283 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:49.283 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:49.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.281+0000 7facb6aa9640 1 Processor -- start 2024-01-20T11:23:49.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 1 -- start start 2024-01-20T11:23:49.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient: init 2024-01-20T11:23:49.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:49.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-01-20T11:23:49.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facb00a96f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:49.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7facb00a9ad0 0x7facb01423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:49.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient(hunting): picked mon.noname-a con 0x7facb00a9ad0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:49.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb0142920 0x7facb014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:49.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient(hunting): picked mon.noname-b con 0x7facb0142920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:49.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb00b25c0 con 0x7facb00a96f0 2024-01-20T11:23:49.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb00aaea0 con 0x7facb00a9ad0 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb013fbe0 con 0x7facb0142920 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.282+0000 7facb6aa9640 10 monclient(hunting): authenticate will time out at 1872.632820s 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7faca7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7facb00a9ad0 0x7facb01423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7facb5046640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb0142920 0x7facb014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7facb4845640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7facb00a96f0 0x7facb013f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:51056/0 (socket says 172.21.15.110:51056) 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7facb4845640 1 -- 172.21.15.110:0/2991491174 learned_addr learned my addr 172.21.15.110:0/2991491174 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7faca7fff640 10 monclient(hunting): get_auth_request con 0x7facb00a9ad0 auth_method 0 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7faca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7faca7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7faca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7facb5046640 10 monclient(hunting): get_auth_request con 0x7facb0142920 auth_method 0 2024-01-20T11:23:49.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7facb5046640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7facb5046640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.283+0000 7facb5046640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.284+0000 7faca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.284+0000 7faca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.284+0000 7faca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.284+0000 7facb5046640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.284+0000 7facb5046640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.284+0000 7facb5046640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca77fe640 1 -- 172.21.15.110:0/2991491174 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554400806 0 0) 0x7facb00b25c0 con 0x7facb00a96f0 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca77fe640 10 monclient(hunting): _negotiate 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca77fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca77fe640 10 monclient(hunting): my global_id is 6247 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca77fe640 1 -- 172.21.15.110:0/2991491174 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac98003520 con 0x7facb00a96f0 2024-01-20T11:23:49.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca7fff640 10 monclient(hunting): handle_auth_done global_id 6138 payload 274 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca7fff640 1 -- 172.21.15.110:0/2991491174 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb0142920 msgr2=0x7facb014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca7fff640 1 --2- 172.21.15.110:0/2991491174 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb0142920 0x7facb014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca7fff640 1 -- 172.21.15.110:0/2991491174 >> v1:172.21.15.134:6789/0 conn(0x7facb00a96f0 legacy=0x7facb013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca7fff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca7fff640 1 monclient: found mon.noname-a 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.285+0000 7faca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.286+0000 7faca7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.286+0000 7faca7fff640 1 -- 172.21.15.110:0/2991491174 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb00ab0a0 con 0x7facb00a9ad0 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.286+0000 7facb5046640 1 --2- 172.21.15.110:0/2991491174 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb0142920 0x7facb014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:49.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.287+0000 7faca7fff640 1 --2- 172.21.15.110:0/2991491174 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb00a9ad0 0x7facb01423e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7faca00097a0 tx=0x7faca002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fd00792861bdcf server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 1 -- 172.21.15.110:0/2991491174 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca002e070 con 0x7facb00a9ad0 2024-01-20T11:23:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 10 monclient: dump: 2024-01-20T11:23:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:49.295 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 1 -- 172.21.15.110:0/2991491174 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca00315a0 con 0x7facb00a9ad0 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 5 monclient: authenticate success, global_id 6138 2024-01-20T11:23:49.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 1 -- 172.21.15.110:0/2991491174 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca0031930 con 0x7facb00a9ad0 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 4 set_mon_vals no callback set 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7faca77fe640 10 monclient: dump: 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:49.297 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 10 monclient: shutdown 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 1 -- 172.21.15.110:0/2991491174 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb00a9ad0 msgr2=0x7facb01423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 1 --2- 172.21.15.110:0/2991491174 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb00a9ad0 0x7facb01423e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7faca00097a0 tx=0x7faca002b840 comp rx=0 tx=0).stop 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 1 -- 172.21.15.110:0/2991491174 shutdown_connections 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 1 --2- 172.21.15.110:0/2991491174 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb0142920 0x7facb014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 1 --2- 172.21.15.110:0/2991491174 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb00a9ad0 0x7facb01423e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.291+0000 7facb6aa9640 1 -- 172.21.15.110:0/2991491174 >> 172.21.15.110:0/2991491174 conn(0x7facb00b7bc0 msgr2=0x7facb00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.292+0000 7facb6aa9640 1 -- 172.21.15.110:0/2991491174 shutdown_connections 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.292+0000 7facb6aa9640 1 -- 172.21.15.110:0/2991491174 wait complete. 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.292+0000 7facb6aa9640 10 monclient: build_initial_monmap 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.292+0000 7facb6aa9640 10 monclient: monmap: 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.292+0000 7facb6aa9640 1 librados: starting msgr at 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 librados: starting objecter 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 Processor -- start 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 -- start start 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 librados: setting wanted keys 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 librados: calling monclient init 2024-01-20T11:23:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient: init 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb00a96f0 0x7facb01c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient(hunting): picked mon.noname-c con 0x7facb00a96f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00a9ad0 0x7facb01c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient(hunting): picked mon.noname-b con 0x7facb00a9ad0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb0142920 0x7facb00b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient(hunting): picked mon.noname-a con 0x7facb0142920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb00abdd0 con 0x7facb0142920 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb01c9890 con 0x7facb00a9ad0 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:49.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb01c9d20 con 0x7facb00a96f0 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.293+0000 7facb6aa9640 10 monclient(hunting): authenticate will time out at 1872.644080s 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb5046640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb0142920 0x7facb00b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb4845640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb00a96f0 0x7facb01c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb5046640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb0142920 0x7facb00b0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43744/0 (socket says 172.21.15.110:43744) 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb5046640 1 -- 172.21.15.110:0/2377376584 learned_addr learned my addr 172.21.15.110:0/2377376584 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb5046640 10 monclient(hunting): get_auth_request con 0x7facb0142920 auth_method 0 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb5046640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb5046640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb5046640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7faca7fff640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00a9ad0 0x7facb01c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb4845640 10 monclient(hunting): get_auth_request con 0x7facb00a96f0 auth_method 0 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb4845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:49.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb4845640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7facb4845640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7faca7fff640 10 monclient(hunting): get_auth_request con 0x7facb00a9ad0 auth_method 0 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7faca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7faca7fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.294+0000 7faca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.295+0000 7facb5046640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.295+0000 7facb5046640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.295+0000 7facb5046640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.295+0000 7facb4845640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.295+0000 7facb4845640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.295+0000 7facb4845640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.295+0000 7faca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.295+0000 7faca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:49.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.295+0000 7faca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:49.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 10 monclient(hunting): handle_auth_done global_id 6141 payload 824 2024-01-20T11:23:49.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 1 -- 172.21.15.110:0/2377376584 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb00a96f0 msgr2=0x7facb01c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:49.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb00a96f0 0x7facb01c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 1 -- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00a9ad0 msgr2=0x7facb01c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00a9ad0 0x7facb01c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 10 monclient: _finish_hunting 0 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 1 monclient: found mon.noname-a 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.297+0000 7facb5046640 1 -- 172.21.15.110:0/2377376584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8009400 con 0x7facb0142920 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.300+0000 7facb5046640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb0142920 0x7facb00b0760 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7faca800e880 tx=0x7faca800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.300+0000 7faca57fa640 10 client.?.objecter ms_handle_connect 0x7facb0142920 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.300+0000 7faca57fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 1 -- 172.21.15.110:0/2377376584 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca8009660 con 0x7facb0142920 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:49.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 10 monclient: dump: 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 10 monclient: _finish_auth 0 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 10 monclient: _check_auth_tickets 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 1 -- 172.21.15.110:0/2377376584 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca8004020 con 0x7facb0142920 2024-01-20T11:23:49.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 1 -- 172.21.15.110:0/2377376584 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca80043b0 con 0x7facb0142920 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca57fa640 10 monclient: dump: 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:49.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7facb6aa9640 5 monclient: authenticate success, global_id 6141 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7facb6aa9640 10 monclient: _renew_subs 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7facb6aa9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca0009400 con 0x7facb0142920 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7faca6ffd640 4 set_mon_vals no callback set 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7facb6aa9640 10 client.6141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7facb6aa9640 10 monclient: _renew_subs 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7facb6aa9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb00b0ca0 con 0x7facb0142920 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.303+0000 7facb6aa9640 1 librados: init done 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.304+0000 7facb6aa9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.304+0000 7facb6aa9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.304+0000 7facb6aa9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:49.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.304+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb0075490 con 0x7facb0142920 2024-01-20T11:23:49.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.549+0000 7faca57fa640 1 -- 172.21.15.110:0/2377376584 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7faca800c8e0 con 0x7facb0142920 2024-01-20T11:23:49.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.549+0000 7faca57fa640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac9c06c860 0x7fac9c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:49.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.549+0000 7facb4845640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac9c06c860 0x7fac9c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:49.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.550+0000 7facb4845640 10 monclient: get_auth_request con 0x7fac9c06c860 auth_method 0 2024-01-20T11:23:49.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.550+0000 7facb4845640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac9c06c860 0x7fac9c06ed20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7facac005c00 tx=0x7facac009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:49.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.550+0000 7faca57fa640 10 client.6141.objecter ms_handle_connect 0x7fac9c06c860 2024-01-20T11:23:49.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.565+0000 7faca57fa640 1 -- 172.21.15.110:0/2377376584 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faca8058820 con 0x7facb0142920 2024-01-20T11:23:49.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.565+0000 7faca57fa640 10 client.6141.objecter ms_dispatch 0x7facb0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:49.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.565+0000 7faca57fa640 3 client.6141.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:49.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.565+0000 7faca57fa640 3 client.6141.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:49.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.565+0000 7faca57fa640 20 client.6141.objecter dump_active .. 0 homeless 2024-01-20T11:23:49.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.565+0000 7faca57fa640 1 -- 172.21.15.110:0/2377376584 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7faca8054b60 con 0x7facb0142920 2024-01-20T11:23:49.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.565+0000 7faca57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:49.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.565+0000 7faca57fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:49.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.744+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7facb00b8380 con 0x7fac9c06c860 2024-01-20T11:23:49.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.746+0000 7faca57fa640 1 -- 172.21.15.110:0/2377376584 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63991 (secure 0 0 0) 0x7facb00b8380 con 0x7fac9c06c860 2024-01-20T11:23:49.747 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:49.747 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 10 librados: watch_flush enter 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 10 librados: watch_flush exit 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 20 client.6141.objecter shutdown clearing up homeless session... 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 10 client.6141.objecter successfully canceled tick 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac9c06c860 msgr2=0x7fac9c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac9c06c860 0x7fac9c06ed20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7facac005c00 tx=0x7facac009040 comp rx=0 tx=0).stop 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 10 monclient: shutdown 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb0142920 msgr2=0x7facb00b0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb0142920 0x7facb00b0760 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7faca800e880 tx=0x7faca800ed40 comp rx=0 tx=0).stop 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 shutdown_connections 2024-01-20T11:23:49.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fac9c06c860 0x7fac9c06ed20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb0142920 0x7facb00b0760 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00a9ad0 0x7facb01c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 --2- 172.21.15.110:0/2377376584 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb00a96f0 0x7facb01c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:49.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 >> 172.21.15.110:0/2377376584 conn(0x7facb00b7bc0 msgr2=0x7facb013f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:49.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 shutdown_connections 2024-01-20T11:23:49.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.748+0000 7facb6aa9640 1 -- 172.21.15.110:0/2377376584 wait complete. 2024-01-20T11:23:49.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:49.749+0000 7facb6aa9640 1 librados: shutdown 2024-01-20T11:23:49.759 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-01-20T11:23:48.816707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002732"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746827+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:19:41.746827+0000","last_peered":"2024-01-20T11:19:41.746827+0000","last_clean":"2024-01-20T11:19:41.746827+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:19:41.746827+0000","last_undegraded":"2024-01-20T11:19:41.746827+0000","last_fullsized":"2024-01-20T11:19:41.746827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746327+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:19:41.746327+0000","last_peered":"2024-01-20T11:19:41.746327+0000","last_clean":"2024-01-20T11:19:41.746327+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:19:41.746327+0000","last_undegraded":"2024-01-20T11:19:41.746327+0000","last_fullsized":"2024-01-20T11:19:41.746327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:19:41.675224+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:19:41.675224+0000","last_peered":"2024-01-20T11:19:41.675224+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:19:41.675224+0000","last_undegraded":"2024-01-20T11:19:41.675224+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:41.746598+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:19:41.746598+0000","last_peered":"2024-01-20T11:19:41.746598+0000","last_clean":"2024-01-20T11:19:41.746598+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:19:41.746598+0000","last_undegraded":"2024-01-20T11:19:41.746598+0000","last_fullsized":"2024-01-20T11:19:41.746598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280621,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":8.9290000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":10.532},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":11.890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.052},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":15.555999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.692},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":7.4950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.335999999999999},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":17.390999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":9.7929999999999993},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":14.356999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313323,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":4.8849999999999998},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":10.643000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":5.7990000000000004},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":7.7949999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":13.513},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":12.332000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":9.0990000000000002},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":11.56}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":8.5139999999999993},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":14.215}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":16.401},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":15.209}]}]},{"osd":3,"up_from":21,"seq":90194313324,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":7.2930000000000001},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":4.5990000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":13.919},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":13.17}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":8.3379999999999992},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":12.797000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":15.055999999999999},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":11.095000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":11.741},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":16.675999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":15.919},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":9.4740000000000002}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.1909999999999998},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":7.8600000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":9.9039999999999999},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.2040000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":10.930999999999999},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":11.569000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.913,"15min":16.091999999999999},"min":{"1min":9.5099999999999998,"5min":7.8200000000000003,"15min":7.3869999999999996},"max":{"1min":17.895,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.897},{"interface":"front","average":{"1min":12.028,"5min":12.231999999999999,"15min":13.638},"min":{"1min":8.1460000000000008,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.597999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":12.207000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":13.525},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.1479999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":8.4909999999999997},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":14.227}]}]},{"osd":1,"up_from":17,"seq":73014444139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":8.6270000000000007},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":10.124000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":5.992},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":12.651999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":11.933},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":13.34}]},{"osd":4,"last update":"Sat Jan 20 11:23:40 2024","interfaces":[{"interface":"back","average":{"1min":9.6929999999999996,"5min":11.172000000000001,"15min":13},"min":{"1min":6.7290000000000001,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":14.914999999999999,"5min":18.899999999999999,"15min":25.541},"last":8.9350000000000005},{"interface":"front","average":{"1min":12.557,"5min":13.332000000000001,"15min":14.596},"min":{"1min":7.0490000000000004,"5min":6.75,"15min":6.75},"max":{"1min":16.565000000000001,"5min":22.280000000000001,"15min":30.204999999999998},"last":16.498000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":12.223000000000001},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":8.0299999999999994}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":14.65},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":15.286}]}]},{"osd":0,"up_from":17,"seq":73014444139,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":3.96},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":5.9640000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.2690000000000001},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.9690000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.776,"5min":12.148999999999999,"15min":21.678999999999998},"min":{"1min":7.8280000000000003,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":18.335999999999999,"5min":27.292000000000002,"15min":358.738},"last":9.2409999999999997},{"interface":"front","average":{"1min":9.9199999999999999,"5min":10.561999999999999,"15min":12.034000000000001},"min":{"1min":5.5839999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":15.347,"5min":20.672000000000001,"15min":21.004999999999999},"last":13.311999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":12.465,"15min":23.204000000000001},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":17.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":11.960000000000001},{"interface":"front","average":{"1min":11.621,"5min":10.532999999999999,"15min":20.757000000000001},"min":{"1min":7.0620000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":16.713000000000001,"5min":21.334,"15min":372.70800000000003},"last":10.694000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":9.9749999999999996},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":14.093999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":15.414,"5min":15.087,"15min":16.367999999999999},"min":{"1min":8.9819999999999993,"5min":8.625,"15min":8.625},"max":{"1min":22.588999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.977},{"interface":"front","average":{"1min":12.353,"5min":11.938000000000001,"15min":12.262},"min":{"1min":8.0250000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":19.954000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.305}]}]},{"osd":6,"up_from":15,"seq":64424509547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":16.888000000000002},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":8.2409999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":12.960000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":7.1559999999999997}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":9.2010000000000005},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":10.217000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":12.141},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":6.4610000000000003}]},{"osd":4,"last update":"Sat Jan 20 11:23:41 2024","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":13.513999999999999,"15min":15.654},"min":{"1min":8.3879999999999999,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":21.539000000000001,"5min":21.539000000000001,"15min":70.254999999999995},"last":11.077999999999999},{"interface":"front","average":{"1min":12.259,"5min":12.333,"15min":15.17},"min":{"1min":7.3079999999999998,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":17.798999999999999,"5min":18.061,"15min":72.670000000000002},"last":15.42}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":14.750999999999999},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":14.093999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:49.907 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~292s 2024-01-20T11:23:52.762 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:52.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.865+0000 7f30a6901640 10 monclient: get_monmap_and_config 2024-01-20T11:23:52.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.865+0000 7f30a6901640 10 monclient: build_initial_monmap 2024-01-20T11:23:52.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.865+0000 7f30a6901640 10 monclient: monmap: 2024-01-20T11:23:52.865 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:52.865 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:52.866243+0000 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:52.866243+0000 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:52.866 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:52.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.866+0000 7f30a6901640 1 Processor -- start 2024-01-20T11:23:52.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.866+0000 7f30a6901640 1 -- start start 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.866+0000 7f30a6901640 10 monclient: init 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f30a006bb00 0x7f30a00b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 10 monclient(hunting): picked mon.noname-a con 0x7f30a006bb00 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30a00af790 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f30a00b0900 0x7f30a0145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 10 monclient(hunting): picked mon.noname-c con 0x7f30a00b0900 addr v2:172.21.15.134:3300/0 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a006c220 con 0x7f30a00af790 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a006bee0 con 0x7f30a006bb00 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a00ada30 con 0x7f30a00b0900 2024-01-20T11:23:52.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f30a6901640 10 monclient(hunting): authenticate will time out at 1876.217277s 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f309ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f30a006bb00 0x7f30a00b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f309ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f30a006bb00 0x7f30a00b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54146/0 (socket says 172.21.15.110:54146) 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f309ffff640 1 -- 172.21.15.110:0/740792570 learned_addr learned my addr 172.21.15.110:0/740792570 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f309ffff640 10 monclient(hunting): get_auth_request con 0x7f30a006bb00 auth_method 0 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f309ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f309ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.867+0000 7f309ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.868+0000 7f30a4e9e640 1 --2- 172.21.15.110:0/740792570 >> v2:172.21.15.134:3300/0 conn(0x7f30a00b0900 0x7f30a0145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.868+0000 7f309ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.868+0000 7f309ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.868+0000 7f309ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.868+0000 7f30a4e9e640 10 monclient(hunting): get_auth_request con 0x7f30a00b0900 auth_method 0 2024-01-20T11:23:52.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.868+0000 7f30a4e9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:52.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.868+0000 7f30a4e9e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:52.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.868+0000 7f30a4e9e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.869+0000 7f30a4e9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.869+0000 7f30a4e9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.869+0000 7f30a4e9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.869+0000 7f309ffff640 10 monclient(hunting): handle_auth_done global_id 6144 payload 274 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.869+0000 7f309ffff640 1 -- 172.21.15.110:0/740792570 >> v2:172.21.15.134:3300/0 conn(0x7f30a00b0900 msgr2=0x7f30a0145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.869+0000 7f309ffff640 1 --2- 172.21.15.110:0/740792570 >> v2:172.21.15.134:3300/0 conn(0x7f30a00b0900 0x7f30a0145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.870+0000 7f309ffff640 1 -- 172.21.15.110:0/740792570 >> v1:172.21.15.134:6789/0 conn(0x7f30a00af790 legacy=0x7f30a0145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.870+0000 7f309ffff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.870+0000 7f309ffff640 1 monclient: found mon.noname-a 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.870+0000 7f309ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.870+0000 7f309ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:52.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.870+0000 7f309ffff640 1 -- 172.21.15.110:0/740792570 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a00add10 con 0x7f30a006bb00 2024-01-20T11:23:52.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.871+0000 7f309ffff640 1 --2- 172.21.15.110:0/740792570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a006bb00 0x7f30a00b03c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f30a00b1fb0 tx=0x7f309002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1813bb200e75d199 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 1 -- 172.21.15.110:0/740792570 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090031870 con 0x7f30a006bb00 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 10 monclient: dump: 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:52.876 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 10 monclient: _finish_auth 0 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 1 -- 172.21.15.110:0/740792570 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30900319d0 con 0x7f30a006bb00 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f30a6901640 5 monclient: authenticate success, global_id 6144 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f30a6901640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f30a6901640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f30a6901640 4 set_mon_vals no callback set 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 1 -- 172.21.15.110:0/740792570 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090031d60 con 0x7f30a006bb00 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.875+0000 7f309effd640 10 monclient: dump: 2024-01-20T11:23:52.877 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 10 monclient: shutdown 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 1 -- 172.21.15.110:0/740792570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a006bb00 msgr2=0x7f30a00b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 1 --2- 172.21.15.110:0/740792570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a006bb00 0x7f30a00b03c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f30a00b1fb0 tx=0x7f309002b910 comp rx=0 tx=0).stop 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 1 -- 172.21.15.110:0/740792570 shutdown_connections 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 1 --2- 172.21.15.110:0/740792570 >> v2:172.21.15.134:3300/0 conn(0x7f30a00b0900 0x7f30a0145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:52.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 1 --2- 172.21.15.110:0/740792570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a006bb00 0x7f30a00b03c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:52.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 1 -- 172.21.15.110:0/740792570 >> 172.21.15.110:0/740792570 conn(0x7f30a0066d60 msgr2=0x7f30a00b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:52.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 1 -- 172.21.15.110:0/740792570 shutdown_connections 2024-01-20T11:23:52.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 1 -- 172.21.15.110:0/740792570 wait complete. 2024-01-20T11:23:52.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 10 monclient: build_initial_monmap 2024-01-20T11:23:52.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.876+0000 7f30a6901640 10 monclient: monmap: 2024-01-20T11:23:52.881 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.877+0000 7f30a6901640 1 librados: starting msgr at 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.877+0000 7f30a6901640 1 librados: starting objecter 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.877+0000 7f30a6901640 1 Processor -- start 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.877+0000 7f30a6901640 1 -- start start 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.877+0000 7f30a6901640 1 librados: setting wanted keys 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.877+0000 7f30a6901640 1 librados: calling monclient init 2024-01-20T11:23:52.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.877+0000 7f30a6901640 10 monclient: init 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006bb00 0x7f30a013bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient(hunting): picked mon.noname-c con 0x7f30a006bb00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00af790 0x7f30a0070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient(hunting): picked mon.noname-a con 0x7f30a00af790 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30a00b0900 0x7f30a013aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient(hunting): picked mon.noname-b con 0x7f30a00b0900 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a011f1f0 con 0x7f30a00af790 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a011f4d0 con 0x7f30a00b0900 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00b5580 con 0x7f30a006bb00 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:52.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a6901640 10 monclient(hunting): authenticate will time out at 1876.228342s 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f309f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00af790 0x7f30a0070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f309ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006bb00 0x7f30a013bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f30a4e9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30a00b0900 0x7f30a013aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f309f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00af790 0x7f30a0070f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54150/0 (socket says 172.21.15.110:54150) 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f309f7fe640 1 -- 172.21.15.110:0/2577961506 learned_addr learned my addr 172.21.15.110:0/2577961506 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f309f7fe640 10 monclient(hunting): get_auth_request con 0x7f30a00af790 auth_method 0 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f309f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f309f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.878+0000 7f309f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f30a4e9e640 10 monclient(hunting): get_auth_request con 0x7f30a00b0900 auth_method 0 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f30a4e9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f30a4e9e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:52.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f30a4e9e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309ffff640 10 monclient(hunting): get_auth_request con 0x7f30a006bb00 auth_method 0 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309ffff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f30a4e9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f30a4e9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f30a4e9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:52.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.879+0000 7f309ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:52.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 10 monclient(hunting): handle_auth_done global_id 6259 payload 824 2024-01-20T11:23:52.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 1 -- 172.21.15.110:0/2577961506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006bb00 msgr2=0x7f30a013bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:52.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006bb00 0x7f30a013bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:52.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 1 -- 172.21.15.110:0/2577961506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00af790 msgr2=0x7f30a0070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:52.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00af790 0x7f30a0070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:52.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 10 monclient: _finish_hunting 0 2024-01-20T11:23:52.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 1 monclient: found mon.noname-b 2024-01-20T11:23:52.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:52.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:52.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.882+0000 7f30a4e9e640 1 -- 172.21.15.110:0/2577961506 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30900094d0 con 0x7f30a00b0900 2024-01-20T11:23:52.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.884+0000 7f30a4e9e640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30a00b0900 0x7f30a013aed0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f309800eba0 tx=0x7f309800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:52.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.884+0000 7f309cff9640 10 client.?.objecter ms_handle_connect 0x7f30a00b0900 2024-01-20T11:23:52.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.884+0000 7f309cff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 1 -- 172.21.15.110:0/2577961506 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309800cd20 con 0x7f30a00b0900 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 10 monclient: dump: 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:52.889 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 10 monclient: _finish_auth 0 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 10 monclient: _check_auth_tickets 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 1 -- 172.21.15.110:0/2577961506 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309800f040 con 0x7f30a00b0900 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f30a6901640 5 monclient: authenticate success, global_id 6259 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f30a6901640 10 monclient: _renew_subs 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f30a6901640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a00b5250 con 0x7f30a00b0900 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 1 -- 172.21.15.110:0/2577961506 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30980146c0 con 0x7f30a00b0900 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309cff9640 10 monclient: dump: 2024-01-20T11:23:52.890 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.888+0000 7f309e7fc640 4 set_mon_vals no callback set 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.889+0000 7f30a6901640 10 client.6259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.889+0000 7f30a6901640 10 monclient: _renew_subs 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.889+0000 7f30a6901640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:52.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.889+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a013b5f0 con 0x7f30a00b0900 2024-01-20T11:23:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.889+0000 7f30a6901640 1 librados: init done 2024-01-20T11:23:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.889+0000 7f30a6901640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.889+0000 7f30a6901640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.889+0000 7f30a6901640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:52.889+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a0075490 con 0x7f30a00b0900 2024-01-20T11:23:53.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.129+0000 7f309cff9640 1 -- 172.21.15.110:0/2577961506 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3098014820 con 0x7f30a00b0900 2024-01-20T11:23:53.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.129+0000 7f309cff9640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f308c06c860 0x7f308c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:53.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.130+0000 7f309ffff640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f308c06c860 0x7f308c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:53.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.130+0000 7f309ffff640 10 monclient: get_auth_request con 0x7f308c06c860 auth_method 0 2024-01-20T11:23:53.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.130+0000 7f309ffff640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f308c06c860 0x7f308c06ed20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3090036000 tx=0x7f309003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:53.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.130+0000 7f309cff9640 10 client.6259.objecter ms_handle_connect 0x7f308c06c860 2024-01-20T11:23:53.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.133+0000 7f309cff9640 1 -- 172.21.15.110:0/2577961506 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3098059470 con 0x7f30a00b0900 2024-01-20T11:23:53.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.133+0000 7f309cff9640 10 client.6259.objecter ms_dispatch 0x7f30a0000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:53.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.133+0000 7f309cff9640 3 client.6259.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:53.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.133+0000 7f309cff9640 3 client.6259.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:53.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.133+0000 7f309cff9640 20 client.6259.objecter dump_active .. 0 homeless 2024-01-20T11:23:53.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.144+0000 7f309cff9640 1 -- 172.21.15.110:0/2577961506 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f30980557b0 con 0x7f30a00b0900 2024-01-20T11:23:53.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.144+0000 7f309cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:53.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.144+0000 7f309cff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:53.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.335+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f30a006c950 con 0x7f308c06c860 2024-01-20T11:23:53.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.340+0000 7f309cff9640 1 -- 172.21.15.110:0/2577961506 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64005 (secure 0 0 0) 0x7f3090044400 con 0x7f308c06c860 2024-01-20T11:23:53.340 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:53.341 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:53.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.341+0000 7f30a6901640 10 librados: watch_flush enter 2024-01-20T11:23:53.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.341+0000 7f30a6901640 10 librados: watch_flush exit 2024-01-20T11:23:53.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.341+0000 7f30a6901640 20 client.6259.objecter shutdown clearing up homeless session... 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.341+0000 7f30a6901640 10 client.6259.objecter successfully canceled tick 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f308c06c860 msgr2=0x7f308c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f308c06c860 0x7f308c06ed20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3090036000 tx=0x7f309003c040 comp rx=0 tx=0).stop 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 10 monclient: shutdown 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30a00b0900 msgr2=0x7f30a013aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30a00b0900 0x7f30a013aed0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f309800eba0 tx=0x7f309800c490 comp rx=0 tx=0).stop 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 shutdown_connections 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f308c06c860 0x7f308c06ed20 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30a00b0900 0x7f30a013aed0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00af790 0x7f30a0070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 --2- 172.21.15.110:0/2577961506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006bb00 0x7f30a013bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 >> 172.21.15.110:0/2577961506 conn(0x7f30a0066d60 msgr2=0x7f30a0142c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:53.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 shutdown_connections 2024-01-20T11:23:53.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 -- 172.21.15.110:0/2577961506 wait complete. 2024-01-20T11:23:53.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.342+0000 7f30a6901640 1 librados: shutdown 2024-01-20T11:23:53.352 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-01-20T11:23:52.817637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002843"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830287+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:21:46.830287+0000","last_peered":"2024-01-20T11:21:46.830287+0000","last_clean":"2024-01-20T11:21:46.830287+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:21:46.830287+0000","last_undegraded":"2024-01-20T11:21:46.830287+0000","last_fullsized":"2024-01-20T11:21:46.830287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.829744+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:21:46.829744+0000","last_peered":"2024-01-20T11:21:46.829744+0000","last_clean":"2024-01-20T11:21:46.829744+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:21:46.829744+0000","last_undegraded":"2024-01-20T11:21:46.829744+0000","last_fullsized":"2024-01-20T11:21:46.829744+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:46.747466+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:21:46.747466+0000","last_peered":"2024-01-20T11:21:46.747466+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:21:46.747466+0000","last_undegraded":"2024-01-20T11:21:46.747466+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830036+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:21:46.830036+0000","last_peered":"2024-01-20T11:21:46.830036+0000","last_clean":"2024-01-20T11:21:46.830036+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:21:46.830036+0000","last_undegraded":"2024-01-20T11:21:46.830036+0000","last_fullsized":"2024-01-20T11:21:46.830036+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280621,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":8.9290000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":10.532},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":11.890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.052},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":15.555999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.692},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":7.4950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.335999999999999},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":17.390999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":9.7929999999999993},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":14.356999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":7.6470000000000002},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":6.8209999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":10.83},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":15.849}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":17.356000000000002},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":11.827}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":16.579000000000001},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":18.015999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":15.436},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":12.462999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":13.255000000000001},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":19.591999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313325,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":5.6230000000000002},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":6.7080000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":15.225}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":10.499000000000001},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":11.723000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":12.657999999999999},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":16.148}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":9.9250000000000007},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":13.577999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":17.189},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":17.855}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.1909999999999998},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":7.8600000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":9.9039999999999999},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.2040000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":10.930999999999999},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":11.569000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.913,"15min":16.091999999999999},"min":{"1min":9.5099999999999998,"5min":7.8200000000000003,"15min":7.3869999999999996},"max":{"1min":17.895,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.897},{"interface":"front","average":{"1min":12.028,"5min":12.231999999999999,"15min":13.638},"min":{"1min":8.1460000000000008,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.597999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":12.207000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":13.525},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.1479999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":8.4909999999999997},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":14.227}]}]},{"osd":1,"up_from":17,"seq":73014444139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":8.6270000000000007},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":10.124000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":5.992},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":12.651999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":11.933},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":13.34}]},{"osd":4,"last update":"Sat Jan 20 11:23:40 2024","interfaces":[{"interface":"back","average":{"1min":9.6929999999999996,"5min":11.172000000000001,"15min":13},"min":{"1min":6.7290000000000001,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":14.914999999999999,"5min":18.899999999999999,"15min":25.541},"last":8.9350000000000005},{"interface":"front","average":{"1min":12.557,"5min":13.332000000000001,"15min":14.596},"min":{"1min":7.0490000000000004,"5min":6.75,"15min":6.75},"max":{"1min":16.565000000000001,"5min":22.280000000000001,"15min":30.204999999999998},"last":16.498000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":12.223000000000001},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":8.0299999999999994}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":14.65},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":15.286}]}]},{"osd":0,"up_from":17,"seq":73014444139,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":3.96},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":5.9640000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.2690000000000001},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.9690000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.776,"5min":12.148999999999999,"15min":21.678999999999998},"min":{"1min":7.8280000000000003,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":18.335999999999999,"5min":27.292000000000002,"15min":358.738},"last":9.2409999999999997},{"interface":"front","average":{"1min":9.9199999999999999,"5min":10.561999999999999,"15min":12.034000000000001},"min":{"1min":5.5839999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":15.347,"5min":20.672000000000001,"15min":21.004999999999999},"last":13.311999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":12.465,"15min":23.204000000000001},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":17.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":11.960000000000001},{"interface":"front","average":{"1min":11.621,"5min":10.532999999999999,"15min":20.757000000000001},"min":{"1min":7.0620000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":16.713000000000001,"5min":21.334,"15min":372.70800000000003},"last":10.694000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":9.9749999999999996},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":14.093999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":15.414,"5min":15.087,"15min":16.367999999999999},"min":{"1min":8.9819999999999993,"5min":8.625,"15min":8.625},"max":{"1min":22.588999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.977},{"interface":"front","average":{"1min":12.353,"5min":11.938000000000001,"15min":12.262},"min":{"1min":8.0250000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":19.954000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.305}]}]},{"osd":6,"up_from":15,"seq":64424509548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":4.5629999999999997},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":6.4130000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":13.973000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":8.0259999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":7.4089999999999998},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":13.159000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.2620000000000005},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.253}]},{"osd":4,"last update":"Sat Jan 20 11:23:41 2024","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":13.513999999999999,"15min":15.654},"min":{"1min":8.3879999999999999,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":21.539000000000001,"5min":21.539000000000001,"15min":70.254999999999995},"last":9.9659999999999993},{"interface":"front","average":{"1min":12.259,"5min":12.333,"15min":15.17},"min":{"1min":7.3079999999999998,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":17.798999999999999,"5min":18.061,"15min":72.670000000000002},"last":14.744999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":15.579000000000001},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":11.894}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:53.354 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:53.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.453+0000 7f23f0e4b640 10 monclient: get_monmap_and_config 2024-01-20T11:23:53.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.453+0000 7f23f0e4b640 10 monclient: build_initial_monmap 2024-01-20T11:23:53.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.453+0000 7f23f0e4b640 10 monclient: monmap: 2024-01-20T11:23:53.453 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:53.454328+0000 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:53.454328+0000 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:53.454 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:53.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.454+0000 7f23f0e4b640 1 Processor -- start 2024-01-20T11:23:53.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.454+0000 7f23f0e4b640 1 -- start start 2024-01-20T11:23:53.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.454+0000 7f23f0e4b640 10 monclient: init 2024-01-20T11:23:53.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23ec066620 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23ec06df70 0x7f23ec145440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 10 monclient(hunting): picked mon.noname-a con 0x7f23ec06df70 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec145980 0x7f23ec14c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 10 monclient(hunting): picked mon.noname-b con 0x7f23ec145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ec0adf80 con 0x7f23ec066620 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ec072880 con 0x7f23ec06df70 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec06e690 con 0x7f23ec145980 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:53.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23f0e4b640 10 monclient(hunting): authenticate will time out at 1876.805324s 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23ead9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec145980 0x7f23ec14c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.455+0000 7f23e3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23ec06df70 0x7f23ec145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23e3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23ec06df70 0x7f23ec145440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54154/0 (socket says 172.21.15.110:54154) 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23e3fff640 1 -- 172.21.15.110:0/1199916789 learned_addr learned my addr 172.21.15.110:0/1199916789 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23e3fff640 10 monclient(hunting): get_auth_request con 0x7f23ec06df70 auth_method 0 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23e3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23ead9d640 10 monclient(hunting): get_auth_request con 0x7f23ec145980 auth_method 0 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23ead9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:53.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:53.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:53.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:53.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.456+0000 7f23ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:53.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.458+0000 7f23e3fff640 10 monclient(hunting): handle_auth_done global_id 6150 payload 274 2024-01-20T11:23:53.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.458+0000 7f23e3fff640 1 -- 172.21.15.110:0/1199916789 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec145980 msgr2=0x7f23ec14c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:53.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.458+0000 7f23e3fff640 1 --2- 172.21.15.110:0/1199916789 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec145980 0x7f23ec14c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.458+0000 7f23e3fff640 1 -- 172.21.15.110:0/1199916789 >> v1:172.21.15.110:6789/0 conn(0x7f23ec066620 legacy=0x7f23ec13bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:53.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.458+0000 7f23e3fff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:53.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.458+0000 7f23e3fff640 1 monclient: found mon.noname-a 2024-01-20T11:23:53.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.458+0000 7f23e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:53.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.458+0000 7f23e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:53.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.458+0000 7f23e3fff640 1 -- 172.21.15.110:0/1199916789 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ec06f650 con 0x7f23ec06df70 2024-01-20T11:23:53.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.461+0000 7f23e3fff640 1 --2- 172.21.15.110:0/1199916789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec06df70 0x7f23ec145440 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009870 tx=0x7f23d402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1da7ad41604f23 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 1 -- 172.21.15.110:0/1199916789 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d40317c0 con 0x7f23ec06df70 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 10 monclient: dump: 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:53.465 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 10 monclient: _finish_auth 0 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 1 -- 172.21.15.110:0/1199916789 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d4031920 con 0x7f23ec06df70 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 1 -- 172.21.15.110:0/1199916789 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d4031cb0 con 0x7f23ec06df70 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23f0e4b640 5 monclient: authenticate success, global_id 6150 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23f0e4b640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23f0e4b640 4 set_mon_vals no callback set 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:53.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.465+0000 7f23e9d9b640 10 monclient: dump: 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:53.467 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 10 monclient: shutdown 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 1 -- 172.21.15.110:0/1199916789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec06df70 msgr2=0x7f23ec145440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 1 --2- 172.21.15.110:0/1199916789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec06df70 0x7f23ec145440 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009870 tx=0x7f23d402b860 comp rx=0 tx=0).stop 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 1 -- 172.21.15.110:0/1199916789 shutdown_connections 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 1 --2- 172.21.15.110:0/1199916789 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec145980 0x7f23ec14c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 1 --2- 172.21.15.110:0/1199916789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec06df70 0x7f23ec145440 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 1 -- 172.21.15.110:0/1199916789 >> 172.21.15.110:0/1199916789 conn(0x7f23ec0b09e0 msgr2=0x7f23ec0b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 1 -- 172.21.15.110:0/1199916789 shutdown_connections 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 1 -- 172.21.15.110:0/1199916789 wait complete. 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 10 monclient: build_initial_monmap 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.466+0000 7f23f0e4b640 10 monclient: monmap: 2024-01-20T11:23:53.471 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 librados: starting msgr at 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 librados: starting objecter 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 Processor -- start 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 -- start start 2024-01-20T11:23:53.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 librados: setting wanted keys 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 librados: calling monclient init 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 10 monclient: init 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23ec066620 0x7f23ec13b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 10 monclient(hunting): picked mon.noname-b con 0x7f23ec066620 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec06df70 0x7f23ec13ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 10 monclient(hunting): picked mon.noname-c con 0x7f23ec06df70 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec145980 0x7f23ec140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 10 monclient(hunting): picked mon.noname-a con 0x7f23ec145980 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec142fc0 con 0x7f23ec145980 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.467+0000 7f23f0e4b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec13f550 con 0x7f23ec066620 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23f0e4b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:53.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23f0e4b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec142550 con 0x7f23ec06df70 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23f0e4b640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23f0e4b640 10 monclient(hunting): authenticate will time out at 1876.818195s 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ead9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec145980 0x7f23ec140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23e3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec06df70 0x7f23ec13ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ea59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23ec066620 0x7f23ec13b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ead9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec145980 0x7f23ec140610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54160/0 (socket says 172.21.15.110:54160) 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ead9d640 1 -- 172.21.15.110:0/818406438 learned_addr learned my addr 172.21.15.110:0/818406438 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ead9d640 10 monclient(hunting): get_auth_request con 0x7f23ec145980 auth_method 0 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ead9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ea59c640 10 monclient(hunting): get_auth_request con 0x7f23ec066620 auth_method 0 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ea59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:53.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23e3fff640 10 monclient(hunting): get_auth_request con 0x7f23ec06df70 auth_method 0 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23e3fff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.468+0000 7f23e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.469+0000 7f23ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.469+0000 7f23ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.469+0000 7f23ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.469+0000 7f23ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.469+0000 7f23ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.469+0000 7f23ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.469+0000 7f23e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.469+0000 7f23e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:53.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.469+0000 7f23e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:53.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 10 monclient(hunting): handle_auth_done global_id 6262 payload 824 2024-01-20T11:23:53.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 1 -- 172.21.15.110:0/818406438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec06df70 msgr2=0x7f23ec13ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:53.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec06df70 0x7f23ec13ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 1 -- 172.21.15.110:0/818406438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec145980 msgr2=0x7f23ec140610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:53.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec145980 0x7f23ec140610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 10 monclient: _finish_hunting 0 2024-01-20T11:23:53.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 1 monclient: found mon.noname-b 2024-01-20T11:23:53.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:53.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:53.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.471+0000 7f23ea59c640 1 -- 172.21.15.110:0/818406438 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d40094d0 con 0x7f23ec066620 2024-01-20T11:23:53.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.474+0000 7f23ea59c640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23ec066620 0x7f23ec13b250 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00c410 tx=0x7f23dc00c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:53.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.474+0000 7f23e2ffd640 10 client.?.objecter ms_handle_connect 0x7f23ec066620 2024-01-20T11:23:53.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.474+0000 7f23e2ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:53.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 1 -- 172.21.15.110:0/818406438 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc0035f0 con 0x7f23ec066620 2024-01-20T11:23:53.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:53.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:53.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 10 monclient: dump: 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 10 monclient: _finish_auth 0 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 1 -- 172.21.15.110:0/818406438 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23dc003770 con 0x7f23ec066620 2024-01-20T11:23:53.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 1 -- 172.21.15.110:0/818406438 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc003b60 con 0x7f23ec066620 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e2ffd640 10 monclient: dump: 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:53.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23f0e4b640 5 monclient: authenticate success, global_id 6262 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23f0e4b640 10 monclient: _renew_subs 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23f0e4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ec1427d0 con 0x7f23ec066620 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23e959a640 4 set_mon_vals no callback set 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23f0e4b640 10 client.6262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23f0e4b640 10 monclient: _renew_subs 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23f0e4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ec140d30 con 0x7f23ec066620 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.477+0000 7f23f0e4b640 1 librados: init done 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.478+0000 7f23f0e4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.478+0000 7f23f0e4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.478+0000 7f23f0e4b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:53.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.478+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ec142bf0 con 0x7f23ec066620 2024-01-20T11:23:53.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.496+0000 7f23e2ffd640 1 -- 172.21.15.110:0/818406438 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f23dc003cc0 con 0x7f23ec066620 2024-01-20T11:23:53.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.497+0000 7f23e2ffd640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f23cc06c860 0x7f23cc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:53.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.497+0000 7f23e3fff640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f23cc06c860 0x7f23cc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:53.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.497+0000 7f23e3fff640 10 monclient: get_auth_request con 0x7f23cc06c860 auth_method 0 2024-01-20T11:23:53.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.497+0000 7f23e3fff640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f23cc06c860 0x7f23cc06ed20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f23d4036000 tx=0x7f23d403d040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:53.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.497+0000 7f23e2ffd640 10 client.6262.objecter ms_handle_connect 0x7f23cc06c860 2024-01-20T11:23:53.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.499+0000 7f23e2ffd640 1 -- 172.21.15.110:0/818406438 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f23dc0575e0 con 0x7f23ec066620 2024-01-20T11:23:53.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.499+0000 7f23e2ffd640 10 client.6262.objecter ms_dispatch 0x7f23ec000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:53.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.499+0000 7f23e2ffd640 3 client.6262.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:53.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.499+0000 7f23e2ffd640 3 client.6262.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:53.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.499+0000 7f23e2ffd640 20 client.6262.objecter dump_active .. 0 homeless 2024-01-20T11:23:53.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.735+0000 7f23e2ffd640 1 -- 172.21.15.110:0/818406438 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f23dc053920 con 0x7f23ec066620 2024-01-20T11:23:53.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.735+0000 7f23e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:53.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.735+0000 7f23e2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:53.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.910+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23ec145e30 con 0x7f23cc06c860 2024-01-20T11:23:53.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.913+0000 7f23e2ffd640 1 -- 172.21.15.110:0/818406438 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+64005 (secure 0 0 0) 0x7f23d4031a90 con 0x7f23cc06c860 2024-01-20T11:23:53.913 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:53.913 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:53.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 10 librados: watch_flush enter 2024-01-20T11:23:53.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 10 librados: watch_flush exit 2024-01-20T11:23:53.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 20 client.6262.objecter shutdown clearing up homeless session... 2024-01-20T11:23:53.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 10 client.6262.objecter successfully canceled tick 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f23cc06c860 msgr2=0x7f23cc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f23cc06c860 0x7f23cc06ed20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f23d4036000 tx=0x7f23d403d040 comp rx=0 tx=0).stop 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 10 monclient: shutdown 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23ec066620 msgr2=0x7f23ec13b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23ec066620 0x7f23ec13b250 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00c410 tx=0x7f23dc00c810 comp rx=0 tx=0).stop 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 shutdown_connections 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f23cc06c860 0x7f23cc06ed20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23ec145980 0x7f23ec140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23ec06df70 0x7f23ec13ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 --2- 172.21.15.110:0/818406438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23ec066620 0x7f23ec13b250 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 >> 172.21.15.110:0/818406438 conn(0x7f23ec0b09e0 msgr2=0x7f23ec0afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 shutdown_connections 2024-01-20T11:23:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 -- 172.21.15.110:0/818406438 wait complete. 2024-01-20T11:23:53.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:53.915+0000 7f23f0e4b640 1 librados: shutdown 2024-01-20T11:23:53.926 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-01-20T11:23:52.817637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002843"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830287+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:21:46.830287+0000","last_peered":"2024-01-20T11:21:46.830287+0000","last_clean":"2024-01-20T11:21:46.830287+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:21:46.830287+0000","last_undegraded":"2024-01-20T11:21:46.830287+0000","last_fullsized":"2024-01-20T11:21:46.830287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.829744+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:21:46.829744+0000","last_peered":"2024-01-20T11:21:46.829744+0000","last_clean":"2024-01-20T11:21:46.829744+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:21:46.829744+0000","last_undegraded":"2024-01-20T11:21:46.829744+0000","last_fullsized":"2024-01-20T11:21:46.829744+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:46.747466+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:21:46.747466+0000","last_peered":"2024-01-20T11:21:46.747466+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:21:46.747466+0000","last_undegraded":"2024-01-20T11:21:46.747466+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830036+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:21:46.830036+0000","last_peered":"2024-01-20T11:21:46.830036+0000","last_clean":"2024-01-20T11:21:46.830036+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:21:46.830036+0000","last_undegraded":"2024-01-20T11:21:46.830036+0000","last_fullsized":"2024-01-20T11:21:46.830036+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280621,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":8.9290000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":10.532},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":11.890000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.052},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":15.555999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.692},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":7.4950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.335999999999999},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":17.390999999999998}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":9.7929999999999993},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":14.356999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":7.6470000000000002},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":6.8209999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":10.83},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":15.849}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":17.356000000000002},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":11.827}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":16.579000000000001},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":18.015999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":15.436},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":12.462999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":13.255000000000001},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":19.591999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313325,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":5.6230000000000002},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":6.7080000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":15.225}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":10.499000000000001},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":11.723000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":12.657999999999999},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":16.148}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":9.9250000000000007},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":13.577999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":17.189},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":17.855}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444138,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.1909999999999998},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":7.8600000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":9.9039999999999999},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.2040000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":10.930999999999999},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":11.569000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.913,"15min":16.091999999999999},"min":{"1min":9.5099999999999998,"5min":7.8200000000000003,"15min":7.3869999999999996},"max":{"1min":17.895,"5min":20.393000000000001,"15min":51.000999999999998},"last":12.897},{"interface":"front","average":{"1min":12.028,"5min":12.231999999999999,"15min":13.638},"min":{"1min":8.1460000000000008,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.597999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":12.207000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":13.525},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":7.1479999999999997}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":8.4909999999999997},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":14.227}]}]},{"osd":1,"up_from":17,"seq":73014444139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":8.6270000000000007},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":10.124000000000001}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":5.992},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":12.651999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":11.933},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":13.34}]},{"osd":4,"last update":"Sat Jan 20 11:23:40 2024","interfaces":[{"interface":"back","average":{"1min":9.6929999999999996,"5min":11.172000000000001,"15min":13},"min":{"1min":6.7290000000000001,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":14.914999999999999,"5min":18.899999999999999,"15min":25.541},"last":8.9350000000000005},{"interface":"front","average":{"1min":12.557,"5min":13.332000000000001,"15min":14.596},"min":{"1min":7.0490000000000004,"5min":6.75,"15min":6.75},"max":{"1min":16.565000000000001,"5min":22.280000000000001,"15min":30.204999999999998},"last":16.498000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":12.223000000000001},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":8.0299999999999994}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":14.65},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":15.286}]}]},{"osd":0,"up_from":17,"seq":73014444139,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":3.96},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":5.9640000000000004}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":7.2690000000000001},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.9690000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.776,"5min":12.148999999999999,"15min":21.678999999999998},"min":{"1min":7.8280000000000003,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":18.335999999999999,"5min":27.292000000000002,"15min":358.738},"last":9.2409999999999997},{"interface":"front","average":{"1min":9.9199999999999999,"5min":10.561999999999999,"15min":12.034000000000001},"min":{"1min":5.5839999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":15.347,"5min":20.672000000000001,"15min":21.004999999999999},"last":13.311999999999999}]},{"osd":4,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":12.465,"15min":23.204000000000001},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":17.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":11.960000000000001},{"interface":"front","average":{"1min":11.621,"5min":10.532999999999999,"15min":20.757000000000001},"min":{"1min":7.0620000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":16.713000000000001,"5min":21.334,"15min":372.70800000000003},"last":10.694000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":9.9749999999999996},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":14.093999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":15.414,"5min":15.087,"15min":16.367999999999999},"min":{"1min":8.9819999999999993,"5min":8.625,"15min":8.625},"max":{"1min":22.588999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":14.977},{"interface":"front","average":{"1min":12.353,"5min":11.938000000000001,"15min":12.262},"min":{"1min":8.0250000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":19.954000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.305}]}]},{"osd":6,"up_from":15,"seq":64424509548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":4.5629999999999997},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":6.4130000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":13.973000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":8.0259999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":7.4089999999999998},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":13.159000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.2620000000000005},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.253}]},{"osd":4,"last update":"Sat Jan 20 11:23:41 2024","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":13.513999999999999,"15min":15.654},"min":{"1min":8.3879999999999999,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":21.539000000000001,"5min":21.539000000000001,"15min":70.254999999999995},"last":9.9659999999999993},{"interface":"front","average":{"1min":12.259,"5min":12.333,"15min":15.17},"min":{"1min":7.3079999999999998,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":17.798999999999999,"5min":18.061,"15min":72.670000000000002},"last":14.744999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":15.579000000000001},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":11.894}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:53.928 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:53.928 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:53.928 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:53.928 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 16, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:46.747466+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:21:46.747466+0000', 'last_peered': '2024-01-20T11:21:46.747466+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:21:46.747466+0000', 'last_undegraded': '2024-01-20T11:21:46.747466+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:53.928 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:53.928 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 16, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:31.815524+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:21:31.815524+0000', 'last_peered': '2024-01-20T11:21:31.815524+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:21:31.815524+0000', 'last_undegraded': '2024-01-20T11:21:31.815524+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:53.929 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:53.929 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 131, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:21:18.077638+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:21:18.077638+0000', 'last_peered': '2024-01-20T11:21:18.077638+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:21:18.077638+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:53.929 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:54.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.028+0000 7fc3f944b640 10 monclient: get_monmap_and_config 2024-01-20T11:23:54.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.028+0000 7fc3f944b640 10 monclient: build_initial_monmap 2024-01-20T11:23:54.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.028+0000 7fc3f944b640 10 monclient: monmap: 2024-01-20T11:23:54.028 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:54.028 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.028 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:54.028972+0000 2024-01-20T11:23:54.028 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:54.028972+0000 2024-01-20T11:23:54.028 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:54.028 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.029 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:54.029 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:54.029 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:54.029 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:54.029 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:54.029 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.028+0000 7fc3f944b640 1 Processor -- start 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.028+0000 7fc3f944b640 1 -- start start 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.028+0000 7fc3f944b640 10 monclient: init 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3f40b4d10 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3f40b1ed0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc3f4146090 0x7fc3f40b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 10 monclient(hunting): picked mon.noname-a con 0x7fc3f4146090 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f40aade0 con 0x7fc3f40b4d10 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f4145460 con 0x7fc3f40b1ed0 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:54.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f40b50f0 con 0x7fc3f4146090 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.029+0000 7fc3f944b640 10 monclient(hunting): authenticate will time out at 1877.379947s 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.030+0000 7fc3f37fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc3f4146090 0x7fc3f40b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.030+0000 7fc3f27fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc3f40b1ed0 0x7fc3f4145980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:50982/0 (socket says 172.21.15.110:50982) 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.030+0000 7fc3f27fc640 1 -- 172.21.15.110:0/3237678545 learned_addr learned my addr 172.21.15.110:0/3237678545 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.030+0000 7fc3f37fe640 10 monclient(hunting): get_auth_request con 0x7fc3f4146090 auth_method 0 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.030+0000 7fc3f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.030+0000 7fc3f37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.030+0000 7fc3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.031+0000 7fc3f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.031+0000 7fc3f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:54.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.031+0000 7fc3f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:54.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f1ffb640 1 -- 172.21.15.110:0/3237678545 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928804119 0 0) 0x7fc3f4145460 con 0x7fc3f40b1ed0 2024-01-20T11:23:54.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f1ffb640 10 monclient(hunting): _negotiate 2024-01-20T11:23:54.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f1ffb640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f1ffb640 10 monclient(hunting): my global_id is 6265 2024-01-20T11:23:54.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f1ffb640 1 -- 172.21.15.110:0/3237678545 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3dc003520 con 0x7fc3f40b1ed0 2024-01-20T11:23:54.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f37fe640 10 monclient(hunting): handle_auth_done global_id 6156 payload 274 2024-01-20T11:23:54.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f37fe640 1 -- 172.21.15.110:0/3237678545 >> v1:172.21.15.134:6789/0 conn(0x7fc3f40b1ed0 legacy=0x7fc3f4145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f37fe640 1 -- 172.21.15.110:0/3237678545 >> v1:172.21.15.110:6789/0 conn(0x7fc3f40b4d10 legacy=0x7fc3f40b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:54.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f37fe640 10 monclient: _finish_hunting 0 2024-01-20T11:23:54.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f37fe640 1 monclient: found mon.noname-a 2024-01-20T11:23:54.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:54.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:54.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.032+0000 7fc3f37fe640 1 -- 172.21.15.110:0/3237678545 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40b2660 con 0x7fc3f4146090 2024-01-20T11:23:54.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.034+0000 7fc3f37fe640 1 --2- 172.21.15.110:0/3237678545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3f4146090 0x7fc3f40b3da0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009dc0 tx=0x7fc3ec02be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f1a071e8f6a43 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 1 -- 172.21.15.110:0/3237678545 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec02cbb0 con 0x7fc3f4146090 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 10 monclient: dump: 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 10 monclient: _finish_auth 0 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 1 -- 172.21.15.110:0/3237678545 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ec02cd10 con 0x7fc3f4146090 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 5 monclient: authenticate success, global_id 6156 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 1 -- 172.21.15.110:0/3237678545 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec034660 con 0x7fc3f4146090 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 4 set_mon_vals no callback set 2024-01-20T11:23:54.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f1ffb640 10 monclient: dump: 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:54.041 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 10 monclient: shutdown 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 1 -- 172.21.15.110:0/3237678545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3f4146090 msgr2=0x7fc3f40b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 1 --2- 172.21.15.110:0/3237678545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3f4146090 0x7fc3f40b3da0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009dc0 tx=0x7fc3ec02be00 comp rx=0 tx=0).stop 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 1 -- 172.21.15.110:0/3237678545 shutdown_connections 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 1 --2- 172.21.15.110:0/3237678545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3f4146090 0x7fc3f40b3da0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 1 -- 172.21.15.110:0/3237678545 >> 172.21.15.110:0/3237678545 conn(0x7fc3f4066590 msgr2=0x7fc3f40669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 1 -- 172.21.15.110:0/3237678545 shutdown_connections 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.039+0000 7fc3f944b640 1 -- 172.21.15.110:0/3237678545 wait complete. 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.040+0000 7fc3f944b640 10 monclient: build_initial_monmap 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.040+0000 7fc3f944b640 10 monclient: monmap: 2024-01-20T11:23:54.045 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.040+0000 7fc3f944b640 1 librados: starting msgr at 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.040+0000 7fc3f944b640 1 librados: starting objecter 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 Processor -- start 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 -- start start 2024-01-20T11:23:54.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 librados: setting wanted keys 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 librados: calling monclient init 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient: init 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3f40b1ed0 0x7fc3f413d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient(hunting): picked mon.noname-c con 0x7fc3f40b1ed0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3f40b4d10 0x7fc3f413dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient(hunting): picked mon.noname-b con 0x7fc3f40b4d10 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3f4146090 0x7fc3f413e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient(hunting): picked mon.noname-a con 0x7fc3f4146090 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40ab7a0 con 0x7fc3f4146090 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:54.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f413c460 con 0x7fc3f40b4d10 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40ab9f0 con 0x7fc3f40b1ed0 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.041+0000 7fc3f944b640 10 monclient(hunting): authenticate will time out at 1877.391846s 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3f40b4d10 0x7fc3f413dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3f40b1ed0 0x7fc3f413d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3f40b4d10 0x7fc3f413dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:48072/0 (socket says 172.21.15.110:48072) 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 1 -- 172.21.15.110:0/3489745663 learned_addr learned my addr 172.21.15.110:0/3489745663 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 10 monclient(hunting): get_auth_request con 0x7fc3f40b4d10 auth_method 0 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f37fe640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3f4146090 0x7fc3f413e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f2ffd640 10 monclient(hunting): get_auth_request con 0x7fc3f40b1ed0 auth_method 0 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f37fe640 10 monclient(hunting): get_auth_request con 0x7fc3f4146090 auth_method 0 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f37fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.042+0000 7fc3f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.043+0000 7fc3f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.043+0000 7fc3f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.043+0000 7fc3f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.043+0000 7fc3f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.043+0000 7fc3f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.043+0000 7fc3f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 10 monclient(hunting): handle_auth_done global_id 6268 payload 824 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 1 -- 172.21.15.110:0/3489745663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3f40b1ed0 msgr2=0x7fc3f413d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3f40b1ed0 0x7fc3f413d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 1 -- 172.21.15.110:0/3489745663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3f4146090 msgr2=0x7fc3f413e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3f4146090 0x7fc3f413e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 10 monclient: _finish_hunting 0 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 1 monclient: found mon.noname-b 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.045+0000 7fc3f27fc640 1 -- 172.21.15.110:0/3489745663 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec009a70 con 0x7fc3f40b4d10 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.047+0000 7fc3f27fc640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3f40b4d10 0x7fc3f413dd10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c400 tx=0x7fc3e400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.047+0000 7fc3db7fe640 10 client.?.objecter ms_handle_connect 0x7fc3f40b4d10 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.047+0000 7fc3db7fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 1 -- 172.21.15.110:0/3489745663 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4005ce0 con 0x7fc3f40b4d10 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 10 monclient: dump: 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 1 -- 172.21.15.110:0/3489745663 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e4005e60 con 0x7fc3f40b4d10 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 1 -- 172.21.15.110:0/3489745663 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e40072f0 con 0x7fc3f40b4d10 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f944b640 5 monclient: authenticate success, global_id 6268 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f944b640 10 monclient: _renew_subs 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f944b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f40abcd0 con 0x7fc3f40b4d10 2024-01-20T11:23:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f0ff9640 4 set_mon_vals no callback set 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3db7fe640 10 monclient: dump: 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f944b640 10 client.6268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f944b640 10 monclient: _renew_subs 2024-01-20T11:23:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f944b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f40ac180 con 0x7fc3f40b4d10 2024-01-20T11:23:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.051+0000 7fc3f944b640 1 librados: init done 2024-01-20T11:23:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.052+0000 7fc3f944b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.052+0000 7fc3f944b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.052+0000 7fc3f944b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.052+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f413c180 con 0x7fc3f40b4d10 2024-01-20T11:23:54.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.292+0000 7fc3db7fe640 1 -- 172.21.15.110:0/3489745663 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc3e4007450 con 0x7fc3f40b4d10 2024-01-20T11:23:54.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.292+0000 7fc3db7fe640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3e006c8b0 0x7fc3e006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.292+0000 7fc3f2ffd640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3e006c8b0 0x7fc3e006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.292+0000 7fc3f2ffd640 10 monclient: get_auth_request con 0x7fc3e006c8b0 auth_method 0 2024-01-20T11:23:54.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.293+0000 7fc3db7fe640 10 client.6268.objecter ms_handle_connect 0x7fc3e006c8b0 2024-01-20T11:23:54.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.293+0000 7fc3f2ffd640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3e006c8b0 0x7fc3e006ed70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80017f0 tx=0x7fc3e8009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:54.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.294+0000 7fc3db7fe640 1 -- 172.21.15.110:0/3489745663 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc3e4057b80 con 0x7fc3f40b4d10 2024-01-20T11:23:54.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.294+0000 7fc3db7fe640 10 client.6268.objecter ms_dispatch 0x7fc3f4000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:54.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.294+0000 7fc3db7fe640 3 client.6268.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:54.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.294+0000 7fc3db7fe640 3 client.6268.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:54.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.294+0000 7fc3db7fe640 20 client.6268.objecter dump_active .. 0 homeless 2024-01-20T11:23:54.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.304+0000 7fc3db7fe640 1 -- 172.21.15.110:0/3489745663 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc3e4053ec0 con 0x7fc3f40b4d10 2024-01-20T11:23:54.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.304+0000 7fc3db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:54.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.304+0000 7fc3db7fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.548+0000 7fc3f944b640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.548+0000 7fc3f944b640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.548+0000 7fc3f944b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.548+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc3f413c180 con 0x7fc3f40b4d10 2024-01-20T11:23:54.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.554+0000 7fc3db7fe640 1 -- 172.21.15.110:0/3489745663 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fc3f413c180 con 0x7fc3f40b4d10 2024-01-20T11:23:54.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.554+0000 7fc3db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:54.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.554+0000 7fc3db7fe640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:54.554 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:54.554 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":569,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 10 librados: watch_flush enter 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 10 librados: watch_flush exit 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 20 client.6268.objecter shutdown clearing up homeless session... 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 10 client.6268.objecter successfully canceled tick 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3e006c8b0 msgr2=0x7fc3e006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3e006c8b0 0x7fc3e006ed70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80017f0 tx=0x7fc3e8009040 comp rx=0 tx=0).stop 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 10 monclient: shutdown 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3f40b4d10 msgr2=0x7fc3f413dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3f40b4d10 0x7fc3f413dd10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c400 tx=0x7fc3e400c8c0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 shutdown_connections 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fc3e006c8b0 0x7fc3e006ed70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3f4146090 0x7fc3f413e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3f40b4d10 0x7fc3f413dd10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 --2- 172.21.15.110:0/3489745663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3f40b1ed0 0x7fc3f413d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 >> 172.21.15.110:0/3489745663 conn(0x7fc3f4066590 msgr2=0x7fc3f40b83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 shutdown_connections 2024-01-20T11:23:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 -- 172.21.15.110:0/3489745663 wait complete. 2024-01-20T11:23:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.556+0000 7fc3f944b640 1 librados: shutdown 2024-01-20T11:23:54.567 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:23:54.567 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:54.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.665+0000 7f503109b640 10 monclient: get_monmap_and_config 2024-01-20T11:23:54.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.665+0000 7f503109b640 10 monclient: build_initial_monmap 2024-01-20T11:23:54.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.665+0000 7f503109b640 10 monclient: monmap: 2024-01-20T11:23:54.665 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:54.666387+0000 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:54.666387+0000 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:54.666 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 1 Processor -- start 2024-01-20T11:23:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 1 -- start start 2024-01-20T11:23:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 10 monclient: init 2024-01-20T11:23:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-01-20T11:23:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c071c60 0x7f502c142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 10 monclient(hunting): picked mon.noname-b con 0x7f502c071c60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f502c0ac540 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f502c0b0930 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f502c0ad610 con 0x7f502c0ac540 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f502c1426c0 con 0x7f502c0b0930 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502c072040 con 0x7f502c071c60 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.666+0000 7f503109b640 10 monclient(hunting): authenticate will time out at 1878.017164s 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.667+0000 7f502a59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f502c0ac540 0x7f502c0ab4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42756/0 (socket says 172.21.15.110:42756) 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.667+0000 7f502a59c640 1 -- 172.21.15.110:0/2245884115 learned_addr learned my addr 172.21.15.110:0/2245884115 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.667+0000 7f502ad9d640 1 --2- 172.21.15.110:0/2245884115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c071c60 0x7f502c142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.668+0000 7f502ad9d640 10 monclient(hunting): get_auth_request con 0x7f502c071c60 auth_method 0 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.668+0000 7f502ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:54.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.668+0000 7f502ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.668+0000 7f502ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.668+0000 7f502ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:54.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.668+0000 7f502ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:54.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.668+0000 7f502ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 1 -- 172.21.15.110:0/2245884115 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204387248 0 0) 0x7f502c0ad610 con 0x7f502c0ac540 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 10 monclient(hunting): my global_id is 6162 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 1 -- 172.21.15.110:0/2245884115 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5014003520 con 0x7f502c0ac540 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 1 -- 172.21.15.110:0/2245884115 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836743640 0 0) 0x7f502c1426c0 con 0x7f502c0b0930 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 10 monclient(hunting): _negotiate 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 10 monclient(hunting): my global_id is 6271 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f5029d9b640 1 -- 172.21.15.110:0/2245884115 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f502c0ad610 con 0x7f502c0b0930 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f502ad9d640 10 monclient(hunting): handle_auth_done global_id 5783 payload 274 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f502ad9d640 1 -- 172.21.15.110:0/2245884115 >> v1:172.21.15.134:6789/0 conn(0x7f502c0b0930 legacy=0x7f502c149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f502ad9d640 1 -- 172.21.15.110:0/2245884115 >> v1:172.21.15.110:6789/0 conn(0x7f502c0ac540 legacy=0x7f502c0ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f502ad9d640 10 monclient: _finish_hunting 0 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f502ad9d640 1 monclient: found mon.noname-b 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f502ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f502ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.670+0000 7f502ad9d640 1 -- 172.21.15.110:0/2245884115 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502c0721b0 con 0x7f502c071c60 2024-01-20T11:23:54.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.672+0000 7f502ad9d640 1 --2- 172.21.15.110:0/2245884115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c071c60 0x7f502c142180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f50240097d0 tx=0x7f502402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e66b7c4b9bafcb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:54.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 1 -- 172.21.15.110:0/2245884115 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024031890 con 0x7f502c071c60 2024-01-20T11:23:54.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 10 monclient: dump: 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 10 monclient: _finish_auth 0 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 10 monclient: _check_auth_tickets 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 1 -- 172.21.15.110:0/2245884115 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50240319f0 con 0x7f502c071c60 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f503109b640 5 monclient: authenticate success, global_id 5783 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f503109b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f503109b640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 1 -- 172.21.15.110:0/2245884115 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024031d80 con 0x7f502c071c60 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f503109b640 4 set_mon_vals no callback set 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.675+0000 7f5029d9b640 10 monclient: dump: 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.677 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.678 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.678 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:54.678 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.678 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:54.678 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:54.678 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:54.678 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.676+0000 7f503109b640 10 monclient: shutdown 2024-01-20T11:23:54.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.676+0000 7f503109b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:54.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.676+0000 7f503109b640 1 -- 172.21.15.110:0/2245884115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c071c60 msgr2=0x7f502c142180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.676+0000 7f503109b640 1 --2- 172.21.15.110:0/2245884115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c071c60 0x7f502c142180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f50240097d0 tx=0x7f502402b930 comp rx=0 tx=0).stop 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.676+0000 7f503109b640 1 -- 172.21.15.110:0/2245884115 shutdown_connections 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.676+0000 7f503109b640 1 --2- 172.21.15.110:0/2245884115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c071c60 0x7f502c142180 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.676+0000 7f503109b640 1 -- 172.21.15.110:0/2245884115 >> 172.21.15.110:0/2245884115 conn(0x7f502c0b3e60 msgr2=0x7f502c0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.676+0000 7f503109b640 1 -- 172.21.15.110:0/2245884115 shutdown_connections 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.676+0000 7f503109b640 1 -- 172.21.15.110:0/2245884115 wait complete. 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.677+0000 7f503109b640 10 monclient: build_initial_monmap 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.677+0000 7f503109b640 10 monclient: monmap: 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.682 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.677+0000 7f503109b640 1 librados: starting msgr at 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.677+0000 7f503109b640 1 librados: starting objecter 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 Processor -- start 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 -- start start 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 librados: setting wanted keys 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 librados: calling monclient init 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient: init 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502c071c60 0x7f502c1387d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient(hunting): picked mon.noname-a con 0x7f502c071c60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:54.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c0ac540 0x7f502c1d6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient(hunting): picked mon.noname-c con 0x7f502c0ac540 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0b0930 0x7f502c1d67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient(hunting): picked mon.noname-b con 0x7f502c0b0930 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c1d6cf0 con 0x7f502c071c60 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c139990 con 0x7f502c0b0930 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502c140b70 con 0x7f502c0ac540 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.678+0000 7f503109b640 10 monclient(hunting): authenticate will time out at 1878.028958s 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c0ac540 0x7f502c1d6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502c071c60 0x7f502c1387d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502b59e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0b0930 0x7f502c1d67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502b59e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0b0930 0x7f502c1d67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:48078/0 (socket says 172.21.15.110:48078) 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502b59e640 1 -- 172.21.15.110:0/2132885548 learned_addr learned my addr 172.21.15.110:0/2132885548 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502b59e640 10 monclient(hunting): get_auth_request con 0x7f502c0b0930 auth_method 0 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502b59e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502b59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502ad9d640 10 monclient(hunting): get_auth_request con 0x7f502c071c60 auth_method 0 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502a59c640 10 monclient(hunting): get_auth_request con 0x7f502c0ac540 auth_method 0 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502a59c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.679+0000 7f502a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:54.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.680+0000 7f502b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:54.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.680+0000 7f502b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:54.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.680+0000 7f502b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:54.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.680+0000 7f502ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:54.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.680+0000 7f502ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:54.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.680+0000 7f502ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:54.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.680+0000 7f502a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:54.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.680+0000 7f502a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:54.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.680+0000 7f502a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:54.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 10 monclient(hunting): handle_auth_done global_id 6274 payload 824 2024-01-20T11:23:54.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 1 -- 172.21.15.110:0/2132885548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c0ac540 msgr2=0x7f502c1d6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c0ac540 0x7f502c1d6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 1 -- 172.21.15.110:0/2132885548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502c071c60 msgr2=0x7f502c1387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:54.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502c071c60 0x7f502c1387d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:54.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 10 monclient: _finish_hunting 0 2024-01-20T11:23:54.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 1 monclient: found mon.noname-b 2024-01-20T11:23:54.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:54.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:23:54.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.682+0000 7f502b59e640 1 -- 172.21.15.110:0/2132885548 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5024009400 con 0x7f502c0b0930 2024-01-20T11:23:54.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.685+0000 7f502b59e640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0b0930 0x7f502c1d67b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f501c00c6e0 tx=0x7f501c00cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:54.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.685+0000 7f50137fe640 10 client.?.objecter ms_handle_connect 0x7f502c0b0930 2024-01-20T11:23:54.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.685+0000 7f50137fe640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 1 -- 172.21.15.110:0/2132885548 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c003b70 con 0x7f502c0b0930 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 10 monclient: dump: 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:54.689 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 10 monclient: _finish_auth 0 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 10 monclient: _check_auth_tickets 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 1 -- 172.21.15.110:0/2132885548 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501c003cd0 con 0x7f502c0b0930 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f503109b640 5 monclient: authenticate success, global_id 6274 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f503109b640 10 monclient: _renew_subs 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f503109b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502c140e50 con 0x7f502c0b0930 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 1 -- 172.21.15.110:0/2132885548 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c0052d0 con 0x7f502c0b0930 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:23:54.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f50137fe640 10 monclient: dump: 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f503109b640 10 client.6274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f503109b640 10 monclient: _renew_subs 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f503109b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502c138d10 con 0x7f502c0b0930 2024-01-20T11:23:54.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f503109b640 1 librados: init done 2024-01-20T11:23:54.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.688+0000 7f502959a640 4 set_mon_vals no callback set 2024-01-20T11:23:54.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.689+0000 7f503109b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:54.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.689+0000 7f503109b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:54.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.689+0000 7f503109b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:23:54.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.689+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502c1396b0 con 0x7f502c0b0930 2024-01-20T11:23:54.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.929+0000 7f50137fe640 1 -- 172.21.15.110:0/2132885548 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f501c005430 con 0x7f502c0b0930 2024-01-20T11:23:54.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.929+0000 7f50137fe640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f501806c8b0 0x7f501806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:54.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.929+0000 7f502ad9d640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f501806c8b0 0x7f501806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:54.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.929+0000 7f502ad9d640 10 monclient: get_auth_request con 0x7f501806c8b0 auth_method 0 2024-01-20T11:23:54.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.930+0000 7f502ad9d640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f501806c8b0 0x7f501806ed70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5024036000 tx=0x7f502403c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:54.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.930+0000 7f50137fe640 10 client.6274.objecter ms_handle_connect 0x7f501806c8b0 2024-01-20T11:23:54.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.933+0000 7f50137fe640 1 -- 172.21.15.110:0/2132885548 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f501c058860 con 0x7f502c0b0930 2024-01-20T11:23:54.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.933+0000 7f50137fe640 10 client.6274.objecter ms_dispatch 0x7f502c000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:54.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.933+0000 7f50137fe640 3 client.6274.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:54.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.933+0000 7f50137fe640 3 client.6274.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:54.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.933+0000 7f50137fe640 20 client.6274.objecter dump_active .. 0 homeless 2024-01-20T11:23:54.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.943+0000 7f50137fe640 1 -- 172.21.15.110:0/2132885548 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f501c058630 con 0x7f502c0b0930 2024-01-20T11:23:54.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.943+0000 7f50137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:54.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:54.943+0000 7f50137fe640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.154+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f502c1396b0 con 0x7f501806c8b0 2024-01-20T11:23:55.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.156+0000 7f50137fe640 1 -- 172.21.15.110:0/2132885548 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63973 (secure 0 0 0) 0x7f502c1396b0 con 0x7f501806c8b0 2024-01-20T11:23:55.157 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:55.157 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 10 librados: watch_flush enter 2024-01-20T11:23:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 10 librados: watch_flush exit 2024-01-20T11:23:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 20 client.6274.objecter shutdown clearing up homeless session... 2024-01-20T11:23:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 10 client.6274.objecter successfully canceled tick 2024-01-20T11:23:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f501806c8b0 msgr2=0x7f501806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f501806c8b0 0x7f501806ed70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5024036000 tx=0x7f502403c040 comp rx=0 tx=0).stop 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 10 monclient: shutdown 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0b0930 msgr2=0x7f502c1d67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0b0930 0x7f502c1d67b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f501c00c6e0 tx=0x7f501c00cba0 comp rx=0 tx=0).stop 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 shutdown_connections 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f501806c8b0 0x7f501806ed70 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0b0930 0x7f502c1d67b0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502c0ac540 0x7f502c1d6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 --2- 172.21.15.110:0/2132885548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502c071c60 0x7f502c1387d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 >> 172.21.15.110:0/2132885548 conn(0x7f502c0b3e60 msgr2=0x7f502c0aa660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.158+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 shutdown_connections 2024-01-20T11:23:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.159+0000 7f503109b640 1 -- 172.21.15.110:0/2132885548 wait complete. 2024-01-20T11:23:55.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:55.159+0000 7f503109b640 1 librados: shutdown 2024-01-20T11:23:55.169 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-01-20T11:23:54.818077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002667"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830287+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:21:46.830287+0000","last_peered":"2024-01-20T11:21:46.830287+0000","last_clean":"2024-01-20T11:21:46.830287+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:21:46.830287+0000","last_undegraded":"2024-01-20T11:21:46.830287+0000","last_fullsized":"2024-01-20T11:21:46.830287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.829744+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:21:46.829744+0000","last_peered":"2024-01-20T11:21:46.829744+0000","last_clean":"2024-01-20T11:21:46.829744+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:21:46.829744+0000","last_undegraded":"2024-01-20T11:21:46.829744+0000","last_fullsized":"2024-01-20T11:21:46.829744+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:46.747466+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:21:46.747466+0000","last_peered":"2024-01-20T11:21:46.747466+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:21:46.747466+0000","last_undegraded":"2024-01-20T11:21:46.747466+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830036+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:21:46.830036+0000","last_peered":"2024-01-20T11:21:46.830036+0000","last_clean":"2024-01-20T11:21:46.830036+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:21:46.830036+0000","last_undegraded":"2024-01-20T11:21:46.830036+0000","last_fullsized":"2024-01-20T11:21:46.830036+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":6.391},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":8.7699999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":14.664},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":8.0920000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.113},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.4210000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.912000000000001},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":10.461}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.565999999999999},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":11.188000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":11.874000000000001},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":15.769}]}]},{"osd":7,"up_from":21,"seq":90194313324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":7.6470000000000002},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":6.8209999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":10.83},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":15.849}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":17.356000000000002},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":11.827}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":16.579000000000001},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":18.015999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":15.436},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":12.462999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":13.255000000000001},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":19.591999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313325,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":5.6230000000000002},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":6.7080000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":15.225}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":10.499000000000001},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":11.723000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":12.657999999999999},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":16.148}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":9.9250000000000007},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":13.577999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":17.189},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":17.855}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":6.2439999999999998},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":8.7309999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":5.5579999999999998},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.8040000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.254},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":7.9470000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.913,"15min":16.091999999999999},"min":{"1min":9.5099999999999998,"5min":7.8200000000000003,"15min":7.3869999999999996},"max":{"1min":17.895,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.289},{"interface":"front","average":{"1min":12.028,"5min":12.231999999999999,"15min":13.638},"min":{"1min":8.1460000000000008,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.597999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.532999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":11.214},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":11.853999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":13.944000000000001},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":16.164999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":4.1529999999999996},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":8.0259999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":9.0449999999999999},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":7.1790000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":12.189},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":10.005000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:40 2024","interfaces":[{"interface":"back","average":{"1min":9.6929999999999996,"5min":11.172000000000001,"15min":13},"min":{"1min":6.7290000000000001,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":14.914999999999999,"5min":18.899999999999999,"15min":25.541},"last":15.234999999999999},{"interface":"front","average":{"1min":12.557,"5min":13.332000000000001,"15min":14.596},"min":{"1min":7.0490000000000004,"5min":6.75,"15min":6.75},"max":{"1min":16.565000000000001,"5min":22.280000000000001,"15min":30.204999999999998},"last":10.603}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":16.689},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":11.602}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":12.952999999999999},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":13.712}]}]},{"osd":0,"up_from":17,"seq":73014444140,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":6.2969999999999997},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":10.323}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":8.5540000000000003},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":16.382999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.776,"5min":12.148999999999999,"15min":21.678999999999998},"min":{"1min":7.8280000000000003,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":18.335999999999999,"5min":27.292000000000002,"15min":358.738},"last":15.526999999999999},{"interface":"front","average":{"1min":9.9199999999999999,"5min":10.561999999999999,"15min":12.034000000000001},"min":{"1min":5.5839999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":15.347,"5min":20.672000000000001,"15min":21.004999999999999},"last":10.933}]},{"osd":4,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":12.465,"15min":23.204000000000001},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":17.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":12.182},{"interface":"front","average":{"1min":11.621,"5min":10.532999999999999,"15min":20.757000000000001},"min":{"1min":7.0620000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":16.713000000000001,"5min":21.334,"15min":372.70800000000003},"last":9.6110000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":17.263000000000002},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":12.763999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":15.414,"5min":15.087,"15min":16.367999999999999},"min":{"1min":8.9819999999999993,"5min":8.625,"15min":8.625},"max":{"1min":22.588999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":13.308999999999999},{"interface":"front","average":{"1min":12.353,"5min":11.938000000000001,"15min":12.262},"min":{"1min":8.0250000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":19.954000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.535}]}]},{"osd":6,"up_from":15,"seq":64424509548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":4.5629999999999997},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":6.4130000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":13.973000000000001},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":8.0259999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":7.4089999999999998},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":13.159000000000001}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":9.2620000000000005},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":11.253}]},{"osd":4,"last update":"Sat Jan 20 11:23:41 2024","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":13.513999999999999,"15min":15.654},"min":{"1min":8.3879999999999999,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":21.539000000000001,"5min":21.539000000000001,"15min":70.254999999999995},"last":9.9659999999999993},{"interface":"front","average":{"1min":12.259,"5min":12.333,"15min":15.17},"min":{"1min":7.3079999999999998,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":17.798999999999999,"5min":18.061,"15min":72.670000000000002},"last":14.744999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":15.579000000000001},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":11.894}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:56.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~298s 2024-01-20T11:23:58.172 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.271+0000 7f037871d640 10 monclient: get_monmap_and_config 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.271+0000 7f037871d640 10 monclient: build_initial_monmap 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.271+0000 7f037871d640 10 monclient: monmap: 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:58.272275+0000 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:58.272275+0000 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:58.272 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:58.273 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 1 Processor -- start 2024-01-20T11:23:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 1 -- start start 2024-01-20T11:23:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 10 monclient: init 2024-01-20T11:23:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-01-20T11:23:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03700b0ba0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:58.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03700adc30 addr v1:172.21.15.110:6789/0 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f03700ac4d0 0x7f03700ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 10 monclient(hunting): picked mon.noname-a con 0x7f03700ac4d0 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.272+0000 7f037871d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03700ad530 con 0x7f03700adc30 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f037871d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03700ad810 con 0x7f03700b0ba0 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f037871d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f037871d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03700af840 con 0x7f03700ac4d0 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f037871d640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f037871d640 10 monclient(hunting): authenticate will time out at 1881.623197s 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f0376cba640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f03700ac4d0 0x7f03700ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f0376cba640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f03700ac4d0 0x7f03700ab630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54212/0 (socket says 172.21.15.110:54212) 2024-01-20T11:23:58.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f0376cba640 1 -- 172.21.15.110:0/555742312 learned_addr learned my addr 172.21.15.110:0/555742312 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:58.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f0376cba640 10 monclient(hunting): get_auth_request con 0x7f03700ac4d0 auth_method 0 2024-01-20T11:23:58.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f0376cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:58.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f0376cba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:58.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.273+0000 7f0376cba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:58.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.274+0000 7f0376cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:58.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.274+0000 7f0376cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:58.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.274+0000 7f0376cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:58.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.276+0000 7f0376cba640 10 monclient(hunting): handle_auth_done global_id 6168 payload 274 2024-01-20T11:23:58.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.276+0000 7f0376cba640 1 -- 172.21.15.110:0/555742312 >> v1:172.21.15.134:6789/0 conn(0x7f03700b0ba0 legacy=0x7f03700af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.276+0000 7f0376cba640 1 -- 172.21.15.110:0/555742312 >> v1:172.21.15.110:6789/0 conn(0x7f03700adc30 legacy=0x7f037013bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:58.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.276+0000 7f0376cba640 10 monclient: _finish_hunting 0 2024-01-20T11:23:58.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.276+0000 7f0376cba640 1 monclient: found mon.noname-a 2024-01-20T11:23:58.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.276+0000 7f0376cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.276+0000 7f0376cba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.276+0000 7f0376cba640 1 -- 172.21.15.110:0/555742312 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03700ab110 con 0x7f03700ac4d0 2024-01-20T11:23:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.278+0000 7f0376cba640 1 --2- 172.21.15.110:0/555742312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 0x7f03700ab630 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f036800d150 tx=0x7f036802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34bff9737c633fe server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 1 -- 172.21.15.110:0/555742312 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036802ea80 con 0x7f03700ac4d0 2024-01-20T11:23:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 10 monclient: dump: 2024-01-20T11:23:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 10 monclient: _finish_auth 0 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 10 monclient: _check_auth_tickets 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f037871d640 5 monclient: authenticate success, global_id 6168 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f037871d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 1 -- 172.21.15.110:0/555742312 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036802ebe0 con 0x7f03700ac4d0 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f037871d640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f037871d640 4 set_mon_vals no callback set 2024-01-20T11:23:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 1 -- 172.21.15.110:0/555742312 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368009600 con 0x7f03700ac4d0 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.282+0000 7f03754b7640 10 monclient: dump: 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:58.284 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 10 monclient: shutdown 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 1 -- 172.21.15.110:0/555742312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 msgr2=0x7f03700ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 1 --2- 172.21.15.110:0/555742312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 0x7f03700ab630 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f036800d150 tx=0x7f036802e0e0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 1 -- 172.21.15.110:0/555742312 shutdown_connections 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 1 --2- 172.21.15.110:0/555742312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 0x7f03700ab630 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 1 -- 172.21.15.110:0/555742312 >> 172.21.15.110:0/555742312 conn(0x7f03700b3e60 msgr2=0x7f03700b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 1 -- 172.21.15.110:0/555742312 shutdown_connections 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 1 -- 172.21.15.110:0/555742312 wait complete. 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 10 monclient: build_initial_monmap 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.283+0000 7f037871d640 10 monclient: monmap: 2024-01-20T11:23:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 librados: starting msgr at 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 librados: starting objecter 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 Processor -- start 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 -- start start 2024-01-20T11:23:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 librados: setting wanted keys 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 librados: calling monclient init 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient: init 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 0x7f037006ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient(hunting): picked mon.noname-a con 0x7f03700ac4d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03700b0ba0 0x7f0370070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient(hunting): picked mon.noname-b con 0x7f03700b0ba0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03701d6390 0x7f03701d6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient(hunting): picked mon.noname-c con 0x7f03701d6390 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0370071270 con 0x7f03700ac4d0 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03700720e0 con 0x7f03700b0ba0 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03701d6d60 con 0x7f03701d6390 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.284+0000 7f037871d640 10 monclient(hunting): authenticate will time out at 1881.635105s 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f03764b9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 0x7f037006ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0376cba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03701d6390 0x7f03701d6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f03764b9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 0x7f037006ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54214/0 (socket says 172.21.15.110:54214) 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f03764b9640 1 -- 172.21.15.110:0/1376660198 learned_addr learned my addr 172.21.15.110:0/1376660198 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f03764b9640 10 monclient(hunting): get_auth_request con 0x7f03700ac4d0 auth_method 0 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f03764b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f03764b9640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f03764b9640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0375cb8640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03700b0ba0 0x7f0370070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0375cb8640 10 monclient(hunting): get_auth_request con 0x7f03700b0ba0 auth_method 0 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0375cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0375cb8640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0375cb8640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0376cba640 10 monclient(hunting): get_auth_request con 0x7f03701d6390 auth_method 0 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0376cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0376cba640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.285+0000 7f0376cba640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.286+0000 7f03764b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.286+0000 7f03764b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.286+0000 7f03764b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.286+0000 7f0375cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.286+0000 7f0375cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.286+0000 7f0375cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.286+0000 7f0376cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.286+0000 7f0376cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.286+0000 7f0376cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 10 monclient(hunting): handle_auth_done global_id 6171 payload 824 2024-01-20T11:23:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 1 -- 172.21.15.110:0/1376660198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03701d6390 msgr2=0x7f03701d6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03701d6390 0x7f03701d6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 1 -- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03700b0ba0 msgr2=0x7f0370070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03700b0ba0 0x7f0370070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 10 monclient: _finish_hunting 0 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 1 monclient: found mon.noname-a 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.289+0000 7f03764b9640 1 -- 172.21.15.110:0/1376660198 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036800bd40 con 0x7f03700ac4d0 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.291+0000 7f03764b9640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 0x7f037006ff60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f036c00c3c0 tx=0x7f036c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.291+0000 7f0366ffd640 10 client.?.objecter ms_handle_connect 0x7f03700ac4d0 2024-01-20T11:23:58.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.291+0000 7f0366ffd640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 1 -- 172.21.15.110:0/1376660198 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c0052d0 con 0x7f03700ac4d0 2024-01-20T11:23:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 10 monclient: dump: 2024-01-20T11:23:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 10 monclient: _finish_auth 0 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 1 -- 172.21.15.110:0/1376660198 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036c005450 con 0x7f03700ac4d0 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 1 -- 172.21.15.110:0/1376660198 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c007260 con 0x7f03700ac4d0 2024-01-20T11:23:58.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f037871d640 5 monclient: authenticate success, global_id 6171 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f037871d640 10 monclient: _renew_subs 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f037871d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03701d6f80 con 0x7f03700ac4d0 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0374cb6640 4 set_mon_vals no callback set 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f0366ffd640 10 monclient: dump: 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f037871d640 10 client.6171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f037871d640 10 monclient: _renew_subs 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f037871d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03701d2ce0 con 0x7f03700ac4d0 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.295+0000 7f037871d640 1 librados: init done 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.296+0000 7f037871d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.296+0000 7f037871d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.296+0000 7f037871d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.296+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0370075490 con 0x7f03700ac4d0 2024-01-20T11:23:58.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.314+0000 7f0366ffd640 1 -- 172.21.15.110:0/1376660198 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f036c0073c0 con 0x7f03700ac4d0 2024-01-20T11:23:58.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.315+0000 7f0366ffd640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f035406c890 0x7f035406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.315+0000 7f0375cb8640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f035406c890 0x7f035406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.315+0000 7f0375cb8640 10 monclient: get_auth_request con 0x7f035406c890 auth_method 0 2024-01-20T11:23:58.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.316+0000 7f0375cb8640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f035406c890 0x7f035406ed50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0370071a50 tx=0x7f0360009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:58.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.316+0000 7f0366ffd640 10 client.6171.objecter ms_handle_connect 0x7f035406c890 2024-01-20T11:23:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.550+0000 7f0366ffd640 1 -- 172.21.15.110:0/1376660198 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f036c007670 con 0x7f03700ac4d0 2024-01-20T11:23:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.550+0000 7f0366ffd640 10 client.6171.objecter ms_dispatch 0x7f0370000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.550+0000 7f0366ffd640 3 client.6171.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.550+0000 7f0366ffd640 3 client.6171.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.550+0000 7f0366ffd640 20 client.6171.objecter dump_active .. 0 homeless 2024-01-20T11:23:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.550+0000 7f0366ffd640 1 -- 172.21.15.110:0/1376660198 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f036c054160 con 0x7f03700ac4d0 2024-01-20T11:23:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.550+0000 7f0366ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.550+0000 7f0366ffd640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:58.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.749+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0370072250 con 0x7f035406c890 2024-01-20T11:23:58.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.753+0000 7f0366ffd640 1 -- 172.21.15.110:0/1376660198 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63936 (secure 0 0 0) 0x7f0370072250 con 0x7f035406c890 2024-01-20T11:23:58.753 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:58.753 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:58.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.754+0000 7f037871d640 10 librados: watch_flush enter 2024-01-20T11:23:58.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.754+0000 7f037871d640 10 librados: watch_flush exit 2024-01-20T11:23:58.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.754+0000 7f037871d640 20 client.6171.objecter shutdown clearing up homeless session... 2024-01-20T11:23:58.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 10 client.6171.objecter successfully canceled tick 2024-01-20T11:23:58.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f035406c890 msgr2=0x7f035406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f035406c890 0x7f035406ed50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0370071a50 tx=0x7f0360009040 comp rx=0 tx=0).stop 2024-01-20T11:23:58.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 10 monclient: shutdown 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 msgr2=0x7f037006ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 0x7f037006ff60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f036c00c3c0 tx=0x7f036c00c880 comp rx=0 tx=0).stop 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 shutdown_connections 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f035406c890 0x7f035406ed50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03701d6390 0x7f03701d6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03700b0ba0 0x7f0370070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 --2- 172.21.15.110:0/1376660198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03700ac4d0 0x7f037006ff60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 >> 172.21.15.110:0/1376660198 conn(0x7f03700b3e60 msgr2=0x7f03700aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 shutdown_connections 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 -- 172.21.15.110:0/1376660198 wait complete. 2024-01-20T11:23:58.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.755+0000 7f037871d640 1 librados: shutdown 2024-01-20T11:23:58.765 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-01-20T11:23:56.818431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002515"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830287+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:21:46.830287+0000","last_peered":"2024-01-20T11:21:46.830287+0000","last_clean":"2024-01-20T11:21:46.830287+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:21:46.830287+0000","last_undegraded":"2024-01-20T11:21:46.830287+0000","last_fullsized":"2024-01-20T11:21:46.830287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.829744+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:21:46.829744+0000","last_peered":"2024-01-20T11:21:46.829744+0000","last_clean":"2024-01-20T11:21:46.829744+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:21:46.829744+0000","last_undegraded":"2024-01-20T11:21:46.829744+0000","last_fullsized":"2024-01-20T11:21:46.829744+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:46.747466+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:21:46.747466+0000","last_peered":"2024-01-20T11:21:46.747466+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:21:46.747466+0000","last_undegraded":"2024-01-20T11:21:46.747466+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830036+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:21:46.830036+0000","last_peered":"2024-01-20T11:21:46.830036+0000","last_clean":"2024-01-20T11:21:46.830036+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:21:46.830036+0000","last_undegraded":"2024-01-20T11:21:46.830036+0000","last_fullsized":"2024-01-20T11:21:46.830036+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":6.391},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":8.7699999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":14.664},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":8.0920000000000005}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":13.113},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":7.4210000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":13.912000000000001},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":10.461}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":16.565999999999999},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":11.188000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":11.874000000000001},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":15.769}]}]},{"osd":7,"up_from":21,"seq":90194313324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":7.6470000000000002},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":6.8209999999999997}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":10.83},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":15.849}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":17.356000000000002},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":11.827}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":16.579000000000001},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":18.015999999999998}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":15.436},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":12.462999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":13.255000000000001},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":19.591999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313325,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":5.6230000000000002},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":6.7080000000000002}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":8.9939999999999998},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":15.225}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":10.499000000000001},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":11.723000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":12.657999999999999},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":16.148}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":9.9250000000000007},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":13.577999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":17.189},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":17.855}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444139,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":6.2439999999999998},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":8.7309999999999999}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":5.5579999999999998},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":9.8040000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":14.254},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":7.9470000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.913,"15min":16.091999999999999},"min":{"1min":9.5099999999999998,"5min":7.8200000000000003,"15min":7.3869999999999996},"max":{"1min":17.895,"5min":20.393000000000001,"15min":51.000999999999998},"last":15.289},{"interface":"front","average":{"1min":12.028,"5min":12.231999999999999,"15min":13.638},"min":{"1min":8.1460000000000008,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.597999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":10.532999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":11.214},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":11.853999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":13.944000000000001},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":16.164999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":4.1529999999999996},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":8.0259999999999998}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":9.0449999999999999},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":7.1790000000000003}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":12.189},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":10.005000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:40 2024","interfaces":[{"interface":"back","average":{"1min":9.6929999999999996,"5min":11.172000000000001,"15min":13},"min":{"1min":6.7290000000000001,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":14.914999999999999,"5min":18.899999999999999,"15min":25.541},"last":15.234999999999999},{"interface":"front","average":{"1min":12.557,"5min":13.332000000000001,"15min":14.596},"min":{"1min":7.0490000000000004,"5min":6.75,"15min":6.75},"max":{"1min":16.565000000000001,"5min":22.280000000000001,"15min":30.204999999999998},"last":10.603}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":16.689},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":11.602}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":12.952999999999999},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":13.712}]}]},{"osd":0,"up_from":17,"seq":73014444140,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":6.2969999999999997},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":10.323}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":8.5540000000000003},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":16.382999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.776,"5min":12.148999999999999,"15min":21.678999999999998},"min":{"1min":7.8280000000000003,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":18.335999999999999,"5min":27.292000000000002,"15min":358.738},"last":15.526999999999999},{"interface":"front","average":{"1min":9.9199999999999999,"5min":10.561999999999999,"15min":12.034000000000001},"min":{"1min":5.5839999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":15.347,"5min":20.672000000000001,"15min":21.004999999999999},"last":10.933}]},{"osd":4,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":12.465,"15min":23.204000000000001},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":17.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":12.182},{"interface":"front","average":{"1min":11.621,"5min":10.532999999999999,"15min":20.757000000000001},"min":{"1min":7.0620000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":16.713000000000001,"5min":21.334,"15min":372.70800000000003},"last":9.6110000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":17.263000000000002},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":12.763999999999999}]},{"osd":7,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":15.414,"5min":15.087,"15min":16.367999999999999},"min":{"1min":8.9819999999999993,"5min":8.625,"15min":8.625},"max":{"1min":22.588999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":13.308999999999999},{"interface":"front","average":{"1min":12.353,"5min":11.938000000000001,"15min":12.262},"min":{"1min":8.0250000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":19.954000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":11.535}]}]},{"osd":6,"up_from":15,"seq":64424509549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":6.1609999999999996},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":5.4189999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.2439999999999998},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":14.52}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":7.7000000000000002},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":8.8290000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.692},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.565}]},{"osd":4,"last update":"Sat Jan 20 11:23:41 2024","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":13.513999999999999,"15min":15.654},"min":{"1min":8.3879999999999999,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":21.539000000000001,"5min":21.539000000000001,"15min":70.254999999999995},"last":9.4730000000000008},{"interface":"front","average":{"1min":12.259,"5min":12.333,"15min":15.17},"min":{"1min":7.3079999999999998,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":17.798999999999999,"5min":18.061,"15min":72.670000000000002},"last":12.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":10.106},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":15.353999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:58.767 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:23:58.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.867+0000 7fd1f2589640 10 monclient: get_monmap_and_config 2024-01-20T11:23:58.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.867+0000 7fd1f2589640 10 monclient: build_initial_monmap 2024-01-20T11:23:58.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.867+0000 7fd1f2589640 10 monclient: monmap: 2024-01-20T11:23:58.867 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:58.868280+0000 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:58.868280+0000 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:58.868 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.868+0000 7fd1f2589640 1 Processor -- start 2024-01-20T11:23:58.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.868+0000 7fd1f2589640 1 -- start start 2024-01-20T11:23:58.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.868+0000 7fd1f2589640 10 monclient: init 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.868+0000 7fd1f2589640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.868+0000 7fd1f2589640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd1ec0b0c00 0x7fd1ec0accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 10 monclient(hunting): picked mon.noname-a con 0x7fd1ec0b0c00 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1ec13d0e0 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec13d520 0x7fd1ec143910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 10 monclient(hunting): picked mon.noname-b con 0x7fd1ec13d520 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1ec06dcc0 con 0x7fd1ec13d0e0 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ec13d9b0 con 0x7fd1ec0b0c00 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec0ad450 con 0x7fd1ec13d520 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f2589640 10 monclient(hunting): authenticate will time out at 1882.219244s 2024-01-20T11:23:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1ebfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd1ec0b0c00 0x7fd1ec0accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f0b26640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec13d520 0x7fd1ec143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1eb7fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd1ec13d0e0 0x7fd1ec143200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:51008/0 (socket says 172.21.15.110:51008) 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1eb7fe640 1 -- 172.21.15.110:0/1645185605 learned_addr learned my addr 172.21.15.110:0/1645185605 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1ebfff640 10 monclient(hunting): get_auth_request con 0x7fd1ec0b0c00 auth_method 0 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1ebfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.869+0000 7fd1f0b26640 10 monclient(hunting): get_auth_request con 0x7fd1ec13d520 auth_method 0 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.870+0000 7fd1f0b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.870+0000 7fd1f0b26640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.870+0000 7fd1f0b26640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.870+0000 7fd1ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.870+0000 7fd1ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.870+0000 7fd1ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.870+0000 7fd1f0b26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.870+0000 7fd1f0b26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.870+0000 7fd1f0b26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1ebfff640 10 monclient(hunting): handle_auth_done global_id 6174 payload 274 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1ebfff640 1 -- 172.21.15.110:0/1645185605 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec13d520 msgr2=0x7fd1ec143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1ebfff640 1 --2- 172.21.15.110:0/1645185605 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec13d520 0x7fd1ec143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1ebfff640 1 -- 172.21.15.110:0/1645185605 >> v1:172.21.15.134:6789/0 conn(0x7fd1ec13d0e0 legacy=0x7fd1ec143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1ebfff640 10 monclient: _finish_hunting 0 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1ebfff640 1 monclient: found mon.noname-a 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1ebfff640 1 -- 172.21.15.110:0/1645185605 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ec0ad8e0 con 0x7fd1ec0b0c00 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1eaffd640 1 -- 172.21.15.110:0/1645185605 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846761522 0 0) 0x7fd1ec06dcc0 con 0x7fd1ec13d0e0 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1eaffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.872+0000 7fd1f0b26640 1 --2- 172.21.15.110:0/1645185605 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec13d520 0x7fd1ec143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.874+0000 7fd1ebfff640 1 --2- 172.21.15.110:0/1645185605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec0b0c00 0x7fd1ec0accf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0097a0 tx=0x7fd1dc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db102250945ea94 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:58.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 1 -- 172.21.15.110:0/1645185605 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0316f0 con 0x7fd1ec0b0c00 2024-01-20T11:23:58.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:58.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:58.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 10 monclient: dump: 2024-01-20T11:23:58.877 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:58.877 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.877 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 10 monclient: _finish_auth 0 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 10 monclient: _check_auth_tickets 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 1 -- 172.21.15.110:0/1645185605 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc031850 con 0x7fd1ec0b0c00 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 1 -- 172.21.15.110:0/1645185605 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc031be0 con 0x7fd1ec0b0c00 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1eaffd640 10 monclient: dump: 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1f2589640 5 monclient: authenticate success, global_id 6174 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1f2589640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.877+0000 7fd1f2589640 4 set_mon_vals no callback set 2024-01-20T11:23:58.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 10 monclient: shutdown 2024-01-20T11:23:58.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:58.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 1 -- 172.21.15.110:0/1645185605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec0b0c00 msgr2=0x7fd1ec0accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 1 --2- 172.21.15.110:0/1645185605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec0b0c00 0x7fd1ec0accf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0097a0 tx=0x7fd1dc02b790 comp rx=0 tx=0).stop 2024-01-20T11:23:58.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 1 -- 172.21.15.110:0/1645185605 shutdown_connections 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 1 --2- 172.21.15.110:0/1645185605 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec13d520 0x7fd1ec143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 1 --2- 172.21.15.110:0/1645185605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec0b0c00 0x7fd1ec0accf0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 1 -- 172.21.15.110:0/1645185605 >> 172.21.15.110:0/1645185605 conn(0x7fd1ec0b3e60 msgr2=0x7fd1ec0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 1 -- 172.21.15.110:0/1645185605 shutdown_connections 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.878+0000 7fd1f2589640 1 -- 172.21.15.110:0/1645185605 wait complete. 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.879+0000 7fd1f2589640 10 monclient: build_initial_monmap 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.879+0000 7fd1f2589640 10 monclient: monmap: 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.879+0000 7fd1f2589640 1 librados: starting msgr at 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.879+0000 7fd1f2589640 1 librados: starting objecter 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.879+0000 7fd1f2589640 1 Processor -- start 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.879+0000 7fd1f2589640 1 -- start start 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.879+0000 7fd1f2589640 1 librados: setting wanted keys 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.879+0000 7fd1f2589640 1 librados: calling monclient init 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.879+0000 7fd1f2589640 10 monclient: init 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec0b0c00 0x7fd1ec138560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient(hunting): picked mon.noname-c con 0x7fd1ec0b0c00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1ec13d0e0 0x7fd1ec1daab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient(hunting): picked mon.noname-b con 0x7fd1ec13d0e0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:58.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec13d520 0x7fd1ec1daff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient(hunting): picked mon.noname-a con 0x7fd1ec13d520 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec149d60 con 0x7fd1ec13d520 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec06eca0 con 0x7fd1ec13d0e0 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec06ef10 con 0x7fd1ec0b0c00 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f2589640 10 monclient(hunting): authenticate will time out at 1882.230600s 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.880+0000 7fd1f0b26640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec13d520 0x7fd1ec1daff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1ebfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec0b0c00 0x7fd1ec138560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1f0b26640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec13d520 0x7fd1ec1daff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54234/0 (socket says 172.21.15.110:54234) 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1f0b26640 1 -- 172.21.15.110:0/1819817197 learned_addr learned my addr 172.21.15.110:0/1819817197 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1f0b26640 10 monclient(hunting): get_auth_request con 0x7fd1ec13d520 auth_method 0 2024-01-20T11:23:58.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1f0b26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1f0b26640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1f0b26640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1ebfff640 10 monclient(hunting): get_auth_request con 0x7fd1ec0b0c00 auth_method 0 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1ebfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1eb7fe640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1ec13d0e0 0x7fd1ec1daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1f0b26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1f0b26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1f0b26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1eb7fe640 10 monclient(hunting): get_auth_request con 0x7fd1ec13d0e0 auth_method 0 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1eb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:58.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:58.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:58.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:58.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.881+0000 7fd1ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:58.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.882+0000 7fd1eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:58.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.882+0000 7fd1eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:58.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.882+0000 7fd1eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 10 monclient(hunting): handle_auth_done global_id 6177 payload 824 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 1 -- 172.21.15.110:0/1819817197 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec0b0c00 msgr2=0x7fd1ec138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec0b0c00 0x7fd1ec138560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 1 -- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1ec13d0e0 msgr2=0x7fd1ec1daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1ec13d0e0 0x7fd1ec1daab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 10 monclient: _finish_hunting 0 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 1 monclient: found mon.noname-a 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.884+0000 7fd1f0b26640 1 -- 172.21.15.110:0/1819817197 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e4009a70 con 0x7fd1ec13d520 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.887+0000 7fd1f0b26640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec13d520 0x7fd1ec1daff0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4009e60 tx=0x7fd1e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:58.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.887+0000 7fd1e8ff9640 10 client.?.objecter ms_handle_connect 0x7fd1ec13d520 2024-01-20T11:23:58.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.887+0000 7fd1e8ff9640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 1 -- 172.21.15.110:0/1819817197 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4009cd0 con 0x7fd1ec13d520 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 10 monclient: dump: 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:58.891 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 10 monclient: _finish_auth 0 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 10 monclient: _check_auth_tickets 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 1 -- 172.21.15.110:0/1819817197 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e400f040 con 0x7fd1ec13d520 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1f2589640 5 monclient: authenticate success, global_id 6177 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1f2589640 10 monclient: _renew_subs 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 1 -- 172.21.15.110:0/1819817197 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4003990 con 0x7fd1ec13d520 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1f2589640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1dc009400 con 0x7fd1ec13d520 2024-01-20T11:23:58.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1e8ff9640 10 monclient: dump: 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1ea7fc640 4 set_mon_vals no callback set 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1f2589640 10 client.6177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:58.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1f2589640 10 monclient: _renew_subs 2024-01-20T11:23:58.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1f2589640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec1db530 con 0x7fd1ec13d520 2024-01-20T11:23:58.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.890+0000 7fd1f2589640 1 librados: init done 2024-01-20T11:23:58.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.891+0000 7fd1f2589640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:58.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.891+0000 7fd1f2589640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:58.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.891+0000 7fd1f2589640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:58.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:58.891+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ec1db750 con 0x7fd1ec13d520 2024-01-20T11:23:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.130+0000 7fd1e8ff9640 1 -- 172.21.15.110:0/1819817197 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd1e4014480 con 0x7fd1ec13d520 2024-01-20T11:23:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.130+0000 7fd1e8ff9640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1d806c8b0 0x7fd1d806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.131+0000 7fd1ebfff640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1d806c8b0 0x7fd1d806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.131+0000 7fd1ebfff640 10 monclient: get_auth_request con 0x7fd1d806c8b0 auth_method 0 2024-01-20T11:23:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.131+0000 7fd1ebfff640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1d806c8b0 0x7fd1d806ed70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc036000 tx=0x7fd1dc008030 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.131+0000 7fd1e8ff9640 10 client.6177.objecter ms_handle_connect 0x7fd1d806c8b0 2024-01-20T11:23:59.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.142+0000 7fd1e8ff9640 1 -- 172.21.15.110:0/1819817197 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd1e40594c0 con 0x7fd1ec13d520 2024-01-20T11:23:59.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.142+0000 7fd1e8ff9640 10 client.6177.objecter ms_dispatch 0x7fd1ec000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:59.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.142+0000 7fd1e8ff9640 3 client.6177.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:59.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.142+0000 7fd1e8ff9640 3 client.6177.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:59.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.142+0000 7fd1e8ff9640 20 client.6177.objecter dump_active .. 0 homeless 2024-01-20T11:23:59.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.142+0000 7fd1e8ff9640 1 -- 172.21.15.110:0/1819817197 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd1e4055800 con 0x7fd1ec13d520 2024-01-20T11:23:59.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.142+0000 7fd1e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:59.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.142+0000 7fd1e8ff9640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:59.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.313+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd1ec14a550 con 0x7fd1d806c8b0 2024-01-20T11:23:59.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.316+0000 7fd1e8ff9640 1 -- 172.21.15.110:0/1819817197 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63983 (secure 0 0 0) 0x7fd1dc042070 con 0x7fd1d806c8b0 2024-01-20T11:23:59.317 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:23:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 10 librados: watch_flush enter 2024-01-20T11:23:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 10 librados: watch_flush exit 2024-01-20T11:23:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 20 client.6177.objecter shutdown clearing up homeless session... 2024-01-20T11:23:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 10 client.6177.objecter successfully canceled tick 2024-01-20T11:23:59.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1d806c8b0 msgr2=0x7fd1d806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1d806c8b0 0x7fd1d806ed70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc036000 tx=0x7fd1dc008030 comp rx=0 tx=0).stop 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 10 monclient: shutdown 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec13d520 msgr2=0x7fd1ec1daff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.318+0000 7fd1f2589640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec13d520 0x7fd1ec1daff0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4009e60 tx=0x7fd1e400c490 comp rx=0 tx=0).stop 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.319+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 shutdown_connections 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.319+0000 7fd1f2589640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fd1d806c8b0 0x7fd1d806ed70 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.319+0000 7fd1f2589640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1ec13d520 0x7fd1ec1daff0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.319+0000 7fd1f2589640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1ec13d0e0 0x7fd1ec1daab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.319+0000 7fd1f2589640 1 --2- 172.21.15.110:0/1819817197 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1ec0b0c00 0x7fd1ec138560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.319+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 >> 172.21.15.110:0/1819817197 conn(0x7fd1ec0b3e60 msgr2=0x7fd1ec0a95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.319+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 shutdown_connections 2024-01-20T11:23:59.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.319+0000 7fd1f2589640 1 -- 172.21.15.110:0/1819817197 wait complete. 2024-01-20T11:23:59.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.319+0000 7fd1f2589640 1 librados: shutdown 2024-01-20T11:23:59.329 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-01-20T11:23:58.818815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002470"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830287+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:21:46.830287+0000","last_peered":"2024-01-20T11:21:46.830287+0000","last_clean":"2024-01-20T11:21:46.830287+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:21:46.830287+0000","last_undegraded":"2024-01-20T11:21:46.830287+0000","last_fullsized":"2024-01-20T11:21:46.830287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.829744+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:21:46.829744+0000","last_peered":"2024-01-20T11:21:46.829744+0000","last_clean":"2024-01-20T11:21:46.829744+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:21:46.829744+0000","last_undegraded":"2024-01-20T11:21:46.829744+0000","last_fullsized":"2024-01-20T11:21:46.829744+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:46.747466+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:21:46.747466+0000","last_peered":"2024-01-20T11:21:46.747466+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:21:46.747466+0000","last_undegraded":"2024-01-20T11:21:46.747466+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830036+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:21:46.830036+0000","last_peered":"2024-01-20T11:21:46.830036+0000","last_clean":"2024-01-20T11:21:46.830036+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:21:46.830036+0000","last_undegraded":"2024-01-20T11:21:46.830036+0000","last_fullsized":"2024-01-20T11:21:46.830036+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280623,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.8570000000000002},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":4.7190000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":10.305},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":8.7840000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":11.678000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":14.603999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":16.123000000000001},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":9.5229999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":13.114000000000001},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":16.920000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":13.757},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":12.477}]}]},{"osd":7,"up_from":21,"seq":90194313325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":10.996},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":4.9930000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":8.0549999999999997},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.308999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":14.002000000000001},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":8.9670000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":11.744},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":15.308}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.657},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":9.6910000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":10.272},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":16.074999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313326,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":4.2199999999999998},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":9.6449999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":11.471},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":8.2479999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":14.143000000000001},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":12.970000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":15.053000000000001},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":8.9649999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":10.452999999999999},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":16.754000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":15.922000000000001},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":12.26}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.9100000000000001},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":11.252000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":11.859999999999999},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":15.792}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":15.01},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":8.6929999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.913,"15min":16.091999999999999},"min":{"1min":9.5099999999999998,"5min":7.8200000000000003,"15min":7.3869999999999996},"max":{"1min":17.895,"5min":20.393000000000001,"15min":51.000999999999998},"last":13.769},{"interface":"front","average":{"1min":12.028,"5min":12.231999999999999,"15min":13.638},"min":{"1min":8.1460000000000008,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.597999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":7.569}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":9.7780000000000005},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":6.9020000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":13.093999999999999},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":16.872}]}]},{"osd":1,"up_from":17,"seq":73014444141,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":4.5890000000000004},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":9.1059999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":9.4399999999999995},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.757999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":13.199999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":8.6709999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:23:40 2024","interfaces":[{"interface":"back","average":{"1min":9.6929999999999996,"5min":11.172000000000001,"15min":13},"min":{"1min":6.7290000000000001,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":14.914999999999999,"5min":18.899999999999999,"15min":25.541},"last":12.253},{"interface":"front","average":{"1min":12.557,"5min":13.332000000000001,"15min":14.596},"min":{"1min":7.0490000000000004,"5min":6.75,"15min":6.75},"max":{"1min":16.565000000000001,"5min":22.280000000000001,"15min":30.204999999999998},"last":11.541}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":15.151999999999999},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":14.272}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":16.667000000000002},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":15.972}]}]},{"osd":0,"up_from":17,"seq":73014444141,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":4.2770000000000001},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":9.0020000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":4.9749999999999996},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.6820000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.776,"5min":12.148999999999999,"15min":21.678999999999998},"min":{"1min":7.8280000000000003,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":18.335999999999999,"5min":27.292000000000002,"15min":358.738},"last":14.178000000000001},{"interface":"front","average":{"1min":9.9199999999999999,"5min":10.561999999999999,"15min":12.034000000000001},"min":{"1min":5.5839999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":15.347,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.6820000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":12.465,"15min":23.204000000000001},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":17.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":10.336},{"interface":"front","average":{"1min":11.621,"5min":10.532999999999999,"15min":20.757000000000001},"min":{"1min":7.0620000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":16.713000000000001,"5min":21.334,"15min":372.70800000000003},"last":8.3710000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":14.946},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":10.874000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":15.414,"5min":15.087,"15min":16.367999999999999},"min":{"1min":8.9819999999999993,"5min":8.625,"15min":8.625},"max":{"1min":22.588999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":11.433999999999999},{"interface":"front","average":{"1min":12.353,"5min":11.938000000000001,"15min":12.262},"min":{"1min":8.0250000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":19.954000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":13.247999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":6.1609999999999996},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":5.4189999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.2439999999999998},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":14.52}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":7.7000000000000002},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":8.8290000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.692},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.565}]},{"osd":4,"last update":"Sat Jan 20 11:23:41 2024","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":13.513999999999999,"15min":15.654},"min":{"1min":8.3879999999999999,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":21.539000000000001,"5min":21.539000000000001,"15min":70.254999999999995},"last":9.4730000000000008},{"interface":"front","average":{"1min":12.259,"5min":12.333,"15min":15.17},"min":{"1min":7.3079999999999998,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":17.798999999999999,"5min":18.061,"15min":72.670000000000002},"last":12.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":10.106},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":15.353999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:23:59.331 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-01-20T11:23:59.331 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:00.860676+0000', 'last_change': '2024-01-20T11:16:50.488485+0000', 'last_active': '2024-01-20T11:21:00.860676+0000', 'last_peered': '2024-01-20T11:21:00.860676+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.488485+0000', 'last_became_peered': '2024-01-20T11:16:50.488485+0000', 'last_unstale': '2024-01-20T11:21:00.860676+0000', 'last_undegraded': '2024-01-20T11:21:00.860676+0000', 'last_fullsized': '2024-01-20T11:16:50.409560+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-20T11:23:59.331 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-01-20T11:23:59.331 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 16, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:46.747466+0000', 'last_change': '2024-01-20T11:16:51.868790+0000', 'last_active': '2024-01-20T11:21:46.747466+0000', 'last_peered': '2024-01-20T11:21:46.747466+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:51.868790+0000', 'last_became_peered': '2024-01-20T11:16:51.868790+0000', 'last_unstale': '2024-01-20T11:21:46.747466+0000', 'last_undegraded': '2024-01-20T11:21:46.747466+0000', 'last_fullsized': '2024-01-20T11:16:51.790002+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-20T11:23:59.331 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-01-20T11:23:59.331 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 16, 'reported_epoch': 30, 'state': 'active+undersized', 'last_fresh': '2024-01-20T11:21:31.815524+0000', 'last_change': '2024-01-20T11:16:50.513192+0000', 'last_active': '2024-01-20T11:21:31.815524+0000', 'last_peered': '2024-01-20T11:21:31.815524+0000', 'last_clean': '2024-01-20T11:15:27.266904+0000', 'last_became_active': '2024-01-20T11:16:50.513192+0000', 'last_became_peered': '2024-01-20T11:16:50.513192+0000', 'last_unstale': '2024-01-20T11:21:31.815524+0000', 'last_undegraded': '2024-01-20T11:21:31.815524+0000', 'last_fullsized': '2024-01-20T11:16:50.490990+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 24, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:27.266904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-20T11:23:59.331 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-01-20T11:23:59.331 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'76", 'reported_seq': 131, 'reported_epoch': 30, 'state': 'active+undersized+degraded', 'last_fresh': '2024-01-20T11:21:18.077638+0000', 'last_change': '2024-01-20T11:16:51.800417+0000', 'last_active': '2024-01-20T11:21:18.077638+0000', 'last_peered': '2024-01-20T11:21:18.077638+0000', 'last_clean': '2024-01-20T11:15:30.540447+0000', 'last_became_active': '2024-01-20T11:16:51.800417+0000', 'last_became_peered': '2024-01-20T11:16:51.800417+0000', 'last_unstale': '2024-01-20T11:21:18.077638+0000', 'last_undegraded': '2024-01-20T11:16:51.780272+0000', 'last_fullsized': '2024-01-20T11:16:51.776630+0000', 'mapping_epoch': 27, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'last_clean_scrub_stamp': '2024-01-20T11:15:13.343599+0000', 'objects_scrubbed': 0, 'log_size': 76, 'log_dups_size': 0, 'ondisk_log_size': 76, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 459280, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 96, 'num_read_kb': 82, 'num_write': 113, 'num_write_kb': 1372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-20T11:23:59.331 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-20T11:23:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.429+0000 7fbb97501640 10 monclient: get_monmap_and_config 2024-01-20T11:23:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.429+0000 7fbb97501640 10 monclient: build_initial_monmap 2024-01-20T11:23:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.429+0000 7fbb97501640 10 monclient: monmap: 2024-01-20T11:23:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:23:59.430328+0000 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:23:59.430328+0000 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:23:59.430 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:59.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.430+0000 7fbb97501640 1 Processor -- start 2024-01-20T11:23:59.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.430+0000 7fbb97501640 1 -- start start 2024-01-20T11:23:59.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.430+0000 7fbb97501640 10 monclient: init 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb90071c60 0x7fbb90142180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 10 monclient(hunting): picked mon.noname-b con 0x7fbb90071c60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb900ac540 0x7fbb900ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 10 monclient(hunting): picked mon.noname-a con 0x7fbb900ac540 addr v2:172.21.15.110:3300/0 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb900b0930 addr v1:172.21.15.134:6789/0 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb900ad610 con 0x7fbb900b0930 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb90072040 con 0x7fbb900ac540 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb901426c0 con 0x7fbb90071c60 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb97501640 10 monclient(hunting): authenticate will time out at 1882.781294s 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb94a9c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb900ac540 0x7fbb900ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb9529d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb90071c60 0x7fbb90142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb95a9e640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbb900b0930 0x7fbb900af130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.110:45594/0 (socket says 172.21.15.110:45594) 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb95a9e640 1 -- 172.21.15.110:0/2435816118 learned_addr learned my addr 172.21.15.110:0/2435816118 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb94a9c640 10 monclient(hunting): get_auth_request con 0x7fbb900ac540 auth_method 0 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb94a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb94a9c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.431+0000 7fbb94a9c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb9529d640 10 monclient(hunting): get_auth_request con 0x7fbb90071c60 auth_method 0 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb9529d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb9529d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb9529d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb94a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb94a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:59.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb94a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:59.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb9529d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:59.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb9529d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:59.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.432+0000 7fbb9529d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb94a9c640 10 monclient(hunting): handle_auth_done global_id 6180 payload 274 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb94a9c640 1 -- 172.21.15.110:0/2435816118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb90071c60 msgr2=0x7fbb90142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb94a9c640 1 --2- 172.21.15.110:0/2435816118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb90071c60 0x7fbb90142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb94a9c640 1 -- 172.21.15.110:0/2435816118 >> v1:172.21.15.134:6789/0 conn(0x7fbb900b0930 legacy=0x7fbb900af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb94a9c640 10 monclient: _finish_hunting 0 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb94a9c640 1 monclient: found mon.noname-a 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb94a9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb94a9c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb94a9c640 1 -- 172.21.15.110:0/2435816118 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb900aba00 con 0x7fbb900ac540 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.434+0000 7fbb9529d640 1 --2- 172.21.15.110:0/2435816118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb90071c60 0x7fbb90142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-20T11:23:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.436+0000 7fbb94a9c640 1 --2- 172.21.15.110:0/2435816118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb900ac540 0x7fbb900ab4c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbb800097a0 tx=0x7fbb8002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbfac11a475b4d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:59.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 1 -- 172.21.15.110:0/2435816118 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8002d040 con 0x7fbb900ac540 2024-01-20T11:23:59.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:59.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 10 monclient: dump: 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 10 monclient: _finish_auth 0 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 10 monclient: _check_auth_tickets 2024-01-20T11:23:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 1 -- 172.21.15.110:0/2435816118 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb80031810 con 0x7fbb900ac540 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 1 -- 172.21.15.110:0/2435816118 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80031ba0 con 0x7fbb900ac540 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb97501640 5 monclient: authenticate success, global_id 6180 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb97501640 10 monclient: get_monmap_and_config success 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb97501640 4 set_mon_vals no callback set 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.439+0000 7fbb87fff640 10 monclient: dump: 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:59.442 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:59.442 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:59.442 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:59.442 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:59.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 10 monclient: shutdown 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 1 -- 172.21.15.110:0/2435816118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb900ac540 msgr2=0x7fbb900ab4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 1 --2- 172.21.15.110:0/2435816118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb900ac540 0x7fbb900ab4c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbb800097a0 tx=0x7fbb8002b8b0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 1 -- 172.21.15.110:0/2435816118 shutdown_connections 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 1 --2- 172.21.15.110:0/2435816118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb900ac540 0x7fbb900ab4c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 1 --2- 172.21.15.110:0/2435816118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb90071c60 0x7fbb90142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 1 -- 172.21.15.110:0/2435816118 >> 172.21.15.110:0/2435816118 conn(0x7fbb900b3e60 msgr2=0x7fbb900b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 1 -- 172.21.15.110:0/2435816118 shutdown_connections 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.440+0000 7fbb97501640 1 -- 172.21.15.110:0/2435816118 wait complete. 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.441+0000 7fbb97501640 10 monclient: build_initial_monmap 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.441+0000 7fbb97501640 10 monclient: monmap: 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:23:59.446 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 librados: starting msgr at 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 librados: starting objecter 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 Processor -- start 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 -- start start 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 librados: setting wanted keys 2024-01-20T11:23:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 librados: calling monclient init 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient: init 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient: _reopen_session rank -1 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb90071c60 0x7fbb901411b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient(hunting): picked mon.noname-a con 0x7fbb90071c60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb900ac540 0x7fbb9006df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient(hunting): picked mon.noname-b con 0x7fbb900ac540 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb900b0930 0x7fbb9006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient(hunting): picked mon.noname-c con 0x7fbb900b0930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90142130 con 0x7fbb90071c60 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9006f800 con 0x7fbb900ac540 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient(hunting): start opening mon connection 2024-01-20T11:23:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9006f9c0 con 0x7fbb900b0930 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient(hunting): _renew_subs 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.442+0000 7fbb97501640 10 monclient(hunting): authenticate will time out at 1882.792966s 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb9529d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb90071c60 0x7fbb901411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb95a9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb900b0930 0x7fbb9006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb94a9c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb900ac540 0x7fbb9006df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb9529d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb90071c60 0x7fbb901411b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51558/0 (socket says 172.21.15.110:51558) 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb9529d640 1 -- 172.21.15.110:0/200127980 learned_addr learned my addr 172.21.15.110:0/200127980 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb9529d640 10 monclient(hunting): get_auth_request con 0x7fbb90071c60 auth_method 0 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb9529d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb9529d640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb9529d640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb95a9e640 10 monclient(hunting): get_auth_request con 0x7fbb900b0930 auth_method 0 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb95a9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:59.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb95a9e640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb95a9e640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb94a9c640 10 monclient(hunting): get_auth_request con 0x7fbb900ac540 auth_method 0 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb94a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb94a9c640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.443+0000 7fbb94a9c640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.444+0000 7fbb9529d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.444+0000 7fbb9529d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.444+0000 7fbb9529d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.444+0000 7fbb95a9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.444+0000 7fbb95a9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.444+0000 7fbb95a9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.444+0000 7fbb94a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.444+0000 7fbb94a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:23:59.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.444+0000 7fbb94a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:23:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 10 monclient(hunting): handle_auth_done global_id 6183 payload 824 2024-01-20T11:23:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 1 -- 172.21.15.110:0/200127980 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb900b0930 msgr2=0x7fbb9006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb900b0930 0x7fbb9006d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 1 -- 172.21.15.110:0/200127980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb900ac540 msgr2=0x7fbb9006df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb900ac540 0x7fbb9006df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 10 monclient: _finish_hunting 0 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 1 monclient: found mon.noname-a 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.447+0000 7fbb9529d640 1 -- 172.21.15.110:0/200127980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c009400 con 0x7fbb90071c60 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.449+0000 7fbb9529d640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb90071c60 0x7fbb901411b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00e960 tx=0x7fbb8c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.449+0000 7fbb85ffb640 10 client.?.objecter ms_handle_connect 0x7fbb90071c60 2024-01-20T11:23:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.449+0000 7fbb85ffb640 10 client.?.objecter resend_mon_ops 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 1 -- 172.21.15.110:0/200127980 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c00ca00 con 0x7fbb90071c60 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 10 monclient: dump: 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:59.454 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 10 monclient: _finish_auth 0 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 10 monclient: _check_auth_tickets 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 1 -- 172.21.15.110:0/200127980 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c00cb60 con 0x7fbb90071c60 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb97501640 5 monclient: authenticate success, global_id 6183 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb97501640 10 monclient: _renew_subs 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb97501640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb80009400 con 0x7fbb90071c60 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 1 -- 172.21.15.110:0/200127980 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c013470 con 0x7fbb90071c60 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:23:59.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb85ffb640 10 monclient: dump: 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.452+0000 7fbb877fe640 4 set_mon_vals no callback set 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.453+0000 7fbb97501640 10 client.6183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:23:59.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.453+0000 7fbb97501640 10 monclient: _renew_subs 2024-01-20T11:23:59.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.453+0000 7fbb97501640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:59.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.453+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb90071000 con 0x7fbb90071c60 2024-01-20T11:23:59.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.453+0000 7fbb97501640 1 librados: init done 2024-01-20T11:23:59.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.454+0000 7fbb97501640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:59.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.454+0000 7fbb97501640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:59.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.454+0000 7fbb97501640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:59.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.454+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9006f250 con 0x7fbb90071c60 2024-01-20T11:23:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.699+0000 7fbb85ffb640 1 -- 172.21.15.110:0/200127980 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fbb8c0135d0 con 0x7fbb90071c60 2024-01-20T11:23:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.700+0000 7fbb85ffb640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb7406c8b0 0x7fbb7406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:23:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.700+0000 7fbb94a9c640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb7406c8b0 0x7fbb7406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:23:59.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.700+0000 7fbb94a9c640 10 monclient: get_auth_request con 0x7fbb7406c8b0 auth_method 0 2024-01-20T11:23:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.701+0000 7fbb85ffb640 10 client.6183.objecter ms_handle_connect 0x7fbb7406c8b0 2024-01-20T11:23:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.701+0000 7fbb94a9c640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb7406c8b0 0x7fbb7406ed70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbb80036000 tx=0x7fbb8003c040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:23:59.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.714+0000 7fbb85ffb640 1 -- 172.21.15.110:0/200127980 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbb8c058c60 con 0x7fbb90071c60 2024-01-20T11:23:59.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.714+0000 7fbb85ffb640 10 client.6183.objecter ms_dispatch 0x7fbb90000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:23:59.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.714+0000 7fbb85ffb640 3 client.6183.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:23:59.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.714+0000 7fbb85ffb640 3 client.6183.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:23:59.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.714+0000 7fbb85ffb640 20 client.6183.objecter dump_active .. 0 homeless 2024-01-20T11:23:59.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.714+0000 7fbb85ffb640 1 -- 172.21.15.110:0/200127980 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fbb8c054fa0 con 0x7fbb90071c60 2024-01-20T11:23:59.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.714+0000 7fbb85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:23:59.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.714+0000 7fbb85ffb640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:23:59.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.969+0000 7fbb97501640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-01-20T11:23:59.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.969+0000 7fbb97501640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:59.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.969+0000 7fbb97501640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-20T11:23:59.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.969+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbb90075490 con 0x7fbb90071c60 2024-01-20T11:23:59.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.995+0000 7fbb85ffb640 1 -- 172.21.15.110:0/200127980 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1283 (secure 0 0 0) 0x7fbb8c021520 con 0x7fbb90071c60 2024-01-20T11:23:59.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.995+0000 7fbb85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-01-20T11:23:59.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.995+0000 7fbb85ffb640 10 monclient: _finish_command 2 = system:0 2024-01-20T11:23:59.995 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:23:59.995 INFO:teuthology.orchestra.run.smithi110.stdout:{"fsid":"16ed1bef-072b-4816-8684-5d140f76194c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 4 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":574,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":8,"num_up_osds":7,"osd_up_since":1705749409,"num_in_osds":8,"osd_in_since":1705749270,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+undersized","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":459280,"bytes_used":224636928,"bytes_avail":772869476352,"bytes_total":773094113280,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-01-20T11:18:32.747765+0000","services":{}},"progress_events":{}} 2024-01-20T11:23:59.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 10 librados: watch_flush enter 2024-01-20T11:23:59.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 10 librados: watch_flush exit 2024-01-20T11:23:59.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 20 client.6183.objecter shutdown clearing up homeless session... 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 10 client.6183.objecter successfully canceled tick 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb7406c8b0 msgr2=0x7fbb7406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb7406c8b0 0x7fbb7406ed70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbb80036000 tx=0x7fbb8003c040 comp rx=0 tx=0).stop 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 10 monclient: shutdown 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb90071c60 msgr2=0x7fbb901411b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb90071c60 0x7fbb901411b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00e960 tx=0x7fbb8c00ee20 comp rx=0 tx=0).stop 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 shutdown_connections 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7fbb7406c8b0 0x7fbb7406ed70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb900b0930 0x7fbb9006d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb900ac540 0x7fbb9006df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 --2- 172.21.15.110:0/200127980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb90071c60 0x7fbb901411b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 >> 172.21.15.110:0/200127980 conn(0x7fbb900b3e60 msgr2=0x7fbb900a9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:23:59.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 shutdown_connections 2024-01-20T11:23:59.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 -- 172.21.15.110:0/200127980 wait complete. 2024-01-20T11:23:59.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:23:59.997+0000 7fbb97501640 1 librados: shutdown 2024-01-20T11:24:00.008 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-01-20T11:24:00.008 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:24:00.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.106+0000 7f25f2f7d640 10 monclient: get_monmap_and_config 2024-01-20T11:24:00.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.106+0000 7f25f2f7d640 10 monclient: build_initial_monmap 2024-01-20T11:24:00.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.106+0000 7f25f2f7d640 10 monclient: monmap: 2024-01-20T11:24:00.106 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:24:00.106 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:24:00.107277+0000 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:24:00.107277+0000 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:24:00.107 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 1 Processor -- start 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 1 -- start start 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 10 monclient: init 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 10 monclient: _reopen_session rank -1 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25ec06a860 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f25ec0af730 0x7f25ec1455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 10 monclient(hunting): picked mon.noname-c con 0x7f25ec0af730 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f25ec145af0 0x7f25ec14c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:00.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 10 monclient(hunting): picked mon.noname-a con 0x7f25ec145af0 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.107+0000 7f25f2f7d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec0afc50 con 0x7f25ec06a860 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f2f7d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f2f7d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec06e450 con 0x7f25ec145af0 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f2f7d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f2f7d640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec0b0170 con 0x7f25ec0af730 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f2f7d640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f2f7d640 10 monclient(hunting): authenticate will time out at 1883.458200s 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f151a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f25ec145af0 0x7f25ec14c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f151a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f25ec145af0 0x7f25ec14c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51566/0 (socket says 172.21.15.110:51566) 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f151a640 1 -- 172.21.15.110:0/11292166 learned_addr learned my addr 172.21.15.110:0/11292166 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f151a640 10 monclient(hunting): get_auth_request con 0x7f25ec145af0 auth_method 0 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f151a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f151a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25f151a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:24:00.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.108+0000 7f25ebfff640 1 --2- 172.21.15.110:0/11292166 >> v2:172.21.15.134:3300/0 conn(0x7f25ec0af730 0x7f25ec1455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:24:00.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.109+0000 7f25ebfff640 10 monclient(hunting): get_auth_request con 0x7f25ec0af730 auth_method 0 2024-01-20T11:24:00.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.109+0000 7f25ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:24:00.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.109+0000 7f25ebfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:24:00.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.109+0000 7f25ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:24:00.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.109+0000 7f25f151a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:24:00.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.109+0000 7f25f151a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:24:00.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.109+0000 7f25f151a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.110+0000 7f25ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.110+0000 7f25ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.110+0000 7f25ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.110+0000 7f25f151a640 10 monclient(hunting): handle_auth_done global_id 6186 payload 274 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.110+0000 7f25f151a640 1 -- 172.21.15.110:0/11292166 >> v2:172.21.15.134:3300/0 conn(0x7f25ec0af730 msgr2=0x7f25ec1455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.110+0000 7f25f151a640 1 --2- 172.21.15.110:0/11292166 >> v2:172.21.15.134:3300/0 conn(0x7f25ec0af730 0x7f25ec1455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.111+0000 7f25f151a640 1 -- 172.21.15.110:0/11292166 >> v1:172.21.15.134:6789/0 conn(0x7f25ec06a860 legacy=0x7f25ec13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.111+0000 7f25f151a640 10 monclient: _finish_hunting 0 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.111+0000 7f25f151a640 1 monclient: found mon.noname-a 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.111+0000 7f25f151a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.111+0000 7f25f151a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-20T11:24:00.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.111+0000 7f25f151a640 1 -- 172.21.15.110:0/11292166 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec13fb60 con 0x7f25ec145af0 2024-01-20T11:24:00.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.112+0000 7f25f151a640 1 --2- 172.21.15.110:0/11292166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25ec145af0 0x7f25ec14c940 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0b57a0 tx=0x7f25e402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2075c25d24806a7a server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 1 -- 172.21.15.110:0/11292166 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e402c9f0 con 0x7f25ec145af0 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 10 monclient: dump: 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:24:00.116 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 10 monclient: _finish_auth 0 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 10 monclient: _check_auth_tickets 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 1 -- 172.21.15.110:0/11292166 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e402cb50 con 0x7f25ec145af0 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25f2f7d640 5 monclient: authenticate success, global_id 6186 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25f2f7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25f2f7d640 10 monclient: get_monmap_and_config success 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 1 -- 172.21.15.110:0/11292166 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e40343f0 con 0x7f25ec145af0 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:24:00.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25f2f7d640 4 set_mon_vals no callback set 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.116+0000 7f25eb7fe640 10 monclient: dump: 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:24:00.118 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:24:00.119 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:24:00.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 10 monclient: shutdown 2024-01-20T11:24:00.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 1 -- 172.21.15.110:0/11292166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25ec145af0 msgr2=0x7f25ec14c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 1 --2- 172.21.15.110:0/11292166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25ec145af0 0x7f25ec14c940 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0b57a0 tx=0x7f25e402be50 comp rx=0 tx=0).stop 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 1 -- 172.21.15.110:0/11292166 shutdown_connections 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 1 --2- 172.21.15.110:0/11292166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25ec145af0 0x7f25ec14c940 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 1 --2- 172.21.15.110:0/11292166 >> v2:172.21.15.134:3300/0 conn(0x7f25ec0af730 0x7f25ec1455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 1 -- 172.21.15.110:0/11292166 >> 172.21.15.110:0/11292166 conn(0x7f25ec0b47d0 msgr2=0x7f25ec0b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 1 -- 172.21.15.110:0/11292166 shutdown_connections 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 1 -- 172.21.15.110:0/11292166 wait complete. 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 10 monclient: build_initial_monmap 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.117+0000 7f25f2f7d640 10 monclient: monmap: 2024-01-20T11:24:00.123 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.noname-b 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.118+0000 7f25f2f7d640 1 librados: starting msgr at 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.118+0000 7f25f2f7d640 1 librados: starting objecter 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.118+0000 7f25f2f7d640 1 Processor -- start 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.118+0000 7f25f2f7d640 1 -- start start 2024-01-20T11:24:00.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.118+0000 7f25f2f7d640 1 librados: setting wanted keys 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.118+0000 7f25f2f7d640 1 librados: calling monclient init 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.118+0000 7f25f2f7d640 10 monclient: init 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient: _reopen_session rank -1 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25ec06a860 0x7f25ec13be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient(hunting): picked mon.noname-c con 0x7f25ec06a860 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25ec0af730 0x7f25ec1d7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient(hunting): picked mon.noname-a con 0x7f25ec0af730 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25ec145af0 0x7f25ec1d7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient(hunting): picked mon.noname-b con 0x7f25ec145af0 addr [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0715b0 con 0x7f25ec0af730 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec1d7bb0 con 0x7f25ec145af0 2024-01-20T11:24:00.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec140c50 con 0x7f25ec06a860 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f2f7d640 10 monclient(hunting): authenticate will time out at 1883.469283s 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25ebfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25ec0af730 0x7f25ec1d7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f151a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25ec145af0 0x7f25ec1d7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f151a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25ec145af0 0x7f25ec1d7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.110:54636/0 (socket says 172.21.15.110:54636) 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f151a640 1 -- 172.21.15.110:0/1945277908 learned_addr learned my addr 172.21.15.110:0/1945277908 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f151a640 10 monclient(hunting): get_auth_request con 0x7f25ec145af0 auth_method 0 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f151a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f151a640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f151a640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25ebfff640 10 monclient(hunting): get_auth_request con 0x7f25ec0af730 auth_method 0 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25ebfff640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:24:00.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.119+0000 7f25f0d19640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25ec06a860 0x7f25ec13be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f0d19640 10 monclient(hunting): get_auth_request con 0x7f25ec06a860 auth_method 0 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f0d19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f0d19640 10 monclient(hunting): _init_auth method 2 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f0d19640 10 monclient(hunting): _init_auth creating new auth 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f151a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f151a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f151a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f0d19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f0d19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.120+0000 7f25f0d19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-20T11:24:00.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 10 monclient(hunting): handle_auth_done global_id 6301 payload 824 2024-01-20T11:24:00.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 1 -- 172.21.15.110:0/1945277908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25ec06a860 msgr2=0x7f25ec13be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:24:00.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25ec06a860 0x7f25ec13be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:00.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 1 -- 172.21.15.110:0/1945277908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25ec0af730 msgr2=0x7f25ec1d7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:24:00.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25ec0af730 0x7f25ec1d7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:00.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 10 monclient: _finish_hunting 0 2024-01-20T11:24:00.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 1 monclient: found mon.noname-b 2024-01-20T11:24:00.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-20T11:24:00.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.134:3300/0 2024-01-20T11:24:00.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.122+0000 7f25f151a640 1 -- 172.21.15.110:0/1945277908 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e4009b30 con 0x7f25ec145af0 2024-01-20T11:24:00.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.124+0000 7f25f151a640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25ec145af0 0x7f25ec1d7670 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f25e4036040 tx=0x7f25e4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.124+0000 7f25e97fa640 10 client.?.objecter ms_handle_connect 0x7f25ec145af0 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.124+0000 7f25e97fa640 10 client.?.objecter resend_mon_ops 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 1 -- 172.21.15.110:0/1945277908 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e403e450 con 0x7f25ec145af0 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 10 monclient: dump: 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:24:00.130 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 10 monclient: _finish_auth 0 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 10 monclient: _check_auth_tickets 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 1 -- 172.21.15.110:0/1945277908 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e403e5d0 con 0x7f25ec145af0 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 1 -- 172.21.15.110:0/1945277908 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e403e980 con 0x7f25ec145af0 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25f2f7d640 5 monclient: authenticate success, global_id 6301 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25f2f7d640 10 monclient: _renew_subs 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25f2f7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec0719d0 con 0x7f25ec145af0 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25eaffd640 4 set_mon_vals no callback set 2024-01-20T11:24:00.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25e97fa640 10 monclient: dump: 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:13:38.005649+0000 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:13:38.005649+0000 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.b 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25f2f7d640 10 client.6301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-20T11:24:00.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25f2f7d640 10 monclient: _renew_subs 2024-01-20T11:24:00.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25f2f7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:24:00.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec071200 con 0x7f25ec145af0 2024-01-20T11:24:00.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.128+0000 7f25f2f7d640 1 librados: init done 2024-01-20T11:24:00.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.129+0000 7f25f2f7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-20T11:24:00.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.129+0000 7f25f2f7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:24:00.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.129+0000 7f25f2f7d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.134:3300/0 2024-01-20T11:24:00.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.129+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec140ac0 con 0x7f25ec145af0 2024-01-20T11:24:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.366+0000 7f25e97fa640 1 -- 172.21.15.110:0/1945277908 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f25e403eae0 con 0x7f25ec145af0 2024-01-20T11:24:00.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.366+0000 7f25e97fa640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f25d006c8b0 0x7f25d006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.367+0000 7f25f0d19640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f25d006c8b0 0x7f25d006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-20T11:24:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.367+0000 7f25f0d19640 10 monclient: get_auth_request con 0x7f25d006c8b0 auth_method 0 2024-01-20T11:24:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.367+0000 7f25f0d19640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f25d006c8b0 0x7f25d006ed70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f25e00016e0 tx=0x7f25e0009040 comp rx=0 tx=0).ready entity=mgr.4164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-20T11:24:00.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.367+0000 7f25e97fa640 10 client.6301.objecter ms_handle_connect 0x7f25d006c8b0 2024-01-20T11:24:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.369+0000 7f25e97fa640 1 -- 172.21.15.110:0/1945277908 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f25e407ba40 con 0x7f25ec145af0 2024-01-20T11:24:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.369+0000 7f25e97fa640 10 client.6301.objecter ms_dispatch 0x7f25ec000b90 osd_map(30..30 src has 1..30) v4 2024-01-20T11:24:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.369+0000 7f25e97fa640 3 client.6301.objecter handle_osd_map got epochs [30,30] > 0 2024-01-20T11:24:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.369+0000 7f25e97fa640 3 client.6301.objecter handle_osd_map decoding full epoch 30 2024-01-20T11:24:00.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.369+0000 7f25e97fa640 20 client.6301.objecter dump_active .. 0 homeless 2024-01-20T11:24:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.379+0000 7f25e97fa640 1 -- 172.21.15.110:0/1945277908 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f25e4077d80 con 0x7f25ec145af0 2024-01-20T11:24:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.379+0000 7f25e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-20T11:24:00.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.379+0000 7f25e97fa640 10 monclient: _finish_command 1 = system:0 2024-01-20T11:24:00.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.555+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 --> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25ec071d10 con 0x7f25d006c8b0 2024-01-20T11:24:00.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.558+0000 7f25e97fa640 1 -- 172.21.15.110:0/1945277908 <== mgr.4164 v2:172.21.15.134:6816/813062206 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+63983 (secure 0 0 0) 0x7f25ec071d10 con 0x7f25d006c8b0 2024-01-20T11:24:00.558 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-20T11:24:00.559 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-20T11:24:00.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 10 librados: watch_flush enter 2024-01-20T11:24:00.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 10 librados: watch_flush exit 2024-01-20T11:24:00.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 20 client.6301.objecter shutdown clearing up homeless session... 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 10 client.6301.objecter successfully canceled tick 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f25d006c8b0 msgr2=0x7f25d006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f25d006c8b0 0x7f25d006ed70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f25e00016e0 tx=0x7f25e0009040 comp rx=0 tx=0).stop 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 10 monclient: shutdown 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25ec145af0 msgr2=0x7f25ec1d7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25ec145af0 0x7f25ec1d7670 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f25e4036040 tx=0x7f25e4004a40 comp rx=0 tx=0).stop 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 shutdown_connections 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:6816/813062206,v1:172.21.15.134:6817/813062206] conn(0x7f25d006c8b0 0x7f25d006ed70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25ec145af0 0x7f25ec1d7670 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25ec0af730 0x7f25ec1d7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 --2- 172.21.15.110:0/1945277908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25ec06a860 0x7f25ec13be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 >> 172.21.15.110:0/1945277908 conn(0x7f25ec0b47d0 msgr2=0x7f25ec0b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-20T11:24:00.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 shutdown_connections 2024-01-20T11:24:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 -- 172.21.15.110:0/1945277908 wait complete. 2024-01-20T11:24:00.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:00.560+0000 7f25f2f7d640 1 librados: shutdown 2024-01-20T11:24:00.571 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-01-20T11:23:58.818815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219372,"kb_used_data":3000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755348,"statfs":{"total":773094113280,"available":772869476352,"internally_reserved":0,"allocated":3072000,"data_stored":1603962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002470"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:51.753038+0000","last_change":"2024-01-20T11:15:29.252292+0000","last_active":"2024-01-20T11:19:51.753038+0000","last_peered":"2024-01-20T11:19:51.753038+0000","last_clean":"2024-01-20T11:19:51.753038+0000","last_became_active":"2024-01-20T11:15:29.242418+0000","last_became_peered":"2024-01-20T11:15:29.242418+0000","last_unstale":"2024-01-20T11:19:51.753038+0000","last_undegraded":"2024-01-20T11:19:51.753038+0000","last_fullsized":"2024-01-20T11:19:51.753038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T16:05:20.122623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:19:48.375583+0000","last_change":"2024-01-20T11:15:29.297428+0000","last_active":"2024-01-20T11:19:48.375583+0000","last_peered":"2024-01-20T11:19:48.375583+0000","last_clean":"2024-01-20T11:19:48.375583+0000","last_became_active":"2024-01-20T11:15:29.269503+0000","last_became_peered":"2024-01-20T11:15:29.269503+0000","last_unstale":"2024-01-20T11:19:48.375583+0000","last_undegraded":"2024-01-20T11:19:48.375583+0000","last_fullsized":"2024-01-20T11:19:48.375583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T12:53:01.582809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830287+0000","last_change":"2024-01-20T11:15:28.958038+0000","last_active":"2024-01-20T11:21:46.830287+0000","last_peered":"2024-01-20T11:21:46.830287+0000","last_clean":"2024-01-20T11:21:46.830287+0000","last_became_active":"2024-01-20T11:15:28.928171+0000","last_became_peered":"2024-01-20T11:15:28.928171+0000","last_unstale":"2024-01-20T11:21:46.830287+0000","last_undegraded":"2024-01-20T11:21:46.830287+0000","last_fullsized":"2024-01-20T11:21:46.830287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T18:33:42.626623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:00.860676+0000","last_change":"2024-01-20T11:16:50.488485+0000","last_active":"2024-01-20T11:21:00.860676+0000","last_peered":"2024-01-20T11:21:00.860676+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.488485+0000","last_became_peered":"2024-01-20T11:16:50.488485+0000","last_unstale":"2024-01-20T11:21:00.860676+0000","last_undegraded":"2024-01-20T11:21:00.860676+0000","last_fullsized":"2024-01-20T11:16:50.409560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T21:49:15.319396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.829744+0000","last_change":"2024-01-20T11:15:29.191103+0000","last_active":"2024-01-20T11:21:46.829744+0000","last_peered":"2024-01-20T11:21:46.829744+0000","last_clean":"2024-01-20T11:21:46.829744+0000","last_became_active":"2024-01-20T11:15:29.181968+0000","last_became_peered":"2024-01-20T11:15:29.181968+0000","last_unstale":"2024-01-20T11:21:46.829744+0000","last_undegraded":"2024-01-20T11:21:46.829744+0000","last_fullsized":"2024-01-20T11:21:46.829744+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T13:17:47.189980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:46.747466+0000","last_change":"2024-01-20T11:16:51.868790+0000","last_active":"2024-01-20T11:21:46.747466+0000","last_peered":"2024-01-20T11:21:46.747466+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:51.868790+0000","last_became_peered":"2024-01-20T11:16:51.868790+0000","last_unstale":"2024-01-20T11:21:46.747466+0000","last_undegraded":"2024-01-20T11:21:46.747466+0000","last_fullsized":"2024-01-20T11:16:51.790002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:15:31.323772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":30,"state":"active+undersized","last_fresh":"2024-01-20T11:21:31.815524+0000","last_change":"2024-01-20T11:16:50.513192+0000","last_active":"2024-01-20T11:21:31.815524+0000","last_peered":"2024-01-20T11:21:31.815524+0000","last_clean":"2024-01-20T11:15:27.266904+0000","last_became_active":"2024-01-20T11:16:50.513192+0000","last_became_peered":"2024-01-20T11:16:50.513192+0000","last_unstale":"2024-01-20T11:21:31.815524+0000","last_undegraded":"2024-01-20T11:21:31.815524+0000","last_fullsized":"2024-01-20T11:16:50.490990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T15:45:49.650275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-20T11:21:46.830036+0000","last_change":"2024-01-20T11:15:29.218591+0000","last_active":"2024-01-20T11:21:46.830036+0000","last_peered":"2024-01-20T11:21:46.830036+0000","last_clean":"2024-01-20T11:21:46.830036+0000","last_became_active":"2024-01-20T11:15:29.211662+0000","last_became_peered":"2024-01-20T11:15:29.211662+0000","last_unstale":"2024-01-20T11:21:46.830036+0000","last_undegraded":"2024-01-20T11:21:46.830036+0000","last_fullsized":"2024-01-20T11:21:46.830036+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:27.266904+0000","last_clean_scrub_stamp":"2024-01-20T11:15:27.266904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T20:05:01.979946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":131,"reported_epoch":30,"state":"active+undersized+degraded","last_fresh":"2024-01-20T11:21:18.077638+0000","last_change":"2024-01-20T11:16:51.800417+0000","last_active":"2024-01-20T11:21:18.077638+0000","last_peered":"2024-01-20T11:21:18.077638+0000","last_clean":"2024-01-20T11:15:30.540447+0000","last_became_active":"2024-01-20T11:16:51.800417+0000","last_became_peered":"2024-01-20T11:16:51.800417+0000","last_unstale":"2024-01-20T11:21:18.077638+0000","last_undegraded":"2024-01-20T11:16:51.780272+0000","last_fullsized":"2024-01-20T11:16:51.776630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-20T11:15:13.343599+0000","last_clean_scrub_stamp":"2024-01-20T11:15:13.343599+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-21T11:31:04.795791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":22,"seq":94489280623,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":7.8159999999999998,"5min":8.1159999999999997,"15min":11.984999999999999},"min":{"1min":3.5430000000000001,"5min":3.444,"15min":3.444},"max":{"1min":12.379,"5min":24.600000000000001,"15min":100.05800000000001},"last":7.8570000000000002},{"interface":"front","average":{"1min":7.6980000000000004,"5min":8.1850000000000005,"15min":12.715999999999999},"min":{"1min":5.4119999999999999,"5min":4.1929999999999996,"15min":4.1929999999999996},"max":{"1min":17.138999999999999,"5min":17.138999999999999,"15min":91.789000000000001},"last":4.7190000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":9.1769999999999996,"5min":9.6199999999999992,"15min":14.427},"min":{"1min":5.79,"5min":4.9640000000000004,"15min":4.9640000000000004},"max":{"1min":14.324999999999999,"5min":22.195,"15min":98.700000000000003},"last":10.305},{"interface":"front","average":{"1min":10.712999999999999,"5min":10.273,"15min":14.021000000000001},"min":{"1min":6.125,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":15.217000000000001,"5min":22.798999999999999,"15min":95.603999999999999},"last":8.7840000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.298999999999999,"5min":11.349,"15min":15.708},"min":{"1min":6.2919999999999998,"5min":5.6879999999999997,"15min":5.6879999999999997},"max":{"1min":18.085000000000001,"5min":19.283999999999999,"15min":131.09700000000001},"last":11.678000000000001},{"interface":"front","average":{"1min":11.364000000000001,"5min":11.865,"15min":15.5},"min":{"1min":7.6449999999999996,"5min":7.5960000000000001,"15min":6.1989999999999998},"max":{"1min":16.286999999999999,"5min":20.315000000000001,"15min":105.43899999999999},"last":14.603999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.1,"5min":11.234999999999999,"15min":17.097999999999999},"min":{"1min":7.524,"5min":5.4660000000000002,"15min":5.4660000000000002},"max":{"1min":16.204000000000001,"5min":27.010999999999999,"15min":94.911000000000001},"last":16.123000000000001},{"interface":"front","average":{"1min":11.275,"5min":12.116,"15min":18.684000000000001},"min":{"1min":6.0750000000000002,"5min":6.0750000000000002,"15min":6.0750000000000002},"max":{"1min":16.061,"5min":18.577000000000002,"15min":100.633},"last":9.5229999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.528}]},{"osd":6,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":11.693,"5min":13.045999999999999,"15min":18.779},"min":{"1min":7.3840000000000003,"5min":6.282,"15min":6.282},"max":{"1min":16.593,"5min":23.946999999999999,"15min":131.70599999999999},"last":13.114000000000001},{"interface":"front","average":{"1min":14.1,"5min":12.811,"15min":17.818999999999999},"min":{"1min":10.657,"5min":6.5250000000000004,"15min":6.5250000000000004},"max":{"1min":16.988,"5min":25.286000000000001,"15min":107.962},"last":16.920000000000002}]},{"osd":7,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":12.728,"5min":11.733000000000001,"15min":16.459},"min":{"1min":7.4829999999999997,"5min":6.5990000000000002,"15min":6.5990000000000002},"max":{"1min":16.849,"5min":20.991,"15min":96.911000000000001},"last":13.757},{"interface":"front","average":{"1min":12.712,"5min":14.061999999999999,"15min":19.725000000000001},"min":{"1min":7.9509999999999996,"5min":7.9509999999999996,"15min":7.9509999999999996},"max":{"1min":17.497,"5min":27.356000000000002,"15min":129.828},"last":12.477}]}]},{"osd":7,"up_from":21,"seq":90194313325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.0660000000000007,"5min":7.4260000000000002,"15min":9.0950000000000006},"min":{"1min":4.8869999999999996,"5min":4.2930000000000001,"15min":3.9820000000000002},"max":{"1min":12.654,"5min":14.342000000000001,"15min":38.005000000000003},"last":10.996},{"interface":"front","average":{"1min":7.7729999999999997,"5min":8.3680000000000003,"15min":11.545999999999999},"min":{"1min":5.6980000000000004,"5min":4.0730000000000004,"15min":3.9289999999999998},"max":{"1min":11.542999999999999,"5min":15.512,"15min":76.718999999999994},"last":4.9930000000000003}]},{"osd":1,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":8.6489999999999991,"5min":8.7989999999999995,"15min":9.8469999999999995},"min":{"1min":5.5229999999999997,"5min":4.3150000000000004,"15min":4.3150000000000004},"max":{"1min":14.186999999999999,"5min":16.486000000000001,"15min":18.859000000000002},"last":8.0549999999999997},{"interface":"front","average":{"1min":11.220000000000001,"5min":10.897,"15min":12.32},"min":{"1min":6.4390000000000001,"5min":5.8860000000000001,"15min":5.8140000000000001},"max":{"1min":15.837999999999999,"5min":16.920000000000002,"15min":43.774000000000001},"last":13.308999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":13.481999999999999,"5min":11.955,"15min":15.044},"min":{"1min":9.1170000000000009,"5min":7.0049999999999999,"15min":6.9370000000000003},"max":{"1min":16.693000000000001,"5min":17.428000000000001,"15min":49.210000000000001},"last":14.002000000000001},{"interface":"front","average":{"1min":11.353999999999999,"5min":10.81,"15min":14.345000000000001},"min":{"1min":6.1289999999999996,"5min":5.6260000000000003,"15min":5.6260000000000003},"max":{"1min":18.021999999999998,"5min":18.021999999999998,"15min":78.548000000000002},"last":8.9670000000000005}]},{"osd":3,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":9.4789999999999992,"5min":10.661,"15min":13.433999999999999},"min":{"1min":6.1959999999999997,"5min":6.1959999999999997,"15min":5.8129999999999997},"max":{"1min":15.954000000000001,"5min":17,"15min":55.771000000000001},"last":11.744},{"interface":"front","average":{"1min":13.935,"5min":13.44,"15min":15.789999999999999},"min":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"max":{"1min":17.521999999999998,"5min":18.385000000000002,"15min":49.829999999999998},"last":15.308}]},{"osd":4,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":11.619999999999999,"5min":12.180999999999999,"15min":15.617000000000001},"min":{"1min":7.6959999999999997,"5min":6.7960000000000003,"15min":6.7960000000000003},"max":{"1min":15.971,"5min":18.501999999999999,"15min":91.667000000000002},"last":14.657},{"interface":"front","average":{"1min":12.353999999999999,"5min":12.096,"15min":15.545999999999999},"min":{"1min":9.0709999999999997,"5min":6.3099999999999996,"15min":6.3099999999999996},"max":{"1min":18.774999999999999,"5min":18.774999999999999,"15min":92.613},"last":9.6910000000000007}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.257999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:36 2024","interfaces":[{"interface":"back","average":{"1min":14.220000000000001,"5min":13.616,"15min":16.809999999999999},"min":{"1min":9.8710000000000004,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":19.864000000000001,"5min":19.864000000000001,"15min":80.102000000000004},"last":10.272},{"interface":"front","average":{"1min":13.468999999999999,"5min":13.837,"15min":17.164999999999999},"min":{"1min":8.9190000000000005,"5min":7.6790000000000003,"15min":7.6790000000000003},"max":{"1min":17.309000000000001,"5min":18.315000000000001,"15min":53.398000000000003},"last":16.074999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313326,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":8.5109999999999992,"5min":7.6420000000000003,"15min":12.962999999999999},"min":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":3.8700000000000001},"max":{"1min":14.254,"5min":18.129000000000001,"15min":155.27799999999999},"last":4.2199999999999998},{"interface":"front","average":{"1min":7.9160000000000004,"5min":8.0790000000000006,"15min":11.877000000000001},"min":{"1min":4.0730000000000004,"5min":3.923,"15min":3.923},"max":{"1min":15.061,"5min":15.65,"15min":144.16499999999999},"last":9.6449999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.9939999999999998,"5min":9.8130000000000006,"15min":13.531000000000001},"min":{"1min":6.4489999999999998,"5min":4.7240000000000002,"15min":4.7240000000000002},"max":{"1min":22.052,"5min":22.052,"15min":149.55000000000001},"last":11.471},{"interface":"front","average":{"1min":11.141,"5min":11.016,"15min":16.215},"min":{"1min":5.6849999999999996,"5min":5.3860000000000001,"15min":5.3860000000000001},"max":{"1min":21.207999999999998,"5min":21.207999999999998,"15min":155.91200000000001},"last":8.2479999999999993}]},{"osd":2,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":9.907,"5min":10.333,"15min":15.125999999999999},"min":{"1min":5.8120000000000003,"5min":5.8120000000000003,"15min":5.1719999999999997},"max":{"1min":14.335000000000001,"5min":16.574999999999999,"15min":145.553},"last":14.143000000000001},{"interface":"front","average":{"1min":10.819000000000001,"5min":11.186,"15min":15.462},"min":{"1min":7.077,"5min":7.0119999999999996,"15min":6.8840000000000003},"max":{"1min":14.951000000000001,"5min":19.343,"15min":147.66300000000001},"last":12.970000000000001}]},{"osd":4,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":13.393000000000001,"5min":12.872999999999999,"15min":20.648},"min":{"1min":8.4819999999999993,"5min":7.9530000000000003,"15min":7.9530000000000003},"max":{"1min":23.436,"5min":23.436,"15min":284.00700000000001},"last":15.053000000000001},{"interface":"front","average":{"1min":12.159000000000001,"5min":12.528,"15min":20.949999999999999},"min":{"1min":7.5540000000000003,"5min":7.0759999999999996,"15min":7.0759999999999996},"max":{"1min":17.370000000000001,"5min":19.489000000000001,"15min":284.62799999999999},"last":8.9649999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.759}]},{"osd":6,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":12.358000000000001,"5min":11.361000000000001,"15min":16.847999999999999},"min":{"1min":7.008,"5min":6.1319999999999997,"15min":6.1319999999999997},"max":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":146.744},"last":10.452999999999999},{"interface":"front","average":{"1min":13.752000000000001,"5min":13.364000000000001,"15min":18.126999999999999},"min":{"1min":10.433,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":24.084,"5min":24.084,"15min":150.542},"last":16.754000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:39 2024","interfaces":[{"interface":"back","average":{"1min":15.451000000000001,"5min":15.26,"15min":20.085000000000001},"min":{"1min":11.426,"5min":9.9540000000000006,"15min":9.9540000000000006},"max":{"1min":25.253,"5min":25.253,"15min":159.179},"last":15.922000000000001},{"interface":"front","average":{"1min":13.234,"5min":13.723000000000001,"15min":19.463999999999999},"min":{"1min":7.2910000000000004,"5min":7.2910000000000004,"15min":7.2910000000000004},"max":{"1min":19.992999999999999,"5min":19.992999999999999,"15min":157.35599999999999},"last":12.26}]}]},{"osd":5,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":610304,"data_stored":464936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1449999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.282}]}]},{"osd":2,"up_from":17,"seq":73014444140,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.4809999999999999,"5min":8.1669999999999998,"15min":9.4060000000000006},"min":{"1min":4.1459999999999999,"5min":3.911,"15min":3.911},"max":{"1min":12.938000000000001,"5min":20.539999999999999,"15min":25.603999999999999},"last":5.9100000000000001},{"interface":"front","average":{"1min":8.8339999999999996,"5min":8.3089999999999993,"15min":8.548},"min":{"1min":5.2380000000000004,"5min":3.9630000000000001,"15min":3.9630000000000001},"max":{"1min":13.708,"5min":17.154,"15min":17.154},"last":11.252000000000001}]},{"osd":1,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":7.8840000000000003,"5min":8.4770000000000003,"15min":11.627000000000001},"min":{"1min":5.3600000000000003,"5min":4.7720000000000002,"15min":4.1660000000000004},"max":{"1min":14.817,"5min":17.024000000000001,"15min":65.064999999999998},"last":11.859999999999999},{"interface":"front","average":{"1min":10.629,"5min":11.146000000000001,"15min":14.222},"min":{"1min":6.8739999999999997,"5min":5.508,"15min":5.0460000000000003},"max":{"1min":14.138,"5min":19.446000000000002,"15min":32.414999999999999},"last":15.792}]},{"osd":3,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":12.534000000000001,"5min":12.755000000000001,"15min":15.407},"min":{"1min":9.2070000000000007,"5min":7.1859999999999999,"15min":7.1859999999999999},"max":{"1min":15.773,"5min":20.829000000000001,"15min":49.237000000000002},"last":15.01},{"interface":"front","average":{"1min":8.8689999999999998,"5min":10.082000000000001,"15min":12.805999999999999},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":13.977,"5min":16.702999999999999,"15min":52.905999999999999},"last":8.6929999999999996}]},{"osd":4,"last update":"Sat Jan 20 11:23:38 2024","interfaces":[{"interface":"back","average":{"1min":13.585000000000001,"5min":13.913,"15min":16.091999999999999},"min":{"1min":9.5099999999999998,"5min":7.8200000000000003,"15min":7.3869999999999996},"max":{"1min":17.895,"5min":20.393000000000001,"15min":51.000999999999998},"last":13.769},{"interface":"front","average":{"1min":12.028,"5min":12.231999999999999,"15min":13.638},"min":{"1min":8.1460000000000008,"5min":6.5140000000000002,"15min":6.5140000000000002},"max":{"1min":16.597999999999999,"5min":21.629999999999999,"15min":48.299999999999997},"last":7.569}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.622}]},{"osd":6,"last update":"Sat Jan 20 11:23:24 2024","interfaces":[{"interface":"back","average":{"1min":13.634,"5min":13.842000000000001,"15min":15.019},"min":{"1min":10.263,"5min":8.1470000000000002,"15min":7.6050000000000004},"max":{"1min":16.736999999999998,"5min":22.48,"15min":35.186999999999998},"last":9.7780000000000005},{"interface":"front","average":{"1min":9.7479999999999993,"5min":10.487,"15min":10.720000000000001},"min":{"1min":6.1319999999999997,"5min":6.0579999999999998,"15min":4.7850000000000001},"max":{"1min":14.023,"5min":16.632000000000001,"15min":32.984000000000002},"last":6.9020000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:35 2024","interfaces":[{"interface":"back","average":{"1min":11.651,"5min":12.295999999999999,"15min":13.125999999999999},"min":{"1min":8.0760000000000005,"5min":7.3449999999999998,"15min":7.3449999999999998},"max":{"1min":17.093,"5min":18.527000000000001,"15min":24.266999999999999},"last":13.093999999999999},{"interface":"front","average":{"1min":14.661,"5min":14.917,"15min":16.617999999999999},"min":{"1min":11.569000000000001,"5min":9.5760000000000005,"15min":9.5760000000000005},"max":{"1min":18.689,"5min":21.884,"15min":28.148},"last":16.872}]}]},{"osd":1,"up_from":17,"seq":73014444141,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":502510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":8.3510000000000009,"5min":7.9770000000000003,"15min":8.9939999999999998},"min":{"1min":4.9039999999999999,"5min":4.2910000000000004,"15min":4.2910000000000004},"max":{"1min":14.999000000000001,"5min":15.151999999999999,"15min":18.547000000000001},"last":4.5890000000000004},{"interface":"front","average":{"1min":7.2839999999999998,"5min":8.4740000000000002,"15min":9.6319999999999997},"min":{"1min":4.9589999999999996,"5min":4.8419999999999996,"15min":4.8419999999999996},"max":{"1min":12.084,"5min":16.992999999999999,"15min":30.439},"last":9.1059999999999999}]},{"osd":2,"last update":"Sat Jan 20 11:23:30 2024","interfaces":[{"interface":"back","average":{"1min":9.9049999999999994,"5min":10.605,"15min":12.279999999999999},"min":{"1min":6.5170000000000003,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":13.898,"5min":17.009,"15min":24.725000000000001},"last":9.4399999999999995},{"interface":"front","average":{"1min":9.9299999999999997,"5min":10.005000000000001,"15min":12.067},"min":{"1min":6.1909999999999998,"5min":5.4969999999999999,"15min":5.4969999999999999},"max":{"1min":15.407999999999999,"5min":15.845000000000001,"15min":23.981999999999999},"last":10.757999999999999}]},{"osd":3,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.137,"5min":12.438000000000001,"15min":20.449000000000002},"min":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"max":{"1min":15.898999999999999,"5min":21.178999999999998,"15min":259.483},"last":13.199999999999999},{"interface":"front","average":{"1min":11.436999999999999,"5min":10.169,"15min":16.289999999999999},"min":{"1min":6.6059999999999999,"5min":5.2519999999999998,"15min":5.2519999999999998},"max":{"1min":17.923999999999999,"5min":19.297999999999998,"15min":243.24199999999999},"last":8.6709999999999994}]},{"osd":4,"last update":"Sat Jan 20 11:23:40 2024","interfaces":[{"interface":"back","average":{"1min":9.6929999999999996,"5min":11.172000000000001,"15min":13},"min":{"1min":6.7290000000000001,"5min":5.9669999999999996,"15min":5.9669999999999996},"max":{"1min":14.914999999999999,"5min":18.899999999999999,"15min":25.541},"last":12.253},{"interface":"front","average":{"1min":12.557,"5min":13.332000000000001,"15min":14.596},"min":{"1min":7.0490000000000004,"5min":6.75,"15min":6.75},"max":{"1min":16.565000000000001,"5min":22.280000000000001,"15min":30.204999999999998},"last":11.541}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3170000000000002}]},{"osd":6,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":11.412000000000001,"5min":11.02,"15min":11.744},"min":{"1min":6.8650000000000002,"5min":5.8780000000000001,"15min":5.8780000000000001},"max":{"1min":16.27,"5min":16.792999999999999,"15min":27.233000000000001},"last":15.151999999999999},{"interface":"front","average":{"1min":12.305999999999999,"5min":13.166,"15min":14.667},"min":{"1min":9.3559999999999999,"5min":7.2859999999999996,"15min":7.2859999999999996},"max":{"1min":17.702000000000002,"5min":21.553000000000001,"15min":31.634},"last":14.272}]},{"osd":7,"last update":"Sat Jan 20 11:23:34 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":12.975,"15min":23.236000000000001},"min":{"1min":8.2300000000000004,"5min":7.5960000000000001,"15min":7.5960000000000001},"max":{"1min":17.57,"5min":20.875,"15min":344.98099999999999},"last":16.667000000000002},{"interface":"front","average":{"1min":14.32,"5min":14.718,"15min":24.286999999999999},"min":{"1min":11.092000000000001,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":18.556000000000001,"5min":22.863,"15min":313.45999999999998},"last":15.972}]}]},{"osd":0,"up_from":17,"seq":73014444141,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":6.7359999999999998,"5min":7.5259999999999998,"15min":11.848000000000001},"min":{"1min":4.4809999999999999,"5min":3.9369999999999998,"15min":3.9369999999999998},"max":{"1min":11.199999999999999,"5min":22.428999999999998,"15min":114.843},"last":4.2770000000000001},{"interface":"front","average":{"1min":7.476,"5min":8.5269999999999992,"15min":12.734},"min":{"1min":4.0439999999999996,"5min":3.6909999999999998,"15min":3.6909999999999998},"max":{"1min":13.010999999999999,"5min":18.504999999999999,"15min":121.044},"last":9.0020000000000007}]},{"osd":2,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":8.9420000000000002,"5min":8.4990000000000006,"15min":20.919},"min":{"1min":5.4550000000000001,"5min":4.5880000000000001,"15min":4.5880000000000001},"max":{"1min":14.103999999999999,"5min":19.783000000000001,"15min":379.17099999999999},"last":4.9749999999999996},{"interface":"front","average":{"1min":10.599,"5min":10.763999999999999,"15min":22.942},"min":{"1min":7.1340000000000003,"5min":4.819,"15min":4.819},"max":{"1min":16.361999999999998,"5min":23.260999999999999,"15min":373.88299999999998},"last":7.6820000000000004}]},{"osd":3,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.776,"5min":12.148999999999999,"15min":21.678999999999998},"min":{"1min":7.8280000000000003,"5min":5.2389999999999999,"15min":5.2389999999999999},"max":{"1min":18.335999999999999,"5min":27.292000000000002,"15min":358.738},"last":14.178000000000001},{"interface":"front","average":{"1min":9.9199999999999999,"5min":10.561999999999999,"15min":12.034000000000001},"min":{"1min":5.5839999999999996,"5min":5.3179999999999996,"15min":5.3179999999999996},"max":{"1min":15.347,"5min":20.672000000000001,"15min":21.004999999999999},"last":9.6820000000000004}]},{"osd":4,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":12.163,"5min":12.465,"15min":23.204000000000001},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":17.949000000000002,"5min":23.949000000000002,"15min":372.02300000000002},"last":10.336},{"interface":"front","average":{"1min":11.621,"5min":10.532999999999999,"15min":20.757000000000001},"min":{"1min":7.0620000000000003,"5min":5.1680000000000001,"15min":5.1680000000000001},"max":{"1min":16.713000000000001,"5min":21.334,"15min":372.70800000000003},"last":8.3710000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.712999999999999}]},{"osd":6,"last update":"Sat Jan 20 11:23:29 2024","interfaces":[{"interface":"back","average":{"1min":13.679,"5min":13.222,"15min":25.547999999999998},"min":{"1min":10.541,"5min":7.2270000000000003,"15min":7.2270000000000003},"max":{"1min":17.52,"5min":28.087,"15min":375.733},"last":14.946},{"interface":"front","average":{"1min":13.837,"5min":13.778,"15min":24.300999999999998},"min":{"1min":9.1850000000000005,"5min":8.2599999999999998,"15min":8.1270000000000007},"max":{"1min":19.565000000000001,"5min":26.254999999999999,"15min":374.654},"last":10.874000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:43 2024","interfaces":[{"interface":"back","average":{"1min":15.414,"5min":15.087,"15min":16.367999999999999},"min":{"1min":8.9819999999999993,"5min":8.625,"15min":8.625},"max":{"1min":22.588999999999999,"5min":25.673999999999999,"15min":27.385999999999999},"last":11.433999999999999},{"interface":"front","average":{"1min":12.353,"5min":11.938000000000001,"15min":12.262},"min":{"1min":8.0250000000000004,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":19.954000000000001,"5min":24.547000000000001,"15min":24.547000000000001},"last":13.247999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":106086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.282,"5min":7.984,"15min":10.031000000000001},"min":{"1min":4.1059999999999999,"5min":3.4180000000000001,"15min":3.4180000000000001},"max":{"1min":14.311999999999999,"5min":14.696,"15min":106.059},"last":6.1609999999999996},{"interface":"front","average":{"1min":7.952,"5min":8.1820000000000004,"15min":11.782},"min":{"1min":4.4900000000000002,"5min":4.1840000000000002,"15min":3.96},"max":{"1min":19.834,"5min":19.834,"15min":157.67699999999999},"last":5.4189999999999996}]},{"osd":1,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":8.8989999999999991,"5min":8.8420000000000005,"15min":20.606999999999999},"min":{"1min":5.4589999999999996,"5min":4.8319999999999999,"15min":4.6319999999999997},"max":{"1min":15.849,"5min":16.399000000000001,"15min":189.37700000000001},"last":8.2439999999999998},{"interface":"front","average":{"1min":11.859999999999999,"5min":11.167999999999999,"15min":22.186},"min":{"1min":6.6219999999999999,"5min":5.2039999999999997,"15min":5.2039999999999997},"max":{"1min":15.178000000000001,"5min":17.751000000000001,"15min":190.304},"last":14.52}]},{"osd":2,"last update":"Sat Jan 20 11:23:27 2024","interfaces":[{"interface":"back","average":{"1min":10.135,"5min":10.819000000000001,"15min":13.032},"min":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":5.3300000000000001},"max":{"1min":15.52,"5min":17.491,"15min":128.67099999999999},"last":7.7000000000000002},{"interface":"front","average":{"1min":11.536,"5min":10.558,"15min":13.744999999999999},"min":{"1min":6.6369999999999996,"5min":5.3650000000000002,"15min":5.3650000000000002},"max":{"1min":15.016999999999999,"5min":16.323,"15min":101.75700000000001},"last":8.8290000000000006}]},{"osd":3,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":12.875,"5min":12.423999999999999,"15min":14.545},"min":{"1min":8.8670000000000009,"5min":6.3780000000000001,"15min":6.3780000000000001},"max":{"1min":17.093,"5min":18.550999999999998,"15min":81.373999999999995},"last":10.692},{"interface":"front","average":{"1min":10.817,"5min":12.17,"15min":14.632999999999999},"min":{"1min":7.1280000000000001,"5min":6.2290000000000001,"15min":6.2290000000000001},"max":{"1min":16.373000000000001,"5min":18.533000000000001,"15min":84.524000000000001},"last":13.565}]},{"osd":4,"last update":"Sat Jan 20 11:23:41 2024","interfaces":[{"interface":"back","average":{"1min":13.547000000000001,"5min":13.513999999999999,"15min":15.654},"min":{"1min":8.3879999999999999,"5min":8.0790000000000006,"15min":8.0790000000000006},"max":{"1min":21.539000000000001,"5min":21.539000000000001,"15min":70.254999999999995},"last":9.4730000000000008},{"interface":"front","average":{"1min":12.259,"5min":12.333,"15min":15.17},"min":{"1min":7.3079999999999998,"5min":6.7080000000000002,"15min":6.7080000000000002},"max":{"1min":17.798999999999999,"5min":18.061,"15min":72.670000000000002},"last":12.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.736000000000001}]},{"osd":7,"last update":"Sat Jan 20 11:23:33 2024","interfaces":[{"interface":"back","average":{"1min":13.132999999999999,"5min":13.334,"15min":15.336},"min":{"1min":7.2549999999999999,"5min":7.2549999999999999,"15min":7.2549999999999999},"max":{"1min":19.141999999999999,"5min":19.141999999999999,"15min":25.303999999999998},"last":10.106},{"interface":"front","average":{"1min":13.634,"5min":12.903,"15min":14.869},"min":{"1min":7.351,"5min":7.149,"15min":7.149},"max":{"1min":22.689,"5min":22.689,"15min":22.838999999999999},"last":15.353999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-20T11:24:02.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~304s 2024-01-20T11:24:02.329 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-01-20T11:24:03.538 INFO:tasks.ceph.osd.0:Sent signal 15 2024-01-20T11:24:03.539 INFO:tasks.ceph.osd.1:Sent signal 15 2024-01-20T11:24:03.539 INFO:tasks.ceph.osd.2:Sent signal 15 2024-01-20T11:24:03.539 INFO:tasks.ceph.osd.3:Sent signal 15 2024-01-20T11:24:03.539 INFO:tasks.ceph.osd.4:Sent signal 15 2024-01-20T11:24:03.540 INFO:tasks.ceph.osd.6:Sent signal 15 2024-01-20T11:24:03.540 INFO:tasks.ceph.osd.7:Sent signal 15 2024-01-20T11:24:03.540 INFO:tasks.ceph.mon.a:Sent signal 15 2024-01-20T11:24:03.540 INFO:tasks.ceph.mon.c:Sent signal 15 2024-01-20T11:24:03.540 INFO:tasks.ceph.mon.b:Sent signal 15 2024-01-20T11:24:03.541 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-01-20T11:24:03.541 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-01-20T11:24:03.542 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:24:03.542+0000 7eff7f26a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 33845) UID: 0 2024-01-20T11:24:03.543 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-20T11:24:03.542+0000 7eff7f26a640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-01-20T11:24:03.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-20T11:24:03.545+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 34427) UID: 0 2024-01-20T11:24:03.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-20T11:24:03.546+0000 b47f640 -1 osd.2 30 *** Got signal Terminated *** 2024-01-20T11:24:03.555 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-20T11:24:03.544+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 34414) UID: 0 2024-01-20T11:24:03.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-20T11:24:03.544+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 34420) UID: 0 2024-01-20T11:24:03.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-20T11:24:03.546+0000 b47f640 -1 osd.1 30 *** Got signal Terminated *** 2024-01-20T11:24:03.556 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-20T11:24:03.546+0000 b47f640 -1 osd.0 30 *** Got signal Terminated *** 2024-01-20T11:24:03.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:24:03.546+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 34429) UID: 0 2024-01-20T11:24:03.561 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:24:03.545+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 34410) UID: 0 2024-01-20T11:24:03.561 INFO:tasks.ceph.osd.4.smithi134.stderr:2024-01-20T11:24:03.547+0000 b47f640 -1 osd.4 30 *** Got signal Terminated *** 2024-01-20T11:24:03.562 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-01-20T11:24:03.545+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 34415) UID: 0 2024-01-20T11:24:03.562 INFO:tasks.ceph.osd.6.smithi134.stderr:2024-01-20T11:24:03.547+0000 b47f640 -1 osd.6 30 *** Got signal Terminated *** 2024-01-20T11:24:03.562 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:24:03.547+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 34417) UID: 0 2024-01-20T11:24:03.563 INFO:tasks.ceph.osd.7.smithi134.stderr:2024-01-20T11:24:03.549+0000 b47f640 -1 osd.7 30 *** Got signal Terminated *** 2024-01-20T11:24:03.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-20T11:24:03.547+0000 b47f640 -1 osd.3 30 *** Got signal Terminated *** 2024-01-20T11:24:03.564 INFO:tasks.ceph.mon.c.smithi110.stderr:2024-01-20T11:24:03.548+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34090) UID: 0 2024-01-20T11:24:03.565 INFO:tasks.ceph.mon.c.smithi110.stderr:2024-01-20T11:24:03.550+0000 9f77640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-01-20T11:24:03.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-01-20T11:24:03.547+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34088) UID: 0 2024-01-20T11:24:03.569 INFO:tasks.ceph.mon.b.smithi134.stderr:2024-01-20T11:24:03.547+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 33843) UID: 0 2024-01-20T11:24:03.573 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:24:03.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-01-20T11:24:03.549+0000 9f77640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-01-20T11:24:03.589 INFO:tasks.ceph.mon.b.smithi134.stderr:2024-01-20T11:24:03.549+0000 9f77640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-01-20T11:24:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.687+0000 7f368024f640 10 monclient: get_monmap_and_config 2024-01-20T11:24:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.687+0000 7f368024f640 10 monclient: build_initial_monmap 2024-01-20T11:24:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.687+0000 7f368024f640 10 monclient: monmap: 2024-01-20T11:24:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:24:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:24:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:24:03.687650+0000 2024-01-20T11:24:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:24:03.687650+0000 2024-01-20T11:24:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.687+0000 7f368024f640 1 Processor -- start 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.687+0000 7f368024f640 1 -- start start 2024-01-20T11:24:03.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.687+0000 7f368024f640 10 monclient: init 2024-01-20T11:24:03.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 10 monclient: _reopen_session rank -1 2024-01-20T11:24:03.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-01-20T11:24:03.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36780b4d10 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 0x7f36780b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 10 monclient(hunting): picked mon.noname-a con 0x7f36780b1ed0 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 0x7f3678145e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 10 monclient(hunting): picked mon.noname-b con 0x7f3678145980 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 msgr2=0x7f3678145e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 msgr2=0x7f36780b3da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 0x7f36780b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f36780b4d10 legacy=0x7f36780b17c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f36780b4d10 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 0x7f3678145e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f36780b1ed0 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:03.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f3678145980 2024-01-20T11:24:03.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:03.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.688+0000 7f368024f640 10 monclient(hunting): authenticate will time out at 1887.038789s 2024-01-20T11:24:03.743 INFO:tasks.ceph.mgr.y.smithi110.stderr:daemon-helper: command crashed with signal 15 2024-01-20T11:24:03.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.889+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 msgr2=0x7f36780b3da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:03.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.889+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 msgr2=0x7f3678145e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:03.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.889+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 0x7f36780b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:03.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.889+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 0x7f3678145e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:03.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:03.889+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f36780b4d10 legacy=0x7f36780b17c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:04.188 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:09:55.194 33847== 2024-01-20T11:24:04.188 INFO:tasks.ceph.mon.b.smithi134.stderr:==00:00:09:55.194 33847== Exit program on first error (--exit-on-first-error=yes) 2024-01-20T11:24:04.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:04.290+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 msgr2=0x7f3678145e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:04.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:04.290+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 msgr2=0x7f36780b3da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:04.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:04.290+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 0x7f3678145e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:04.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:04.290+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 0x7f36780b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:04.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:04.290+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f36780b4d10 legacy=0x7f36780b17c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:04.326 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:55.351 34116== 2024-01-20T11:24:04.327 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:55.351 34116== Exit program on first error (--exit-on-first-error=yes) 2024-01-20T11:24:04.329 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:55.353 34120== 2024-01-20T11:24:04.329 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:55.353 34120== Exit program on first error (--exit-on-first-error=yes) 2024-01-20T11:24:04.343 INFO:tasks.ceph.mon.b.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-01-20T11:24:04.344 INFO:tasks.ceph.mon.a.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-01-20T11:24:04.344 INFO:tasks.ceph.mon.c.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-01-20T11:24:05.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:05.091+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 msgr2=0x7f36780b3da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:05.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:05.091+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 msgr2=0x7f3678145e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:05.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:05.091+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 0x7f36780b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:05.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:05.091+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 0x7f3678145e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:05.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:05.091+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f36780b4d10 legacy=0x7f36780b17c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:06.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:06.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:06.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:06.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:06.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 msgr2=0x7f3678145e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:06.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3678145980 0x7f3678145e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:06.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 msgr2=0x7f36780b3da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:06.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36780b1ed0 0x7f36780b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f36780b4d10 legacy=0x7f36780b17c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 10 monclient: _add_conns ranks=[0,4] 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0036c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 0x7f365c009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c007370 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c0036c0 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c007370 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.688+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0036c0 legacy=0x7f365c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.689+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:06.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.689+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:06.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.889+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0036c0 legacy=0x7f365c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:06.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.890+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:06.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:06.890+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:07.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:07.290+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0036c0 legacy=0x7f365c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:07.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:07.291+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:07.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:07.291+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:08.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:08.091+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0036c0 legacy=0x7f365c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:08.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:08.092+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:08.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:08.092+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:09.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.688+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.688+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.688+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.688+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.688+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.688+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c007370 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0036c0 legacy=0x7f365c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 0x7f365c009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c006270 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:09.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0066f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c005bb0 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c0066f0 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c005bb0 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c006270 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:09.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.689+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.890+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.890+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.890+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.890+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:09.890+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:10.291+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:10.291+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:10.291+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:10.291+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:10.291+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:11.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:11.092+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:11.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:11.092+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:11.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:11.092+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:11.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:11.092+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:11.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:11.092+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:12.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006270 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,1] 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 0x7f365c00da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c00d620 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c00e100 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c00e100 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00d620 2024-01-20T11:24:12.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:12.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00e100 legacy=0x7f365c0127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:12.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 msgr2=0x7f365c00da30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:12.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.689+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 0x7f365c00da30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:12.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.890+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 msgr2=0x7f365c00da30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:12.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.890+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00e100 legacy=0x7f365c0127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:12.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:12.890+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 0x7f365c00da30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:13.291+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 msgr2=0x7f365c00da30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:13.291+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00e100 legacy=0x7f365c0127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:13.291+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 0x7f365c00da30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:14.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:14.092+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 msgr2=0x7f365c00da30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:14.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:14.092+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00e100 legacy=0x7f365c0127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:14.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:14.092+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 0x7f365c00da30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:15.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 msgr2=0x7f365c00da30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00d620 0x7f365c00da30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00e100 legacy=0x7f365c0127a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c013f20 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:24:15.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:15.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c005bb0 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:15.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0066f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:15.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c0066f0 2024-01-20T11:24:15.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c013f20 2024-01-20T11:24:15.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:15.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c005bb0 2024-01-20T11:24:15.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:15.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:15.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:15.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:15.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.690+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:15.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.891+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:15.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.891+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:15.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.891+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:15.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:15.891+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:16.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:16.292+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:16.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:16.292+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:16.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:16.292+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:16.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:16.292+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:17.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:17.093+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:17.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:17.093+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:17.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:17.093+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:17.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:17.093+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:18.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:18.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:18.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:18.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:18.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:18.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 10 monclient: _add_conns ranks=[2,4] 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c00da00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 0x7f365c010ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c00e780 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00e780 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c00da00 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:18.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:18.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 msgr2=0x7f365c010ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:18.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:18.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.689+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 0x7f365c010ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:18.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.890+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:18.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.890+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:18.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.890+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 msgr2=0x7f365c010ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:18.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:18.890+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 0x7f365c010ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:19.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:19.291+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:19.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:19.291+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:19.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:19.291+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 msgr2=0x7f365c010ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:19.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:19.291+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 0x7f365c010ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:20.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:20.092+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:20.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:20.092+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:20.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:20.092+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 msgr2=0x7f365c010ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:20.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:20.092+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 0x7f365c010ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:21.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:21.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 msgr2=0x7f365c010ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00e780 0x7f365c010ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c013f20 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c00da00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0066f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c0066f0 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c013f20 2024-01-20T11:24:21.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00da00 2024-01-20T11:24:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:21.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.689+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:21.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.890+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:21.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.890+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:21.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.890+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:21.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:21.890+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:22.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:22.291+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:22.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:22.291+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:22.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:22.291+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:22.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:22.291+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:23.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:23.092+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:23.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:23.092+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:23.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:23.092+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:23.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:23.092+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0066f0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,3] 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c005bb0 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:24.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c0130f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c0130f0 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:24.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.690+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:24.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.891+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:24.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.891+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:24.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.891+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:24.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:24.891+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:25.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:25.292+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:25.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:25.292+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:25.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:25.292+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:25.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:25.292+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:26.093+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:26.093+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:26.093+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:26.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:26.093+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:27.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:27.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:27.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:27.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:27.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c00d070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:27.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c00d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:27.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:27.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:27.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c013f20 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c0130f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c005bb0 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c005bb0 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c013f20 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c0130f0 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:27.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:27.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.690+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:27.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.891+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:27.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.891+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:27.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.891+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:27.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:27.891+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:28.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:28.292+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:28.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:28.292+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:28.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:28.292+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:28.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:28.292+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:29.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:29.093+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:29.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:29.093+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:29.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:29.093+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:29.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:29.093+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013f20 legacy=0x7f365c014360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,2] 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 0x7f365c00d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c0066f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c00da00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c0066f0 2024-01-20T11:24:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c00da00 2024-01-20T11:24:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 msgr2=0x7f365c00d030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.690+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 0x7f365c00d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:30.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.891+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:30.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.891+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:30.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.891+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 msgr2=0x7f365c00d030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:30.891+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 0x7f365c00d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:31.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:31.292+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:31.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:31.292+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:31.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:31.292+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 msgr2=0x7f365c00d030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:31.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:31.292+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 0x7f365c00d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:32.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:32.093+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:32.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:32.093+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:32.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:32.093+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 msgr2=0x7f365c00d030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:32.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:32.093+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 0x7f365c00d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:33.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:33.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 msgr2=0x7f365c009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00da00 0x7f365c009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 msgr2=0x7f365c00d030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0066f0 0x7f365c00d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c0130f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c005bb0 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c013f20 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:33.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c013f20 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c0130f0 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c013f20 legacy=0x7f365c0137b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:33.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.690+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:33.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.891+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c013f20 legacy=0x7f365c0137b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:33.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.891+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:33.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.891+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:33.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.891+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:33.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:33.891+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:34.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:34.292+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c013f20 legacy=0x7f365c0137b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:34.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:34.292+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:34.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:34.292+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:34.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:34.292+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:34.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:34.292+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:35.093+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:35.093+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c013f20 legacy=0x7f365c0137b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:35.093+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:35.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:35.093+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:35.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:35.093+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:36.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:36.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:36.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c013f20 legacy=0x7f365c0137b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 10 monclient: _add_conns ranks=[2,1] 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c014300 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c00afe0 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c00afe0 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:36.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c014300 2024-01-20T11:24:36.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:36.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:36.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:36.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.691+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:36.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.892+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:36.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.892+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:36.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:36.892+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:37.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:37.293+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:37.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:37.293+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:37.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:37.293+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:38.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:38.094+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:38.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:38.094+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:38.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:38.094+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:39.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c005bb0 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c0130f0 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c014300 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c0130f0 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365c014700 con 0x7f365c014300 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:39.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:39.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:39.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:39.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.691+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:39.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.892+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:39.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.892+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:39.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.892+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:39.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.892+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:39.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.892+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:39.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:39.892+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:40.293+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:40.293+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:40.293+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:40.293+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:40.293+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:40.293+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:41.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:41.094+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:41.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:41.094+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:41.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:41.094+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:41.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:41.094+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:41.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:41.094+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:41.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:41.094+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c014300 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:42.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,0] 2024-01-20T11:24:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c013ef0 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c00afe0 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c00afe0 2024-01-20T11:24:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c013ef0 2024-01-20T11:24:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:42.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.691+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013ef0 legacy=0x7f365c013770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:42.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.892+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:42.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:42.892+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013ef0 legacy=0x7f365c013770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:43.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:43.293+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:43.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:43.293+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013ef0 legacy=0x7f365c013770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:44.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:44.095+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:44.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:44.095+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013ef0 legacy=0x7f365c013770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:45.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.691+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:45.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c013ef0 legacy=0x7f365c013770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c00afe0 legacy=0x7f365c014fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0166c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c005bb0 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c013ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c013740 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c0166c0 2024-01-20T11:24:45.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:45.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c013740 2024-01-20T11:24:45.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:45.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:24:45.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:45.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c018ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:45.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:45.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:45.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:45.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.692+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:45.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.893+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:45.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.893+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c018ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:45.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.893+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:45.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.893+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:45.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:45.893+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:46.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:46.294+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:46.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:46.294+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c018ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:46.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:46.294+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:46.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:46.294+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:46.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:46.294+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:47.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:47.095+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:47.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:47.095+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c018ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:47.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:47.095+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:47.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:47.095+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:47.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:47.095+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c018ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,2] 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c009ec0 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c0130f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:48.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c009ec0 2024-01-20T11:24:48.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:48.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c0130f0 2024-01-20T11:24:48.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:48.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:48.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:48.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.692+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.893+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.893+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:48.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:48.893+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:49.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:49.294+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:49.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:49.294+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:49.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:49.294+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:50.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:50.095+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:50.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:50.095+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:50.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:50.095+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0166c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:51.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c013740 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c009ec0 addr v1:172.21.15.134:6789/0 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c0166c0 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c009ec0 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c013740 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:51.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.692+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:51.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.893+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:51.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.893+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:51.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.893+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:51.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:51.893+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:52.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:52.294+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:52.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:52.294+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:52.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:52.294+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:52.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:52.294+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:53.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:53.095+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:53.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:53.095+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:53.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:53.095+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:53.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:53.095+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:24:54.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:54.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:54.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:54.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:54.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c009ec0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 10 monclient: _add_conns ranks=[2,4] 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c005bb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c0130f0 addr v2:172.21.15.134:3300/0 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c0130f0 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:24:54.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:54.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:54.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.692+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:54.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.693+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:54.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.693+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.893+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.893+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.894+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:54.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:54.894+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:55.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:55.294+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:55.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:55.294+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:55.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:55.295+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:55.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:55.295+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:56.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:56.095+0000 7f367dfeb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:56.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:56.095+0000 7f367dfeb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:56.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:56.096+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:24:56.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:56.096+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:24:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:24:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:24:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:24:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c0130f0 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0166c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c005bb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c013740 addr v2:172.21.15.110:3300/0 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c0166c0 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c013740 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:24:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:24:57.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:24:57.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:57.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:57.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:57.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:57.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.693+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:24:57.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.894+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:57.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.894+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:57.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.894+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:57.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.894+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:57.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:57.894+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:24:58.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:58.295+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:58.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:58.295+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:58.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:58.295+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:58.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:58.295+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:58.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:58.295+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:24:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:59.096+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:24:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:59.096+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:24:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:59.096+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:24:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:59.096+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:24:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:24:59.096+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:00.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:00.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:00.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c013740 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,1] 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c009ec0 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c0130f0 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c0130f0 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c009ec0 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c0130f0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:00.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:00.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.693+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.894+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c0130f0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.894+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:00.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:00.894+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:01.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:01.295+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c0130f0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:01.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:01.295+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:01.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:01.295+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:02.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:02.096+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c0130f0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:02.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:02.096+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:02.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:02.096+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:03.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:03.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:03.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:03.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:03.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:03.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:03.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c0130f0 legacy=0x7f365c00a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:03.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-01-20T11:25:03.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c00aa40 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c005bb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c009ec0 addr v2:172.21.15.110:3300/0 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c00aa40 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c009ec0 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:03.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.693+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c019ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:03.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.894+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:03.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.894+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:03.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.894+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:03.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.894+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:03.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:03.894+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c019ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:04.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:04.295+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:04.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:04.295+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:04.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:04.295+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:04.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:04.295+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:04.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:04.295+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c019ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:05.096+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:05.096+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:05.096+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:05.096+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:05.096+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c019ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:06.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:06.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:06.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:06.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:06.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:06.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:06.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:06.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c019ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,0] 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c00a330 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0166c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c0166c0 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00a330 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:06.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.694+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:06.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.895+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:06.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.895+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:06.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:06.895+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:07.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:07.296+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:07.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:07.296+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:07.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:07.296+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:08.097+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:08.097+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:08.097+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:09.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:09.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:09.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:09.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:09.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:09.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:09.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:09.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-01-20T11:25:09.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c00aa40 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c010b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c00a330 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0166c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c0166c0 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c00aa40 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00a330 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:09.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.694+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.895+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.895+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.895+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:09.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:09.895+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:10.296+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:10.296+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:10.296+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:10.296+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:11.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:11.097+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:11.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:11.097+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:11.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:11.097+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:11.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:11.097+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c010b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c010b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c016ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 10 monclient: _add_conns ranks=[3,2] 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c009ec0 addr v2:172.21.15.110:3300/0 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c005bb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c009ec0 2024-01-20T11:25:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:25:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.694+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.895+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.895+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.895+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:12.895+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:13.296+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:13.296+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:13.296+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:13.296+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:14.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:14.097+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:14.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:14.097+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:14.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:14.097+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:14.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:14.097+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c00aa40 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:15.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c005bb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c009ec0 addr v2:172.21.15.110:3300/0 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c00aa40 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c009ec0 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:15.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:15.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.694+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:15.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.895+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:15.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.895+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:15.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.895+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:15.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.895+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:15.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:15.895+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:16.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:16.296+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:16.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:16.296+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:16.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:16.296+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:16.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:16.296+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:16.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:16.296+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:17.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:17.097+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:17.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:17.097+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:17.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:17.097+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:17.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:17.097+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:17.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:17.097+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:18.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:18.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:18.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:18.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:18.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:18.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00aa40 legacy=0x7f365c015f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,0] 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c016aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c00a330 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c0166c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c0166c0 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00a330 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:18.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c016aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:18.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.695+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c016aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:18.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.896+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:18.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.896+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c016aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:18.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:18.896+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c016aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:19.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:19.297+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:19.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:19.297+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c016aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:19.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:19.297+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c016aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:20.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:20.098+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:20.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:20.098+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c016aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:20.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:20.098+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c016aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:21.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c016aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c016aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c0166c0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c012b80 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c005bb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:21.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c00a330 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c012b80 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00a330 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f367e7ec640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:21.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.694+0000 7f367e7ec640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:21.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.895+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:21.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.895+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:21.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.895+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:21.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.895+0000 7f367e7ec640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:21.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:21.895+0000 7f367e7ec640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:22.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:22.296+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:22.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:22.296+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:22.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:22.296+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:22.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:22.296+0000 7f367e7ec640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:22.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:22.296+0000 7f367e7ec640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:23.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:23.097+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:23.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:23.097+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:23.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:23.097+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:23.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:23.097+0000 7f367e7ec640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:23.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:23.097+0000 7f367e7ec640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:24.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 10 monclient: _add_conns ranks=[3,1] 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c009ec0 addr v2:172.21.15.110:3300/0 2024-01-20T11:25:24.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c00a750 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:24.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c00a750 2024-01-20T11:25:24.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:24.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c009ec0 2024-01-20T11:25:24.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:24.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:24.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:24.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.694+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00a750 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:24.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.895+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:24.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.895+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:24.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:24.895+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00a750 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:25.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:25.296+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:25.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:25.296+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:25.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:25.296+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00a750 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:26.097+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:26.097+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:26.097+0000 7f367d7ea640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00a750 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:27.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:27.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:27.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:27.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:27.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:27.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:27.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c00a750 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:27.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-01-20T11:25:27.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c016aa0 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c005bb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c0136d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c009ec0 addr v2:172.21.15.110:3300/0 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c016aa0 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c009ec0 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c005bb0 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:27.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:27.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:27.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.695+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.896+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.896+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.896+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.896+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:27.896+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:28.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:28.297+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:28.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:28.297+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:28.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:28.297+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:28.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:28.297+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:28.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:28.297+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:29.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:29.098+0000 7f367e7ec640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:29.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:29.098+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:29.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:29.098+0000 7f367e7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:29.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:29.098+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:29.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:29.098+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,0] 2024-01-20T11:25:30.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c00a330 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c012b80 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c012b80 2024-01-20T11:25:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00a330 2024-01-20T11:25:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:30.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.695+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:30.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.896+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:30.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.896+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:30.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:30.896+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:31.297+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:31.297+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:31.297+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:32.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:32.098+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:32.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:32.098+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:32.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:32.098+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:33.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c013ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c016aa0 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c012b80 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c005bb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:33.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c012b80 2024-01-20T11:25:33.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c016aa0 2024-01-20T11:25:33.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:33.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c005bb0 2024-01-20T11:25:33.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:33.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:33.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:33.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:33.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.695+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:33.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.896+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:33.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.896+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:33.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.896+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:33.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:33.896+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:34.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:34.297+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:34.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:34.297+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:34.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:34.297+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:34.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:34.297+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:35.098+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:35.098+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:35.098+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:35.098+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c016e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,3] 2024-01-20T11:25:36.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c009ec0 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 0x7f365c0136d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c00a330 addr v2:172.21.15.110:3300/0 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00a330 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c009ec0 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:36.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.696+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:36.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.897+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:36.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.897+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:36.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.897+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:36.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:36.897+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:37.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:37.298+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:37.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:37.298+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:37.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:37.298+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:37.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:37.298+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:38.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:38.099+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:38.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:38.099+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:38.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:38.099+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:38.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:38.099+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c009ec0 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c0136d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c00a330 0x7f365c0136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c016aa0 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:39.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c005bb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c012b80 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c012b80 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c016aa0 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c005bb0 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.696+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:39.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.897+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:39.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.897+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:39.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.897+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:39.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:39.897+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:40.298+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:40.298+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:40.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:40.298+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:40.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:40.298+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:41.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:41.099+0000 7f367e7ec640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:41.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:41.099+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:41.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:41.099+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:41.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:41.099+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:42.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c005bb0 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c012b80 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,3] 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c00a330 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c009ec0 addr v2:172.21.15.110:3300/0 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c009ec0 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c00a330 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.695+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:42.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.696+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.696+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:42.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.696+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:42.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.897+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:42.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.897+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:42.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.897+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:42.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:42.897+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:43.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:43.298+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:43.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:43.298+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:43.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:43.298+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:43.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:43.298+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:44.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:44.099+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:44.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:44.099+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:44.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:44.099+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:44.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:44.099+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:45.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:45.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:45.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 msgr2=0x7f365c013690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c009ec0 0x7f365c013690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c01a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c017e90 addr v2:172.21.15.110:3300/0 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c00a330 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c016aa0 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c016aa0 2024-01-20T11:25:45.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:45.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c017e90 2024-01-20T11:25:45.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:45.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c00a330 2024-01-20T11:25:45.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:45.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c01a280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:45.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c01a280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:45.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:45.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:45.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.697+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:45.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.898+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c01a280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:45.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.898+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c01a280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:45.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.898+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:45.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.898+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:45.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:45.898+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:46.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:46.299+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c01a280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:46.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:46.299+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c01a280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:46.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:46.299+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:46.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:46.299+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:46.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:46.299+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:47.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:47.100+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c01a280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:47.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:47.100+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c01a280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:47.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:47.100+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:47.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:47.100+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:47.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:47.100+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c01a280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c01a280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:48.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 10 monclient: _add_conns ranks=[0,2] 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c005bb0 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c006ad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c005bb0 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c006ad0 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:48.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.697+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:48.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.898+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.898+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:48.898+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:49.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:49.299+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:49.299+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:49.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:49.299+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:50.100+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:50.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:50.100+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:50.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:50.100+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c005bb0 legacy=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c010b70 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c014f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c017e90 addr v2:172.21.15.110:3300/0 2024-01-20T11:25:51.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c016aa0 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c010b70 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c014b60 con 0x7f365c016aa0 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c017e90 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c01a7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c014f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:51.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.697+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:51.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.898+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:51.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.898+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c01a7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:51.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.898+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c014f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:51.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:51.898+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:52.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:52.299+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c01a7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:52.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:52.299+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:52.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:52.299+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c014f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:52.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:52.299+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:53.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:53.100+0000 7f367d7ea640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:25:53.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:53.100+0000 7f367dfeb640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c01a7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:53.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:53.100+0000 7f367d7ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c014f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:53.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:53.100+0000 7f367e7ec640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c014f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c014f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:54.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c01a7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 10 monclient: _add_conns ranks=[4,2] 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c006ad0 addr v2:172.21.15.134:3300/0 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c00a330 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c006ad0 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c00a330 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.697+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.698+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:54.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.698+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.898+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.898+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:54.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.899+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:54.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:54.899+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:55.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:55.299+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:55.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:55.299+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:55.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:55.300+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:55.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:55.300+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:56.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:56.100+0000 7f367d7ea640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:56.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:56.100+0000 7f367d7ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:56.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:56.101+0000 7f367dfeb640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:25:56.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:56.101+0000 7f367dfeb640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:57.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.697+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:25:57.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.697+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:25:57.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.697+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.697+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.697+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.697+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 msgr2=0x7f365c013660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c006ad0 0x7f365c013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f365c016aa0 addr v1:172.21.15.134:6789/0 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f367e7ec640 1 -- reap_dead start 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f365c010b70 addr v1:172.21.15.110:6789/0 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f365c00a330 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678146370 con 0x7f365c010b70 2024-01-20T11:25:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3678145460 con 0x7f365c016aa0 2024-01-20T11:25:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:25:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c00a330 2024-01-20T11:25:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:25:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c0154a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:25:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.698+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.899+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c0154a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.899+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.899+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:25:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:57.899+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:58.300+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:58.300+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c0154a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:58.300+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:25:58.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:58.300+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:25:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:59.101+0000 7f367e7ec640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:25:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:59.101+0000 7f367d7ea640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c0154a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:25:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:59.101+0000 7f367e7ec640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:25:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:25:59.101+0000 7f367dfeb640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:00.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 10 monclient(hunting): tick 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 msgr2=0x7f365c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f365c00a330 0x7f365c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f365c016aa0 legacy=0x7f365c013ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f365c010b70 legacy=0x7f365c0154a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 10 monclient: _add_conns ranks=[3,4] 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c006ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f365c017e90 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c014f60 0x7f365c013630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f365c014f60 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:00.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780b50f0 con 0x7f365c017e90 2024-01-20T11:26:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780aade0 con 0x7f365c014f60 2024-01-20T11:26:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f366f7fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c014f60 msgr2=0x7f365c013630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:00.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.698+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c014f60 0x7f365c013630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:00.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.899+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:00.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.899+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:00.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.899+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c014f60 msgr2=0x7f365c013630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:00.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:00.899+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c014f60 0x7f365c013630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:01.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:01.300+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:01.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:01.300+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:01.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:01.300+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c014f60 msgr2=0x7f365c013630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:01.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:01.300+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c014f60 0x7f365c013630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:02.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:02.101+0000 7f367dfeb640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 msgr2=0x7f365c006ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:02.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:02.101+0000 7f367dfeb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f365c017e90 0x7f365c006ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:02.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:02.101+0000 7f367d7ea640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f365c014f60 msgr2=0x7f365c013630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:02.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:02.101+0000 7f367d7ea640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f365c014f60 0x7f365c013630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-20T11:26:03.611 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-20T11:26:03.611 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-20T11:26:03.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.720+0000 7f5e34141640 10 monclient: get_monmap_and_config 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.720+0000 7f5e34141640 10 monclient: build_initial_monmap 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.720+0000 7f5e34141640 10 monclient: monmap: 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 16ed1bef-072b-4816-8684-5d140f76194c 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-20T11:26:03.720801+0000 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-20T11:26:03.720801+0000 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.134:6789/0 mon.noname-c-legacy 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.134:3300/0 mon.noname-c 2024-01-20T11:26:03.720 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:26:03.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.720+0000 7f5e34141640 1 Processor -- start 2024-01-20T11:26:03.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 1 -- start start 2024-01-20T11:26:03.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 10 monclient: init 2024-01-20T11:26:03.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 10 monclient: _reopen_session rank -1 2024-01-20T11:26:03.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-01-20T11:26:03.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e2c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:03.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 10 monclient(hunting): picked mon.noname-a con 0x7f5e2c0b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:03.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e2c0b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:03.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e2c145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e2c145980 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e2c0b17c0 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e2c0b4d10 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e34141640 10 monclient(hunting): authenticate will time out at 2007.071910s 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e2c0b3da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e2c145e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e2c0b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:03.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.721+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e2c0b1c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.922+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e2c145e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.922+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e2c0b3da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.922+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e2c0b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:03.922+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e2c0b1c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:04.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:04.323+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e2c145e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:04.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:04.323+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e2c0b3da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:04.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:04.323+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e2c0b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:04.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:04.323+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e2c0b1c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:05.124+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e2c145e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:05.124+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e2c0b3da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:05.124+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e2c0b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:05.124+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e2c0b1c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e2c0b3da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e2c0b3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e2c0b1c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e2c145e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 10 monclient: _add_conns ranks=[4,2] 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 0x7f5e10005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e100036a0 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 0x7f5e100085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e10006110 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100036a0 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e10006110 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.721+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.722+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 msgr2=0x7f5e10005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:06.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.722+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 0x7f5e10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:06.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.923+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:06.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.923+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:06.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.923+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 msgr2=0x7f5e10005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:06.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:06.923+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 0x7f5e10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:07.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:07.324+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:07.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:07.324+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:07.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:07.324+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 msgr2=0x7f5e10005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:07.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:07.324+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 0x7f5e10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:08.125+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:08.125+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:08.125+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 msgr2=0x7f5e10005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:08.125+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 0x7f5e10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:09.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:09.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:09.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:09.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:09.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:09.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10006110 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 msgr2=0x7f5e10005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100036a0 0x7f5e10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e1000afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e2c0b4d10 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e2c145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e2c0b17c0 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e2c145980 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e2c0b17c0 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e2c0b4d10 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e1000afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:09.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e1000afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:09.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.722+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:09.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.923+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:09.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.923+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e1000afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:09.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.923+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e1000afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:09.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:09.923+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:10.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:10.324+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:10.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:10.324+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e1000afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:10.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:10.324+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e1000afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:10.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:10.324+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:11.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:11.125+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:11.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:11.125+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e1000afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:11.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:11.125+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e1000afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:11.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:11.125+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:12.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:12.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 msgr2=0x7f5e1000afe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c0b4d10 0x7f5e1000afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e2c0b17c0 legacy=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 10 monclient: _add_conns ranks=[2,4] 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e100124f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10012a50 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10012a50 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100124f0 2024-01-20T11:26:12.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:12.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:12.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:12.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:12.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.722+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.923+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.923+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.923+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:12.923+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:13.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:13.324+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:13.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:13.324+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:13.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:13.324+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:13.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:13.324+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:14.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:14.125+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:14.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:14.125+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:14.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:14.125+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:14.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:14.125+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e2c145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:15.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e100124f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10012a50 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e2c145980 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10012a50 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100124f0 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:15.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.722+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:15.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.923+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:15.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.923+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:15.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.923+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:15.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.923+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:15.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:15.923+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:16.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:16.324+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:16.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:16.324+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:16.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:16.324+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:16.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:16.324+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:16.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:16.324+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:17.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:17.125+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:17.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:17.125+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:17.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:17.125+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:17.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:17.125+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:17.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:17.125+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100124f0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e1000d550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:18.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,3] 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10005c20 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 0x7f5e10010cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100042a0 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e10005c20 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100042a0 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:18.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.723+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005c20 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.924+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.924+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:18.924+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005c20 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:19.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:19.325+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:19.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:19.325+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:19.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:19.325+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005c20 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:20.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:20.126+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:20.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:20.126+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:20.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:20.126+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005c20 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100042a0 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005c20 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10012a50 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:26:21.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10004750 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e2c145980 addr v1:172.21.15.110:6789/0 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e2c145980 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e10004750 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10012a50 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10011210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:21.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.723+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:21.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.924+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:21.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.924+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:21.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.924+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:21.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:21.924+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10011210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:22.325+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:22.325+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:22.325+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10011210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:22.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:22.325+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:23.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:23.126+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:23.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:23.126+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10011210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:23.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:23.126+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:23.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:23.126+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:24.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:24.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:24.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:24.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:24.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:24.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:24.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10011210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:24.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e2c145980 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:24.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,2] 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 0x7f5e10010cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e10011920 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 0x7f5e100085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e10011d80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10011920 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e10011d80 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:24.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.723+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:24.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.924+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:24.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.924+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:24.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.924+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:24.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:24.924+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:25.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:25.324+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:25.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:25.324+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:25.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:25.324+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:25.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:25.324+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:26.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:26.125+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:26.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:26.125+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:26.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:26.125+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:26.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:26.125+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10011d80 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 msgr2=0x7f5e10010cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:27.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10011920 0x7f5e10010cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e10004cb0 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10012a50 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10004750 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e10004750 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10004cb0 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e10012a50 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:27.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:27.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:27.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:27.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.723+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:27.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.924+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:27.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.924+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:27.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.924+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:27.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.924+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:27.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:27.924+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:28.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:28.325+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:28.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:28.325+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:28.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:28.325+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:28.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:28.325+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:28.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:28.325+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:29.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:29.126+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:29.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:29.126+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:29.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:29.126+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:29.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:29.126+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:29.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:29.126+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:30.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:30.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:30.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:30.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 msgr2=0x7f5e10012e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012a50 0x7f5e10012e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 10 monclient: _add_conns ranks=[0,2] 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e10011d80 addr v1:172.21.15.110:6789/0 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 0x7f5e100085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e1000d870 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e10011d80 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:30.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e1000d870 2024-01-20T11:26:30.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:30.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:30.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:30.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.722+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:30.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.923+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:30.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.923+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:30.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:30.923+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:31.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:31.324+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:31.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:31.324+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:31.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:31.324+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:32.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:32.125+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:32.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:32.125+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:32.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:32.125+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:33.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000d870 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e10004cb0 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10004750 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e10010cd0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e10004750 2024-01-20T11:26:33.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:33.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10004cb0 2024-01-20T11:26:33.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:33.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e10010cd0 2024-01-20T11:26:33.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:33.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:33.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:33.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:33.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:33.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.723+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:33.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.924+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.924+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.924+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.924+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:33.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:33.924+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:34.325+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:34.325+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:34.325+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:34.325+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:34.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:34.325+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:35.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:35.126+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:35.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:35.126+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:35.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:35.126+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:35.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:35.126+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:35.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:35.126+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:36.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:36.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000d260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 10 monclient: _add_conns ranks=[0,4] 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e10011d80 addr v1:172.21.15.110:6789/0 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e100110b0 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e10011d80 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:36.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100110b0 2024-01-20T11:26:36.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:36.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:36.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:36.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.724+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:36.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.925+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:36.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.925+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:36.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:36.925+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:37.326+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:37.326+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:37.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:37.326+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:38.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:38.127+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:38.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:38.127+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:38.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:38.127+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:39.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:39.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:39.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:39.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:39.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:39.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:39.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10011d80 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:39.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e10012f60 addr v1:172.21.15.110:6789/0 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e10004cb0 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e100110b0 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e10012f60 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10004cb0 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100110b0 2024-01-20T11:26:39.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:39.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e100184f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:39.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:39.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:39.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:39.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.724+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:39.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.925+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:39.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.925+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e100184f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:39.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.925+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:39.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.925+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:39.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:39.925+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:40.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:40.326+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:40.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:40.326+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e100184f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:40.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:40.326+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:40.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:40.326+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:40.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:40.326+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:41.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:41.127+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:41.127+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e100184f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:41.127+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:41.127+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:41.127+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e100184f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,2] 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10004750 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:42.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e10010cd0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e10004750 2024-01-20T11:26:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10010cd0 2024-01-20T11:26:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:42.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.724+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:42.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.925+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:42.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.925+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:42.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:42.925+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:43.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:43.326+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:43.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:43.326+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:43.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:43.326+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:44.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:44.127+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:44.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:44.127+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:44.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:44.127+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:45.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:45.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 msgr2=0x7f5e100085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10010cd0 0x7f5e100085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 0x7f5e1000d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e10005090 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e100110b0 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:45.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e10004cb0 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10004cb0 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100110b0 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10011610 con 0x7f5e10005090 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 msgr2=0x7f5e1000d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e326de640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 0x7f5e1000d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e326de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:45.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.724+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:45.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.925+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 msgr2=0x7f5e1000d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:45.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.925+0000 7f5e326de640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:45.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.925+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 0x7f5e1000d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:45.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.925+0000 7f5e326de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:45.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.925+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:45.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:45.925+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:46.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:46.326+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 msgr2=0x7f5e1000d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:46.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:46.326+0000 7f5e326de640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:46.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:46.326+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 0x7f5e1000d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:46.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:46.326+0000 7f5e326de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:46.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:46.326+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:46.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:46.326+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:47.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:47.127+0000 7f5e326de640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:47.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:47.127+0000 7f5e326de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:47.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:47.127+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 msgr2=0x7f5e1000d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:47.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:47.127+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 0x7f5e1000d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:47.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:47.127+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:47.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:47.127+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:48.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:48.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:48.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:48.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 msgr2=0x7f5e1000d170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10005090 0x7f5e1000d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100110b0 0x7f5e10012a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,0] 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10012f60 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e10004750 addr v1:172.21.15.110:6789/0 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e10004750 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e10012f60 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e1000f1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:48.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.725+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:48.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.926+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e1000f1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:48.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:48.926+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:49.327+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e1000f1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:49.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:49.327+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:50.128+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e1000f1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:50.128+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10012f60 legacy=0x7f5e10003b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e1000f1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e10004cb0 addr v2:172.21.15.110:3300/0 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10004750 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:51.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e100110b0 addr v1:172.21.15.110:6789/0 2024-01-20T11:26:51.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e100110b0 2024-01-20T11:26:51.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e10004750 2024-01-20T11:26:51.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:51.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10004cb0 2024-01-20T11:26:51.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:51.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:51.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:51.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100110b0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:51.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.725+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.926+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100110b0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.926+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.926+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:51.926+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:52.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:52.327+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100110b0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:52.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:52.327+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:52.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:52.327+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:52.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:52.327+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:53.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:53.128+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:26:53.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:53.128+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100110b0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:26:53.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:53.128+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:53.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:53.128+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:54.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:54.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:54.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:54.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:54.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 msgr2=0x7f5e1000dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:54.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e10004cb0 0x7f5e1000dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:54.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:54.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100110b0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:54.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 10 monclient: _add_conns ranks=[4,2] 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e1000d170 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 0x7f5e10012e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e100129f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.725+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e1000d170 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.726+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.726+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100129f0 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.726+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.726+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.726+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.726+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:54.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.726+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:54.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.927+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:54.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.927+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:54.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.927+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:54.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:54.927+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:55.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:55.328+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:55.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:55.328+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:55.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:55.328+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:55.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:55.328+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:56.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:56.129+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:56.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:56.129+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:56.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:56.129+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:56.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:56.129+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:57.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:26:57.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:26:57.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:26:57.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 msgr2=0x7f5e10012e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100129f0 0x7f5e10012e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10010590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e100101b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10004750 addr v1:172.21.15.134:6789/0 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e1000d170 addr v2:172.21.15.134:3300/0 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e10004750 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e1000d170 2024-01-20T11:26:57.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:26:57.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100101b0 2024-01-20T11:26:57.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:26:57.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:57.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10010590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:57.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:57.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:57.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.726+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:26:57.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.927+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:57.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.927+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10010590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:57.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.927+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:57.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.927+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:57.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:57.927+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:26:58.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:58.328+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:58.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:58.328+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10010590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:58.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:58.328+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:58.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:58.328+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:58.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:58.328+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:26:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:59.129+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:26:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:59.129+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10010590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:26:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:59.129+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:26:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:59.129+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:26:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:26:59.129+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:00.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:00.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10010590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,0] 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100110b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e10004cb0 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e10004cb0 2024-01-20T11:27:00.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100110b0 2024-01-20T11:27:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:00.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.726+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004cb0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:00.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.927+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004cb0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:00.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.927+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:00.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:00.927+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:01.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:01.328+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004cb0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:01.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:01.328+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:01.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:01.328+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:02.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:02.129+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004cb0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:02.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:02.129+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:02.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:02.129+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:03.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004cb0 legacy=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e1000dc60 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:03.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100110b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e100101b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e1000dc60 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100110b0 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100101b0 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000dc60 legacy=0x7f5e100194f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.726+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:03.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.727+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:03.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.727+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:03.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.927+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000dc60 legacy=0x7f5e100194f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:03.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.928+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:03.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.928+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.928+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:03.928+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:04.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:04.328+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000dc60 legacy=0x7f5e100194f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:04.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:04.329+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:04.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:04.329+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:04.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:04.329+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:04.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:04.329+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:05.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:05.129+0000 7f5e31edd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000dc60 legacy=0x7f5e100194f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:05.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:05.130+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:05.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:05.130+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:05.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:05.130+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:05.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:05.130+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000dc60 legacy=0x7f5e100194f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,4] 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e1000d170 addr v1:172.21.15.134:6789/0 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10004750 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e1000d170 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10004750 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.727+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:06.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.928+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:06.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.928+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:06.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:06.928+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:07.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:07.329+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:07.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:07.329+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:07.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:07.329+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:08.130+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:08.130+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:08.130+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e100101b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:09.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10004750 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100110b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100110b0 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e10004750 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10011610 con 0x7f5e100101b0 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:09.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e326de640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:09.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:09.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e326de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:09.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:09.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:09.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.727+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.928+0000 7f5e326de640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.928+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.928+0000 7f5e326de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.928+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.928+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:09.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:09.928+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:10.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:10.329+0000 7f5e326de640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:10.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:10.329+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:10.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:10.329+0000 7f5e326de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:10.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:10.329+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:10.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:10.329+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:10.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:10.329+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:11.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:11.130+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:11.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:11.130+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:11.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:11.130+0000 7f5e326de640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:11.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:11.130+0000 7f5e326de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:11.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:11.130+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:11.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:11.130+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:12.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:12.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e100085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e100085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,0] 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e1000d170 addr v1:172.21.15.134:6789/0 2024-01-20T11:27:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e10010590 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e10010590 2024-01-20T11:27:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e1000d170 2024-01-20T11:27:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10010590 legacy=0x7f5e100075b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.727+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.928+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10010590 legacy=0x7f5e100075b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:12.928+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:13.329+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10010590 legacy=0x7f5e100075b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:13.329+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:14.130+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10010590 legacy=0x7f5e100075b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:14.130+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000d170 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10010590 legacy=0x7f5e100075b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100110b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:15.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e100101b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e10004750 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e10004750 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100110b0 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100101b0 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:15.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.728+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:15.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.929+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.929+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.929+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.929+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:15.929+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:16.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:16.330+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:16.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:16.330+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:16.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:16.330+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:16.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:16.330+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:16.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:16.330+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:17.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:17.131+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:17.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:17.131+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:17.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:17.131+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:17.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:17.131+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:17.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:17.131+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 msgr2=0x7f5e10005090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100101b0 0x7f5e10005090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 10 monclient: _add_conns ranks=[4,1] 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e1000fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:18.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e1000d170 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:18.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10010590 addr v1:172.21.15.134:6789/0 2024-01-20T11:27:18.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e10010590 2024-01-20T11:27:18.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:18.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e1000d170 2024-01-20T11:27:18.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:18.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10010590 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:18.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:18.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.728+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.929+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.929+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10010590 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:18.929+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:19.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:19.330+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10010590 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:19.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:19.330+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:19.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:19.330+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:20.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:20.131+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:20.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:20.131+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:20.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:20.131+0000 7f5e316dc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10010590 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:21.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e1000d170 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10010590 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100110b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e10004750 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:21.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e100085b0 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e10004750 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100110b0 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100085b0 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:21.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.728+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.929+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.929+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.929+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.929+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:21.929+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:22.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:22.330+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:22.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:22.330+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:22.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:22.330+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:22.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:22.330+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:22.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:22.330+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:23.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:23.131+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:23.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:23.131+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:23.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:23.131+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:23.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:23.131+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:23.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:23.131+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:24.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,2] 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10005090 addr v1:172.21.15.134:6789/0 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e100164e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e10005090 2024-01-20T11:27:24.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:24.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100164e0 2024-01-20T11:27:24.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:24.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:24.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:24.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.728+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005090 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:24.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.929+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:24.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.929+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:24.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:24.929+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005090 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:25.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:25.330+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:25.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:25.330+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:25.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:25.330+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005090 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:26.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:26.131+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:26.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:26.131+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:26.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:26.131+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005090 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:27.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:27.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:27.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:27.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:27.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:27.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:27.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10005090 legacy=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e100085b0 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e10004750 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e100101b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e10004750 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e100101b0 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100085b0 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.728+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.729+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.729+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:27.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.729+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:27.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.929+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:27.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.930+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:27.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.930+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:27.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:27.930+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:28.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:28.330+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:28.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:28.331+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:28.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:28.331+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:28.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:28.331+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:29.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:29.130+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:29.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:29.131+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:29.131+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:29.131+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:30.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.727+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:30.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.727+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:30.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.727+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:30.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.727+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:30.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.727+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e1000a6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:30.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.727+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e1000a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e10004750 legacy=0x7f5e10003b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,2] 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100110b0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 0x7f5e10009fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e1000ac30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100110b0 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e1000ac30 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:30.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 msgr2=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:30.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.728+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 0x7f5e10009fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:30.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.929+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:30.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.929+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:30.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.929+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 msgr2=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:30.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:30.929+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 0x7f5e10009fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:31.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:31.330+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 msgr2=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:31.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:31.330+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:31.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:31.330+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 0x7f5e10009fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:31.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:31.330+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:32.131+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:32.131+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:32.131+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 msgr2=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:32.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:32.131+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 0x7f5e10009fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 msgr2=0x7f5e10009fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000ac30 0x7f5e10009fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100110b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e100101b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100164e0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:33.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e10012e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e100085b0 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e100101b0 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100164e0 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e100085b0 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:33.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.728+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.929+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.929+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.929+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:33.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.929+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:33.929+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:34.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:34.330+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:34.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:34.330+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:34.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:34.330+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:34.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:34.330+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:34.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:34.330+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:35.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:35.131+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:35.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:35.131+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:35.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:35.131+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:35.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:35.131+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:35.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:35.131+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:36.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:36.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 msgr2=0x7f5e10012e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e100085b0 0x7f5e10012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 10 monclient: _add_conns ranks=[2,0] 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e10004750 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e1000a660 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e1000a660 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10004750 2024-01-20T11:27:36.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:36.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:36.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000a660 legacy=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:36.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.729+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:36.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.930+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000a660 legacy=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:36.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.930+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:36.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:36.930+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:37.331+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000a660 legacy=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:37.331+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:37.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:37.331+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:38.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:38.132+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000a660 legacy=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:38.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:38.132+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:38.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:38.132+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:39.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:39.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:39.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:39.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:39.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:39.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:39.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000a660 legacy=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:39.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-01-20T11:27:39.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e100101b0 addr v1:172.21.15.134:6789/0 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100164e0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e1000aa40 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e1000aa40 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e100101b0 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100164e0 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e10009680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.729+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:39.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.930+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:39.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.930+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e10009680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:39.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.930+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:39.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:39.930+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:40.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:40.331+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:40.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:40.331+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e10009680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:40.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:40.331+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:40.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:40.331+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:41.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:41.132+0000 7f5e316dc640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:41.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:41.132+0000 7f5e326de640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e10009680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:41.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:41.132+0000 7f5e316dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:41.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:41.132+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:42.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:42.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:42.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e100101b0 legacy=0x7f5e10010590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e10009680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 10 monclient: _add_conns ranks=[2,4] 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e10004750 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10012e00 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10012e00 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:42.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e10004750 2024-01-20T11:27:42.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:42.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:42.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:42.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:42.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.729+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:42.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.930+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:42.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.930+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:42.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.930+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:42.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:42.930+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:43.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:43.331+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:43.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:43.331+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:43.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:43.331+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:43.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:43.331+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:44.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:44.132+0000 7f5e31edd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:44.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:44.132+0000 7f5e31edd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:44.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:44.132+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:44.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:44.132+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:45.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:45.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:45.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100164e0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e10004750 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10012e00 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:45.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100164e0 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e10012e00 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10011610 con 0x7f5e10004750 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:45.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.729+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:45.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.930+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:45.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.930+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:45.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.930+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:45.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.930+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:45.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.930+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:45.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:45.930+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:46.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:46.331+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:46.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:46.331+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:46.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:46.331+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:46.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:46.331+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:46.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:46.331+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:46.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:46.331+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:47.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:47.132+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:47.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:47.132+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:47.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:47.132+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:47.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:47.132+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:47.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:47.132+0000 7f5e326de640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:47.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:47.132+0000 7f5e326de640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,0] 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e1000aa40 addr v1:172.21.15.134:6789/0 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e100108c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e100108c0 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0b2660 con 0x7f5e1000aa40 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.730+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:48.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.931+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:48.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:48.931+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:49.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:49.332+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:49.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:49.332+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:50.133+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:50.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:50.133+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100164e0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:51.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e10004750 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e1000aa40 addr v1:172.21.15.134:6789/0 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e1000aa40 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100164e0 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e10004750 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:51.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.730+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.931+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.931+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.931+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.931+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:51.931+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:52.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:52.332+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:52.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:52.332+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:52.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:52.332+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:52.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:52.332+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:52.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:52.332+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:53.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:53.133+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:53.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:53.133+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:53.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:53.133+0000 7f5e316dc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:53.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:53.133+0000 7f5e316dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:53.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:53.133+0000 7f5e326de640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e1000aa40 legacy=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:54.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 10 monclient: _add_conns ranks=[4,0] 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10012e00 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e100108c0 addr v1:172.21.15.110:6789/0 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e100108c0 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e10012e00 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:54.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.730+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:54.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.931+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:54.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.931+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:54.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:54.931+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:55.332+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:55.332+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:55.332+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:56.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:56.134+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:27:56.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:56.134+0000 7f5e31edd640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:56.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:56.134+0000 7f5e31edd640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:57.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:27:57.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 msgr2=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10012e00 0x7f5e10008f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100108c0 legacy=0x7f5e100094b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e326de640 1 -- reap_dead start 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5e100164e0 addr v2:172.21.15.110:3300/0 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5e10004750 addr v2:172.21.15.134:3300/0 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 0x7f5e100101b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-20T11:27:57.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5e1000aa40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b50f0 con 0x7f5e100164e0 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 -- --> v2:172.21.15.134:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e10008cb0 con 0x7f5e10004750 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): start opening mon connection 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10011610 con 0x7f5e1000aa40 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 msgr2=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 0x7f5e100101b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:57.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.731+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-20T11:27:57.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.932+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 msgr2=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:57.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.932+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:57.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.932+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 0x7f5e100101b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:57.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.932+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:57.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.932+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:57.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:57.932+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-20T11:27:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:58.333+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 msgr2=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:58.333+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:58.333+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:58.333+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 0x7f5e100101b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:58.333+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:58.333+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-20T11:27:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:59.134+0000 7f5e326de640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 msgr2=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-20T11:27:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:59.134+0000 7f5e31edd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-20T11:27:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:59.134+0000 7f5e326de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 0x7f5e100101b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:59.134+0000 7f5e31edd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:27:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:59.134+0000 7f5e316dc640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:3300/0 2024-01-20T11:27:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:27:59.134+0000 7f5e316dc640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-20T11:28:00.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 10 monclient(hunting): tick 2024-01-20T11:28:00.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 10 monclient(hunting): _check_auth_tickets 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 1 monclient(hunting): continuing hunt 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 10 monclient(hunting): _reopen_session rank -1 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 msgr2=0x7f5e100101b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e1000aa40 0x7f5e100101b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 1 -- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 msgr2=0x7f5e10003af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 1 --2- >> v2:172.21.15.134:3300/0 conn(0x7f5e10004750 0x7f5e10003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 msgr2=0x7f5e1000fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e100164e0 0x7f5e1000fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 10 monclient: _add_conns ranks=[1,0] 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e10012e00 addr v1:172.21.15.134:6789/0 2024-01-20T11:28:00.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e100106f0 addr v1:172.21.15.110:6789/0 2024-01-20T11:28:00.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0aade0 con 0x7f5e100106f0 2024-01-20T11:28:00.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c145460 con 0x7f5e10012e00 2024-01-20T11:28:00.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e237fe640 10 monclient(hunting): _renew_subs 2024-01-20T11:28:00.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100106f0 legacy=0x7f5e10009680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:28:00.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.731+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10012e00 legacy=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:28:00.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.932+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100106f0 legacy=0x7f5e10009680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:28:00.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:00.932+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10012e00 legacy=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:28:01.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:01.333+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100106f0 legacy=0x7f5e10009680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:28:01.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:01.333+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10012e00 legacy=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:28:02.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:02.134+0000 7f5e316dc640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f5e100106f0 legacy=0x7f5e10009680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-20T11:28:02.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-20T11:28:02.134+0000 7f5e31edd640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f5e10012e00 legacy=0x7f5e10008f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-01-20T11:28:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-20T11:28:03.649 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1474, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2775, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2711, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-01-20T11:28:03.651 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-20T11:28:03.652 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-20T11:28:03.652 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-20T11:28:03.652 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.653 INFO:tasks.ceph.osd.0:Stopped 2024-01-20T11:28:03.653 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-20T11:28:03.653 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.653 INFO:tasks.ceph.osd.1:Stopped 2024-01-20T11:28:03.653 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-20T11:28:03.653 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.653 INFO:tasks.ceph.osd.2:Stopped 2024-01-20T11:28:03.653 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-20T11:28:03.654 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.654 INFO:tasks.ceph.osd.3:Stopped 2024-01-20T11:28:03.654 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-20T11:28:03.654 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.654 INFO:tasks.ceph.osd.4:Stopped 2024-01-20T11:28:03.654 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-20T11:28:03.654 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-20T11:28:03.654 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1474, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2775, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2711, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-01-20T11:28:03.655 INFO:tasks.ceph.osd.5:Stopped 2024-01-20T11:28:03.655 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-20T11:28:03.655 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.655 INFO:tasks.ceph.osd.6:Stopped 2024-01-20T11:28:03.655 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-20T11:28:03.655 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.656 INFO:tasks.ceph.osd.7:Stopped 2024-01-20T11:28:03.656 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-20T11:28:03.656 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-20T11:28:03.656 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-20T11:28:03.656 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1474, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2775, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2711, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y' 2024-01-20T11:28:03.657 INFO:tasks.ceph.mgr.y:Stopped 2024-01-20T11:28:03.657 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-20T11:28:03.657 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.657 INFO:tasks.ceph.mgr.x:Stopped 2024-01-20T11:28:03.657 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-20T11:28:03.657 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-20T11:28:03.657 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.657 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-20T11:28:03.658 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1474, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2775, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2711, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-01-20T11:28:03.658 INFO:tasks.ceph.mon.a:Stopped 2024-01-20T11:28:03.658 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-20T11:28:03.658 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-20T11:28:03.658 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1474, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2775, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2711, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-01-20T11:28:03.659 INFO:tasks.ceph.mon.c:Stopped 2024-01-20T11:28:03.659 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-20T11:28:03.659 INFO:teuthology.orchestra.run:waiting for 300 2024-01-20T11:28:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-20T11:28:03.659 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1474, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2775, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2711, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-01-20T11:28:03.660 INFO:tasks.ceph.mon.b:Stopped 2024-01-20T11:28:03.660 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-20T11:28:03.660 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-01-20T11:28:03.698 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:28:03.698 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-20T11:28:03.814 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:28:03.814 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-20T11:28:03.878 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:28:03.878 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-20T11:28:03.935 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:28:03.935 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-20T11:28:04.012 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:28:04.012 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-20T11:28:04.073 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:28:04.074 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-20T11:28:04.129 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:28:04.129 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-20T11:28:04.187 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:28:04.203 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-20T11:28:04.261 INFO:tasks.ceph:Archiving mon data... 2024-01-20T11:28:04.263 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/data/mon.a.tgz 2024-01-20T11:28:04.263 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-01-20T11:28:04.280 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.8NcJVJdbfm 2024-01-20T11:28:04.280 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.8NcJVJdbfm 2024-01-20T11:28:04.411 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.8NcJVJdbfm 2024-01-20T11:28:04.496 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.8NcJVJdbfm is 250KB 2024-01-20T11:28:04.590 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.8NcJVJdbfm 2024-01-20T11:28:04.604 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/data/mon.c.tgz 2024-01-20T11:28:04.604 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-01-20T11:28:04.658 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.xDWYmXkgEY 2024-01-20T11:28:04.658 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xDWYmXkgEY 2024-01-20T11:28:04.766 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.xDWYmXkgEY 2024-01-20T11:28:04.851 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.xDWYmXkgEY is 251KB 2024-01-20T11:28:04.944 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.xDWYmXkgEY 2024-01-20T11:28:04.958 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/data/mon.b.tgz 2024-01-20T11:28:04.959 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-01-20T11:28:04.977 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.OMFfBcLFVb 2024-01-20T11:28:04.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.OMFfBcLFVb 2024-01-20T11:28:05.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.OMFfBcLFVb 2024-01-20T11:28:05.185 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.OMFfBcLFVb is 250KB 2024-01-20T11:28:05.278 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.OMFfBcLFVb 2024-01-20T11:28:05.292 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-20T11:28:05.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-20T11:28:05.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-20T11:28:05.614 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-20T11:28:05.848 INFO:tasks.ceph:Archiving crash dumps... 2024-01-20T11:28:05.849 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/remote/smithi110/crash 2024-01-20T11:28:05.850 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-20T11:28:05.877 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/remote/smithi134/crash 2024-01-20T11:28:05.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-20T11:28:05.906 INFO:tasks.ceph:Compressing logs... 2024-01-20T11:28:05.906 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-20T11:28:05.922 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr: 67.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-01-20T11:28:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34001.log 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34252.log 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.34001.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.34252.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34001.log.gz 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.audit.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34252.log.gz 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34161.log 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34121.log 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.34161.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.34121.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34161.log.gz 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34041.log 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34121.log.gz 2024-01-20T11:28:05.999 INFO:teuthology.orchestra.run.smithi134.stderr: -5 90.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34081.log 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.34041.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.34081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34292.log 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34041.log.gz 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34081.log.gz 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.34292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34205.log 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34292.log.gz 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.34205.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34205.log.gz 2024-01-20T11:28:06.000 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-20T11:28:06.030 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mon.b.log: 91.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-20T11:28:06.069 INFO:teuthology.orchestra.run.smithi134.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-20T11:28:06.088 INFO:teuthology.orchestra.run.smithi134.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-20T11:28:06.088 INFO:teuthology.orchestra.run.smithi134.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-20T11:28:06.101 INFO:teuthology.orchestra.run.smithi134.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51127.log 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43103.log 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47137.log 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47347.log 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51127.log.gz 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49279.log 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43103.log.gz 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51085.log 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47137.log.gz 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39491.log 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47347.log.gz 2024-01-20T11:28:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51085.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49279.log.gz 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49111.log 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51085.log.gz 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38148.log 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39491.log.gz 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48481.log 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51967.log 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38148.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49111.log.gz 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41717.log 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38148.log.gz 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43943.log 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48481.log.gz 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4%/var/log/ceph/ceph-client.admin.41717.log: -- replaced with /var/log/ceph/ceph-client.admin.51967.log.gz 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46968.log 2024-01-20T11:28:06.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48817.log 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41717.log.gz 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43859.log 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43943.log.gz 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34576.log 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46968.log.gz 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43859.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48817.log.gz 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38314.log 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43607.log 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43859.log.gz 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42389.log 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34576.log.gz 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43607.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38314.log.gz 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46170.log 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47095.log 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42389.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43607.log.gz 2024-01-20T11:28:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45330.log 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42725.log 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42389.log.gz 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46170.log.gz 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42599.log 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47095.log.gz 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52303.log 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45330.log.gz 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40205.log 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42725.log.gz 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44237.log 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42599.log.gz 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38493.log 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52303.log.gz 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47221.log 2024-01-20T11:28:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44237.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40205.log.gz 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38490.log 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51799.log 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44237.log.gz 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38493.log.gz 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40751.log 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47221.log.gz 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47389.log 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40751.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38490.log.gz 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46926.log 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51799.log.gz 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40751.log.gz 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50833.log 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47389.log.gz 2024-01-20T11:28:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39827.log 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42473.log 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46926.log.gz 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41423.log 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.50833.log.gz 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44279.log 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42473.log: /var/log/ceph/ceph-client.admin.41423.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39827.log.gz 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44279.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42473.log.gz 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41423.log.gz 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/mon.c.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44279.log.gz 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-01-20T11:28:06.110 INFO:teuthology.orchestra.run.smithi110.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr: 64.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr: 64.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43565.log 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52009.log 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39869.log 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47263.log 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43565.log.gz 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45750.log 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52009.log.gz 2024-01-20T11:28:06.111 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39533.log 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47263.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39869.log.gz 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42347.log 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47263.log.gz 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46590.log 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45750.log.gz 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48565.log 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42347.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39533.log.gz 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41171.log 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46380.log 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42347.log.gz 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46590.log.gz 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50749.log 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48565.log.gz 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47809.log 2024-01-20T11:28:06.112 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41171.log.gz 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42977.log 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46380.log.gz 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42935.log 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50749.log.gz 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42137.log 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47809.log.gz 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45624.log 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42935.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42977.log.gz 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40961.log 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49405.log 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42935.log.gz 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42137.log.gz 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49909.log 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45624.log.gz 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39575.log 2024-01-20T11:28:06.113 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40961.log.gz 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39659.log 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49405.log.gz 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46086.log 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49909.log.gz 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44783.log 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39575.log.gz 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48775.log 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39659.log.gz 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49783.log 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46086.log.gz 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38033.log 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44783.log.gz 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38491.log 2024-01-20T11:28:06.114 INFO:teuthology.orchestra.run.smithi110.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48775.log.gz 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38495.log 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49783.log.gz 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45161.log 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38033.log.gz 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41549.log 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38491.log.gz 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47179.log 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38495.log.gz 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50623.log 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45161.log.gz 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47179.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.41549.log.gz 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40037.log 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46464.log 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50623.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47179.log.gz 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42095.log 2024-01-20T11:28:06.115 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51673.log 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46464.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40037.log.gz 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50623.log.gz 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50791.log 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51673.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46464.log.gz 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43691.log 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52135.log 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42095.log.gz 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50791.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51673.log.gz 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52135.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50791.log.gz 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50161.log 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43691.log.gz 2024-01-20T11:28:06.116 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40079.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52135.log.gz 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51337.log 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50287.log 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49573.log 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50161.log.gz 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48901.log 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51337.log.gz 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42431.log 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50287.log.gz 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52840.log 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49573.log.gz 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44531.log 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48901.log.gz 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39449.log 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42431.log.gz 2024-01-20T11:28:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43271.log 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45666.log 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44531.log.gz 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52051.log 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39449.log.gz 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44405.log 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43271.log.gz 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52177.log 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52051.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45666.log.gz 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46674.log 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr: 95.3% -- replaced with /var/log/ceph/ceph-client.admin.52840.log.gz 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43187.log 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52177.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52051.log.gz 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41633.log 2024-01-20T11:28:06.118 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44405.log.gz 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43187.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52177.log.gz 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40541.log 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46674.log.gz 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38092.log 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41633.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43187.log.gz 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40499.log 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42557.log 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40541.log.gz 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41633.log.gz 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44111.log 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38092.log.gz 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51295.log 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42557.log: /var/log/ceph/ceph-client.admin.44111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49657.log 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40499.log.gz 2024-01-20T11:28:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51295.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42557.log.gz 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50203.log 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44111.log.gz 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51169.log 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49657.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51295.log.gz 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42893.log 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38188.log 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49657.log.gz 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50203.log.gz 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48985.log 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51169.log.gz 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49489.log 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38188.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42893.log.gz 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44447.log 2024-01-20T11:28:06.120 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38188.log.gz 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40163.log 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48985.log.gz 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52597.log 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44447.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49489.log.gz 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43355.log 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47683.log 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44447.log.gz 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40163.log.gz 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51463.log 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52597.log.gz 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47557.log 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43355.log.gz 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51463.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47683.log.gz 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49321.log 2024-01-20T11:28:06.121 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51463.log.gz 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40625.log 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47557.log.gz 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44069.log 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51841.log 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49321.log.gz 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48229.log 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40625.log.gz 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51589.log 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44069.log.gz 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51631.log 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51841.log.gz 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr: 94.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45918.log 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51589.log: /var/log/ceph/ceph-client.admin.51631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50497.log 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48229.log.gz 2024-01-20T11:28:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45918.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51589.log.gz 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:gzip 87.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.51631.log.gz 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.38494.log 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47725.log 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45918.log.gz 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42305.log 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46548.log 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50497.log.gz 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45456.log 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47725.log.gz 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43397.log 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42305.log.gz 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43649.log 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46548.log.gz 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40415.log 2024-01-20T11:28:06.123 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45456.log.gz 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46296.log 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43397.log.gz 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51001.log 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43649.log.gz 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34461.log 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40415.log.gz 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43313.log 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46296.log.gz 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48061.log 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51001.log.gz 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40289.log 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34461.log.gz 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48061.log: gzip 87.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.43313.log.gz --verbose 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.46212.log 2024-01-20T11:28:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42263.log 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48061.log.gz 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50413.log 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40289.log.gz 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48943.log 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46212.log.gz 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50455.log 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42263.log.gz 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44363.log 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50413.log.gz 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39123.log 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48943.log.gz 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46716.log 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50455.log.gz 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45582.log 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44363.log.gz 2024-01-20T11:28:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49195.log 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39123.log.gz 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44615.log 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46716.log.gz 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48439.log 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45582.log.gz 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40667.log 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49195.log.gz 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50035.log 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48439.log: /var/log/ceph/ceph-client.admin.40667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45245.log 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44615.log.gz 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50035.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.48439.log.gz 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46632.log 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40667.log.gz 2024-01-20T11:28:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38775.log 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45245.log: /var/log/ceph/ceph-client.admin.46632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45540.log 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50035.log.gz 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38775.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45245.log.gz 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46758.log 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46632.log.gz 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50959.log 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38775.log.gz 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44867.log 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40247.log 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45540.log.gz 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46758.log.gz 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49153.log 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50959.log.gz 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49615.log 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40247.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44867.log.gz 2024-01-20T11:28:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45372.log 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40247.log.gz 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45708.log 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49153.log.gz 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48607.log 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45372.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49615.log.gz 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47473.log 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45372.log.gz 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43901.log 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48607.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45708.log.gz 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51043.log 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42641.log 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48607.log.gz 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47473.log.gz 2024-01-20T11:28:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38860.log 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43901.log.gz 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52261.log 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42641.log: /var/log/ceph/ceph-client.admin.38860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39323.log 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51043.log.gz 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52261.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42641.log.gz 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49447.log 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38860.log.gz 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39323.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52261.log.gz 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49447.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43019.log 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49447.log.gz 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39323.log.gz 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47599.log 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2024-01-20T11:28:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42809.log 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43019.log: /var/log/ceph/ceph-client.admin.47599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44321.log 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34434.log 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43019.log.gz 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47599.log.gz 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50371.log 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42809.log.gz 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34130.log 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50371.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44321.log.gz 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41087.log 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34434.log.gz 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41297.log 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50371.log.gz 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39280.log 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49027.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41087.log.gz 2024-01-20T11:28:06.130 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39280.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41297.log.gz 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48145.log 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.34130.log.gz 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44699.log 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39280.log.gz 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49363.log 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49027.log.gz 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44699.log: gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48145.log.gz 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.52471.log 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49363.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44699.log.gz 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38746.log 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50077.log 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49363.log.gz 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41003.log 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52471.log.gz 2024-01-20T11:28:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42515.log 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38746.log.gz 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49237.log 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42515.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50077.log.gz 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41003.log.gz 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41213.log 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42515.log.gz 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38230.log 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41213.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49237.log.gz 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52639.log 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47641.log 2024-01-20T11:28:06.132 INFO:teuthology.orchestra.run.smithi110.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41213.log.gz 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37960.log 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38230.log.gz 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44195.log 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52639.log.gz 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40373.log 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47641.log.gz 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39617.log 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37960.log.gz 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41381.log 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.39365.log 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44195.log.gz 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39617.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40373.log.gz 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43145.log 2024-01-20T11:28:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46044.log 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39365.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39617.log.gz 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50581.log 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41381.log.gz 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39365.log.gz 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39911.log 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43145.log.gz 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49069.log 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50581.log: /var/log/ceph/ceph-client.admin.39911.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46044.log.gz 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46422.log 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39911.log.gz 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50581.log.gz 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40877.log 2024-01-20T11:28:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46422.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49069.log.gz 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49741.log 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38986.log 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40877.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46422.log.gz 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39191.log 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38986.log: gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40877.log.gz 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52765.log 87.7% 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49741.log.gz 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46002.log 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38986.log.gz 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52765.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39191.log.gz 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49699.log 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44027.log 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46002.log: /var/log/ceph/ceph-client.admin.49699.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52555.log 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52765.log.gz 2024-01-20T11:28:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44741.log 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46002.log.gz 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49699.log.gz 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47851.log 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52555.log: 88.1%/var/log/ceph/ceph-client.admin.44741.log: -- replaced with /var/log/ceph/ceph-client.admin.44027.log.gz 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36381.log 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52555.log.gz 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42851.log 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44741.log.gz 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44951.log 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36381.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47851.log.gz 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44951.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36381.log.gz 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39076.log 2024-01-20T11:28:06.136 INFO:teuthology.orchestra.run.smithi110.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42851.log.gz 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44951.log.gz 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38732.log 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44909.log 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39076.log.gz 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48313.log 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38732.log.gz 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49951.log 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44909.log.gz 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48103.log 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2024-01-20T11:28:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49825.log 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48313.log.gz 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51757.log 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49951.log.gz 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39407.log 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48103.log.gz 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39743.log 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49825.log.gz 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37721.log 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51757.log.gz 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52429.log 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39407.log.gz 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48355.log 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39743.log.gz/var/log/ceph/ceph-client.admin.52429.log: 2024-01-20T11:28:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48859.log 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48355.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50917.log 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37721.log.gz 87.9% 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52429.log.gz 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41129.log 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48355.log.gz 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52681.log 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48859.log.gz 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38791.log 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50917.log.gz 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47053.log 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41129.log.gz 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51883.log 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52681.log.gz 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43817.log 2024-01-20T11:28:06.139 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38791.log.gz 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/ceph-client.admin.51883.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45077.log 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47053.log.gz 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43817.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51883.log.gz 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46884.log 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43817.log.gz 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45498.log 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46884.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45077.log.gz 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39006.log 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43985.log 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39006.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46884.log.gz 2024-01-20T11:28:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45498.log.gz 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.50875.log 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43985.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39006.log.gz 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44657.log 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50875.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43985.log.gz 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46800.log 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50875.log.gz 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47515.log 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/ceph-client.admin.46800.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48019.log 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44657.log.gz/var/log/ceph/ceph-client.admin.47515.log: 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43229.log 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46800.log.gz 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48019.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47515.log.gz 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46254.log 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48019.log.gz 2024-01-20T11:28:06.141 INFO:teuthology.orchestra.run.smithi110.stderr: -5/var/log/ceph/ceph-client.admin.43229.log: --verbose -- /var/log/ceph/ceph-client.admin.48523.log 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41675.log 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0%/var/log/ceph/ceph-client.admin.48523.log: -- replaced with /var/log/ceph/ceph-client.admin.43229.log.gz 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48187.log 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46254.log.gz 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:gzip 88.4% -5 --verbose -- /var/log/ceph/ceph-client.admin.39953.log 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48523.log.gz/var/log/ceph/ceph-client.admin.41675.log: 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52807.log 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5%/var/log/ceph/ceph-client.admin.39953.log: -- replaced with /var/log/ceph/ceph-client.admin.41675.log.gz 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45414.log 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48187.log.gz 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5/var/log/ceph/ceph-client.admin.52807.log: --verbose 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39953.log.gz 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.45203.log 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49867.log 2024-01-20T11:28:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45203.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.45414.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.39785.log 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49867.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45203.log.gz 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38745.log 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49867.log.gz 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40709.log 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39785.log: /var/log/ceph/ceph-client.admin.38745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42011.log 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr: 95.3% 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52807.log.gz 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48691.log 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38745.log.gz 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39785.log.gz 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40709.log.gz 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41045.log 2024-01-20T11:28:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45834.log 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48691.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42011.log.gz/var/log/ceph/ceph-client.admin.41045.log: 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51715.log 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48691.log.gz 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39995.log 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41045.log.gz 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/ceph-client.admin.51715.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41759.log 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45834.log.gz 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41885.log 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51715.log.gz 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41759.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39995.log.gz 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40457.log 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50707.log 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8%/var/log/ceph/ceph-client.admin.40457.log: -- replaced with /var/log/ceph/ceph-client.admin.41759.log.gz 2024-01-20T11:28:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38744.log 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41885.log.gz 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50707.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40457.log.gz 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50245.log 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38744.log: gzip -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50707.log.gz 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.51505.log 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38744.log.gz 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50665.log 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51505.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50245.log.gz 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45876.log 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40583.log 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50665.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51505.log.gz 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47305.log 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40583.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50665.log.gz 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr: -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45876.log.gz 2024-01-20T11:28:06.145 INFO:teuthology.orchestra.run.smithi110.stderr: /var/log/ceph/ceph-client.admin.38496.log 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42683.log 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40583.log.gz 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38496.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47305.log.gz 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42179.log 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42683.log: /var/log/ceph/ceph-osd.0.log: gzip -5 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38496.log.gz 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43481.log 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42179.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42683.log.gz 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47431.log 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43481.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42179.log.gz 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48649.log 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47431.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43481.log.gz 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40835.log 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48649.log: gzip -5 87.8% --verbose -- /var/log/ceph/ceph-client.admin.47893.log 2024-01-20T11:28:06.146 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47431.log.gz 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44153.log 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48649.log.gz 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47893.log: -- /var/log/ceph/ceph-client.admin.46842.log 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40835.log.gz 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47893.log.gz 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46842.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44153.log.gz 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51379.log 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28279.log 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3%/var/log/ceph/ceph-client.admin.51547.log: -- replaced with /var/log/ceph/ceph-client.admin.46842.log.gz 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45792.log 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.tmp-client.admin.28279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52723.log 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28279.log.gz 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2024-01-20T11:28:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45792.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51379.log.gz 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38272.log 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45792.log.gz 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48397.log 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38272.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48397.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52723.log.gz 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45287.log 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38272.log.gz 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/ceph.log: -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48397.log.gz 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39701.log 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52093.log 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39701.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45287.log.gz 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40121.log 2024-01-20T11:28:06.148 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39701.log.gz 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39124.log 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.40121.log: 94.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52093.log.gz 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44489.log 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40121.log.gz 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41927.log 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44489.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39124.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43439.log 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44489.log.gz 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41927.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41927.log.gz 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41255.log 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47935.log 2024-01-20T11:28:06.149 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43439.log.gz 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41801.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41255.log.gz 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45960.log 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47010.log 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47935.log.gz 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45960.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45960.log.gz 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44825.log 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52219.log 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51925.log 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47010.log.gz 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52219.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44825.log.gz 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52219.log.gz 2024-01-20T11:28:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38487.log 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51211.log 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45119.log 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51925.log.gz 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51211.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38487.log.gz 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51211.log.gz 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52513.log 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42221.log 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52387.log 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45119.log.gz 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42221.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52513.log.gz 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42221.log.gz 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46338.log 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46506.log 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50329.log 2024-01-20T11:28:06.151 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52387.log.gz 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46506.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46338.log.gz 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46506.log.gz 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51421.log 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40331.log 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43733.log 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50329.log.gz 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40331.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51253.log 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51421.log.gz 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35391.log 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40331.log.gz 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.51253.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46128.log 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43733.log.gz 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51253.log.gz 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35391.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35391.log.gz 2024-01-20T11:28:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52345.log 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.46128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38492.log 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.52345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49531.log 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46128.log.gz 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38492.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52345.log.gz 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42767.log 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50119.log 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38492.log.gz/var/log/ceph/ceph-client.admin.42767.log: 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49531.log.gz 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42767.log.gz 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr: /var/log/ceph/ceph-client.admin.43775.log 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48271.log 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.43775.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50119.log.gz 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44993.log 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48733.log 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9%/var/log/ceph/ceph-client.admin.48271.log: -- replaced with /var/log/ceph/ceph-client.admin.43775.log.gz 2024-01-20T11:28:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44993.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.48271.log.gz 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44993.log.gz 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41465.log 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.48733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41591.log 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43523.log 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48733.log.gz 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41591.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41465.log.gz 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41591.log.gz 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49993.log 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/ceph-client.admin.43523.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47977.log 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.49993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41843.log 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43523.log.gz 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47977.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49993.log.gz 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50539.log 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47977.log.gz 2024-01-20T11:28:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47767.log 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40919.log 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.50539.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41843.log.gz 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.47767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45035.log 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50539.log.gz 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40919.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47767.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42053.log 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40919.log.gz 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.45035.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45035.log.gz 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44573.log 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.c.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42053.log.gz 2024-01-20T11:28:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.44573.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44573.log.gz 2024-01-20T11:28:06.156 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-20T11:28:06.156 INFO:teuthology.orchestra.run.smithi110.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-20T11:28:06.156 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-20T11:28:06.162 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-20T11:28:06.189 INFO:teuthology.orchestra.run.smithi134.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-20T11:28:06.191 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-20T11:28:06.191 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m0.225s 2024-01-20T11:28:06.191 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m0.660s 2024-01-20T11:28:06.192 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m0.040s 2024-01-20T11:28:06.273 INFO:teuthology.orchestra.run.smithi110.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-20T11:28:06.340 INFO:teuthology.orchestra.run.smithi110.stderr: 89.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-20T11:28:06.342 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-20T11:28:06.343 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m0.409s 2024-01-20T11:28:06.343 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m1.274s 2024-01-20T11:28:06.343 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m0.268s 2024-01-20T11:28:06.343 INFO:tasks.ceph:Archiving logs... 2024-01-20T11:28:06.343 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/remote/smithi110/log 2024-01-20T11:28:06.344 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-20T11:28:06.893 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/remote/smithi134/log 2024-01-20T11:28:06.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-20T11:28:07.029 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1474, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2775, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2711, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-20T11:28:07.240 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=95d0a83841994618ab70022ecc08a6a1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1474, in healthy manager.wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2775, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2711, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a00c9940991c5e1e9ab595faa3f6e1281d8ae167/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-20T11:28:07.249 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-20T11:28:07.271 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-20T11:28:07.298 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-20T11:28:07.299 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-20T11:28:07.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-20T11:28:07.393 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-20T11:28:07.401 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-20T11:28:07.401 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-20T11:28:07.403 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-20T11:28:07.417 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-01-20T11:28:07.419 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-01-20T11:28:07.421 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-20T11:28:07.422 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-01-20T11:28:07.422 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +461us[ +461us] +/- 75ms 2024-01-20T11:28:07.422 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv02.front.sepia.ceph.com 3 7 377 65 -799us[ -770us] +/- 70ms 2024-01-20T11:28:07.422 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 70 +781us[ +810us] +/- 90ms 2024-01-20T11:28:07.422 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:28:07.424 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-20T11:28:07.424 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-01-20T11:28:07.424 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 +380us[ +398us] +/- 75ms 2024-01-20T11:28:07.424 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -866us[ -848us] +/- 71ms 2024-01-20T11:28:07.424 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +663us[ +682us] +/- 91ms 2024-01-20T11:28:07.424 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-20T11:28:07.424 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-20T11:28:07.465 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-20T11:28:07.466 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-20T11:28:07.507 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-20T11:28:07.583 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-20T11:28:07.653 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-20T11:28:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-20T11:28:07.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-20T11:28:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-20T11:28:07.716 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-20T11:28:07.726 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-20T11:28:07.756 INFO:teuthology.task.internal:Duration was 1801.352887 seconds 2024-01-20T11:28:07.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-20T11:28:07.799 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-20T11:28:07.799 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-20T11:28:07.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-20T11:28:07.842 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-20T11:28:07.846 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-20T11:28:08.177 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-20T11:28:08.177 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-01-20T11:28:08.177 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-20T11:28:08.197 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-01-20T11:28:08.197 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-20T11:28:08.224 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-20T11:28:08.224 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-20T11:28:08.239 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-20T11:28:08.293 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-20T11:28:08.294 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-20T11:28:08.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-20T11:28:08.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-20T11:28:08.443 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-20T11:28:08.443 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-20T11:28:08.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-20T11:28:08.496 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-20T11:28:08.505 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-20T11:28:08.512 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-20T11:28:08.535 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-01-20T11:28:08.561 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-01-20T11:28:08.588 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-20T11:28:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-20T11:28:08.611 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-20T11:28:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-20T11:28:08.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-20T11:28:08.652 INFO:teuthology.task.internal:Transferring archived files... 2024-01-20T11:28:08.652 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/remote/smithi110 2024-01-20T11:28:08.652 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-20T11:28:08.688 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621/remote/smithi134 2024-01-20T11:28:08.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-20T11:28:08.722 INFO:teuthology.task.internal:Removing archive directory... 2024-01-20T11:28:08.722 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-20T11:28:08.725 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-20T11:28:08.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-20T11:28:08.782 INFO:teuthology.task.internal:Not uploading archives. 2024-01-20T11:28:08.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-20T11:28:08.835 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-20T11:28:08.835 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-20T11:28:08.838 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-20T11:28:08.849 INFO:teuthology.orchestra.run.smithi110.stdout: 266028 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 20 11:28 /home/ubuntu/cephtest 2024-01-20T11:28:08.851 INFO:teuthology.orchestra.run.smithi134.stdout: 265295 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 20 11:28 /home/ubuntu/cephtest 2024-01-20T11:28:08.852 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-20T11:28:08.901 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-20T11:28:08.982 INFO:teuthology.nuke:Checking targets against current locks 2024-01-20T11:28:09.008 DEBUG:teuthology.nuke:shortname: smithi110 2024-01-20T11:28:09.009 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-20T11:28:09.031 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-20 10:49:57.727375', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtYQuegvVyO6UZG27OP9qpqVkjKdMu9D5GDJXb4WYgvaW0gqlywdzyv+IURvYFDwwphGS9JaMvWDYVc0Logf+c='} 2024-01-20T11:28:09.034 DEBUG:teuthology.nuke:shortname: smithi134 2024-01-20T11:28:09.034 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-20T11:28:09.054 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_15:25:09-rados-main-distro-default-smithi/7522621', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-20 10:49:57.726009', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNf/dmXNPIdSY8E4I0n7Bmhwc9mJyxV0TODuUn9bQpvSl5ohdkYiIji26ylANr2L59U4qZ8itxKut7Q+UOPlBjU='} 2024-01-20T11:28:09.083 INFO:teuthology.orchestra.console.smithi110:Power off 2024-01-20T11:28:09.083 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-20T11:28:09.106 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2024-01-20T11:28:09.106 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-20T11:28:09.120 INFO:teuthology.orchestra.console.smithi134:Power off 2024-01-20T11:28:09.120 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-20T11:28:09.128 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-01-20T11:28:09.141 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-01-20T11:28:09.141 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-20T11:28:09.162 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-20T11:28:13.129 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-20T11:28:13.163 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-20T11:28:13.239 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-01-20T11:28:13.272 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-20T11:28:17.245 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-20T11:28:17.273 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-20T11:28:17.357 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2024-01-20T11:28:17.385 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-20T11:28:17.458 INFO:teuthology.orchestra.console.smithi110:Power off completed 2024-01-20T11:28:21.390 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-20T11:28:21.500 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-01-20T11:28:21.600 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-01-20T11:28:21.705 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 1801.352886915207 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=95d0a83841994618ab70022ecc08a6a1 status: fail success: false 2024-01-20T11:28:21.705 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-20T11:28:21.782 INFO:teuthology.run:FAIL